When the All Under Heaven update released we shared an update on our performance work here: (Dev Diary #187 - Performance & Optimization). In that update we also included some hopes and dreams for Crusader Kings III going forward and this is the Dev Diary where we add up the status as we reach the By God Alone update.
Hello again, I am Carl-Henrik, Principal Programmer on the Crusader Kings III team. Around the release of All under Heaven I discussed speeding up the load screen and improving various other systems soon. That is what this Dev Diary is about, along with where the day by day performance is heading, and how much memory the game is holding on to while you play.
This article lands a few weeks before the release of By God Alone and we are still working hard to improve the systems even further. Anything can happen in this period so be prepared for any final numbers being wrong but this is where we are right now.
If you only want the summary:
Simulation speed: we are aiming to match 1.19.0 “Scribe”, our previous quality of life update, on day to day performance. We are close to that today and expect to land near it.
The game loads a lot less up front. On our slow test machine for instance, streaming meshes and illustrations instead of loading everything at startup cuts time to the main menu by a significant amount, on my computer from 84 seconds to 16.
Video memory use in the same scene is down substantially - somewhere between a third and a half less, depending on your texture quality setting.
System memory a decade into a game is down by roughly a sixth.
Everything below is the longer version.
In order to know where we are and whether our optimizations are valid we need to measure the performance reliably.
Simulation speed is measured by the average time to simulate one day over 100 years. To make the test consistent we start observing in 1066 and keep going until at least 1166. Occasionally we run the game overnight to get a picture of the late game behavior as there are more and more things updating as time goes.
Each run will be a little different from the next due to random seed and changes that occur throughout the day from all the developers. That means there is some variation in each capture and we take that into account.
\[“Before” performance capture loading the game to the main menu in the current release at 84 seconds]
Every time we get the daily tick performance we also look at the startup time. One thing we noticed is that Linux is far superior to Windows if you want to get into the game and start playing faster!
So how long does a day take?
We use a fairly low-spec reference machine - an eight core CPU from 2012 with 16 GB of RAM - a day later in the game takes a fraction of a second. Most days are quicker than the average, and a small portion of those takes significantly longer. Those are the days when something big happens, and they are what you notice as a stutter rather than as general slowness.
The amount of processing increases with the age in the game. Between the 1070s and the 1160s the average day is a bit over one and a half times more expensive.
Where does the time go? Roughly, on a hundred year run:
AI decision making, a bit over a quarter of the day
The parallel pre-update, where systems calculate what they are about to change is about a quarter
The sequential update, where those changes are applied is roughly a fifth of the time
And firing events is a bit over a tenth of the time.
The target for By God Alone is to give you the same day to day speed as the “Scribe” patch, our previous quality of life update, and as of writing we are close enough to it that I am comfortable saying so out loud. What we will not do is put a number on it here, because the last weeks before a release move that number in both directions.
\[Average day duration by category over 100 years]
Each day is processed as a pre-update, then update and AI update.
For pre-update, characters and modifiers are the most significant portion, and that is expected - that’s who you are playing with and against! Within the AI update, evaluating character interactions is the biggest cost, and it is the one that grew the most during this project: a large amount of new ecclesiastical interaction content landed over a few weeks in July, and each new interaction is another thing every character considers doing every month. This is not something we can remove, since the interactions are the feature; we re-focus on making each evaluation cheaper, which the design and script side has been doing with good results.
Finding optimizations by deleting code is a wonderful feeling! Contributing to great projects checked that a count would only consider it every so many months. The intent was correct but the check occurred after the game calculated "could this ruler fund any of this" check first, and only then asked whether it was even supposed to consider it this month. Simply removing the funding check meant the whole process went down to a quarter of the previous cost which we could verify easily.
Thanks to the detail we log for performance tests we can find curious actions that might be very fast but run very many times.
Comparing two captures just a few days apart the character manager got noticeably slower, with no obvious reason. It turned out to be a single function that checks whether a ruler is allowed to keep one of their realm laws. In the Scribe release, that check happens around fifteen hundred times a month. In the new version it was being checked about eight hundred and fifty thousand times a month.
There wasn’t anything wrong with the function. A script change had simply moved it somewhere that ran a lot more often, and not obvious by just looking at the change - it looks quite reasonable on the page. Narrowing it to a single commit gave the script side exactly what they needed to restructure it, and the result was just a few percent of the entire daily tick. This by itself validates the effort of measuring continuously rather than just at the end: the difference between finding that in two days and finding it in two months is that after two months it is baked into the cake and nobody remembers why.
We frequently find that we do a lot more calculation for things than is necessary to make a decision.
The most obvious case was a check that comes up all the time in the new ecclesiastical content: is this particular ruler inside this clerical region? To answer it, the game built a list of every ruler in the region, sorted it, removed the duplicates, handed the finished list back to the caller, and the caller then looked to see whether the one ruler it cared about was in there. Everything about that is correct and none of it is necessary.
Asking the question directly instead - is this ruler in this region, yes or no - made it more than ten times cheaper.
The same shape turned up in comparing two faiths. The game used to build a table of the doctrines of both faiths and then walk the whole thing looking for differences; building a table for one faith and walking the other against it does the same job about a third faster. Neither of these is clever. They are both just doing less.
Here is one where the interesting part is why the cost existed in the first place.
Things in the game refer to each other - a character to their liege, a title to its holder - and a common cause of a game crash is to follow a reference to something that has been destroyed. Rather than let that crash a player's game, every object carries a small identifying stamp, what we call a “nullobject”, and checking whether a reference is still valid means checking that the stamp is intact. That check is asked an enormous number of times.
It was implemented in a way that required the machine to look up which kind of object it was dealing with before it could ask the question at all. Removing that indirection - the check does the same work and gives the same answer, it just no longer has to ask for directions first - was worth around three percent of everything, across a full hundred year run. It is one of the larger single wins in the project and it changed no behaviour whatsoever.
A crash was fixed some time ago by taking a piece of cleanup work - clearing out temporary character modifiers each tick - out of the parallel section and running it on a single thread. That is a completely reasonable way to make a crash go away, and it worked. It also quietly costs around four percent of overall performance, because that cleanup is super cheap and limited to one CPU thread for its work.
The underlying reason it could not run in parallel was that the memory allocator it used was not safe to call from multiple threads at once. Making the allocator safe, and reusing the objects rather than destroying and recreating them, let the work go back to running in parallel without the crash coming back. It nearly halved the time spent in that part of the tick.
The lesson here is to check if a crash fix has performance implications, because they are usually made under time pressure and the immediate solution might not be the best.
While in there I found several places - character memories, dynasties, opinions - where the game removes a lot of entries from a very long list one at a time, and each removal shuffles everything after it along by one. Removing them all in a single pass instead is the sort of change that takes an afternoon, appears nowhere in a graph, and is free from then on.
The load screen was where I said I would start, and the headline change for performance is that the game no longer loads everything before letting you play.
Previously, every 3D model in the game was loaded, constructed and uploaded to the graphics card during startup, whether you were about to see it or not. Now a model is loaded the first time something actually needs it, and released again only if the space is needed and nothing has drawn it for a while.
So for all of you solipsists out there then you can be happy to know that we now make sure the game accurately simulates the correct metaphysics of the world where object permanence makes sure that when something is out of sight it no longer exists (in memory). ; )
This is the single biggest change to loading time in this update. On our slow test machine, streaming cuts the time to the main menu by around two thirds. Measured a different way, the game builds about a third as many materials during startup as it used to - the rest are built later, if they are ever needed at all.
\[The Mesh Streaming debugger: what is loaded, how long since it was last drawn. This is only for internal builds of the game.]
Streaming graphics introduces a risk of popping things in: assets appearing a moment after you look at them. We address that in two ways. First, models are now kept in memory as long as there is video memory to spare, rather than being released on a fixed timer regardless of how much room the card has. In a normal session that settles well below what a modern card can spare, and it means an asset that scrolls in and out of view repeatedly is not rebuilt every time. Second, in various places we make sure the assets are in place before showing for instance the throne rooms.
Textures got the same treatment, from the other end.
Crusader Kings III has a very large amount of two-dimensional art - event illustrations, artifact icons, portrait detail masks - and historically all of it was resident in video memory for the whole session, whether or not it was ever shown. Illustration streaming loads those images when something needs them and releases them again shortly after they stop being drawn. An event picture is on screen for as long as you are reading the event, and that is all the time it needs to exist.
\[A visual video memory explorer we added to track down usage]
Two specific cases were worth the effort by themselves. The artifact icons were all loaded and in VRAM, not because they needed to be displayed, but to report the status to an error to the log if one was missing. Checking whether the file exists gives exactly the same error and costs almost nothing. And the portrait pattern masks, close to a gigabyte of source art used for clothing and heraldry detail, are now streamed as needed rather than permanently kept in active memory.
This is what caused a lot of hard to find slowdowns and even crashes for certain hardware configurations.
We kept adding assets after All Under Heaven, and all of them were loaded up front. Content grows with every update because the game itself becomes larger, but preloading means the cost lands on the graphics card whether or not any of it is on screen. Past a certain point a graphics card with less video memory than the game is asking for has to start bringing data back and forth across the bus every frame, and when that happens performance does not degrade gently - it falls off a cliff. On lower end machines that turned the framerate into the single digits. Occasionally it might even crash to the desktop without warning. This is not a playable experience.
That is the thing the two streaming systems fix, and why they matter more than any individual optimization in this Dev Diary. Measuring the same scene with them on and off, they take somewhere between a third and a half of the video memory usage away, and the proportion holds at every texture quality setting.
The saving is not really about texture resolution, it is about how much of the game's content is resident at once, and that is roughly the same amount of stuff whichever quality setting you are on. It also means the benefit lands hardest exactly where it is needed - not on the machines that had headroom, but on the ones that had run out of it.
As a matter of fact there doesn’t seem to be any difference between medium and low quality textures in VRAM usage. Feel free to play around with the graphics options and let us know what you think.
Render Targets are VRAM space reserved for drawing into, such as static portraits or the frame buffers for the screen. This is a place we can optimize VRAM separate from streaming.
When the game renders a frame it works in a set of full-screen buffers - the scene itself, the ambient occlusion pass, various blur and post-processing steps. These were all in a high precision format that stores each pixel as four 16-bit floating point values, which at a high resolution with anti-aliasing adds up to a substantial amount of video memory that exists purely as scratch space.
Most of that precision was not being used. The map is rendered and then brightened afterwards, so the values being stored sit in the lower half of the available range, and a format at half the size that spends its precision where the data actually is turns out to be enough. Halving the largest of those buffers is worth a meaningful share of what the renderer holds, and unlike streaming it costs nothing at all in pop-in.
It was not a one-line change, and this is the part I would tell you about over a coffee. Switching the format broke the fog of war - it went blocky in exactly the places where the fog is darkest. The obvious explanation was that we had lost precision in the dark, and that explanation was wrong; we spent a while disproving it. What was actually happening is that the map draws its cloud shadows and its clouds as two separate passes over the same pixels, so the result of the first pass was being written to the buffer at reduced precision and then read back as the starting point for the second. At high precision that round trip is free. At the smaller format it quantises twice, and does it right where the fog is darkest.
\[Before and after: the map at the reduced memory footprint]
The fix was to merge the two passes into one, so the composite is calculated at full precision inside the shader and stored once. That also removes a full screen draw, so it is slightly cheaper than what it replaced. A separate, older banding artifact in the ground fog - present at high precision too, we simply had not noticed it - was fixed by adding a tiny amount of dither at the very end of the frame.
Occasionally I get questions about how I work and normally it is not all that interesting tracking the current daily tick rate going up because the update is quickly adding more features but as we are nearing the release of the update the day gets more intense. I start by grabbing the log files from running the game overnight which might be anywhere from 600 to 1000 years.
The logs give me the 100 year graph (and more if I need to look at the bigger perspective) and it helps me find system categories that have changed since last time. A system usually has both scripts and code, so the next step is figuring out what could have changed the numbers.
Once I have a list of potential suspects it is time to do a round of interviews, who knows whether the change is something intentional or not? Who can give me more information to help me find my next steps?
From that I can often identify where in the code to look for more information. A quick glance rarely helps since one line of code can hide great complexity, but at least I’m familiar with most of it to gather basic information.
If the code can be isolated to a unique thing it is time to measure it locally. An external sampling profiler such as Very Sleepy or similar can often pinpoint bottlenecks, but I also have an internal in-game instrumented profiler. The instrumentation means that it measures the exact time a function takes and then the result can be added up and I can tell both how many times a piece of code was called and also how much total time was spent there. And in most cases I go over to a designer and explain what I found and they always know exactly what I am talking about (even if I don’t).
Once in a rare while I even get to rewrite a little bit of code to make it faster!
On the RAM side, when measured a decade into the game on the same machine, we are down by roughly a sixth against where we started this project. Not a huge number, but it is more than a gigabyte on a game where players might run alongside a browser and a chat client, and it is the difference between comfortable and swapping on a machine with the minimum amount of memory.
One fix was in animation data, and it turns out we stored the same movement data many times over. When an animation is retargeted from one skeleton to another, the retargeted copy used to duplicate all of its keyframe data, even though it is the same motion data stored the same way; it now reads the original's data and applies the differences in limb proportions as it samples. Separately, the same animation file loaded by several different skeletons was being read from disk once per skeleton, each producing its own identical copy in memory.
Another saving was animation metadata that was being treated in much the same way and usage went from 1200 MB to around 12 MB so reduced by a factor of 100.
One interesting note from all the statistics is that memory usage flattens out around the fifth year of a campaign. Almost all of what the game holds is loaded at startup rather than accumulated during play, which means a long campaign is not meaningfully heavier than a short one.
There are various other interesting opportunities in animation data and other systems that we will look at in the future.
Streaming only what is needed did most of the heavy lifting to reduce startup time, but it is not the whole story, and the rest of it was mostly finding work the game did not need to do.
The largest single item was not a system at all, it was one function. Loading the localization - the text of the game, spread over more than a thousand files - was done one file at a time on a single thread while the other CPU cores sat idle. On the slow machine it was the biggest single entry in the whole startup profile, a sizable fraction of the time you spent staring at the loading screen. Reading those files in parallel and merging them in a fixed order afterwards takes that time down to a rounding error.
The game also used to calculate its file checksum twice in a row, waiting for the first one to finish before throwing the result away and starting the second. That is fixed, and the remaining single pass now runs after the main menu is up rather than in front of it, so you reach the menu sooner and can start a single player game while it finishes. The multiplayer game buttons wait for it, since the checksum is what tells two players they are running the same version of the game.
[p align="center"]
\[Before and after startup time profiling (Scribe at 84s VS By God Alone at 16s)]
Large directories were scanned for files more than once because each scan was asking a slightly different question, and all nine localization languages are enumerated at startup even though you are only going to read one.\[Before and after startup time profiling (Scribe at 84s VS By God Alone at 16s)]
The Vulkan renderer was around 1.8 times slower to start than DirectX 11 on the same machine, and it turned out to be caused by a single line. When one thread uploaded a texture to the graphics card, it waited for the graphics queue to empty rather than for its own upload to finish - so each of the eight asset threads absorbed all seven others' work. Waiting on the correct thing brings the two backends closer to parity.
Another finding was that on Linux we seem to consistently load into the main menu twice as fast as the same computer running Windows. There is no concrete difference in the file system code so this is just one of those things where one is better than another.
We are leaving the new streaming options in the settings. If there are unexpected problems it is still possible to use the pre-load everything system as it used to be. But we would prefer to solve any issues and make the new faster system work for everyone.
We want to keep the simulation speed stable while the game keeps growing, and to start giving back the memory and loading time that had accumulated over five years of development. On the first, we are close to the Scribe update on the machine we measure and expect to land near it. On the second, the game asks for considerably less video memory, noticeably less system memory, and reaches the main menu in a fraction of the time it used to on a slow machine (also on a fast machine but it is harder to notice).
We are nowhere near finished with all this work! Some of it is still being worked on right now, between me writing this and you playing it. The character interaction cost is down but not yet back to where it was at the start of the summer, and there is a pile of animation data waiting for a change we already know how to make. And the rest of it is business as usual: a game that is still being developed will keep acquiring new things to optimize, which is not a problem at all. It will keep me busy for a while longer!
The real numbers arrive with the release, measurable by all players and we are excited to hear your results whether they are awesome, great or just ok.
Play fast!
"Divine" the Tech Lead sneaking in here with a small update on system requirements for the upcoming patches.
So Carl-Henrik has above outlined numbers shrinking at impressive rates in various forms and what does this then all mean for our minimum hardware specs for the game? The nice thing is that despite all of the content and features added in the new expansions we will keep our low minimum hardware specs the same with previous patch versions. All of the cool techniques explained above play a big part in what enables us to keep decade old hardware viable to play modern games.
For Silk and Silver we are starting to future proof the game and our codebase to better support future development cycles. We're moving over to more modern compiler toolsets to allow us to write the codebase compatible with the C++20 standards. With this we will need to update the required OS versions for our Linux and Mac players. So later this year when the Silk and Silver patch releases the game will require Ubuntu 24.04 LTS (up from Ubuntu 18.04 LTS) and MacOS 15 Sequoia (up from MacOS Catalina). The new OS versions we intend to support for several years before we start to look at new compiler toolchain updates.
For a few years Studio Black has held a game jam called Black Forge Jam. In the interest of company history my own Paradox path begins with developing Mega Drive games so what could be more appropriate than starting Mega Crusader Kings?
An ultimate challenge for performance and memory optimization!
I would like to report that I have made a start. Last December I spent a few days “porting” Crusader Kings to the Sega Mega Drive, in 68000 assembly from scratch, and the ridiculous part is that it reads real CK3 data. The landed titles file and the province map are converted offline into compact tables that get baked straight into the cartridge, and the console reconstructs the political map of Europe from them at boot - 1028 counties across 59 countries, each snapped to the nearest of sixteen available colours, and you can scroll around it with the D-pad. There is a HUD frame and a portrait of Harold Godwinson looking out at you.
\[Mega Crusader Kings, running in an emulator. The map is real CK3 data]
There is no game. No simulation, no text, no sound - the sound chip is switched off at boot and never switched back on. And I ran into a wall that will be familiar to anyone who read the video memory section above: the Mega Drive can hold 1536 unique tiles at once, the map needs about six thousand, and so it simply stops drawing partway through Europe. The note I left on the last commit reads, in full, "Need to save more VRAM."
Technically all the country and region names are in memory ready to be displayed, counties can be moved from one country to another while the game runs, but there is no gameplay in place yet. Most of a work week turns out is not enough even for a 16 bit game. I’m not sure there will be more time working on this project but I’m open to suggestions for what it could feasibly do!
I can use about 4 MB of ROM, or trade some of it for RAM with a custom cartridge. There is 64 kb of RAM and 64 kb of VRAM. Currently the ROM is 104024 bytes. It is a little different from trying to get the common low-end PC trying to run Crusader Kings III at full speed.
