Last month we were very secretive and cryptic about what we were working on. The short version is we were trying to figure something out and didn't want to talk about it until we knew more. We will talk about it some now, but with the premise that this is not an imminent feature announcement. It's something we've had planned for after the 1.0 release, and that hasn't really changed, but for various reasons, some still a bit secret, we thought it was important to assess its feasibility.
Our team is largely comprised of childhood friends, and our passion and dedication for this game was born from week long sessions playing multiplayer games together, closed in a room and surviving on delivery food. It wasn't the convenient and accessible live service games of today, it was more traditional and deep RPGs which allowed co-op play via LAN. That experience is something that doesn't really fit today's gaming landscape, but it's what drove us and we believe there is at least a middle ground. A deep, persistent and interactive world is our priority, but exploring it with friends has always been our ambition.
Our plans for the future of the game hinge on multiplayer features, but we have created a very complex game indeed and to this day we didn't know if multiplayer was really possible. Some have said that it wasn't, and with good reason. The bar for co-op games is pretty low, people expect some funky and janky behaviour, if they even expect to be seeing the same thing at all. But when physics are involved it's a whole different problem, and complex physics make it a very big problem. Exanima is a game where players interact with and react instantly to the physics, where every motion is governed entirely by physics. Also, the tiniest glitch can have disastrous consequences. Creating a high quality multiplayer experience under these requirements is to say the least a challenge.
THE BIG PROBLEM
To talk about this we need to understand what the fundamental problem is: time. Two computers on a network are trying to simulate the same thing, but different things are happening on each (players doing stuff), and communication takes time. Simulating the same thing at the same time is therefore not really possible. Exanima runs entirely on exact physics and collisions, even a small difference can mean drastically different results. If fast moving objects collide, or don't collide, or collide differently, still fast, they move in very different ways. This creates a "butterfly effect", where a small difference quickly causes the simulations to diverge completely.
But surely it can't be impossible, games have multiplayer, and some of those games have physics, right? Well, yes and no. Many games don't really synchronise, you and your friend are probably seeing very different things, but most of the time you wouldn't know. Physics are more often for show rather than critical to gameplay, they don't have a dynamic and dramatic cascading effect. Multiplayer games that involve physics are designed around constraints and limitations, such as not having truly dynamic environment interactions that can affect other players, or limiting interactions to one player at a time, and generally very simplified physics that are easy to solve and fairly inconsequential if things go wrong. Accepting limitations and taking the easy path isn't something we're very good at though, and so here we are with our big problem.
You generally see this one concept: the simulation occurs on a server, or on one player's computer at a time, (other) players observe the results with a delay. Many supposedly peer to peer games actually have one player (the "host") acting as a server. More advanced solutions dynamically put one player in control of objects they are interacting with, making them a temporary server for those objects. So, either you must have your inputs delayed so the server can receive them, process them and send back the results, or someone else has control while you purely observe the results with a delay. Or you're just not really seeing the same thing at all, and it doesn't really matter for each player to have a functional gameplay experience.
But Exanima relies on specific and precise inputs and interactions which makes delays unacceptable. We go to great lengths to get the lowest possible input lag even in single player. This means host/server solutions are automatically out. There's other reasons why these solutions are bad, so we're happy to discard them. Delaying inputs can also give other approaches a way out, but for players it's the same basic problem.
The "dynamic authority" solution where you control the objects you're interacting with may seem good at first. There's no one server, you fully control your own character so it's responsive, and objects your character is interacting with are also responsive. But what happens when more players interact with the same object, even indirectly, or with each other? If you were to bump into an object that's not currently under your control it would be completely immovable. In terms of physics this means it has an infinite mass. A stray hit from another player's infinite mass poleaxe moving at high speeds is bad news for the game's physics. This happens both ways and when any objects under different control interact. It may not always be as bad as it sounds, but bad things can and will happen.
There is another solution adopted by fighting games that is a lot better. They establish a time synchronisation and roll back time to when another player gave an input, running everything that happened in the game since then again, giving you a more accurate up to date result. Besides causing rubber banding, this has one huge issue though: for a simulation as complex as Exanima's, the cost of rewinding time and re-simulating it all is absolutely prohibitive. It would make the already demanding simulation 6 times as demanding or more depending on your ping.
Of course all these problems are proportional to latency. On LAN or a very fast connection to someone close by, the input lag wouldn't be massive, the infinite mass objects would stop sooner because everything is less laggy and desynched, the cost of rerunning the simulation would be lower because there's less time to catch up on. But even under ideal conditions these problems do not go away, even on LAN they fall outside the processing pipeline and delays are much higher than the low ping might suggest. As latency increases, the problems very quickly spiral out of control.
A MOUNTAIN TOO HIGH?
So, we've established the basic problems and why none of the "solutions" work. We want no input lag, the same fluid physics that you interact with so closely for gameplay and control, all objects to have responsive physics for all players at the same time and interactions with other players to play out naturally and without glitches, just to name the most basic requirements. At first this just seems impossible, clearly we needed a different solution of our own, and to be completely fair, that's what we do.
Multiplayer is not an afterthought for us, ideas to tackle these problems have been building for years, and we've tried to design our game and our physics engine with it in mind. But these were just blind hopes, no better than wishes, until we could actually implement them and put them to the test.
What we're describing is pretty much just how a single player game works, without any of the problems of network delays. That becomes the basis of our solution: everything still works the way it does in single player. The problem now becomes how do we get these multiple single player games and world states with different inputs to do the same thing at the same time? Well, we can't, but if we could get really, really close, if the differences were small, we could maybe correct them quickly and seamlessly, and produce outcomes that don't look weird to any player.
This has been our plan all along, and we were hopeful enough to tell people that we believed low ping multiplayer could be done. But even with a low ping these problems are very obvious, as some early basic prototypes proved. Even on LAN, without good synchronisation, the results were hilariously bad if we started exchanging blows.
How do we keep our complex and very dynamic character motion synchronised in time? How do we get objects to move naturally and respond to collisions etc. while multiple simulations are telling them to do different things? We had some theories, but they involved complex solutions with many things at play that remained untested and unproven until now.
TESTING AND RESULTS
We have been developing and testing our so far theoretical approaches. We don't have a complete multiplayer mode yet, there's many individual problems to solve, they all take work, but unlike these core problems we've discussed, we know they are solvable. While incomplete, what we have so far is not a rough prototype, but a carefully designed, properly and thoroughly developed architecture to support all our future multiplayer needs.
Our focus is on co-op, but we felt the best way to test the accuracy of precise time synchronisation and the ability to handle player interactions was a PvP duelling mode. These interactions are very important, we can't have a stray hit from a friend or tangling with them causing the simulation to explode. Bonus, if we can make this extremely difficult case actually work, we would have functional PvP combat. The timing was also good, as we're doing a final pass on combat and multiplayer implications couldn't be ignored.
The results have been pretty damn good. There's no way something like this can support cross continental play, but with pings of up to ~100 ms it makes for a very functional combat experience, if at times slightly quirky. The issues are very minor and forgivable, and we think we can improve on them. Most of the time it feels just the same as fighting someone in single player.
This is a really great outcome, and we now feel we can confidently say that a high quality multiplayer experience is possible. Of course networking is always a fickle thing, lag spikes, disconnects and connection issues are an inevitable reality that's outside our control, but as much as possible we are trying to account for these things and handle them gracefully.
https://www.youtube.com/watch?v=8lClep9ojng
HOW IT WORKS
The best solutions are always bespoke. One size fits all solutions have their place, but something that is specifically tailored to our game's features and our engine is going to give the best results. This goes both ways, our design and our engine's capabilities must support the solution. Some of what we did involves fundamental physics problems and is not that specific, but it's many things working together that produce the final outcome.
The first thing that needs to work very well is getting locally simulated characters that are remotely controlled to match exactly. We call this the "prediction layer". Characters in Exanima don't play canned animations, they make thousands of dynamic adjustments to muscle tensions per second and generate movements on the fly based on forces, points of balance, analogue inputs, points of contact and collisions etc. Predicting exactly how they will move is not possible, and inputs are received with a delay. Obviously though we do still know a lot about how characters behave, and if we know exactly when those inputs occurred, if we consider how they've been changing over time, we can make some pretty damn good guesses.
Some dynamic behaviours are predictable enough that the best way to get a time accurate outcome is to let them be done locally, without a delayed remote outcome messing with them. A lot comes down to very detailed state matching to keep all the little factors lined up, and teaching characters to quickly and smoothly make corrections when they might have mispredicted something. The more every little thing is synchronised, the more accurate the predictions. Brief hitches when things are slightly off get naturally filtered out by the physics, as forces need time to act.
Basically this involved a lot of specific extrapolation and prediction methods for every different action, input and dynamic behaviour or motion. It took a lot of testing and tweaking to find the right methods and balance how they interact with each other, the physics and the other synchronisation layers to get the best results. There's probably room for improvement, but it's already very good. We often thought we had it about as good as it could get, only to see a big improvement with some additional adjustment.
This entire "prediction layer" concept is very useful on its own. The character can continue to act and function without exact physics and state synchronisation. If there's lag spikes, brief network interruptions, computer freezes etc. it will continue going with its best guess. This is kind of huge really. It also allows characters to function with barely any synchronisation at all, providing a very light weight fallback method. In situations where exact synchronisation is not necessary, such as maybe in a lobby or for a distant or inactive NPC, only basic instructions are required to keep characters behaving very closely to its target, despite the physics.
That's a great place to start and a critical part of the puzzle, but then the raw physics need to get involved. Everything unexpected that can't be predicted. This part is even more important, but involves a lot of technical details and maths. The short version is that the initially good synchronisation usually produces the same or very close physical interactions, but if things diverge, by analysing differences and changes in motion and with some maths tricks, we're able to quickly correct them without creating any sudden unwanted forces or obvious effects. If there's a major desync, we have a fix that is basically the opposite of the infinite mass effect of other methods, we make objects weightless so they can snap smoothly into a new place without turning into a wrecking ball.
It's hard to explain why all of this is necessary, but simulations are fragile things that rely on an evolving stable state. If objects suddenly move where they're not supposed to, violating the simulation's constraints, things can react very violently, get permanently stuck or "explode". Even if they were rare, in a game like Exanima these would be game breaking events.
Our unique physics engine provides a solid foundation here. It can take a serious beating and remain stable, it will never miss or skip a collision no matter what happens. Objects in our physics engine will actually deform rather than react violently to impossible pressures. Our recent physics engine overhaul improved all this, but we still have some improvements we want to make which will benefit the game in general.
There is one final problem though: slight differences in simulations are inevitable, and in this game that could mean the difference between a miss and a deadly slice to the face. The effects of these events are also dramatic, dynamic and a result of physics rather than some simple feedback or visual front end to what is really a game of numbers. Because these are physical interactions, we can't really just wait around for messages to go back and forth so we know everyone agrees on what happened. This is not a problem that can be completely solved, but it can be mitigated. This involves a complex system for negotiation and concessions that can resolve into a fair outcome and provide feedback on what actually happened to both players. Usually there's no delays, or they are very brief with the effects of non mutual outcomes being emulated by peers.
COMBAT IMPLICATIONS
Some of these problems can also be improved by adjusting the game's mechanics. The main issue here is binary events that can't be adjusted for time. The interesting thing here is that such binary events are already at odds with the game's fluid physical interactions, and disliked by players. A well designed solution can both help solve networking issues and improve the game overall. Binary parries are a good example of this, but this can be applied to most effects and character states that transition instantly with an input or event.
In recent diaries we've talked a lot about additions and improvements to combat. This is supposed to be a really proper final pass of core combat mechanics, and a time to revisit many neglected things. If it has to be a definitive pass, we felt that it was important to understand the implications of multiplayer as well.
Fighting other players has been a very different experience from fighting AI. Every player has come at it with a different approach, doing all the things the AI would never do, and not falling for the same two tricks. Before trying it we were a little concerned that the combat might not flow well, that players might be too hesitant, or too aggressive and break the rules and mechanics too much, but after a little adjustment to a specific player's tactics this wasn't an issue. It was great fun and engaging. It did however really highlight some shortcomings and points of imbalance, but that's a very good thing, just in the nick of time for our combat revision. The lack of bashes in a clinch and the absolute state of parries stood out like a sore thumb. You learn to do without these things against the AI, but with players it's a whole different story. We're very eager to make parrying actually good, when it works it makes for some really fun and cinematic exchanges.
We have also been testing this with a couple of notoriously very skilled members of the community, and will continue to do so. Their feedback has also been very positive. We're maybe not quite at the level of perfect emergent grapples with a high ping, but it's not too far off. Time will tell how much better we can make it, but we're already happy with this baseline. Regardless of PvP specific concerns, things are incredibly stable and responsive.
NETWORKING
When you really get into it networking is a huge pain. The internet is a diseased dinosaur, fundamentally outdated and insufficient for today's requirements and hastily patched by different manufacturers and providers in a myriad different ways that just break something else. You also have a layer of custom security that breaks even more things. Traffic must physically travel across the world through many nodes and everything about it is unpredictable. This means lots of workarounds for all the different possible problems, and designing everything around the fact that you don't know when or if data will arrive. It makes even apparently simple problems almost unsolvable. Anything could go wrong with computers, but with networking it's the norm, not the exception.
In typical BM style, we also developed our own networking solution completely from scratch. We wanted something lightweight and optimised for the best possible latency. There is no server / host, and everything is designed around the hope rather than expectation that communication is possible. This focus on independent player function should also eventually allow for players to be in different parts of the game world and drop in near each other, or easily join and re-join sessions. I think we're all familiar with the woes of peer to peer games, woes we want to avoid.
We were also able to achieve ideal time synchronisation dynamically without a dedicated synch stage, and even accounting for asymmetrical latency and differences in PC speeds and simulation loads. That was really hard to do and quite an accomplishment, we even built a network simulator to develop the algorithm and validate the results. This started as a need to synchronise on the fly and with our minimal networking setup, but ended up working better than known approaches that we had previously considered.
We also wanted something that doesn't rely on online services, and that can work on LAN with no internet connection, in keeping with the DRM free nature of the game. This is a play with friends thing, no matchmaking type features are planned for now. We do not want competitive matchmaking, we feel it just encourages toxic behaviour, cheating and all sorts. We do hope to eventually provide tools for people to host their own community tournaments and things of that nature.
THE NIGHTMARE OF OFCOM COMPLIANCE
Unfortunately we live in an age where governments feel they can put blanket bans on any kind of human interaction or entire artistic mediums, and that mass surveillance trumps basic human rights. The Online Safety Act puts utterly ridiculous restrictions on any kind of online interaction, considering even user names and arranging objects to spell out words a form of banned communication. Compliance requires age verification and strict (and expensive) moderation, failure to comply comes with an 18 million minimum fine. Small niche online communities, even just specialist forums, are being destroyed by this utter crap and online gaming is becoming very complicated, especially for small indie developers.
We are exploring possible workarounds for the future, but they're tightening the noose around these. For now, luckily and incidentally, our desire to make something completely free and forever unrestricted first by not relying on online services also means that as long as we do it this way we're completely exempt from these regulations. No service, no problem.
Right now gamers are facing a new threat in the form of the "EU Kids Act" proposal. This is even more out of touch and sweeping, promising to make distributing games in the EU impossibly expensive and difficult for indie devs, even single player games. If this proposal passes it could mean the death of indie games and basically make playing most games illegal in the EU, old and new, without of course doing anything whatsoever to "protect children". The EU qualifier means little for indie devs who already struggle financially, this would be the end. The big players in the industry are doing nothing, this would just eliminate their irksome indie competitors, so it's up to us, the public, to raise awareness and let these oblivious lawmakers know how outrageous this is. Spread the word, leave your feedback on the EU commission website and if you're in the EU consider presenting your case to your local representative.
Sorry to end on that awful note. Multiplayer is coming, and we have big plans for what we want to do with it after 1.0. It's too early to talk about all that, but we've been quietly working towards them for quite some time now, and they involve a lot more than just adding multiplayer. We needed to be sure that we could make those a reality, but multiplayer will not be a development focus in the short term. We want to finally get Exanima to 1.0, and our next step is implementing the still missing skills, which inevitably comes with some general improvements and additions to combat. This has quickly become our preferred platform for testing combat and gives us a new perspective, so to a small extent they will go hand in hand. We do hope to introduce at least a fleshed out duelling mode with the 1.0 release.
Best,
Bare Mettle
