The unlikely story of Teardown Multiplayer

At least in the early 2000s, Bloomberg had strict requirements about this. Their financial terminal has a ton of math calculations. The requirement was that they always had live servers running with two different hardware platforms with different operating systems and different CPU architectures and different build chains. The math had to agree to the same bitwise results. They had to turn off almost all compiler optimisations to achieve this, and you had to handle lots of corner cases in code: can't trust NaN or Infinity or underflow to be portable.

They could transparently load balance a user from one different backend platform to the other with zero visible difference to the user.

> If you want this to work across ARM and x86 (or even multiple ARM vendors), you are screwed, and need to restrict yourself to using only the basic arithmetic operations and reimplement everything else yourself.

Is this problematic for WASM implementations? The WASM spec requires IEEE 754-2019 compliance with the exception of NaN bits. I guess that could be problematic if you're branching on NaN bits, or serializing, but ideally your code is mostly correct and you don't end up serializing NaN anyway.

I'm sure you know, but for others reading: even on the same architecture, there is more to floating point determinism than just running the same "x = a + b" code on each system. There's also the state of the FPU (eg: rounding modes) that can affect results.

On older versions of DirectX (maybe even in some modern Windows APIs?) there were cases where it would internally change the FPU mode, causing chaos for callers trying to use floats deterministically[1].

[1] https://gafferongames.com/post/floating_point_determinism/ (see the Elijah quote, especially)

We use floating point operations with deterministic lockstep with a server compiled on GCC in Linux a windows client compiled with MSVC in windows, and an iOS client running on ARM which I believe is compiled with clang.

Works fine.

This is a not a small code base, and no particular care has been taken with the floating point operations used.

As far as I know, the ARM (at least aarch64) situation should be about the same as x86-64. Anything specific that's bad about it? (there's aarch32 NEON with no subnormal support or whatever, but you can just not use it if determinism is the goal)

that RECIP14 link is AVX-512, i.e. not available on a bunch of hardware (incl. the newest Intel client CPUs), so you wouldn't ever use it in a deterministic-simulation multiplayer game anyway, even if you restrict yourself to x86-64-only; so you're still stuck to the basic IEEE-754 ops even on x86-64.

x86-64 is worse than aarch64 is a very important aspect - baseline x86-64 doesn't have fused multiply-add, whereas aarch64 does (granted, the x86-64 FMA extension came out around not far from aarch64/armv8, but it's still a concern, such is life). Of course you can choose to not use fma, but that's throwing perf away. (regardless you'll want -ffp-contract=off or equivalent to make sure compiler optimizations don't screw things up, so any such will need to be manual fma calls anyway)

I'm pretty sure he is talking about deterministic output.

It seems like Coffee Stain generally handles the games they publish pretty well. They also have Valheim and Deep Rock Galactic published under them, both of which have been reasonably successful and long running games

Not about multiplayer but still interesting.

Dennis Gustafsson – Parallelizing the physics solver – BSC 2025

https://youtu.be/Kvsvd67XUKw

He is already talking about the new engine at the end, so maybe that means a new game/version.

Anyway I recently bought it because of multiplayer. Can’t wait to try it out.

(I’m not a Teardown dev!)

I tried this on a local project. It looks very jank and the math falls apart quickly. Unfortunately, using a fixed axis-aligned grid for rotating reference frames is not practical.

One to thing I wanted to try but didn’t, was to use dynamic axes. So once an entity is created (that is, a group of voxels not attached to the world grid), it gets its own grid that can rotate relative to the world grid. The challenge would be collision detection between two unaligned grids of voxels. Converting the group to a mesh, like Teardown does, would probably be the easiest and most effective way, unless you want to invent some new game-physics math!

This sounds like a fun thing to do simply for the pleasing global consistency, but the price you will pay is that the physics will inevitably look weird since all our intuition is for smooth space. In this sense it's like those games that try to put you into a 4D space, where the weirdness is sort of the point.

Not sure what you mean with the claim that Newtonian Mechanics doesn't work in discretised space? I'm know there are plenty of codes that discretise space and solve fluid mechanical problems, and that's all Newtonian physics.

Of course you need a quite high resolution (compared to the voxel grid in teardown) when you discretise for it to come out like it does in reality, but if you truly want discretised physics on the same coarse scale as the voxels in teardown you can just run these methods and accept it looks weird.

(not a teardown dev)

i had brainstormed a bit a similar problem (non world aligned voxels "dynamic debris" in a destructible environment. One of the ideas that came through was to have a physics solver like the physX Flex sdk.

https://developer.nvidia.com/flex * 12 years old, but still runs in modern gpus and is quite interesting on itself as a demo * If you run it, consider turning on the "debug view", it will show the colision primitives intead of the shapes.

General purpose physics engine solvers arent that much gpu friendly, but if the only physical primitive shape being simulated are spheres (cubes are made of a few small spheres, everything is a bunch of spheres) the efficiency of the simulation improves quite a bit. (no need for conditional treatment of collisions like sphere+cube, cube+cylinder, cylinder+sphere and so on)

wondered if it could be solved by having a single sphere per voxel, considering only the voxels at the surface of the physically simulated object.

You could project your dynamic objects to world coordinates, but it would look pretty wonky for small objects. A grid is just fundamentally not going to look very physical.

Maybe you could simulate physics but completely constrain any rotation? Then you’d have falling stuff, and it could move linearly (still moving in 3d space but snapping to the world grid for display purposes)?

Did you check out the "about" section and the timeline of his career on the main page of the linked blog?

https://www.voxagon.se/

Because it looks like your opponent is a Swedish former demoscener who started programming at age 12 on the C64 and Amiga computers in 1990, quickly moving on to writing games and demos in assembly, then professionally developing physics engines since 2001, specializing in game performance profiling and squeezing performance out of optimized mobile games.

As far as game dev stereotypes go you basically picked a Final Boss fight. Good luck, you'll need it :p

Part of the problem with [1] is that you still end up needing [3] anyhow, because even if you've got a fiber-to-fiber connection, while the transfer was occurring the game world has moved on and you'll need to replay that anyhow.

But if you've got a solution for [3] that works completely correctly anyhow, then writing lots of code for [1] becomes redundant to that anyhow, even with save/load code sitting right there. Might as well start from the beginning and replay it anyhow.

One of the things I will often do that I sometimes have to explain to my fellow engineers is bound the rate at which we'll have to handle certain changes based on what is making them. If you know that you've got a human on the other end of some system, they can only click and type and enter text so quickly. Yes, you still need to account for things like copy & paste if that's an issue for your system, where they may suddenly roll up with a novel's worth of text, but you know they can't be sending you War and Peace sixty times a second. You can make a lot of good scaling decisions around how many resources a user needs when you remember that. The bitrate coming out of a human being is generally fairly small; we do our human magic with the concatenation of lots of bits over lots of time but the bitrate itself is generally small. For all that Teardown is amazingly more technically complicated than Doom, the "list of instructions the humans gave to the game" is not necessarily all that much larger than a Doom demo (which is itself a recording of inputs that gets played back, not a video file), because even Doom was already brushing up on the limits of how many bits-per-second we humans can emit.

fme, it's only kind of inconvenient. By the time the scene gets to the point where join-in-progress is disabled it's complete chaos anyway. Might as well restart the scene.

That said I haven't played any of the more intricate mods out there, but I can how it would become more of an issue.

They didn't explain those in detail because they didn't implement these concepts themselves. The GameNetworkingSockets library (and most likely also the internal network backend that was mentioned) provides them: > A reliability layer significantly more sophisticated than a basic TCP-style sliding window. It is based on the "ack vector" model from DCCP (RFC 4340, section 11.4) and Google QUIC and discussed in the context of games by Glenn Fiedler. The basic idea is for the receiver to efficiently communicate to the sender the status of every packet number (whether or not a packet was received with that number). By remembering which segments were sent in each packet, the sender can deduce which segments need to be retransmitted. https://github.com/ValveSoftware/GameNetworkingSockets

There really doesn't seem to be anything new or unique to their solution. I'm personally not surprised, because it is what has worked for thirty years.

I presume you know this, but maybe for others: judging by what was written in that paragraph, I'd indeed assume he means the same paradigm that has been driving replicated real-time simulations since at least QuakeWorld - some world-state updates (in this case _"object transforms, velocities, and player positions"_, among others) don't have to be reliable, because in case you miss one and need a retransmit, by the time you receive it, it's already invalid - the next tick of the simulation has already been processed and you will get updated info anyway (IIRC this is essentially John Carmack's insight from his .plan files).

The command messages (player operations and events) _need_ to be reliable, because they essentially serve as the source of truth when it comes to the state of the world. Each game client depends on them to maintain a state that is the same for everyone (assuming determinism the Teardown team has been working on ensuring).

I would imagine it’s all UDP and the reliable+ordered is just a different mode which does the re-sending etc.

I would be surprised if they actually had TCP at all

Seems like the wrong link to me. Just a dude working on a car.

This seems to get brought up in every hn post on video game mulriplayer, and it makes me wonder: do you play video games? I dont know of any video games that do multiplayer that way, and i would think that alone suggests its not a good idea.

Who wants to play a game with 50ms+ keypress to screen update delay? Sounds miserable.

Video streams are not known for their low bandwidth needs, let alone adding in RTT latency for inputs.

In addition to what others have said: even where remote play works, some games are worse candidates than others, and I expect Teardown would be towards the "worse" side of the set.

Teardown, visually speaking, is a pretty noisy game at times, and doesn't give a great visual clarity when streamed at real-time-encoding-type bitrates during these noisy moments.

FPS mouse+keyboard is also one of the worst-case scenarios for Moonlight/GFNow/etc. remote play, because first person aiming with a mouse relies very heavily on a tight input-vision feedback loop, and first person camera movement is much harder to encode while preserving detail relative to, say, a static-camera overhead view, or even third person games where full-scene panning is slower and more infrequent.

Might have been a key differentiator for Stadia.

Other comments are worrying about the streaming and latency but local split screen could also be another use case here.