SFS2X-RV - Complete networking solution (Tech Showcase)

[CCC200 3]

SFS2X-RV: Complete networking solution

A completely new way to create multiplayer/networking content inside Arma 3, without using native multiplayer.

 

What is this & why does it exist?

This is a bit of a strange project, so I'll do my best to explain exactly what I started roughly a year ago, and why.

 

Like a lot of you, I find myself often frustrated with Arma's server performance - since the entire SQF layer is being executed on 1 core, performance suffers significantly even with a dedicated server. It's an inescapable bottleneck that usually means either your player/AI count is low or you will be playing with sub-30 FPS. I have some experience with SmartFoxServer, a networking solution meant for large scale online projects; it's been used in a bunch of games, they have a showcase here. So, after some research, I figured I would try to integrate this into Arma as a sidestep away from the standard multiplayer, to both get around the performance limitations as well as create experiences normally not possible. After a lot of work, I have a demo!

 

How it works

In order for this to even run, there are quite a few moving parts that needed to be built. I will say now that this wouldn't even be possible without the excellent Intercept modification that allows for very flexible C++ integration into the engine. In short:

• There's an SFS2X "master server", this runs Java and handles most of the actual logic, database integrations, request handling etc. It is more comparable to an MMO server than your standard Arma server. It is multi-threaded and the data being passed back and forth is very light and abstract, meaning it performs quite well.
• There is a small application on the user's machine that runs parallel alongside Arma. This parses any incoming information such as when another player spawns, their location, chat messages etc. It is then converted into a custom pseudo-language I created and dumped to a queue on a consistent cycle.
• The Intercept mod loaded into Arma reads the language and converts that data into actual game code, creating units, moving them around, rendering UI etc.

 

This process also works in reverse, meaning the mod will read player inputs, convert them into the pseudo-language, and then the parallel application reads that and forwards it to the master server. I made a basic graph to illustrate this process:

 

So, in short, as far as your instance of Arma is concerned, you are playing single-player, and the things happening on screen are simply AI being spawned and moved around via SQF. Because of this, a much tighter "bubble" can be built around the player; when something is too far away for the game to care, it ceases to exist entirely and stops bogging down the logic thread. All of the important bits are exeuted on the server or via the parallel app/mod DLL, with SQF only being used to execute the necessary code pertaining to things inside the player's bubble.

 

This is all very heady I realize, and I apologize if it's confusing. I will attempt to explain why I think this tech has a lot of potential, what I've done so far and what I plan to do.

 

What can this do right now?

• A basic connection/disconnection/error handling process is in place: When you boot into a mission, you are connected to the server and logged into a "world" that contains other users. If there's a fatal issue, you'll be booted out.
• I've created a basic chat client similar to the one in standard multiplayer, though it is missing a lot of features right now. You can type messages and anyone else logged in will see them regardless of distance.
• Other players who log in will have a unit spawn on your screen if they are within the 200m bubble of your player unit within the world space; this value can be expanded or shrunk, this was just an easy default to test with.
• If a player leaves the radius of your bubble, they will despawn from your game world.
• Basic movement data, such as where you're moving, at what speed, and what animations you're playing will sync for players in your bubble.
• There is a "rubberbanding" mechanic at play for handling desyncs; if a unit's real position in your game is too far from what the server says it should be, it will snap to the correct position.

 

Here is a video of some of this in action:

 

As you can see there is some delay, however I am confident I can continue to bring it down as it's already much better than it was initially. It will likely never be as good as native MP, but good enough for slower gamemodes? I think so.

 

What will this be able to do?

The #1 thing I am interested in is raising the scale of possible simulation within a gamemode. If you're in the same bubble as 100 other players, the game will still perform exactly the same as having 100 AI on screen right now, it isn't magic by any means. But if they're spread out? Your game will function as if they never existed, and will be as smooth as an empty map. Since the data being passed around is very light (and SmartFox is optimized for large scale networking), you could have hundreds or thousands of units moving around, and since there isn't a single SQF thread trying to calculate and maintain everything, it will run much better, only slowing down on your end if you're in an area of high density.

 

SFS2X also has support for many types of databases, and can be used to set and pull large amounts of user data; I have personally already used it for a game with thousands of accounts and it worked like a charm.

 

Basically, I want to make an Arma MMO. It likely will not be able to compete with the speed of intensity of a normal MP match, but I am more interested in a world with thousands of AI and players moving around, without the game chugging to a halt just because there are things happening across the map.

 

Can this do all that right now? No, but I strongly believe it can, and I have carved the first steps on that path.

 

Will this be released publicly?

Honestly, probably not. The setup is complicated, and this can't just effortlessly slide into an existing mod or mission - experiences need to be built around it. I am interested in eventually pitching a full project utilizing this tech to Bohemia, but if that falls through I will probably open-source it just so my efforts weren't wasted. I mainly wanted to post this to see what the community thinks so far, to gauge interest and gather feedback on the concept/demonstration.

 

However, if you are a server owner or modder who thinks you could benefit from parts of this tech - I am open to discussion. There are some aspecs of this I could definitely see being integrated into existing content, such as:

• A global chat that spans across any type of content, allowing things like communication between different servers, or a chat inside a singleplayer mission.
• A database solution that's integrated into multiple MP servers, or even between SP content and MP servers; since this can be executed anywhere, you could say, have a single-player mission that adds to progress on a multiplayer server, or share data between multiple servers however you see fit.
• A leaderboard or similar system within singleplayer content.

 

Here's a GIF of the chat system in action, for example:

 

As far as the big things go, such as the player syncing, or planned things I haven't showed off like vehicle sync, AI etc; those features aren't production-ready at all, and as I said I plan on keeping those parts closed until further developement and (fingers crossed) some official discussions with BI.

 

I hope all this made sense, and I am genuienly very excited about sharing this with you all - I am happy to answer questions or divulge more details. Cheers 😊

[Melody_Mike 130]

11 hours ago, CCC200 said:

Basically, I want to make an Arma MMO.

😎

[Melody_Mike 130]

In more seriousness, I have many questions-

This is going to read as critical, but please believe me that I am trying to understand. What would be the advantage of this setup over caching things that are outside the player's "bubble" on their own machine? Such as this effort:

Although I suppose that with enough players (spread far over the grids), the above solution would become less effective, and the 'MMO' caching would scale better, yes?

How does the parallel application and Intercept mod deal with all the physics aspects of the game? I'm thinking of the latency causing cluster bomb explosions and hilarious vehicle collisions and such.

...Again, however, so very cool!

[chops 111]

I can't say that I understand much of what you're doing, but the potential results sound extremely impressive! Will be following with interest. Best of luck!

[CCC200 3]

On 3/21/2023 at 3:18 AM, Melody_Mike said:

In more seriousness, I have many questions-

This is going to read as critical, but please believe me that I am trying to understand. What would be the advantage of this setup over caching things that are outside the player's "bubble" on their own machine? Such as this effort:

Although I suppose that with enough players (spread far over the grids), the above solution would become less effective, and the 'MMO' caching would scale better, yes?

How does the parallel application and Intercept mod deal with all the physics aspects of the game? I'm thinking of the latency causing cluster bomb explosions and hilarious vehicle collisions and such.

...Again, however, so very cool!

Happy to answer!

 

As for the grid thing, that is interesting and I actually have not seen that before, but this project definitely serves a different purpose. Mainly that nothing here is being cached or rendered inactive when it leaves your bubble: Everything is still happening via the master server, but your client simply doesn't know about it when it isn't relevant. A dedicated server for Arma is still using a singular thread for SQF, which means even if there's no density, performance gets exponentially worse the more people or units are active. From what I can tell, this grid caching method is trying to work within the limitations of the engine to try and rectify it as much as possible (which is awesome, absolutely no shade), but you are correct that with enough activity across enough area, that method will have diminishing returns.

 

The performance saving method is actually kind of reversed here; with a grid cache, the more spread out players are, the more grids need to be active, therefore the less performance is saved. With this, client performance will decrease in areas of density since your game is moving more units around, but otherwise will run as if nothing is happening at all. In a full project with actual gameplay, I would probably also implement some type of garbage collection and caching system to save on server performance. But the big difference here is very much that this isn't an Arma server, there's no single-thread logic restriction and because of that you can have far, far more going on at a macro level.

 

As for the physics, I have not even started on combat yet, so I can't speak beyond hypotheticals. Any data that's exposed to the scripting engine I can access and share across the server, which I already did for position/velocity etc for player movement, and I'm sure I will have to do that for combat as well (like where exactly bullets end up) to even register a basic firefight correctly. Since my current goal is to build an actual game with this at some point, I plan on implementing more advanced combat options as I need them rather than trying to get the whole scope of the sandbox working out the gate.

 

However, I will be upfront in saying that I do not think this will ever be able to compete with a dedicated server in terms of combat scale. There will always be more delay and less reliable physics because of the way this has to be implemented, and I think it will work much better with civilian-focused content or small scale shootouts/vehicle play than traditional OPs that are about using everything the military has. Since the benefit here is how many units I could in theory have active at once, my ideal gamemode would probably be some type of simulation with lots of civy AI going about their daily business, and players can insert themselves into that world and interact with it in lots of ways. Not that I'm anywhere close to that yet though 😛

 

There are actually some more benefits to this method which I did not mention as I haven't implemented them yet, such as instancing. Since there's no strict format for how the server is setup, you could have players enter an entire new "room" within the server seamlessly, which would partition them off from other players or with a smaller group. For example, a hub world with lots of players moving around could seamlessly transition to a closed instance with only you or your friends when you accept a mission, without any loading screens.

 

Since I'm most interested in mechanics or ideas that couldn't exist in the engine normally without heavy concessions, I think it is worth pursuing just for that. Basically, I see solutions like grid caching or other performance mods as a way to work within the existing sandbox and improve it - and I am trying to build an entirely new sandbox with different pros and cons. It's different, not strictly better in every sense.

[Melody_Mike 130]

4 hours ago, CCC200 said:

There are actually some more benefits to this method which I did not mention as I haven't implemented them yet, such as instancing. Since there's no strict format for how the server is setup, you could have players enter an entire new "room" within the server seamlessly, which would partition them off from other players or with a smaller group. For example, a hub world with lots of players moving around could seamlessly transition to a closed instance with only you or your friends when you accept a mission, without any loading screens.

Could you apply this 'hub' concept to connecting different servers?

In ~2014, a developer managed to write an extension that would let you transition between terrains in Singleplayer in a single play session. Unfortunately, the site that hosted the mod (Armaholic) is offline, and I couldn't track down the showcase YouTube video, because Google makes it very difficult to search for videos more than a few years old. A counterpoint towards "forever storage" of the internet. But such tech has been used.

There's also an old post by BI developer KillZoneKid where he explains a method of transferring seamlessly between servers: http://killzonekid.com/farewell-my-arma-friends/
I wonder if your 'instance' tech could simplify that. Many use cases to imagine. Like players running around in a large outdoor terrain of Altis, then transitioning to an underground "VR" or "Cytech" terrain in an 'instance'. Players could have their cake (expansive outdoor terrains) and eat it too (detailed indoor areas).

I'm very excited to read more on this very large undertaking. And your explanations.

 

[CCC200 3]

21 hours ago, Melody_Mike said:

Could you apply this 'hub' concept to connecting different servers?

I would say probably not the way you're imagining it. When I talk of instancing, I mean on the server side; since it functions more like an MMO server and not an Arma server, the way players are grouped/divided after connection is abstract and can be applied however I see fit. It could be all players in 1 large instance, or divide them strictly into 16 player matches, or even dynamically spinup/spindown instances as needed, as per my "mission that sections off your friendgroup" example. It's seamless because it has nothing to do with the game or engine itself, it's happening on the backend SmartFox server.

 

Transitioning between terrains is something tied to the engine itself, which I don't have much control over. I am familiar with that KillZoneKid post, as that is actually what started my brain on the idea of getting around engine limitations many years ago when I read it. That works by manipulating the UI into automatically transitioning between the necessary menus while hiding that from the player. You could still do that if all you wanted was a transitionary method between multiple standard Arma servers.

 

I'm not sure if I've been totally clear, so I will reiterate that there is no Arma server running or managing anything here at all. The mission file itself is just an empty map with the player that's loaded in single-player. Everything onscreen, including a future gamemode with actual missions or objectives, is handled by the master server, which has no idea what an Arma is or how to run SQF. Arma is just the conduit that renders the data and captures your inputs. Since the network connection can be executed anywhere, for an idea like terrain switching I would probably just link an empty mission on each terrain together into a campaign, and tell the game to switch missions when you wanna move over to the other map.

 

Because this is a fairly unique project in terms of how it interacts with the game, I believe it will function best on its own and not as an integration into existing dedicated servers. If you were really looking for ways to make it serve that purpose, I would say that you could definitely use it to connect data between different standard servers or gamemodes, which I mentioned in my initial post. That is probably not the "connection" you're thinking of, but for example, if you had an org that hosts 3 dedicated servers, this could provide a global chat that could be synchronized between all of them, or a database that all servers can access.

 

Frankly though, I don't think that's very appealing, because it requires a much more complicated setup for whoever wants to utilize it. You'd need to distribute the client app, the necessary mods for the game as well as a custom DLL that needs to be approved by BattleEye if you plan on having it on. Not to mention the master server itself, which if you're not using wholly as a replacement, is on top of the dedicated Arma server you're already running. It's a lot of extra work, enough so that I saw no point in releasing it now when it doesn't yet have the capabilities to fully replace a dedicated server. When it (hopefully) reaches that point, I would recommend using it solely instead of an Arma server if you want the features this tech will provide. A hybrid solution as I've described is possible, but probably not worth the headache.

 

I hope I've been making sense! And I very much appreciate the questions, it is cool to see how other people interpret and feel about something I've kept under wraps for so long 😄