Multi-player Connectivity and Performance

[GT Phantom 0]

Dear Folks at Codemasters,

I, Bill Reister aka Phantom, am a member of "Golden Triad", an online group dedicated to both competitive and recreational multi-player gaming and currently ranked #1 in our competition league. We represent a significant number of purchasers of your game, and would like to request a few specific enhancements to the dedicated server which would both enhance our multi-user gaming experience while providing you great appreciation and loyalty from your customer base.

CONNECTIVITY, COMPETITION & PERFORMANCE: Oftentimes our players have difficulty connecting or get disconnected before or during gameplay even if they have stable machines and good connections to the internet. We would like your help with the following:

1) Publish (not "refer to user groups") a specific set of recommendations for your users to help insure the maximum likelyhood of obtaining / retaining a connection to the server.

2) Publish (not "refer to user groups") a specific set of recommendations for server setup to help insure the maximum likelyhood of obtaining / retaining a connections with players.

3) Enhance your code (perhaps using multi-threaded / asynchronous processing?) so that if a player is having difficulty connecting / remaining connected that that does not cause performance degradation amoung the remaining players (i.e. everyone gets the Red Square).

4) Enhance your code so that if a player with the SAME NAME and SAME LICENSE OR PLAYER ID loses connection during a match that they might re-connect to the server during the same match, subject to limitations described below.

5) Enhance your code so that match reconnection parameters may be set/specified at the server / mission level. Some possibilities that come to mind are:

A) If a player disconnects, the avatar is ...

- replaced by AI (current default)

- removed

- frozen in place "invulnerable"

- frozen but vulnerable - won't respawn if killed

- frozen and teleported to the respawn area

B) ...within the playing field until / unless the player reconnects, at which point the player will

- reconnection may be disabled (current default)

- reactivate in original position

- reactivate, reappearing teleported a random distance / direction from original position

- respawn

- be allowed to observe the remaining game as a "bird"

6) Enhance your code so that the full power of multi-processor servers is harnessed (or provide instructions as to how to accomplish this - our dual XP 1600+ server running Windows XP never goes above 53% CPU, even if the process priority is set to "Realtime")

7) Enhance your server code so that the final scorecard may be automatically saved to disk (in case the final screen is closed too quickly - people are currently relying on "screen shots" of the scoreboard, which also fails if there are too many players for one "screen")

8) Provide instructions to run the dedicated server as a service while writing the screen log to a file.

We thank you for the tremendous job you have done so far at creating an excellant game!

Bill Reister

Golden Triad

[UKPUSHER 0]

Definetly this is correct , since 1.42 it has made connectivity even worse it sometimes takes 3 minutes for everyone to see there own names when launching from gamespy for example with 8 ppl on a fast machine with an ace connection

i think there should be an option to disable pinging clients at the start of the mp game  this seems to be whats causing the pausing in ppls text  

[RN Malboeuf 12]

I saw great inprovements in the last few patches

if your having problems it sounds like your servers are just not up to the task

this game requires higher end servers over 1 ghz

we are running a 1 ghz (133 fsb) 384 megs o ram now and know that past 100 AI or 40 players we have problems

next week we have 16 machines in bound all 1.6 ghz Athlons for a Net gaming store we are opening up

our main server will jump to a 1.8 ghz 266 fsb Athlon with 2 gigs DDR ram, this means we can handle 300 AI or 150 players

and then have 16 more servers availble at any given time

I have not seen any problems with players hanging on our server. every thing looks good, we just have a dam memory leak I would like to kill off

One thing we can be sure of is that BIS wil still keep on patching

[GT Phantom 0]

On 10 Feb. RN Nalboeuf said:

"if your having problems it sounds like your servers are just not up to the task

this game requires higher end servers over 1 ghz"

Hmmm, Mal, are you one of Codemaster's guys? Either you didn't bother to read my post, or you are trying to draw attention away from the problems.

So let's clear this up.

Server Specs:

Motherboard - Tyan Thunder K7 dual Athalon MP

-- latest bios

-- latest drivers

-- 1GB Registered ECC DDR RAM

-- 2x MP 1600+ Athalon processors

-- Volcano cooling fans - temp never goes above 110F

Windows XP Professional

-- All critical updates

-- No unnecessary services

-- Basic TCP/IP Only

Radeon 8500 AGP

-- latest drivers, but shouldn't matter since server is non-graphic

3Com 9xx NIC

-- Latest drivers

-- attached to T-1 with no other load during gaming times

-- latency to Denmark about 50ms

-- latency to New York about 40ms

-- located in Atlanta

Now, since we've ruled out lack of horsepower, here are some log results:

-- Bandwidth used NEVER exceeds 20% T-1 capacity

-- never goes above 53% CPU, even if the process priority is set to "Realtime"

If anyone out there has some USEFUL feedback, PLEASE resopnd!!!

Otherwise, I would REALLY appreciate it if Codemasters would consider these issues.

[BigQEd 0]

There is a problem... without a doubt. We have VERY high end servers and more Bandwidth than I believe anyone operating OFP Servers. (Multiple OC3s)

AGAIN WE DEFINITELY SEE A PROBLEM. Especially when loading a server with more than 24 people. The Netcode needs work... IMO

Honestly Phantom we see almost exactly the same things you mentioned...

Malboeuf is simply trying to draw attention to his projects  

[suma 8]

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote </td></tr><tr><td id="QUOTE">Now, since we've ruled out lack of horsepower, here are some log results:

-- Bandwidth used NEVER exceeds 20% T-1 capacity

-- never goes above 53% CPU, even if the process priority is set to "Realtime"

<span id='postcolor'>

OFP is not designed to use dual CPU, therefore what you see is probably one CPU running 100 % and the other one sitting idle. As a result bandwidth is not a problem at all, because CPU is not able to provide data fast enough.

- What mission are you running?

- Are you trying to monitor server frame rate with "#monitor" command? This will give you good indication how much is your sever loaded.

The server never runs at more than 50 fps. When running slower, it always uses all available CPU processing power to maintain the smoothest possible gameplay. When running at less than 15 fps, you can consider the server overloaded – the mission currently played is probably too complex for given server.

[Espectro (DayZ) 0]

Hey there.

I have 1 thing to add, if Suma or any other BIS is reading this.

Would it be hard to get a "kicked" message instead of just disconnected?

Its a pain in the ass to see, that you were gonna win, when the player carrying the flag gets "disconnected", or the one in the tank gets "disconnected".

If we could have a "Player Espectro was killed by Admin" message, instead of just "Player Espectro was disconnected".

BIS, i dont think that would take too much work, but it would make certain things easier.

[Ronbo256 0]

Suma, thanks for your reply.  I am a Admin on the server that Phantom is describing. we do use the monitor command and it rarely drops below 50 fps regardless of the mission, and we do play some complicated ones.  The problem he is having is that the server will quit and lockup the machine, which means he has to physically go to where the machine is located and reset it (the machine stops responding to any input) I know he has sent you log files from the crashes. Now if this were my server which is located in my house, maybe that wouldnt' be such a big deal, but he doesn't live where the server is located (as I am sure a lot of servers are located where the T-1 line is, not at someone's house) The really odd thing is my server has yet to crash since the 1.42 patch, but we don't have as many people on it as we do the T-1, simply because my machine is slower and my upload speed is much lower (just barely 256k on a good day) I have a Via 694t chipset mobo, with a ATI Rage IIc vid card, 256 meg of pc 133 sdram and a realtek 8029a NIC connected to cable, processor is a PIII @667 mhz, running win2k AS, it's  usuable for 4-8 players in simple missions only really. The reason he built that machine was so we could have larger games with lower latency and more  complex missions. Any help you guys can give would be much appreciated. You have done a great job with this game, we would like it tweaked just a bit to become an even better game. Kudos for all your hard work, and thanks for listening to us, which is better than 90% of the software companies out there.

GT-Ronbo, Personnel officer and server guru,

Golden Triad

www.ericchung.com/golden.html

P.S.  could it have to do with the fact he is running winXP Pro, and I am running Win2k AS? the two are very similar, but different, I know since I run Win XP Pro on my desktop.

[GT Phantom 0]

Suma,

Thanks for responding!  Yeah, I had guessed that the program was running single-threaded - hence my suggestion to thread out some of the sub-processes.  I understand the problem (I manage a Java shop):  you all did not probably set out to allow asynchronous threads in your archetecture (harder to code; uses more memory; adds overhead and slows single-player game for users with slower machines), and adding it now would be problematic.  If I could chat with you and / or your fellows perhaps we could devise a simple strategy which would not require a lot of recoding but would make the SERVER much more scalable.  My frontline web app has to handle 10,000+ simultaneous users with realtime database updates, so I understand the problems facing you (especially the ugly Budget Monster!!!  ).

In answer to your specific questions:

- We run EVERY mission!  And, we use lots of Add-Ons.

- Server Framerate as reported by #monitor rarely drops below 15 unless there are one or more particularly bad connections or when we have 20+ players connected.

- We are not memory bound:  I rarely see > 256MB of RAM used by app (until the memory leak sets in - but that's a separate issue...)

- We are not bandwidth limited.

My basic thought is this (and I'm guessing here, so don't slam me too hard if I'm off base):  the server performs several functions, among which are arbitration and connection communication.  Now, you (Codemasters) chose not to allow UDP; commendable for reliability and trying to be "fair" to all parties, but reduces the game to the "least common denominator" of slowest connected user.  I also suspect that each connection is processed sequentially, which is why the Red Square comes on for everyone when one machine is not responding in a timely manner.

What I might propose would to make each connection an asynchronous object with communication functions between the server (arbiter) and client.  The benefit here woud be threefold:  

A)  The arbiter function would have much less to do, so a much larger game could be processed by the same CPU;  

B)  If done with native threads (I also presume you used C++), XP or 2000 should be smart enough to load balance the separate threads across all CPU's (thus gaining cheap scalability);  

C)    The clientConnection class could have limited smarts - it could do a quick stress test on connection for client ping and bandwidth - and might intentionally drop or average frame information to a particular connection based both on server and client available bandwidth.  An asynchronous connection would not bring the game to a halt for everyone while waiting for a slowpoke to "catch up."

Each clientConnection class would be persistent for the duration of a match, thus allowing the object to negotiate reconnection with the original player mid-match in the event of a dropped connection.  In the event of a disconnect, an instance of playerAI could be seamlessly substituted for the connection to the player; on re-connect (if the mission allows) the process could be reversed.  For fairness to players with slower connections, "hits" might be recorded based on the client "view of the world" at time of impact (i.e. if it "looks" like you hit it, then you did).

Anyway, that's all based on speculation about how you architected the server and the bottom line is I LOVE THIS GAME! - and want to see huge 50v50 matches!  I have antied up a good server, and I would be willing to bet that (without compromising the integrety of proprietary portions of the game engine) Codemasters could EASILY solicit some help from the gaming community to reconstruct one small area of the codebase if it would improve the online experience!

Thanks for taking the time to read all this!

[heyvern69 22]

Wow. I'm not a programmer, so I didn't understand half of what you just said. But it sounded good to me.

I like anything that could improve multiplayer. Good luck guys. Wishing you the best.

P.S. I read in an online article that BIS used DirectPlay in OFP. Ouch. Again, I'm NOT a programmer, but every time I've had a flight sim that used DirectPlay for the networking code, I've read web articles later on, where the flight sim developers have said they regretted using DirectPlay for the networking code. They say DirectPlay is NOT very good.

So for future multiplayer games, perhaps BIS should re-consider using DirectPlay for the netcode???

[suma 8]

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote </td></tr><tr><td id="QUOTE">- Server Framerate as reported by #monitor rarely drops below 15 unless there are one or more particularly bad connections or when we have 20+ players connected.

- We are not memory bound: I rarely see > 256MB of RAM used by app (until the memory leak sets in - but that's a separate issue...)

- We are not bandwidth limited.<span id='postcolor'>

I am not sure what the problem is. Do you experience much lag or desync on the server even when running on 15 or better framerate? If yes, did you try set value MaxMsgSend higher? (Default is 128 for dedicated server, but I think in your case value 1024 could be better)

If you are not seeing any lag or desync, I can see no point in using more bandwidth, because if it is not really needed.

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote </td></tr><tr><td id="QUOTE">I also suspect that each connection is processed sequentially, which is why the Red Square comes on for everyone when one machine is not responding in a timely manner

What I might propose would to make each connection an asynchronous object with communication functions between the server (arbiter) and client. <span id='postcolor'>

Currently we are using DirectPlay to perform this communication. On our side there is nothing that would process connections sequentially, so if you see this happening, it is probably problem of DirectPlay. We are designing non-DirectPlay implementation for OFP networking, which we expect to be performing much better, but it will probably take quite long before it will be available.

[PitViper 0]

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote (Suma @ Feb. 14 2002,06:55)</td></tr><tr><td id="QUOTE"></span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote </td></tr><tr><td id="QUOTE">- Server Framerate as reported by #monitor rarely drops below 15 unless there are one or more particularly bad connections or when we have 20+ players connected.

- We are not memory bound:  I rarely see > 256MB of RAM used by app (until the memory leak sets in - but that's a separate issue...)

- We are not bandwidth limited.<span id='postcolor'>

I am not sure what the problem is. Do you experience much lag or desync on the server even when running on 15 or better framerate? If yes, did you try set value MaxMsgSend higher? (Default is 128 for dedicated server, but I think in your case value 1024 could be better)

If you are not seeing any lag or desync, I can see no point in using more bandwidth, because if it is not really needed.

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote </td></tr><tr><td id="QUOTE">I also suspect that each connection is processed sequentially, which is why the Red Square comes on for everyone when one machine is not responding in a timely manner

What I might propose would to make each connection an asynchronous object with communication functions between the server (arbiter) and client. <span id='postcolor'>

Currently we are using DirectPlay to perform this communication. On our side there is nothing that would process connections sequentially, so if you see this happening, it is probably problem of DirectPlay. We are designing non-DirectPlay implementation for OFP networking, which we expect to be performing much better, but it will probably take quite long before it will be available.<span id='postcolor'>

great! Are you going to leave it open/modular enough for a possible future online persistent world?

Look forward to seeing the new code.

BTW, we have no problems with the dedicated server with up to 8 players in our gaming group.

[GT Phantom 0]

PitViper,

Thanks for joining the conversation!  The news that you are already working on improved multiplayer performance is terriffic!  

I did not start this post because of a specific "problem" with my server (although I HAD hoped my server would be able to support larger matches!) - my HOPE for this post is that CodeMasters will address several issues which will enhance overall multiplayer performance and playability (see detailed list, first entry in this thread).  Primarily I would like to see improvements in people's ability to connect and stay connected (which it looks like is already being addressed);  enhance the program to permit re-connection during a match; and increase the total number of players which a server can reasonably support in one match (currently most people agree that the game becomes unsatisfactory and drop rate increases when over 20 people are in the same match, even on a powerful server).  

My long-winded dissertation on HOW to do all that was just a suggestion.  

Once again, I LOVE     this game and it is far and away better than other multiplayer games of it's genre.  Keep it up, guys!

[GT Phantom 0]

Suma,

Sorry for addressing that last post to PitViper - I see now he was quoting you!

[DarkJedi 0]

Nice to see that someone from BIS is taking this serious

There is 1 thing I would love to see new, I think most of you guys would.

When you not in the game, you see this list who is playing and it very anoying when you must wait 30 Min and stare at that list.

Can't players who aren't in the game be birds and fly around?

This is much better then starring at a boring list, maybe you put a editor command in so the editors can edit where these birds can respam.

I hope to see this in the new patch

Maybe my english is a bit bad but you know what I mean

[vipermaul 246]

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote (DarkJedi @ Feb. 14 2002,11:26)</td></tr><tr><td id="QUOTE">Nice to see that someone from BIS is taking this serious

There is 1 thing I would love to see new, I think most of you guys would.

When you not in the game, you see this list who is playing and it very anoying when you must wait 30 Min and stare at that list.

Can't players who aren't in the game be birds and fly around?

This is much better then starring at a boring list, maybe you put a editor command in so the editors can edit where these birds can respam.

I hope to see this in the new patch

Maybe my english is a bit bad but you know what I mean <span id='postcolor'>

On the GT server, we've had 10 to 20 people waiting on a server for the next game. If all those people where birds and not contibuting to the battle, wouldn't that take up much of the network bandwidth. Heck I almost want an admin option of *if killed* you are kicked back to the waiting area where it release the bandwidth contraints. But I would like it as an option and not locked in as a feature.

I would like a 'Server Full' feature as well - to limit the number of people who can connect.

[suma 8]

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote </td></tr><tr><td id="QUOTE">Can't players who aren't in the game be birds and fly around?

<span id='postcolor'>

Unfortunatelly from technical point of view this is the same problem as "Join in progress". When connecting to the game that is in progress, we are currently not able to transfer you complete world state, and therefore you cannot view the world and you cannot play in it.

[Espectro (DayZ) 0]

Suma, as Viper asked.

When you get the new way of comunication ready, will you make it open so join in progress is available.

1 more ting that could be good was, that instead of crashing, the client would be able to download addons (maybe below 500 kb) from the server.

This is because sometimes, the clients crashes do to a small object, witch is a pain.

thanks

btw. You rock!! Ive never seen a company support their game like you do. Im glad that you still work on your "little child" and that you try to improve multiplayer. As some1 allrdy stated, you have now god-ess to all us ofp fans

thanks

[GT Phantom 0]

DarkJedi,

Thanks for joining the discussion.

I think if they were to consider a feature like that then they should also make the option mission-selectable. For instance, in many competition matches teams use a 3rd-party voice program (to distribute the server loads). If a team member killed in a no-spawn game (or simply not included in a match to begin with) were to exit then rejoin as a bird, then they would become an extremely effective spy changing the outcome of the match. I think that many people would be upset in those circumstances.

[heyvern69 22]

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote (Suma @ Feb. 14 2002,12:55)</td></tr><tr><td id="QUOTE">[Currently we are using DirectPlay to perform this communication. On our side there is nothing that would process connections sequentially, so if you see this happening, it is probably problem of DirectPlay. We are designing non-DirectPlay implementation for OFP networking, which we expect to be performing much better, but it will probably take quite long before it will be available.<span id='postcolor'>

Fantastic news! Keep up the great work. You guys really are the absolute BEST ! ! !

[Ronbo256 0]

Suma, thanks, you guys are the best game coders out there for responding to problems and suggestions, most of the other coders just say "nope, can't be done" or "nope, we're not gonna do that" or even worse, they just ignore people who have good ideas or problems. Thanks again, I absolutely LOVE this game, and really appreciate all the work you guys have done to make it more playable. Thanks for working on the non-directplay networking (M$ doesn't always know best, they don't even use their own code for their internal networks, that should say something ) Thanks again, you guys ROCK!

[RN Malboeuf 12]

</span><table border="0" align="center" width="95%" cellpadding="3" cellspacing="1"><tr><td>Quote (Suma @ Feb. 13 2002,05:37)</td></tr><tr><td id="QUOTE">OFP is not designed to use dual CPU, therefore what you see is probably one CPU running 100 % and the other one sitting idle. As a result bandwidth is not a problem at all, because CPU is not able to provide data fast enough.<span id='postcolor'>

that we knew, but a Dual will still use multy threads for OS appz and other things that it may be running, it wont take all the load of the CPU running OFP but it will help

any help is good help, but if you are using say a dual 500 mhz for 1 ghz total then it's just better to play with a 1.4 ghz or above

A Dual P4 2 ghz (each cpu @ 2 ghz) with PC800 and scuzys would be wicked for a server

I have 8 of those comming to the office woo hooo, but those are for work stations heh, not used for servers heh, I was alloted only $22,000 CND for gaming servers, 16 1.6 ghz will be all we get for gaming servers for the next year

[GT Phantom 0]

Ah, the forum is back up!

One thing I didn't see in y'all's posts: with the new multi-player code, are you already planning on allowing mid-match rejoin?

[rekrul 7]

Excellent Suma.

In light of this, I'd like to suggest that you stop bugfixing MP in the current OFP version, put all resources on making an OFP Online game and make this an add-on for OFP and, for those who don't have OFP, a stand-alone version.

Those who want to play single player can play the current OFP version, those who wants a better online version than the current OFP can buy the Online add-on/stand alone version.

Sell the add-on for Å20 and the standalone for Å35. Dedicated servers should be avail in both Window and Linux versions and free to DL.

-Rekrul's two cents

[Vincent Vega 0]

@SUMA:

thank god you gave up this directplay mess! I have good hope OFP could beat all existing multiplayergames if you people manage to implement a solid, working netcode! take your time, better late than never:)

I stopped playing OFP online/on a lan cause I'm really annoyed of this directplay crap... one small example: simple coopmission, friend and me and a few AI enemies >

Me: enemy helicopter incoming from the north!

Friend: I can't see any helicopter!

Me: ####, take this hind down! you got the AA Launcher, open up your f****** eyes!

Friend: I can't see any enemies!

Me: arggg, this hind has spotted us! fire,fire... *BOOOMMM*

everybody dead.

playing like this is no fun at all, you get angry about your buddies cause you think: man, how blind is this lama??? they don't know what you're talking about cause the game runs too often out of sync, this sucks bad...