Jump to content
Sign in to follow this  
gonk

Fennec.ws network traffic

Recommended Posts

Yeah this is actually a pretty big issue. According to my box provider, we've used 1TB of traffic this month compared to 448.64GB from last month.

Share this post


Link to post
Share on other sites

To summarize: the servers are flooded with incoming packets which look like a Gamespy query. They respond by sending back their complete details. Can you confirm my understanding of the problem is correct?

What I miss: How does the incoming packet look like? What kind of query it is?

Share this post


Link to post
Share on other sites
To summarize: the servers are flooded with incoming packets which look like a Gamespy query. They respond by sending back their complete details. Can you confirm my understanding of the problem is correct?

What I miss: How does the incoming packet look like? What kind of query it is?

That's correct.

The requests also come from multiple hosts so the flooding is multiplied by multiple hosts.

The request looks like:

0000 00 1e 8c 0a b7 41 00 50 7f cd 02 b0 08 00 45 00 .....A.P ......E.

0010 00 26 00 01 00 00 79 11 14 c4 53 de e6 7a c0 a8 .&....y. ..S..z..

0020 32 01 1e 61 09 34 00 12 07 9a fe fd 00 48 4c 53 2..a.4.. .....HLS

0030 57 ff 00 00 00 00 00 00 00 00 00 00 W....... ....

Screenshot @ http://dev-heaven.net/attachments/13246/wireshark_filtered_icmp.png

Edited by Sickboy

Share this post


Link to post
Share on other sites

FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

---------- Post added at 09:54 ---------- Previous post was at 09:44 ----------

Correction: The actual UDP packet payload (data) looks like:

0000 fe fd 00 48 4c 53 57 ff 00 00 ...HLSW...

Share this post


Link to post
Share on other sites

Nice, I guess that gives us some options for now - hopefully BIS implements something for it as well - seeing not everyone runs BattlEye.

Share this post


Link to post
Share on other sites

My guess is that the arma servers are being used as DOS flood amplifiers. The addresses you see in the from fields are the targets/victims of this.

The big idea is to have the victim receive loads more data than the attacker is able to send, by using other hosts as amplifiers. It also hides the true identity of the attacker to all but his ISP. Basically, the attackers would be sending these ~100byte packets with a false "from" header. If the "to" header host responds to the false from (victim) with a packet significantly bigger than the original packet...

Rate limit the responses arma2 servers give to requests to a certain IP (else they'll just cycle ports and be back in business) (might use small burst value to compensate for NATed clients) and the problem will go away.

Share this post


Link to post
Share on other sites
My guess is that the arma servers are being used as DOS flood amplifiers. The addresses you see in the from fields are the targets/victims of this.

The big idea is to have the victim receive loads more data than the attacker is able to send, by using other hosts as amplifiers. It also hides the true identity of the attacker to all but his ISP. Basically, the attackers would be sending these ~100byte packets with a false "from" header. If the "to" header host responds to the false from (victim) with a packet significantly bigger than the original packet...

I have the same thought, but this might just be an attack against the game servers themselves (simply causing excessive bandwidth usage). All those machines supposedly sending these requests might not even exist. Maybe this is the work of some ArmA hacker that found a new way to annoy server admins.

Share this post


Link to post
Share on other sites

i came to same assumption that the game servers itself aren't the victims but abused for DDOS against other servers on internet ...

ofcourse overally victims are both the DDOS targets and the abused servers as it eats bw, increase CPU usage due I/O increase

Share this post


Link to post
Share on other sites
but this might just be an attack against the game servers themselves

Implausible.

Exhibit 1) If they are not spoofing their source address, they would be DOSing themselves, particularly from hitting so many servers in parallel. If so, it would stop rather quickly, and wouldn't change source often; we can therefore assume they are spoofing.

Exhibit 2) If they really wanted to hit arma servers rather than the return address, they would* spoof it such that each packet came from a random IP each time. That would make it practically impossible to block outside of a handshake like tcp, (low-size initial reply) or making the gamespy master the only redistribution point for that info - it probably does not do that for several reasons.

It would also have made it far harder to detect what exactly was going on with all that traffic in the first place, and the causes would have remained unexplained (but "annoying") for far longer.

* I'm assuming that they wouldn't be so dumb as to be unable to figure that out. They are, after all, smart enough to create the attack software, or at least set it up, in the first place.

Exhibit 3) There's far more motive to do a proper DOS against a third party than to cause some minor load against arma servers. To then believe (if you really do) this is really directed at us, not recognizing we're actually quite insignificant, is egocentric. That said, to bring it up as a mere possibility is appropriate.

I call it minor because the network isn't the bottleneck for arma servers. At least those that aren't home-hosted will likely have connections good enough to withstand the extra traffic without problems. Until caps are reached.

Exhibit 4) If we assume the source addresses are the target, this would merely be the latest in a loooooooooong tradition of dos amplification. Inflating the meaning of an event ("Blowing things out of proportion") rarely gets you closer to the truth.

Potential exhibit 5) Guessing from dwarden's signature, and the packet content (HLSW), this seems to use a protocol not specific to arma. Are other games being hit as well? (Have they rate limited it already?)

If we assume the others aren't already doing rate limiting, are other types of servers using the same protocol also affected? If yes, that'll immediately kill the theory of this hitting us specifically.

[targets] might not even exist

Then we're still hitting the network connections up to the network on which that address belongs. There's very few unicast ipv4 addresses that aren't routed somewhere. And we also cannot say with any certainty it doesn't exist; even if it would respond to pings in the first place, it may be unreachable because of the ongoing DOS.

Also, that there was a ventrilo server running on one of the targets gives a few hints to the type of person perpetrating this.

Edited by MaHuJa_

Share this post


Link to post
Share on other sites
Potential exhibit 5) Guessing from dwarden's signature, and the packet content (HLSW), this seems to use a protocol not specific to arma. Are other games being hit as well?

Yes, there has been similar exploits performed using other game servers. Reference: for example

Possible solution: Make use of the new "v3" challenge-response variant of the GameSpy query protocol?

Edited by Killswitch

Share this post


Link to post
Share on other sites
FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

Good news $able :)

Share this post


Link to post
Share on other sites
Exhibit 1) If they are not spoofing their source address, they would be DOSing themselves, particularly from hitting so many servers in parallel. If so, it would stop rather quickly, and wouldn't change source often; we can therefore assume they are spoofing.

I tend to think the same.

Exhibit 2) If they really wanted to hit arma servers rather than the return address, they would* spoof it such that each packet came from a random IP each time. That would make it practically impossible to block outside of a handshake like tcp, (low-size initial reply) or making the gamespy master the only redistribution point for that info - it probably does not do that for several reasons.

Maybe they just want to confuse and cause innocent hosts to be accused? No one knows why they are doing this, I just know that hackers always loved to annoy this community (and especially server admins). I am merely pointing out a possibility.

Exhibit 3) There's far more motive to do a proper DOS against a third party than to cause some minor load against arma servers. To then believe (if you really do) this is really directed at us, not recognizing we're actually quite insignificant, is egocentric. That said, to bring it up as a mere possibility is appropriate.

Again, I am simply pointing out another possibility. It has nothing to do with being egocentric.

Potential exhibit 5) Guessing from dwarden's signature, and the packet content (HLSW), this seems to use a protocol not specific to arma. Are other games being hit as well? (Have they rate limited it already?)

If we assume the others aren't already doing rate limiting, are other types of servers using the same protocol also affected? If yes, that'll immediately kill the theory of this hitting us specifically.

It's the GameSpy query protocol.

Then we're still hitting the network connections up to the network on which that address belongs. There's very few unicast ipv4 addresses that aren't routed somewhere. And we also cannot say with any certainty it doesn't exist; even if it would respond to pings in the first place, it may be unreachable because of the ongoing DOS.

I was referring to those machines actually sending these requests, which is not the case if IP spoofing is used.

---------- Post added at 17:25 ---------- Previous post was at 16:51 ----------

FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

Released now.

Share this post


Link to post
Share on other sites
FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

Released now.
Thanks $able!

---------- Post added at 17:54 ---------- Previous post was at 17:29 ----------

The new BE (v119) seems to block genuine gamespy requests that request the player info.

So the server info request seems to work (0xFF, 0x00, 0x00) but player packet fails (0x00, 0xFF, 0x00). I have not tested the full info packet (0xFF, 0xFF, 0xFF).

Im using Six Updater, fetching server info happens by two requests, first the server info request, and then immediately following the seperate player request.

Edited by Sickboy

Share this post


Link to post
Share on other sites
The new BE (v119) seems to block genuine gamespy requests that request the player info.

So the server info request seems to work (0xFF, 0x00, 0x00) but player packet fails (0x00, 0xFF, 0x00). I have not tested the full info packet (0xFF, 0xFF, 0xFF).

Im using Six Updater, fetching server info happens by two requests, first the server info request, and then immediately following the seperate player request.

Right now the BE Server allows one query packet (per IP) every 0.5 seconds. I will change the implementation to fix your problem.

Share this post


Link to post
Share on other sites

Thanks! Perhaps it would be nice to be able to configure these limits, if it wouldn't involve a lot of work :)

Share this post


Link to post
Share on other sites

It looks like the changes for gamespy requests are working $able, I've yet to come across the excessive bandwidth usage (fingers crossed).

Thanks again!

Share this post


Link to post
Share on other sites

Looking back at our server traffic stats, it would seem that this thing began on our server somewhere around after first week / middle of june. up until june our server was outputting somewhere around 100gb / month, but on june the end result was 385gb where as in july it is already at 595gb! No more running arma servers without someone actually playing there....

Share this post


Link to post
Share on other sites
No more running arma servers without someone actually playing there....
There's updated BE that mitigates the problem, as well as changes made to gamespy master to prevent the issues as well;

http://dev-heaven.net/issues/22808#note-55

Edited by Sickboy

Share this post


Link to post
Share on other sites

we enabled the security feature on Gamespy protocol, all queries now need validate ...

also BE will be updated soon to support both data and player info query

Share this post


Link to post
Share on other sites
I will change the implementation to fix your problem.

Fixed now.

Share this post


Link to post
Share on other sites

Sable,

Linux servers are now having issues reponsing to game tracker.

A while a go i also wrote a tool for our admin that queries the server for stats too.

The packet looks like following:

pack("c*",0xFE,0xFD,0x00,0x04,0x05,0x06,0x07,0xFF,0xFF,0xFF);

The server no longer responds to that packet. As well as packets sent by GameTracker.

Share this post


Link to post
Share on other sites
Fixed now.
Thanks! Will confirm later tonight once SU v3 support is up.

@nomad_man:

Probably nothing to do with BattEye but with the new GameSpy v3 protocol activated by BIS: http://dev-heaven.net/issues/22808#note-55

The v3 should become active after restarting the server since today.

You'll need to send challenge request, and include the proper response in the info query packet.

GameQ has support for it for instance:

https://github.com/Austinb/GameQ/blob/v2/gameq/protocols/gamespy3.php

https://github.com/Austinb/GameQ/blob/v2/gameq/protocols/bf2.php

In essence, it comes down to:

base_packet = "\xFE\xFD\x00"
challenge_packet = "\x[size=2][color=#000080][size=2][color=#000080]FE\[/color][/size][/color][/size][size=2][color=#000080][size=2][color=#000080]xFD\[/color][/size][/color][/size][size=2][color=#000080][size=2][color=#000080]x09"[/color][/size][/color][/size]
random_id = "\x10\x20\x30\x40"
info_packet = "\xFF\xFF\xFF\x01"

1. Send the challenge request packet (challenge_request_packet = challenge_packet + random_id)

2. Receive response, parse the response: only take the numbers after 0@. Do some bitwise shifting:

challenge_response_packet = sprintf("%c%c%c%c", challenge_response >> 24, challenge_response >> 16, challenge_response >> 8, challenge_response >> 0)

3. Send the info request packet with the new calculated challenge response.

full_packet = base_packet + random_id + challenge_response_packet + info_packet

Determining if the server requires a challenge response:

If the response to the challenge request contains at the end: 0@0 then it does not require the challenge response.

Note for non-php users, e.g Ruby:

Negative numbers need to be converted first, php does this automatically, ruby does not:

[size=2][b]def[/b] handle_chr[/size][size=2][color=#800000][size=2][color=#800000]([/color][/size][/color][/size][size=2]number[/size][size=2][color=#800000][size=2][color=#800000])[/color][/size][/color][/size]
[size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]= [/color][/size][/color][/size][/b][size=2][color=#800000][size=2][color=#800000](([/color][/size][/color][/size][size=2]number [/size][size=2][color=#0000ff][size=2][color=#0000ff]% 256)+256) [/color][/size][/color][/size][b][size=2]if[/size][/b][size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]< [/color][/size][/color][/size][/b][size=2][color=#000080][size=2][color=#000080]0[/color][/size][/color][/size]
[size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]=[/color][/size][/color][/size][/b][size=2] number [/size][size=2][color=#0000ff][size=2][color=#0000ff]% 256 [/color][/size][/color][/size][b][size=2]if[/size][/b][size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]> [/color][/size][/color][/size][/b][size=2][color=#000080][size=2][color=#000080]255[/color][/size][/color][/size]
[size=2] number[/size]
[size=2][b]end[/b][/size]
[size=2][b][size=2][color=#008000][size=2][color=#008000][font=Consolas][size=2][font=Consolas][size=2]challenge_response [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]=[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] sprintf[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][color=#a31515][font=Consolas][size=2][color=#a31515][font=Consolas][size=2][color=#a31515]"%c%c%c%c"[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080],[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 24[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]),[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 16[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]),[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 8[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]),[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 0[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]))[/color][/size][/font][/color][/size][/font][/color][/size][/font][/color][/size][/color][/size][/b]
[/size]

Edited by Sickboy

Share this post


Link to post
Share on other sites

What is the latest version of BE server?

Also, i can not change the exchange the GameTracker queries :). It seems a lot of popular servers are having the same issue.

Edited by nomad_man

Share this post


Link to post
Share on other sites

See my updated post.

Latest BE seems to be v1.120 currently.

Share this post


Link to post
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
Sign in to follow this  

×