Fennec.ws network traffic

[sickboy 13]

Updated the ticket with more ips and info http://dev-heaven.net/issues/22808

[gossamersolid 155]

Yeah this is actually a pretty big issue. According to my box provider, we've used 1TB of traffic this month compared to 448.64GB from last month.

[suma 8]

To summarize: the servers are flooded with incoming packets which look like a Gamespy query. They respond by sending back their complete details. Can you confirm my understanding of the problem is correct?

What I miss: How does the incoming packet look like? What kind of query it is?

[sickboy 13]

To summarize: the servers are flooded with incoming packets which look like a Gamespy query. They respond by sending back their complete details. Can you confirm my understanding of the problem is correct?

What I miss: How does the incoming packet look like? What kind of query it is?

That's correct.

The requests also come from multiple hosts so the flooding is multiplied by multiple hosts.

The request looks like:

0000 00 1e 8c 0a b7 41 00 50 7f cd 02 b0 08 00 45 00 .....A.P ......E.

0010 00 26 00 01 00 00 79 11 14 c4 53 de e6 7a c0 a8 .&....y. ..S..z..

0020 32 01 1e 61 09 34 00 12 07 9a fe fd 00 48 4c 53 2..a.4.. .....HLS

0030 57 ff 00 00 00 00 00 00 00 00 00 00 W....... ....

Screenshot @ http://dev-heaven.net/attachments/13246/wireshark_filtered_icmp.png

Edited July 27, 2011 by Sickboy

[$able 2]

FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

---------- Post added at 09:54 ---------- Previous post was at 09:44 ----------

The request looks like:

Screenshot @ http://dev-heaven.net/attachments/13246/wireshark_filtered_icmp.png

Correction: The actual UDP packet payload (data) looks like:

0000 fe fd 00 48 4c 53 57 ff 00 00 ...HLSW...

[sickboy 13]

Nice, I guess that gives us some options for now - hopefully BIS implements something for it as well - seeing not everyone runs BattlEye.

[mahuja 12]

My guess is that the arma servers are being used as DOS flood amplifiers. The addresses you see in the from fields are the targets/victims of this.

The big idea is to have the victim receive loads more data than the attacker is able to send, by using other hosts as amplifiers. It also hides the true identity of the attacker to all but his ISP. Basically, the attackers would be sending these ~100byte packets with a false "from" header. If the "to" header host responds to the false from (victim) with a packet significantly bigger than the original packet...

Rate limit the responses arma2 servers give to requests to a certain IP (else they'll just cycle ports and be back in business) (might use small burst value to compensate for NATed clients) and the problem will go away.

[$able 2]

My guess is that the arma servers are being used as DOS flood amplifiers. The addresses you see in the from fields are the targets/victims of this.

The big idea is to have the victim receive loads more data than the attacker is able to send, by using other hosts as amplifiers. It also hides the true identity of the attacker to all but his ISP. Basically, the attackers would be sending these ~100byte packets with a false "from" header. If the "to" header host responds to the false from (victim) with a packet significantly bigger than the original packet...

I have the same thought, but this might just be an attack against the game servers themselves (simply causing excessive bandwidth usage). All those machines supposedly sending these requests might not even exist. Maybe this is the work of some ArmA hacker that found a new way to annoy server admins.

[Dwarden 1125]

i came to same assumption that the game servers itself aren't the victims but abused for DDOS against other servers on internet ...

ofcourse overally victims are both the DDOS targets and the abused servers as it eats bw, increase CPU usage due I/O increase

[mahuja 12]

but this might just be an attack against the game servers themselves

Implausible.

Exhibit 1) If they are not spoofing their source address, they would be DOSing themselves, particularly from hitting so many servers in parallel. If so, it would stop rather quickly, and wouldn't change source often; we can therefore assume they are spoofing.

Exhibit 2) If they really wanted to hit arma servers rather than the return address, they would* spoof it such that each packet came from a random IP each time. That would make it practically impossible to block outside of a handshake like tcp, (low-size initial reply) or making the gamespy master the only redistribution point for that info - it probably does not do that for several reasons.

It would also have made it far harder to detect what exactly was going on with all that traffic in the first place, and the causes would have remained unexplained (but "annoying") for far longer.

* I'm assuming that they wouldn't be so dumb as to be unable to figure that out. They are, after all, smart enough to create the attack software, or at least set it up, in the first place.

Exhibit 3) There's far more motive to do a proper DOS against a third party than to cause some minor load against arma servers. To then believe (if you really do) this is really directed at us, not recognizing we're actually quite insignificant, is egocentric. That said, to bring it up as a mere possibility is appropriate.

I call it minor because the network isn't the bottleneck for arma servers. At least those that aren't home-hosted will likely have connections good enough to withstand the extra traffic without problems. Until caps are reached.

Exhibit 4) If we assume the source addresses are the target, this would merely be the latest in a loooooooooong tradition of dos amplification. Inflating the meaning of an event ("Blowing things out of proportion") rarely gets you closer to the truth.

Potential exhibit 5) Guessing from dwarden's signature, and the packet content (HLSW), this seems to use a protocol not specific to arma. Are other games being hit as well? (Have they rate limited it already?)

If we assume the others aren't already doing rate limiting, are other types of servers using the same protocol also affected? If yes, that'll immediately kill the theory of this hitting us specifically.

[targets] might not even exist

Then we're still hitting the network connections up to the network on which that address belongs. There's very few unicast ipv4 addresses that aren't routed somewhere. And we also cannot say with any certainty it doesn't exist; even if it would respond to pings in the first place, it may be unreachable because of the ongoing DOS.

Also, that there was a ventrilo server running on one of the targets gives a few hints to the type of person perpetrating this.

Edited July 28, 2011 by MaHuJa_

[killswitch 19]

Potential exhibit 5) Guessing from dwarden's signature, and the packet content (HLSW), this seems to use a protocol not specific to arma. Are other games being hit as well?

Yes, there has been similar exploits performed using other game servers. Reference: for example

Possible solution: Make use of the new "v3" challenge-response variant of the GameSpy query protocol?

Edited July 28, 2011 by Killswitch

[MJK-Ranger 0]

FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

Good news $able :)

[$able 2]

Exhibit 1) If they are not spoofing their source address, they would be DOSing themselves, particularly from hitting so many servers in parallel. If so, it would stop rather quickly, and wouldn't change source often; we can therefore assume they are spoofing.

I tend to think the same.

Exhibit 2) If they really wanted to hit arma servers rather than the return address, they would* spoof it such that each packet came from a random IP each time. That would make it practically impossible to block outside of a handshake like tcp, (low-size initial reply) or making the gamespy master the only redistribution point for that info - it probably does not do that for several reasons.

Maybe they just want to confuse and cause innocent hosts to be accused? No one knows why they are doing this, I just know that hackers always loved to annoy this community (and especially server admins). I am merely pointing out a possibility.

Exhibit 3) There's far more motive to do a proper DOS against a third party than to cause some minor load against arma servers. To then believe (if you really do) this is really directed at us, not recognizing we're actually quite insignificant, is egocentric. That said, to bring it up as a mere possibility is appropriate.

Again, I am simply pointing out another possibility. It has nothing to do with being egocentric.

Potential exhibit 5) Guessing from dwarden's signature, and the packet content (HLSW), this seems to use a protocol not specific to arma. Are other games being hit as well? (Have they rate limited it already?)

If we assume the others aren't already doing rate limiting, are other types of servers using the same protocol also affected? If yes, that'll immediately kill the theory of this hitting us specifically.

It's the GameSpy query protocol.

Then we're still hitting the network connections up to the network on which that address belongs. There's very few unicast ipv4 addresses that aren't routed somewhere. And we also cannot say with any certainty it doesn't exist; even if it would respond to pings in the first place, it may be unreachable because of the ongoing DOS.

I was referring to those machines actually sending these requests, which is not the case if IP spoofing is used.

---------- Post added at 17:25 ---------- Previous post was at 16:51 ----------

FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

Released now.

[sickboy 13]

FYI, a BE Server update will be released shortly that firewalls your server against these excessive requests.

Released now.

Thanks $able!

---------- Post added at 17:54 ---------- Previous post was at 17:29 ----------

The new BE (v119) seems to block genuine gamespy requests that request the player info.

So the server info request seems to work (0xFF, 0x00, 0x00) but player packet fails (0x00, 0xFF, 0x00). I have not tested the full info packet (0xFF, 0xFF, 0xFF).

Im using Six Updater, fetching server info happens by two requests, first the server info request, and then immediately following the seperate player request.

Edited July 28, 2011 by Sickboy

[$able 2]

The new BE (v119) seems to block genuine gamespy requests that request the player info.

So the server info request seems to work (0xFF, 0x00, 0x00) but player packet fails (0x00, 0xFF, 0x00). I have not tested the full info packet (0xFF, 0xFF, 0xFF).

Im using Six Updater, fetching server info happens by two requests, first the server info request, and then immediately following the seperate player request.

Right now the BE Server allows one query packet (per IP) every 0.5 seconds. I will change the implementation to fix your problem.

[sickboy 13]

Thanks! Perhaps it would be nice to be able to configure these limits, if it wouldn't involve a lot of work :)

[sickboy 13]

It looks like the changes for gamespy requests are working $able, I've yet to come across the excessive bandwidth usage (fingers crossed).

Thanks again!

[Sekra 10]

Looking back at our server traffic stats, it would seem that this thing began on our server somewhere around after first week / middle of june. up until june our server was outputting somewhere around 100gb / month, but on june the end result was 385gb where as in july it is already at 595gb! No more running arma servers without someone actually playing there....

[sickboy 13]

No more running arma servers without someone actually playing there....

There's updated BE that mitigates the problem, as well as changes made to gamespy master to prevent the issues as well;

http://dev-heaven.net/issues/22808#note-55

Edited July 29, 2011 by Sickboy

[Dwarden 1125]

we enabled the security feature on Gamespy protocol, all queries now need validate ...

also BE will be updated soon to support both data and player info query

[$able 2]

I will change the implementation to fix your problem.

Fixed now.

[nomad_man 10]

Sable,

Linux servers are now having issues reponsing to game tracker.

A while a go i also wrote a tool for our admin that queries the server for stats too.

The packet looks like following:

pack("c*",0xFE,0xFD,0x00,0x04,0x05,0x06,0x07,0xFF,0xFF,0xFF);

The server no longer responds to that packet. As well as packets sent by GameTracker.

[sickboy 13]

Fixed now.

Thanks! Will confirm later tonight once SU v3 support is up.

@nomad_man:

Probably nothing to do with BattEye but with the new GameSpy v3 protocol activated by BIS: http://dev-heaven.net/issues/22808#note-55

The v3 should become active after restarting the server since today.

You'll need to send challenge request, and include the proper response in the info query packet.

GameQ has support for it for instance:

https://github.com/Austinb/GameQ/blob/v2/gameq/protocols/gamespy3.php

https://github.com/Austinb/GameQ/blob/v2/gameq/protocols/bf2.php

In essence, it comes down to:

base_packet = "\xFE\xFD\x00"
challenge_packet = "\x[size=2][color=#000080][size=2][color=#000080]FE\[/color][/size][/color][/size][size=2][color=#000080][size=2][color=#000080]xFD\[/color][/size][/color][/size][size=2][color=#000080][size=2][color=#000080]x09"[/color][/size][/color][/size]
random_id = "\x10\x20\x30\x40"
info_packet = "\xFF\xFF\xFF\x01"

1. Send the challenge request packet (challenge_request_packet = challenge_packet + random_id)

2. Receive response, parse the response: only take the numbers after 0@. Do some bitwise shifting:

challenge_response_packet = sprintf("%c%c%c%c", challenge_response >> 24, challenge_response >> 16, challenge_response >> 8, challenge_response >> 0)

3. Send the info request packet with the new calculated challenge response.

full_packet = base_packet + random_id + challenge_response_packet + info_packet

Determining if the server requires a challenge response:

If the response to the challenge request contains at the end: 0@0 then it does not require the challenge response.

Note for non-php users, e.g Ruby:

Negative numbers need to be converted first, php does this automatically, ruby does not:

[size=2][b]def[/b] handle_chr[/size][size=2][color=#800000][size=2][color=#800000]([/color][/size][/color][/size][size=2]number[/size][size=2][color=#800000][size=2][color=#800000])[/color][/size][/color][/size]
[size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]= [/color][/size][/color][/size][/b][size=2][color=#800000][size=2][color=#800000](([/color][/size][/color][/size][size=2]number [/size][size=2][color=#0000ff][size=2][color=#0000ff]% 256)+256) [/color][/size][/color][/size][b][size=2]if[/size][/b][size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]< [/color][/size][/color][/size][/b][size=2][color=#000080][size=2][color=#000080]0[/color][/size][/color][/size]
[size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]=[/color][/size][/color][/size][/b][size=2] number [/size][size=2][color=#0000ff][size=2][color=#0000ff]% 256 [/color][/size][/color][/size][b][size=2]if[/size][/b][size=2] number [/size][b][size=2][color=#008000][size=2][color=#008000]> [/color][/size][/color][/size][/b][size=2][color=#000080][size=2][color=#000080]255[/color][/size][/color][/size]
[size=2] number[/size]
[size=2][b]end[/b][/size]
[size=2][b][size=2][color=#008000][size=2][color=#008000][font=Consolas][size=2][font=Consolas][size=2]challenge_response [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]=[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] sprintf[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][color=#a31515][font=Consolas][size=2][color=#a31515][font=Consolas][size=2][color=#a31515]"%c%c%c%c"[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080],[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 24[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]),[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 16[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]),[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 8[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]),[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] handle_chr[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]([/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2]str [/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]>>[/color][/size][/font][/color][/size][/font][/color][/size][/font][font=Consolas][size=2][font=Consolas][size=2] 0[/size][/font][/size][/font][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080][font=Consolas][size=2][color=#008080]))[/color][/size][/font][/color][/size][/font][/color][/size][/font][/color][/size][/color][/size][/b]
[/size]

Edited July 30, 2011 by Sickboy

[nomad_man 10]

What is the latest version of BE server?

Also, i can not change the exchange the GameTracker queries :). It seems a lot of popular servers are having the same issue.

Edited July 29, 2011 by nomad_man

[sickboy 13]

See my updated post.

Latest BE seems to be v1.120 currently.