Targeted DDoS attacks of the 15/04/2021

[Nervous]

As usual with every security events impacting the server, I always try to make a write-up on what happened to give you full transparency over the issue. You'll find below a timeline (CET) of the events :

 

11:15 : OVH alert about a DDoS impacting our main game server. Impact : Players cannot connect anymore with the following error : 

 

11:20 : Our internal monitoring triggers an alert regarding an unusual decrease of player compared to the average, I arrive on computer and check discord and notice this error.

11:25 : Initial thought is that the DDoS attack was targeting Rage services directly, and OVH decided to fully block Rage's ports until their mitigation blocks the attack.

11:30 : I notice that Eclipse Roleplay are also impacted and immediately create a group discussion with NBDY (their owner) to investigate the issue

12:00 : OVH mitigation stops, I decide to restart the server to see if the ports would be available after a restart and to allow everyone to login : big fail, no one can connect.

12:15 : Since OVH mitigation wasn't responsible for the error, we decide to fully analyze the network traffic and discover that we're being attacked on two front :

• 1) Thousands of Amazon AWS EC2 instances, Google Cloud and dozens of other providers are massively DDoSing Rage's 22006 port which is used to download files by downloading the files on their instances and thus completely block the traffic.
  • 
• 2) A denial of service attack is also ongoing on Rage's 22005 game port, this time with UDP requests in order to spam the gameserver and ensure that players cannot properly connect to it / play.
• The analysis confirms that this is an advanced targeted attack by someone who knew perfectly how to attack RageMP and prepared its attack with cloud instances to download files.
• RageMP team is also immediately contacted for support on the issue.

14:00 : We blacklist every IP ranges of known cloud providers from being able to access port 22006 with no luck : the amount of random hosts is still too high and players cannot download files at all.

15:00 : Rage's main developer joins our discord group and gives us multiple advices in how to handle it.

15:30 : 16 servers are purchased from OVH to split the incoming traffic and redirect the traffic in case one server goes down.

16:30 : The new mitigation infrastructure is deployed, along with an UDP packet filtering to only whitelist valide UDP rage packets and thus prevent a denial of service on the gameserver service itself. Players can finally start to login slowly. A rage security setting is also deployed on port 22006 to blacklist DDoSing IP.

18:00 : Players who do not need to download any files can properly login, but others are stuck in "Loading server resources" phase, the download is ultra slow due to the security measures, thus no one can really download the files.

21:00 : We disable the Rage's security setting on file download and recode our own thanks to Rage's tips, the game server is restarted and everyone can finally login properly / download files.

21:30 : The DDoS is still ongoing but no impacts at all anymore for players.

7:00 am the next day : Finally caught up with all my IRL work I had to put aside for few hours.

8:00 am: Back to IRL work of the next day!

 

This has been the most advanced DDoS attack we've received since GTMP's era in 2017 where we fought for months with an attack. I can't thank Rage's owner enough for all the help which allowed me to fully block the attack. With this new mitigation infrastructure we're ready to face major future attacks that might happen due to new competitors.

[Yoak]

Thanks to yourself and the team for managing to get the server back up 💪

[Shvag]

Theme song for this thread:

 

 

[i dont wanna od in LA]

Great job. Really glad to see such a professional approach and dedication in order to tackle the issues.

[orca112]

Thanks for the in-sight on the events and what happened "up there" at the times.

[Euphoria]

I say we band together, purchase an orbital beacon and BLAST them away.

 

In reality, thanks for being on it so quickly, you guys rock.

[Lomadias]

Good work, appreciate it!

[Nexxt]

Good job, how pathetic if you need a DDoS-attack to weaken your competitors and even fail.... 🙂 

[knppel]

 

[Daylong]

doing amazing work Nervous, thank you