Detecting and preventing SYN Flood attacks on web servers running Linux

The other day I helped a client deal with a SYN flood denial of service attack. This article describes the symptoms, diagnosis and solution from a Linux server point of view.

Definition

A SYN flood attack exploits one of the properties of the TCP/IP protocol: by sending SYN requests, and then never following up with an ACK, this leaves the server using one network "slot" and waiting for the other side for some time. Doing this many times ties up network resources and the server becomes unresponsive.

Symptom: web site visitor

The symptom to an end user, a web site visitor, is that a site takes a long time to load, or loads some elements of a page but not others. The general impression is that the site is slow or down.

Symptom: on the server

From a server administrator perspective, you will not see high CPU utilization, nor memory utilization. The server will be lightly loaded when this happens.

In order to check whether your server is under a SYN flood attack, you use the wget command times how much it takes for your site to load, from the same box (to eliminate the network as a cause).

For a normal site, you get something like this instantly:

$ time wget -O /dev/null www.example.com
Resolving www.example.com... 1.1.1.1
Connecting to www.example.com|1.1.1.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
...
real    0m0.018s

The request takes just 18 milliseconds, which is quite normal for a well configured server.

Compare that to when a SYN flood attack was active:

$ time wget -O /dev/null www.example.com
Resolving www.example.com... 1.1.1.1
Connecting to www.example.com|1.1.1.1|:80... connected.

And this is where things hang for a long time ... The server takes a very long time to open a network socket.

Then after a long time, you would get this:

HTTP request sent, awaiting response... 200 OK
...
real 0m45.002s

Ouch! 45 seconds for the home page to load locally!

This can vary from a few seconds, to several minutes, and running it several times will give different results, but all or most of them will be over 1 or 2 seconds.

Diagnosis

Then when you look at netstat's output, you find that there are lots of connections in the SYN_RECV state:

netstat -tuna | grep :80 | grep SYN_RECV

The output will look like this:

tcp  0  0 1.1.1.1:80  70.56.83.204:1609       SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1723            SYN_RECV
tcp  0  0 1.1.1.1:80  209.112.192.126:4988    SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1724            SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1727            SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1733            SYN_RECV
tcp  0  0 1.1.1.1:80  24.158.121.0:3337       SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1753            SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1811            SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1821            SYN_RECV
tcp  0  0 1.1.1.1:80  2.2.2.2:1831            SYN_RECV
tcp  0  0 1.1.1.1:80  24.7.27.61:52142        SYN_RECV
tcp  0  0 1.1.1.1:80  207.118.0.58:50819      SYN_RECV
tcp  0  0 1.1.1.1:80  115.64.40.38:52865      SYN_RECV

You will see a lot of SYN requests from the same addresses. Do not bother with tracing what this address is, because it is easily faked, and the attacker is probably using fake addresses.

Solution

So, how do you solve this?

The solution varies, but the best one is to enable SYN cookies on your load balancer or the server itself.

To enable that on a current Linux kernel, you enter the following command:

sysctl -w net.ipv4.tcp_syncookies=1

And then add the following line to the /etc/sysctl.conf file to make make it persist across reboots:

net.ipv4.tcp_syncookies = 1

You may optionally want to increase the size of the SYN backlog queue as well, from a default of 1024, to 2048, using the following command:

sysctl -w net.ipv4.tcp_max_syn_backlog=2048

And you add this to /etc/sysctl.conf:

net.ipv4.tcp_max_syn_backlog = 2048

Results

After you do the above, SYN Flood attacks will continue, but it will not affect the server negatively. You will see a message like this in your logs:

[1116377.589736] possible SYN flooding on port 80. Sending cookies.
[1116439.567828] possible SYN flooding on port 80. Sending cookies.
[1116500.631623] possible SYN flooding on port 80. Sending cookies.

But there will be no ill effect from it ...

And there you have it, attackers will still send SYN requests, but the server will remain responsive ...

Resources

For more reading on this issue, check the following links:

• RFC 4987: TCP SYN Flooding Attacks and Common Mitigations
• Hardening the TCP/IP stack against SYN flood attacks
• IBM X-Force: Syn Flood Denial Of Service
• Ross Oliver presentation (PDF): Countering SYN flood Denial of service attacks, dated, and hence doesn't cover SYN cookies, but describes the problem well.