A SYN flood attack exploits one of the properties of the TCP/IP protocol: by sending SYN requests, and then never following up with an ACK, this leaves the server using one network "slot" and waiting for the other side for some time. Doing this many times ties up network resources and the server becomes unresponsive.
Symptom: web site visitor
The symptom to an end user, a web site visitor, is that a site takes a long time to load, or loads some elements of a page but not others. The general impression is that the site is slow or down.
Symptom: on the server
From a server administrator perspective, you will not see high CPU utilization, nor memory utilization. The server will be lightly loaded when this happens.
In order to check whether your server is under a SYN flood attack, you use the wget command times how much it takes for your site to load, from the same box (to eliminate the network as a cause).
For a normal site, you get something like this instantly:
$ time wget -O /dev/null www.example.com Resolving www.example.com... 126.96.36.199 Connecting to www.example.com|188.8.131.52|:80... connected. HTTP request sent, awaiting response... 200 OK ... real 0m0.018s
The request takes just 18 milliseconds, which is quite normal for a well configured server.
Compare that to when a SYN flood attack was active:
$ time wget -O /dev/null www.example.com Resolving www.example.com... 184.108.40.206 Connecting to www.example.com|220.127.116.11|:80... connected.
And this is where things hang for a long time ... The server takes a very long time to open a network socket.
Then after a long time, you would get this:
HTTP request sent, awaiting response... 200 OK ... real 0m45.002s
Ouch! 45 seconds for the home page to load locally!
This can vary from a few seconds, to several minutes, and running it several times will give different results, but all or most of them will be over 1 or 2 seconds.
Then when you look at netstat's output, you find that there are lots of connections in the SYN_RECV state:
netstat -tuna | grep :80 | grep SYN_RECV
The output will look like this:
tcp 0 0 18.104.22.168:80 22.214.171.124:1609 SYN_RECV tcp 0 0 126.96.36.199:80 188.8.131.52:1723 SYN_RECV tcp 0 0 184.108.40.206:80 220.127.116.11:4988 SYN_RECV tcp 0 0 18.104.22.168:80 22.214.171.124:1724 SYN_RECV tcp 0 0 126.96.36.199:80 188.8.131.52:1727 SYN_RECV tcp 0 0 184.108.40.206:80 220.127.116.11:1733 SYN_RECV tcp 0 0 18.104.22.168:80 22.214.171.124:3337 SYN_RECV tcp 0 0 126.96.36.199:80 188.8.131.52:1753 SYN_RECV tcp 0 0 184.108.40.206:80 220.127.116.11:1811 SYN_RECV tcp 0 0 18.104.22.168:80 22.214.171.124:1821 SYN_RECV tcp 0 0 126.96.36.199:80 188.8.131.52:1831 SYN_RECV tcp 0 0 184.108.40.206:80 220.127.116.11:52142 SYN_RECV tcp 0 0 18.104.22.168:80 22.214.171.124:50819 SYN_RECV tcp 0 0 126.96.36.199:80 188.8.131.52:52865 SYN_RECV
You will see a lot of SYN requests from the same addresses. Do not bother with tracing what this address is, because it is easily faked, and the attacker is probably using fake addresses.
So, how do you solve this?
The solution varies, but the best one is to enable SYN cookies on your load balancer or the server itself.
To enable that on a current Linux kernel, you enter the following command:
sysctl -w net.ipv4.tcp_syncookies=1
And then add the following line to the /etc/sysctl.conf file to make make it persist across reboots:
net.ipv4.tcp_syncookies = 1
You may optionally want to increase the size of the SYN backlog queue as well, from a default of 1024, to 2048, using the following command:
sysctl -w net.ipv4.tcp_max_syn_backlog=2048
And you add this to /etc/sysctl.conf:
net.ipv4.tcp_max_syn_backlog = 2048
After you do the above, SYN Flood attacks will continue, but it will not affect the server negatively. You will see a message like this in your logs:
[1116377.589736] possible SYN flooding on port 80. Sending cookies. [1116439.567828] possible SYN flooding on port 80. Sending cookies. [1116500.631623] possible SYN flooding on port 80. Sending cookies.
But there will be no ill effect from it ...
And there you have it, attackers will still send SYN requests, but the server will remain responsive ...
For more reading on this issue, check the following links:
- RFC 4987: TCP SYN Flooding Attacks and Common Mitigations
- Hardening the TCP/IP stack against SYN flood attacks
- IBM X-Force: Syn Flood Denial Of Service
- Ross Oliver presentation (PDF): Countering SYN flood Denial of service attacks, dated, and hence doesn't cover SYN cookies, but describes the problem well.