1 (edited by grouchy 2013-02-01 04:44:53)

Topic: Large I/O on server (DDoS or whatever...)

- iRedMail version: from 0.7.0 updated to 0.8.3
- Store mail accounts in which backend (MySQL):
- Linux/BSD distribution name and version: Debian Squeeze
====

Today I was faced with strange problem. Catastrophic response from my server.

iostat showed me a large I/O. Tps on disk (600-800) and avg-cpu %idle parameter high CPU usage (0.03). That situation was constantly for hours.

$ iostat
avg-cpu:  %user      %nice    %system    %iowait     %steal      %idle
                   42.93       0.00       42.35            14.69       0.00          0.03

Device:            tps         Blk_read/s     Blk_wrtn/s        Blk_read        Blk_wrtn
sda              727.68        3763.78           8132.43         17387617     37569536

I checked incoming sources to my server and was terrified. 50+ IPs from around the world sending requests. (it is impossible to be a visitors, because the web presentation is from small country with local domain and have 5-15 visits per day)

$ netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

      1 108.23.241.78
      1 113.208.32.112
      1 115.114.58.8
      1 12.154.55.40
      1 12.34.246.167
      1 130.206.18.132
      1 138.141.100.73
      1 158.109.36.179
      1 174.132.149.218
      1 193.222.78.6
      1 193.222.84.70
      1 193.38.113.62
      1 195.186.227.50
      1 195.186.99.50
      1 199.250.129.2
      1 200.230.71.10
      1 200.230.71.55
      1 205.166.218.186
      1 205.178.149.7
      1 205.188.100.58
      1 205.188.159.42
      1 205.188.190.2
      1 205.188.59.193
      1 207.115.17.26
      1 207.172.157.20
      1 207.46.163.30
      1 208.36.123.165
      1 208.70.88.10
      1 208.91.197.128
      1 209.145.111.61
      1 209.33.205.5
      1 212.166.70.250
      1 212.170.233.86
      1 212.59.199.125
      1 213.221.143.235
      1 213.46.255.200
      1 213.55.128.8
      1 213.55.128.9
      1 216.99.131.15
      1 216.99.131.16
      1 217.114.0.53
      1 217.76.128.34
      1 220.181.15.194
      1 50.19.104.123
      1 62.14.4.83
      1 62.201.1.2
      1 64.12.90.34
      1 64.12.90.65
      1 64.12.90.97
      1 64.12.90.98
      1 65.205.78.58
      1 65.55.39.12
      1 65.55.92.136
      1 66.232.205.163
      1 66.246.235.44
      1 66.40.20.29
      1 67.220.48.93
      1 68.232.135.213
      1 69.64.147.249
      1 72.167.238.201
      1 72.9.240.201
      1 74.54.41.162
      1 80.64.32.18
      1 80.67.172.24
      1 82.98.86.161
      1 82.98.86.167
      1 89.108.104.4
      1 94.189.240.125
      1 98.139.214.154
      1 Address
      1 servers)
      2
      2 12.102.252.75
      2 205.188.103.1
      2 205.188.103.2
      2 205.188.146.193
      2 205.188.156.193
      2 209.202.254.14
      2 212.40.2.32
      2 212.82.111.207
      2 213.133.98.98
      2 64.12.137.162
      2 64.12.90.33
      2 64.38.116.12
      2 71.74.56.244
      2 77.238.177.9
      3 205.188.155.110
      3 213.133.100.100
      3 64.12.137.161
      3 65.54.188.110
      3 65.55.37.104
      3 65.55.37.120
      3 65.55.37.88
      3 65.55.92.152
      3 65.55.92.184
      3 71.74.56.243
      3 77.238.184.241
      3 80.57.35.169
      4 204.127.208.75
      4 205.188.146.194
      4 64.12.139.193
      4 65.54.188.126
      4 65.55.37.72
      5 65.54.188.72
      5 65.54.188.94
      6 65.55.92.168
      8 68.1.17.3
     11 61.19.251.167
     13 115.78.225.198
     13 68.6.19.3
     13 98.138.206.39
     14 173.194.70.27
     14 66.216.129.171
     16 115.115.125.50
     18 98.136.217.192
     23 190.69.78.140
     57 91.119.178.10

So after I tried to stop apache, turn off server for a while, drop packets from couple IPs:

$ /sbin/iptables -I INPUT -s x.x.x.x -p tcp -j DROP
$ /sbin/iptables -I INPUT -s x.x.x.x -p udp -j DROP

result was the same.

I installed iRedMail 15 days ago, so I decided to try uninstall and to my surprise that solved a problem tps on disk (2.07) and avg-cpu %idle parameter high CPU usage (97.81).

Really liked iRedMail solution so I would like to consult with someone on this forum about which configuration of iRedMail could cause this situation at all.

2

Re: Large I/O on server (DDoS or whatever...)

No idea why it happened on your server since there's no many info in your description. Here are my questions:

- Was it caused by email deliveries on local server?
- Do you have Fail2ban installed? Any info in its log file?

3

Re: Large I/O on server (DDoS or whatever...)

I wanted to know if someone experienced the same problem. iRedMail is uninstalled from the server now and I will install it again on another one these days. Also I hope that I will make it run in full capability this time.

Thank you for assistance anyway.