1

Topic: Apache coring and no login until reboot

==== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.6
- Linux/BSD distribution name and version: Ubuntu 16.04.2 LTS
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro? No
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.

log/apache2error.log:

[Fri Jul 28 11:27:56.478971 2017] [mpm_prefork:notice] [pid 977] AH00169: caught SIGTERM, shutting down
[Fri Jul 28 11:27:57.462387 2017] [ssl:warn] [pid 10276] AH01906: mail.xx.xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Fri Jul 28 11:27:57.501745 2017] [ssl:warn] [pid 10277] AH01906: mail.xx.xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Fri Jul 28 11:27:57.501881 2017] [wsgi:warn] [pid 10277] mod_wsgi: Compiled for Python/2.7.11.
[Fri Jul 28 11:27:57.501888 2017] [wsgi:warn] [pid 10277] mod_wsgi: Runtime using Python/2.7.12.
[Fri Jul 28 11:27:57.505920 2017] [mpm_prefork:notice] [pid 10277] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations
[Fri Jul 28 11:27:57.505945 2017] [core:notice] [pid 10277] AH00094: Command line: '/usr/sbin/apache2'
[Fri Jul 28 11:29:06.338969 2017] [mpm_prefork:notice] [pid 10277] AH00169: caught SIGTERM, shutting down
[Fri Jul 28 11:29:09.987179 2017] [ssl:warn] [pid 12148] AH01906: mail.xx.xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Fri Jul 28 11:29:10.022569 2017] [ssl:warn] [pid 12149] AH01906: mail.xx.xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Fri Jul 28 11:29:10.022716 2017] [wsgi:warn] [pid 12149] mod_wsgi: Compiled for Python/2.7.11.
[Fri Jul 28 11:29:10.022721 2017] [wsgi:warn] [pid 12149] mod_wsgi: Runtime using Python/2.7.12.
[Fri Jul 28 11:29:10.026371 2017] [mpm_prefork:notice] [pid 12149] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations
[Fri Jul 28 11:29:10.026396 2017] [core:notice] [pid 12149] AH00094: Command line: '/usr/sbin/apache2'
[Fri Jul 28 11:29:59.282819 2017] [mpm_prefork:notice] [pid 12149] AH00169: caught SIGTERM, shutting down
[Fri Jul 28 11:31:20.608258 2017] [ssl:warn] [pid 874] AH01906: mail.xx.xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Fri Jul 28 11:31:20.880025 2017] [ssl:warn] [pid 928] AH01906: mail.xx.xxx:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Fri Jul 28 11:31:20.884746 2017] [wsgi:warn] [pid 928] mod_wsgi: Compiled for Python/2.7.11.
[Fri Jul 28 11:31:20.884756 2017] [wsgi:warn] [pid 928] mod_wsgi: Runtime using Python/2.7.12.
[Fri Jul 28 11:31:20.888090 2017] [mpm_prefork:notice] [pid 928] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations
[Fri Jul 28 11:31:20.888110 2017] [core:notice] [pid 928] AH00094: Command line: '/usr/sbin/apache2'
[Fri Jul 28 11:33:29.046931 2017] [:error] [pid 1002] [client 91.196.50.33:35173] script '/var/www/html/testproxy.php' not found or unable to stat

====



Anyone have any clue as to what's going on here?   Things were fairly stable after install until the last week or so when it can't seem to stay up more than 24 hours or so.
What's with the testproxy.php script?

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: Apache coring and no login until reboot

It looks like, judging by the line;

server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)

that it is saying that you are using a CA certificate (usually bundled with your SSL certificates that you purchase), instead of the acutal certificate for your server. This would cause a failure, because your server can't use a CA certificate for this purpose...

I hope that this helps!

3

Re: Apache coring and no login until reboot

Excuse me, what's your issue? Login to which application?

4 (edited by iRedDale 2017-07-30 00:36:13)

Re: Apache coring and no login until reboot

ZhangHuangbin wrote:

Excuse me, what's your issue? Login to which application?

I'm aware of the certificate issue. the problem is that the "no login" means RoundCube doesn't present a login interface - presumably because of the apache core.

When attempting to login to the Roundcube  interface you get a "no response from server" message in the browser.
Oddly enough it is still up from the last reboot yesterday or I would post the browser message.

Edit -- when this happens, there are still apache processes alive on the server.

5

Re: Apache coring and no login until reboot

Any error in Apache log file?

6 (edited by iRedDale 2017-07-31 00:23:29)

Re: Apache coring and no login until reboot

ZhangHuangbin wrote:

Any error in Apache log file?

That IS the apache error log file in the original post...


The Roundcube interface isn't responding again with apache instances still in the process table, and the iRedAdmin interface doesn't respond, leading me to think it's an issue with Apache, but I really have no clue.

Now even a reboot won't bring it up and remote IMAP clients are being refused access.


$ ps -ef | grep apache
root       928     1  0 Jul28 ?        00:00:05 /usr/sbin/apache2 -k start
iredadm+ 20422   928  0 07:35 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 20423   928  0 07:35 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 20424   928  0 07:35 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 20426   928  0 07:35 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 20429   928  0 07:35 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 20431   928  0 07:35 ?        00:00:00 /usr/sbin/apache2 -k start
www-data 20610   928  0 07:41 ?        00:00:00 /usr/sbin/apache2 -k start

Firefox error is this --

Firefox can’t establish a connection to the server at mail.xx.xxx.

    The site could be temporarily unavailable or too busy. Try again in a few moments.
    If you are unable to load any pages, check your computer’s network connection.
    If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

Current apache access and error log :

Error log:

[Sun Jul 30 07:35:44.693154 2017] [mpm_prefork:notice] [pid 928] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g mod_wsgi/4.3.0 Python/2.7.12 configured -- resuming normal operations
[Sun Jul 30 07:35:44.693176 2017] [core:notice] [pid 928] AH00094: Command line: '/usr/sbin/apache2'
[Sun Jul 30 12:53:12.694493 2017] [core:error] [pid 20423] [client 84.108.1.124:57409] AH00126: Invalid URI in request GET login.cgi HTTP/1.0

access log for the same time:

# cat access.log
45.55.21.148 - - [30/Jul/2017:07:41:49 +0000] "GET / HTTP/1.1" 200 354 "-" "Mozilla/5.0 zgrab/0.x"
91.230.47.3 - - [30/Jul/2017:07:47:06 +0000] "GET / HTTP/1.0" 200 331 "-" "-"
144.48.243.115 - - [30/Jul/2017:07:57:46 +0000] "GET / HTTP/1.1" 200 312 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
74.82.47.2 - - [30/Jul/2017:09:00:29 +0000] "GET / HTTP/1.1" 200 1919 "-" "-"
45.79.223.204 - - [30/Jul/2017:10:09:02 +0000] "GET / HTTP/1.0" 400 0 "-" "-"
91.230.47.3 - - [30/Jul/2017:10:32:40 +0000] "GET / HTTP/1.0" 200 331 "-" "-"
91.230.47.3 - - [30/Jul/2017:11:34:16 +0000] "GET / HTTP/1.0" 200 331 "-" "-"
141.212.122.128 - - [30/Jul/2017:12:14:28 +0000] "GET /x HTTP/1.1" 400 0 "-" "Telesphoreo"
84.108.1.124 - - [30/Jul/2017:12:53:12 +0000] "GET login.cgi HTTP/1.0" 400 457 "-" "-"
91.230.47.3 - - [30/Jul/2017:13:37:19 +0000] "GET / HTTP/1.0" 200 331 "-" "-"
66.96.206.68 - - [30/Jul/2017:14:03:36 +0000] "GET / HTTP/1.1" 200 331 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.81 Safari/537.36"
91.230.47.3 - - [30/Jul/2017:14:37:21 +0000] "GET / HTTP/1.0" 200 331 "-" "-"
91.196.50.33 - - [30/Jul/2017:15:00:37 +0000] "GET http://testp3.pospr.waw.pl/testproxy.php HTTP/1.1" 404 425 "-" "Mozilla/5.0 (Windows NT 5.1; rv:32.0) Gecko/20100101 Firefox/31.0"
177.182.4.208 - - [30/Jul/2017:15:02:19 +0000] "GET /cgi/common.cgi HTTP/1.0" 404 439 "-" "Wget(linux)"
177.182.4.208 - - [30/Jul/2017:15:02:20 +0000] "GET /stssys.htm HTTP/1.0" 404 435 "-" "Wget(linux)"
177.182.4.208 - - [30/Jul/2017:15:02:20 +0000] "GET / HTTP/1.0" 200 331 "-" "Wget(linux)"
177.182.4.208 - - [30/Jul/2017:15:02:21 +0000] "POST /command.php HTTP/1.0" 404 436 "-" "Wget(linux)"

Looks like I'm getting vulnerability scans that are causing the login interface to cure?

7

Re: Apache coring and no login until reboot

Was your client IP address blocked in iptables?

8 (edited by iRedDale 2017-08-02 05:18:08)

Re: Apache coring and no login until reboot

Was your client IP address blocked in iptables?

No.

Edit -- wait - let me restate that -- not that I am aware of nor do I see any evidence of it in any logs.
But just because I don't see it doesn't mean it isn't true.

I am however able to ssh directly the mail server while it is not responding so I assume the IP has not been blocked...


It's down again.  Whatever is happening is taking the whole mail server down -- my clients inside my NAT's networwk can't connect, neither can my android phone on the carriers network. 

I do have one server on my internal network acting as a forwarder for all the internal hosts to send out monitoring script output to the iRedmail server.  I haven't yet been able to correlate the stop-functioning time with any undue load from forwards from that host yet.

9

Re: Apache coring and no login until reboot

iRedDale wrote:

I am however able to ssh directly the mail server while it is not responding so I assume the IP has not been blocked...

This assume is not correct.
Fail2ban bans mail services related ports, but not ssh. So you still need to check iptables.

10 (edited by iRedDale 2017-08-03 21:18:35)

Re: Apache coring and no login until reboot

OK, I see the domain where all of my clients reside is on the list as

REJECT     all    ---    mydomain.com     anywhere    reject-with icmp-port-unreachable 

All of my clients resolve to a different domain because of sitting behind a NAT'd firewall.
But it works sometimes, just not others...

I never really caught this because the fail2ban.log file is always 0 bytes.


Is the (right) fix to put the mail server in the same domain as all the clients?

Edit 2:
I removed the iptables line - what would be the most secure method to alleviate this?
Or determine perhaps why it's getting tripped now in the last couple of weeks when it had never been before.
I suspect some monitoring scripts that output mails that I want to send to that server are the issue - the mail terminates at that mail server, so is it just volume/frequency from the same source doing it?


In the long term -- the mail server sits in one domain and handles mail for that domain and another.What is the safest way to handle that sort of configuration?

11

Re: Apache coring and no login until reboot

iRedDale wrote:

I never really caught this because the fail2ban.log file is always 0 bytes.

If no log in /var/log/fail2ban/fail2ban.log, check /var/log/syslog instead.

12

Re: Apache coring and no login until reboot

ZhangHuangbin wrote:
iRedDale wrote:

I never really caught this because the fail2ban.log file is always 0 bytes.

If no log in /var/log/fail2ban/fail2ban.log, check /var/log/syslog instead.

I added the second domain in iredadmin. 
I'm still seeing "helo rejected: Host not founf; from=<root@seconddomain.com>" entries in the syslog, followed by "fail2ban.filter[1104]: INFO [postfix-iredmail] Found my ip address".

13 (edited by iRedDale 2017-08-04 00:17:20)

Re: Apache coring and no login until reboot

Now I'm seeing an "Error unbanning my.ip.add.ress".

No more Apache cores though -- so an easy way to bring the mail sever down is repeated attempts to send mail to it?

Edit 1:
The domain name (and not IP address) only ever appeared in the f2b-postfix chain.
It never appeared anywhere else like in the f2b-roundcube chain, but access to roundcube was still blocked.

Given that a number of hosts (not that many, I would expect the mail server to easily handle the volume, even from a single IP address) are submitting monitor and tracking emails in close sequence and all appear to be coming from the same IP address, I assume this is triggering the ban?  How do I adjust the ban values for just this ip or domain?

14

Re: Apache coring and no login until reboot

iRedDale wrote:
ZhangHuangbin wrote:
iRedDale wrote:

I never really caught this because the fail2ban.log file is always 0 bytes.

If no log in /var/log/fail2ban/fail2ban.log, check /var/log/syslog instead.

I added the second domain in iredadmin. 
I'm still seeing "helo rejected: Host not founf; from=<root@seconddomain.com>" entries in the syslog, followed by "fail2ban.filter[1104]: INFO [postfix-iredmail] Found my ip address".

I have tried adding the ip address and then the ip address and domain name to the whitelist -- "python wblist_admin.py --add --whitelist " and still the emails are rejected.  Waiting to see if the domain still gets added to an iptables REJECT rule.

"whitelisting" doesn't do what the name implies?

15

Re: Apache coring and no login until reboot

This is augering into unusability... I have even added the hoist name and IP address to the local hosts table so forward and reverse lookups work.     

Aug  3 20:37:45 mail postfix/smtpd[5476]: NOQUEUE: reject: RCPT from otherdomain.com[n.n.n.n]: 450 4.7.1 <www.otherdomain.com>: Helo command rejected: Host not found; from=<root@otherdomain.com> to=<dale@maindomain.us> proto=ESMTP helo=<www.otherdomain.com>
Aug  3 20:37:45 mail postfix/smtpd[5476]: disconnect from otherdomain.com[n.n.n.n] ehlo=1 mail=1 rcpt=0/1 data=0/1 rset=1 quit=1 commands=4/6
Aug  3 20:37:46 mail fail2ban.filter[1104]: INFO [postfix-iredmail] Found n.n.n.n
Aug  3 20:37:46 mail fail2ban.actions[1104]: NOTICE [postfix-iredmail] Unban n.n.n.n
Aug  3 20:37:46 mail fail2ban.action[1104]: ERROR iptables -w -D f2b-postfix -s n.n.n.n -j REJECT --reject-with icmp-port-unreachable -- stdout: b''
Aug  3 20:37:46 mail fail2ban.action[1104]: ERROR iptables -w -D f2b-postfix -s n.n.n.n -j REJECT --reject-with icmp-port-unreachable -- stderr: b'iptables: No chain/target/match by that name.\n'
Aug  3 20:37:46 mail fail2ban.action[1104]: ERROR iptables -w -D f2b-postfix -s n.n.n.n -j REJECT --reject-with icmp-port-unreachable -- returned 1
Aug  3 20:37:46 mail fail2ban.actions[1104]: ERROR Failed to execute unban jail 'postfix-iredmail' action 'iptables-multiport' info '{'failures': 5, 'matches': 'Aug  3 19:22:45 mail postfix/smtpd[5102]: NOQUEUE: reject: RCPT from

16 (edited by RikuS 2017-08-04 05:53:10)

Re: Apache coring and no login until reboot

First of all, message

Helo command rejected: Host not found; ... helo=<www.otherdomain.com>

means that DNS query cannot resolve hostname www.otherdomain.com, so this exact domain needs to have a working DNS A record pointing to an IP address.

Second, white and blacklists are based on sender email address and IP address, not HELO id. So whitelisting would not affect HELO errors.

Third, you should turn off fail2ban while setting up new domains, migrating or whatever testing that involves creating new domains or mailboxes. You can also adjust fail2ban limits to your needs.

You're most likely having issues with your network, DNS and/or with other configs that are not related to iredmail. You also need to fix SSL issues, they possibly lead to other problems. Iredmail works extremely well on a clean server with regular connections and valid DNS records.

17

Re: Apache coring and no login until reboot

RikuS wrote:

First of all, message

Helo command rejected: Host not found; ... helo=<www.otherdomain.com>

means that DNS query cannot resolve hostname www.otherdomain.com, so this exact domain needs to have a working DNS A record pointing to an IP address.

Second, white and blacklists are based on sender email address and IP address, not HELO id. So whitelisting would not affect HELO errors.

Third, you should turn off fail2ban while setting up new domains, migrating or whatever testing that involves creating new domains or mailboxes. You can also adjust fail2ban limits to your needs.

You're most likely having issues with your network, DNS and/or with other configs that are not related to iredmail. You also need to fix SSL issues, they possibly lead to other problems. Iredmail works extremely well on a clean server with regular connections and valid DNS records.

I'm sure whatever the issue is, it exists because I caused. Somehow.
I only started messing with the whitelists and adding domains because it stopped working. 
I found the Apache core in the logs and things have gone downhill from there.
The host/domain that keeps getting banned has forward and reverse lookups working on the mail server. 

I'd turn off fail2ban, but I am a bit paranoid -- this installation is in a vps outside of my direct control and I see constant probes - failures all so far, but not having a lot of experience with mail servers ...   anyway.

The initial setup and the transfer of the handful of mail accounts I have - less than about 25GB of mail altogether - when far easier than I thought it would.  It was so easy I'm still a bit worried I didn't do it right.
Then a week or so ago the mail forwards out of my LAN started failing and the mail server became inaccessible.

18

Re: Apache coring and no login until reboot

iRedDale wrote:

The domain name (and not IP address) only ever appeared in the f2b-postfix chain.
It never appeared anywhere else like in the f2b-roundcube chain, but access to roundcube was still blocked.

Given that a number of hosts (not that many, I would expect the mail server to easily handle the volume, even from a single IP address) are submitting monitor and tracking emails in close sequence and all appear to be coming from the same IP address, I assume this is triggering the ban?  How do I adjust the ban values for just this ip or domain?

pls. try this:
A).Postfix
/etc/postfix/main.cf

smtpd_sender_restrictions = 
    reject_unknown_sender_domain
    reject_non_fqdn_sender
    reject_unlisted_sender
    permit_mynetworks
    permit_sasl_authenticated
    check_sender_access pcre:/etc/postfix/sender_access.pcre         

[...OTHER RESTRICTIONS HERE...]

/etc/postfix/sender_access.pcre

/root\@mx\.second\.com/   ok
/\@mail\.mydomain\.com/    ok
/10\.xx\.xx\.0\/24/ ok
/23\.xx\.xx\.5/     ok
/23\.xx\.xx\.6/     ok

B). iRedAPD
# /opt/iredapd/settings.py

plugins = ["reject_null_sender", "reject_sender_login_mismatch", "greylisting", "throttle", "amavisd_wblist", "ldap_maillist_access_policy"]

# bypass sender email address
     
ALLOWED_FORGED_SENDERS = ['root@web.mydomain.com , root@ns1.mydomain.com , @mail.second.com ']


ALLOWED_LOGIN_MISMATCH_SENDERS = ['postmaster@mail.mydomain.com' ,'@mx.second.com']

# bypass sender IP address
MYNETWORKS = ['203.xx.xx.5', '23.xx.xx.6', '23.xx.xx.6','10.xx.xx.0/24', 219.xx.xx.6, ]

C). fail2ban ignoreip   
/etc/fail2ban/jail.local

ignoreip    = 127.0.0.1 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 23.xx.xx.5 23.xx.xx.6  219.xx.xx.6

d) system reboot