1 Topic by mac

  • mac
  • Member
  • Offline
  • Registered: 2014-10-28
  • Posts: 15

Topic: iRedAPD Whitelist not working / still graylisting?

==== Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.6
- Linux/BSD distribution name and version: CentOS 6.5
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): LDAP / Active Directory
- Web server (Apache or Nginx): Apache
- Manage mail accounts with iRedAdmin-Pro? No (Active Directory)
- Related log if you're reporting an issue:
====

Under the old cluebringer system, adding an e-mail or domain to the whitelist in the iRedAdmin-Pro web interface would indeed cause e-mail from that sender to be whitelisted (no more graylisting).  Ever since we upgraded to iRedAPD, it seems that entries in the web interface have literally no effect.

Currently seeing this in my log files:

May  5 12:11:51 mail postfix/smtpd[6312]: NOQUEUE: reject: RCPT from smtprelay0208.hostedemail.com[216.40.44.208]: 451 4.7.1 <sales@mydomain.com>: Recipient address rejected: Intentional policy rejection, please try again later; from=<pete@customer.net> to=<sales@mydomain.com> proto=ESMTP helo=<smtprelay.hostedemail.com>

May  5 12:31:46 mail postfix/smtpd[7327]: NOQUEUE: reject: RCPT from smtprelay0178.hostedemail.com[216.40.44.178]: 451 4.7.1 <sales@mydomain.com>: Recipient address rejected: Intentional policy rejection, please try again later; from=<pete@customer.net> to=<sales@mydomain.com> proto=ESMTP helo=<smtprelay.hostedemail.com>

I have explicitly whitelisted @customer.net in iRedAPD, but it is being ignored.  To make matters worse, the customer's mail provider (hostedemail.com) sends from a wide range of SMTP servers, so the sending IP address is always different, making it impossible to ever get e-mail past the graylisting service.

My question is, why is my whitelist not taking?  I can't afford to be missing e-mails from customers. smile

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: iRedAPD Whitelist not working / still graylisting?

*) Did you migrate greylisting settings after migrating from Cluebringer?
*) Show us your greylisting settings with commands below:

cd /opt/iredapd/tools/
python greylisting_admin.py --list

*) Turn on debug mode in iRedAPD and send testing email to reproduce this issue. Extract related log from iRedAPD log file and paste here so that others can help troubleshoot.

3 Reply by mac

  • mac
  • Member
  • Offline
  • Registered: 2014-10-28
  • Posts: 15

Re: iRedAPD Whitelist not working / still graylisting?

*) Did you migrate greylisting settings after migrating from Cluebringer?

I'm not sure, honestly; however, the example I posted is from a whitelist I added after the upgrade.

My graylisting settings look pretty minimal:

[root@mail tools]# python greylisting_admin.py --list
Status   Sender                             -> Local Account                 
------------------------------------------------------------------------------
enabled  @. (anyone)                        -> @. (anyone)                   
[root@mail tools]#

Anticipating this request, I had previously turned on debugging.  Here is the relevant debug log:

2017-05-05 09:51:46 DEBUG smtp session: request=smtpd_access_policy
2017-05-05 09:51:46 DEBUG smtp session: protocol_state=RCPT
2017-05-05 09:51:46 DEBUG smtp session: protocol_name=ESMTP
2017-05-05 09:51:46 DEBUG smtp session: client_address=216.40.44.103
2017-05-05 09:51:46 DEBUG smtp session: client_name=smtprelay0103.hostedemail.com
2017-05-05 09:51:46 DEBUG smtp session: reverse_client_name=smtprelay0103.hostedemail.com
2017-05-05 09:51:46 DEBUG smtp session: helo_name=smtprelay.hostedemail.com
2017-05-05 09:51:46 DEBUG smtp session: sender=pete@customer.net
2017-05-05 09:51:46 DEBUG smtp session: recipient=sales@mydomain.com
2017-05-05 09:51:46 DEBUG smtp session: recipient_count=0
2017-05-05 09:51:46 DEBUG smtp session: queue_id=
2017-05-05 09:51:46 DEBUG smtp session: instance=e63.590cada2.dbc2.0
2017-05-05 09:51:46 DEBUG smtp session: size=50483
2017-05-05 09:51:46 DEBUG smtp session: etrn_domain=
2017-05-05 09:51:46 DEBUG smtp session: stress=
2017-05-05 09:51:46 DEBUG smtp session: sasl_method=
2017-05-05 09:51:46 DEBUG smtp session: sasl_username=
2017-05-05 09:51:46 DEBUG smtp session: sasl_sender=
2017-05-05 09:51:46 DEBUG smtp session: ccert_subject=
2017-05-05 09:51:46 DEBUG smtp session: ccert_issuer=
2017-05-05 09:51:46 DEBUG smtp session: ccert_fingerprint=
2017-05-05 09:51:46 DEBUG smtp session: ccert_pubkey_fingerprint=
2017-05-05 09:51:46 DEBUG smtp session: encryption_protocol=TLSv1.2
2017-05-05 09:51:46 DEBUG smtp session: encryption_cipher=ECDHE-RSA-AES256-GCM-SHA384
2017-05-05 09:51:46 DEBUG smtp session: encryption_keysize=256
2017-05-05 09:51:46 DEBUG LDAP connection initialied success.
2017-05-05 09:51:46 DEBUG LDAP bind success.
2017-05-05 09:51:46 DEBUG --> Apply plugin: reject_null_sender
2017-05-05 09:51:46 DEBUG <-- Result: DUNNO
2017-05-05 09:51:46 DEBUG --> Apply plugin: greylisting
2017-05-05 09:51:46 DEBUG [SQL] Query greylisting whitelists:
SELECT id, sender, comment
               FROM greylisting_whitelists
              WHERE account IN ('sales@mydomain.com', '@mydomain.com', '@.')
2017-05-05 09:51:46 DEBUG No whitelist found.
2017-05-05 09:51:46 DEBUG [SQL] query greylisting settings:
SELECT id, account, sender, sender_priority, active
               FROM greylisting
              WHERE account IN ('sales@mydomain.com', '@mydomain.com', '@.')
              ORDER BY priority DESC, sender_priority DESC
2017-05-05 09:51:46 DEBUG [SQL] query result: [(1L, '@.', '@.', 0, 1)]
2017-05-05 09:51:46 DEBUG Greylisting should be applied according to SQL record: (id=1, account='@.', sender='@.')
2017-05-05 09:51:46 DEBUG [SQL] check whether client address (216.40.44.103) passed greylisting:
SELECT id
               FROM greylisting_tracking
              WHERE client_address='216.40.44.103'
                    AND passed=1
              LIMIT 1
2017-05-05 09:51:46 DEBUG Client address (216.40.44.103) didn't pass greylisting.
2017-05-05 09:51:46 DEBUG [SQL] query greylisting tracking:
SELECT init_time, blocked_count, block_expired, record_expired
               FROM greylisting_tracking
              WHERE sender='pete@customer.net'
                    AND recipient='sales@mydomain.com'
                    AND client_address='216.40.44.103'
              LIMIT 1
2017-05-05 09:51:46 INFO [216.40.44.103] Client has not been seen before, greylisted.
2017-05-05 09:51:46 DEBUG [SQL] New tracking:
INSERT INTO greylisting_tracking (sender, sender_domain,
                                                   recipient, rcpt_domain,
                                                   client_address,
                                                   init_time,
                                                   block_expired, record_expired,
                                                   blocked_count)
                      VALUES ('pete@customer.net', 'customer.net', 'sales@mydomain.com', 'mydomain.com', '216.40.44.103', 1494003106, 1494004006, 1494089506, 1)
2017-05-05 09:51:46 DEBUG <-- Result: 451 4.7.1 Intentional policy rejection, please try again later
2017-05-05 09:51:46 DEBUG Session ended.
2017-05-05 09:51:46 INFO 216.40.44.103 RCPT, pete@customer.net -> sales@mydomain.com, 451 4.7.1 Intentional policy rejection, please try again later [0.0099s]
2017-05-05 09:51:46 DEBUG Close LDAP connection.

I have a bunch of similar entries, as hostedemail.com sends from multiple IP addresses.

The admin log does show that I whitelisted the @customer.net at 9:42 on May 5th, so the system should have acted on that whitelist.  I don't see anything in the logs where it checks against any whitelist other than the standard client / mailto graylist.

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: iRedAPD Whitelist not working / still graylisting?

mac wrote:

My graylisting settings look pretty minimal:

According to the output, you didn't whitelist the sender at all.

mac wrote:

2017-05-05 09:51:46 DEBUG --> Apply plugin: greylisting
2017-05-05 09:51:46 DEBUG [SQL] Query greylisting whitelists:
SELECT id, sender, comment
               FROM greylisting_whitelists
              WHERE account IN ('sales@mydomain.com', '@mydomain.com', '@.')
2017-05-05 09:51:46 DEBUG No whitelist found.

no whitelist found.

The question is: How did you whitelist this sender?

We have tutorial for you: http://www.iredmail.org/docs/manage.iredapd.html

5 Reply by mac

  • mac
  • Member
  • Offline
  • Registered: 2014-10-28
  • Posts: 15

Re: iRedAPD Whitelist not working / still graylisting?

ZhangHuangbin wrote:

According to the output, you didn't whitelist the sender at all.

That's odd, because in iRedAdmin Pro, at this URL:
https://mail.mydomain.com/iredadmin/system/wblist

... it shows all my whitelist and blacklist entries, including the specific @customer.net one.  In addition, the admin log at https://mail.mydomain.com/iredadmin/system/log has the following entry:

2017-05-05 16:42:07     postmaster@mydomain.com     192.168.254.21     Add whitelists for @.: @customer.net.
ZhangHuangbin wrote:

The question is: How did you whitelist this sender?

I'm using the iRedAdmin Pro (LDAP) web interface at https://mail.mydomain.com/iredadmin/create/wblist

ZhangHuangbin wrote:

We have tutorial for you: http://www.iredmail.org/docs/manage.iredapd.html

This is the first I've seen that tutorial, which gives command-line instructions as opposed to iRedAdmin Pro web interface instructions.  So that raises a few questions:

1. Is the/opt/iredapd/tools/wblist_admin.py script the only way to manage white/blacklists?

2. If the answer to #1 is "YES", then why does the web interface still exist?

3. Where is the url at /iredadmin/system/wblist reading its data from?

4. Where is greylisting_admin.py reading its data from?  (Since that must be different than #3)

It's becoming clear that there are two different interfaces for whitelisting, that are apparently storing data in two different locations, only one of which is being used by the iredapd checker.

Thoughts?

6 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: iRedAPD Whitelist not working / still graylisting?

There're 2 kinds of whitelists:

1) white/blacklisting based on sender email address, and server IP address. This is the URL: /iredadmin/system/wblist
2) You're trying to whitelist a sender for greylisting service. This is the URL: /iredadmin/system/greylisting

And 2) is what you're looking for.

7 Reply by mac

  • mac
  • Member
  • Offline
  • Registered: 2014-10-28
  • Posts: 15

Re: iRedAPD Whitelist not working / still graylisting?

ZhangHuangbin wrote:

There're 2 kinds of whitelists:

1) white/blacklisting based on sender email address, and server IP address. This is the URL: /iredadmin/system/wblist
2) You're trying to whitelist a sender for greylisting service. This is the URL: /iredadmin/system/greylisting

And 2) is what you're looking for.

Thank you, I will look into that.  In the meantime, can you give more detail as to exactly what the purpose is for the white/blacklist in #1 above (the one I had been trying to use before)?  What, exactly, is it "whitelisting" (or blacklisting) against?

Would it not make more sense to have just one master white/blacklist?

8 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: iRedAPD Whitelist not working / still graylisting?

mac wrote:

can you give more detail as to exactly what the purpose is for the white/blacklist in #1 above (the one I had been trying to use before)?

There're some data we can use for white/blacklisting, for example, sender email address (and sender domain name), sender IP address, reverse DNS name of sender server IP address, sender HELO hostname, etc. They have different syntax/format, and got in different phases of mail flow, that's why we have to use different white/blacklists.

9 Reply by mac

  • mac
  • Member
  • Offline
  • Registered: 2014-10-28
  • Posts: 15

Re: iRedAPD Whitelist not working / still graylisting?

OK, following up on this...

In my iredadmin pro, you mentioned these two links:

ZhangHuangbin wrote:

There're 2 kinds of whitelists:

1) white/blacklisting based on sender email address, and server IP address. This is the URL: /iredadmin/system/wblist
2) You're trying to whitelist a sender for greylisting service. This is the URL: /iredadmin/system/greylisting

And 2) is what you're looking for.

When i pull up both URLs, I find entries for customer.net on both pages.  More precisely, /iredadmin/system/wblist has @customer.net under the "For Inbound Mails / Whitelisted senders", and /iredadmin/system/greylisting has "customer.net" in the box for "Do not apply greylisting on emails sent from domains listed below".

However, it is STILL graylisting pete@customer.net!

2017-05-23 13:34:12 DEBUG smtp session: request=smtpd_access_policy
2017-05-23 13:34:12 DEBUG smtp session: protocol_state=RCPT
2017-05-23 13:34:12 DEBUG smtp session: protocol_name=ESMTP
2017-05-23 13:34:12 DEBUG smtp session: client_address=216.40.44.228
2017-05-23 13:34:12 DEBUG smtp session: client_name=smtprelay0228.hostedemail.com
2017-05-23 13:34:12 DEBUG smtp session: reverse_client_name=smtprelay0228.hostedemail.com
2017-05-23 13:34:12 DEBUG smtp session: helo_name=smtprelay.hostedemail.com
2017-05-23 13:34:12 DEBUG smtp session: sender=pete@customer.net
2017-05-23 13:34:12 DEBUG smtp session: recipient=sales@mydomain.com
2017-05-23 13:34:12 DEBUG smtp session: recipient_count=0
2017-05-23 13:34:12 DEBUG smtp session: queue_id=
2017-05-23 13:34:12 DEBUG smtp session: instance=144.59249cc4.d38e7.0
2017-05-23 13:34:12 DEBUG smtp session: size=43347
2017-05-23 13:34:12 DEBUG smtp session: etrn_domain=
2017-05-23 13:34:12 DEBUG smtp session: stress=
2017-05-23 13:34:12 DEBUG smtp session: sasl_method=
2017-05-23 13:34:12 DEBUG smtp session: sasl_username=
2017-05-23 13:34:12 DEBUG smtp session: sasl_sender=
2017-05-23 13:34:12 DEBUG smtp session: ccert_subject=
2017-05-23 13:34:12 DEBUG smtp session: ccert_issuer=
2017-05-23 13:34:12 DEBUG smtp session: ccert_fingerprint=
2017-05-23 13:34:12 DEBUG smtp session: ccert_pubkey_fingerprint=
2017-05-23 13:34:12 DEBUG smtp session: encryption_protocol=TLSv1.2
2017-05-23 13:34:12 DEBUG smtp session: encryption_cipher=ECDHE-RSA-AES256-GCM-SHA384
2017-05-23 13:34:12 DEBUG smtp session: encryption_keysize=256
2017-05-23 13:34:12 DEBUG LDAP connection initialied success.
2017-05-23 13:34:12 DEBUG LDAP bind success.
2017-05-23 13:34:12 DEBUG --> Apply plugin: reject_null_sender
2017-05-23 13:34:12 DEBUG <-- Result: DUNNO
2017-05-23 13:34:12 DEBUG --> Apply plugin: greylisting
2017-05-23 13:34:12 DEBUG [SQL] Query greylisting whitelists:
SELECT id, sender, comment
               FROM greylisting_whitelists
              WHERE account IN ('sales@mydomain.com', '@mydomain.com', '@.')
2017-05-23 13:34:12 DEBUG No whitelist found.
2017-05-23 13:34:12 DEBUG [SQL] query greylisting settings:
SELECT id, account, sender, sender_priority, active
               FROM greylisting
              WHERE account IN ('sales@mydomain.com', '@mydomain.com', '@.')
              ORDER BY priority DESC, sender_priority DESC
2017-05-23 13:34:12 DEBUG [SQL] query result: [(1L, '@.', '@.', 0, 1)]
2017-05-23 13:34:12 DEBUG Greylisting should be applied according to SQL record: (id=1, account='@.', sender='@.')
2017-05-23 13:34:12 DEBUG [SQL] check whether client address (216.40.44.228) passed greylisting:
SELECT id
               FROM greylisting_tracking
              WHERE client_address='216.40.44.228'
                    AND passed=1
              LIMIT 1
2017-05-23 13:34:12 DEBUG Client address (216.40.44.228) didn't pass greylisting.
2017-05-23 13:34:12 DEBUG [SQL] query greylisting tracking:
SELECT init_time, blocked_count, block_expired, record_expired
               FROM greylisting_tracking
              WHERE sender='pete@customer.net'
                    AND recipient='sales@mydomain.com'
                    AND client_address='216.40.44.228'
              LIMIT 1
2017-05-23 13:34:12 INFO [216.40.44.228] Client has not been seen before, greylisted.
2017-05-23 13:34:12 DEBUG [SQL] New tracking:
INSERT INTO greylisting_tracking (sender, sender_domain,
                                                   recipient, rcpt_domain,
                                                   client_address,
                                                   init_time,
                                                   block_expired, record_expired,
                                                   blocked_count)
                      VALUES ('pete@customer.net', 'customer.net', 'sales@mydomain.com', 'mydomain.com', '216.40.44.228', 1495571652, 1495572552, 1495658052, 1)
2017-05-23 13:34:12 DEBUG <-- Result: 451 4.7.1 Intentional policy rejection, please try again later
2017-05-23 13:34:12 DEBUG Session ended.
2017-05-23 13:34:12 INFO 216.40.44.228 RCPT, pete@customer.net -> sales@mydomain.com, 451 4.7.1 Intentional policy rejection, please try again later [0.0110s]
2017-05-23 13:34:12 DEBUG Close LDAP connection.

That is the latest iredapd debug log.  I don't see any query where it checks whitelisted senders.  Why?

10 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: iRedAPD Whitelist not working / still graylisting?

Could you please show me which version of iRedAPD you're running? Command:

ls -l /opt

11 Reply by mac

  • mac
  • Member
  • Offline
  • Registered: 2014-10-28
  • Posts: 15

Re: iRedAPD Whitelist not working / still graylisting?

Sorry for the delay, was out of town...

[root@mail ~]# ls -l /opt
total 12
lrwxrwxrwx  1 root    root      11 Mar 13 22:35 iredapd -> iRedAPD-2.0
dr-xr-xr-x  5 iredapd iredapd 4096 Nov  5  2014 iRedAPD-1.4.3
dr-x------  8 root    root    4096 May  1 18:17 iRedAPD-2.0
drwxr-xr-x. 2 root    root    4096 Mar 26  2015 rh
[root@mail ~]#

12 Reply by mac

  • mac
  • Member
  • Offline
  • Registered: 2014-10-28
  • Posts: 15

Re: iRedAPD Whitelist not working / still graylisting?

Bump... does anyone have any insights?  It's starting to feel like a bug in iRedAPD that it is storing, but not querying any white or blacklists during mail checking.

Is there anyone successfully using white/blacklists in iRedAPD?

13 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: iRedAPD Whitelist not working / still graylisting?

According to your iRedAPD log, it doesn't query whitelists at all. iRedAPD-2.0 is supposed to query whitelists, i'm not sure why it doesn't work as expected on your server, could you please try to upgrade iRedAPD again?

FYI: http://www.iredmail.org/docs/upgrade.iredapd.html