1

Topic: living with greylisting

============ Required information ====
- iRedMail version (check /etc/iredmail-release): 0.9.5-1
- Linux/BSD distribution name and version: Ubuntu 16.04
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): mariadb
- Web server (Apache or Nginx): nginx
- Manage mail accounts with iRedAdmin-Pro? yes
- Related log if you're reporting an issue:
====

Greylisting has absolutely destroyed the spammers. In the month I've been using it, I've gone from 50 spams per day down to maybe 2 spams per week... and those two spams usually go into my junk folder.  Delaying receipt of a spam run by just 10-15 minutes really allows the blacklists time to catch the bad guys. 

Now I'm trying to tweak it so that users will be willing to use it, reduce user heartburn, etc.


Suggestion #1:  Let's say joe@outside.com sends me an email, it passes greylisting, and is delivered to my mailbox. Future emails from joe are not delayed.  However, a new email from bob@outside.com will be greylisted all over again because it is a new 'triple'. Instead, after the first successful email from joe@outside.com, how about exempting all mail from outside.com from greylisting? Still subject to expiry of course. The server for outside.com has already demonstrated it will pass greylisting, so delaying mail from other outside.com addresses doesn't help.  Thoughts?  I think this would make greylisting easier to live with without hurting its effectiveness.


Suggestion #2:  I recently added dnswl.org to my postscreen config. It is a *whitelist* of well-behaved mail servers. So far I've been impressed with its contributions to my postscreen scores.  If greylisting.py could do lookups with dnswl.org, it would avoid delaying (hopefully) good mailservers.  I've had python on my need-to-learn list for a long time, but haven't done it yet, or I'd take a crack at modifying greylisting.py myself  wink

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: living with greylisting

About #1:

*) sender domain is easy to forge, so you may receive spams from forged sender domain which is whitelisted for greylisting service.
*) How about whitelist sender IP address directly? (Concern: you will get more and more IP addresses in sql table.)
*) Or, submit sender domain to SQL table `iredapd.greylisting_whitelist_domains`, the cron job '/opt/iredapd/tools/spf_to_greylist_whitelists.py' will query SPF DNS record of this domain, and whitelist the IP addresses/networks in SPF record. (Concern: you will get more and more IP addresses in sql table, and the DNS query time by spf_to_greylist_whitelists.py will be longer - you'd better have a local DNS server on localhost or LAN.)

3

Re: living with greylisting

ZhangHuangbin wrote:

sender domain is easy to forge, so you may receive spams from forged sender domain which is whitelisted for greylisting service.

Agreed, spammers always forge the sender address. However, I'm willing to lose a small amount of greylisting effectiveness if I can reduce the collateral damage it causes. Otherwise users won't want greylisting at all.

ZhangHuangbin wrote:

How about whitelist sender IP address directly? (Concern: you will get more and more IP addresses in sql table.)

I thought about that and it is a valid alternative. It has the side effect of also whitelisting every other domain handled by that particular server. Thinking about it, whitelisting by IP would be easier to implement within the current framework. I could likely do it with a simple and frequent cronjob along the lines of:

  delete all my custom records from greylisting_whitelists
  select client_address from greylisting_tracking where passed>0
  add results to greylisting_whitelists, with a unique tag in the comment to find my custom records next time

More rows in the SQL table does not bother me, SQL can handle it. And, my new entries will still expire as they are removed from greylisting_tracking.

ZhangHuangbin wrote:

Or, submit sender domain to SQL table `iredapd.greylisting_whitelist_domains`, the cron job '/opt/iredapd/tools/spf_to_greylist_whitelists.py' will query SPF DNS record of this domain, and whitelist the IP addresses/networks in SPF record.

I like your feature of using SPF records to generate IP whitelists, and it does help. Unfortunately, in my experience, many servers don't bother with SPF records, or the ones they publish are wrong or syntactically invalid. 


Much more complicated is adding dns whitelist lookups (i.e. dnswl.org) to greylisting.py, suggestion #2 above.  Any thoughts or suggestions on that?


ps - For me, local DNS resolver/cache is an absolute requirement for a mail server.

4

Re: living with greylisting

I forgot to introduce the new plugin in upcoming iRedAPD release: whitelist_outbound_recipient. It automatically whitelists recipient or recipient domain (configureable) for greylisting service. For example, if your user sent an email to user@gmail.com, this plugin will either whitelist 'user@gmail.com' directly (this is default setting, and it's per-user whitelists), or submit domain 'gmail.com' to SQL table 'iredapd.greylisting_whitelist_domains' and wait for the cron job 'spf_to_greylist_whitelists.py' to dump its IP addresses/networks as whitelists (again, for greylisting service, and server-wide whitelists).

deltatango wrote:

Thinking about it, whitelisting by IP would be easier to implement within the current framework.

How about this:

- With default greylisting behaviour, if one sender passed the greylisting service, iRedAPD stores a tracking record in sql table `greylisting_tracking` with sender, recipient, client_address, and column `passed=1` to indicate the sender from this ip address passed greylisting service.

- The idea is, how about use the IP address in this (passed) tracking record as a whitelist directly? If this sender server bypassed once, it's supposed to always pass because it will try to re-deliver email.

- If tracking record expired, another mail delivery attempt will be made by sender server, this will cause some delay as usual, but once it's bypassed, it's whitelisted again.

This way should be more effective.

5

Re: living with greylisting

ZhangHuangbin wrote:

I forgot to introduce the new plugin in upcoming iRedAPD release: whitelist_outbound_recipient. It automatically whitelists recipient or recipient domain (configureable) for greylisting service. For example, if your user sent an email to user@gmail.com, this plugin will either whitelist 'user@gmail.com' directly (this is default setting, and it's per-user whitelists), or submit domain 'gmail.com' to SQL table 'iredapd.greylisting_whitelist_domains' and wait for the cron job 'spf_to_greylist_whitelists.py' to dump its IP addresses/networks as whitelists (again, for greylisting service, and server-wide whitelists).

This would be a very nice feature. I think Amavis had something similar for its spam filtering called 'penpals'.   One suggestion: if localuser@localdomain.com sends email to somebody@outside.com, how about adding outside.com to a configurable (by me) choice of either "localuser@localdomain.com", or "@localdomain.com", or "@." in greylisting_whitelists.account

I've only had marginal success using the spf-to-greylist-whitelist feature.  Too many domains do not publish correct SPF records.

ZhangHuangbin wrote:

- The idea is, how about use the IP address in this (passed) tracking record as a whitelist directly? If this sender server bypassed once, it's supposed to always pass because it will try to re-deliver email.

That would be perfect!! Much cleaner than what I described. If iRedAPD was in Perl, I'd try to make a patch for you.  Embarrassingly, my Python is nonexistent. Someday I *will* make time to learn some Python.

6

Re: living with greylisting

deltatango wrote:

One suggestion: if localuser@localdomain.com sends email to somebody@outside.com, how about adding outside.com to a configurable (by me) choice of either "localuser@localdomain.com", or "@localdomain.com", or "@." in greylisting_whitelists.account

Do you mean you want to whitelist the 'outside.com' directly without querying SPF record?

Default action is submitting the domain 'outside.com' to `greylisting_whitelist_domains`, then wait for `spf_to_greylist_whitelists.py`.

deltatango wrote:

That would be perfect!! Much cleaner than what I described. If iRedAPD was in Perl, I'd try to make a patch for you.  Embarrassingly, my Python is nonexistent. Someday I *will* make time to learn some Python.

Already implemented before you replied:
https://bitbucket.org/zhb/iredapd/commi … 604cd57ccc

7

Re: living with greylisting

ZhangHuangbin wrote:

Do you mean you want to whitelist the 'outside.com' directly without querying SPF record?

Yes. Someday SPF records will be more widely implemented. But for now I need to whitelist by actual domain name most of the time. And that's what users will understand.


ZhangHuangbin wrote:

Already implemented before you replied:
https://bitbucket.org/zhb/iredapd/commi … 604cd57ccc

Now you're just showing off  smile smile


Almost midnight for me, I'm calling it a day.   But when you have time, I'd still appreciate your thoughts on my Suggestion #2 way back at the beginning.

8

Re: living with greylisting

deltatango wrote:

Yes. Someday SPF records will be more widely implemented. But for now I need to whitelist by actual domain name most of the time. And that's what users will understand.

Done. In upcoming iRedAPD release with plugin 'whitelist_outbound_recipient', 3 new options available for this purpose:

# Whitelist sender directly (no SPF query).
# Requires options `WL_RCPT_LOCAL_ACCOUNT` and `WL_RCPT_RCPT` listed below.
WL_RCPT_WITHOUT_SPF = False

# Whitelist recipient directly (no SPF query) for local account:
#   - user: for the sender (per-user whitelist)
#   - domain: for the sender domain (per-domain whitelist)
#   - global: for global whitelist
WL_RCPT_LOCAL_ACCOUNT = 'user'

# Whitelist which recipient for local account
#   - user: the recipient (single email address)
#   - domain: the recipient domain
WL_RCPT_RCPT = 'user'

9

Re: living with greylisting

deltatango wrote:

But when you have time, I'd still appreciate your thoughts on my Suggestion #2 way back at the beginning.

I will do some research, thanks for sharing. smile

10

Re: living with greylisting

ZhangHuangbin wrote:

In upcoming iRedAPD release with plugin 'whitelist_outbound_recipient'

If possible, vacation auto-responses should not trigger a whitelist entry. However, I'm not sure how you would differentiate between auto-replies versus actual user-originated email.

11

Re: living with greylisting

ZhangHuangbin wrote:
deltatango wrote:

But when you have time, I'd still appreciate your thoughts on my Suggestion #2 way back at the beginning.

I will do some research, thanks for sharing. smile

Here's a great example of the usefulness of the DNSWL whitelist:

sorbs.net has apparently blacklisted a gmail ip address. But, dnswl.org has the same ip in their whitelist.  Postscreen did this:

Jun 29 16:10:17 mail postfix/postscreen[14241]: CONNECT from [74.125.82.46]:34554 to [1.2.3.4]:25
Jun 29 16:10:17 mail postfix/dnsblog[14247]: addr 74.125.82.46 listed by domain list.dnswl.org as 127.0.5.1
Jun 29 16:10:17 mail postfix/dnsblog[14243]: addr 74.125.82.46 listed by domain dnsbl.sorbs.net as 127.0.0.6
Jun 29 16:10:17 mail postfix/postscreen[14241]: PASS NEW [74.125.82.46]:34554

My postscreen config allowed dnswl to overrule sorbs by assigning a negative score:

postscreen_dnsbl_sites =
    zen.spamhaus.org*3
    b.barracudacentral.org*3
    bl.spameatingmonkey.net*2
    dnsbl-1.uceprotect.net*2
    bl.spamcop.net
    dnsbl.sorbs.net
    psbl.surriel.com
    bl.mailspike.net
    swl.spamhaus.org*-4
    list.dnswl.org=127.[0..255].[0..255].0*-2
    list.dnswl.org=127.[0..255].[0..255].1*-3
    list.dnswl.org=127.[0..255].[0..255].[2..255]*-4

Thus, if greylisting checked with one or more configurable dns whitelists, it would help avoid delaying good mail.