1 (edited by bmackay 2014-07-03 10:48:14)

Topic: Popular Sites To Whitelist From Greylisting

Here is a quick way to whitelist common email sources that really shouldn't be greylisted:   Google, Outlook.com, Twitter, Facebook.   Add any additional IP blocks that you might need such as Mailchimp, ZenDesk, etc... by editing the SQL file before importing.   After that it's much easier to make minor edits as the Cluebringer web interface UX is horrendous for adding a lot of entries.

Take the following SQL code, save it as greylist-whitelist.sql  and import into your cluebringer database.

mysql -p cluebringer <greylist-whitelist.sql

Contents of greylist-whitelist.sql

INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:216.239.32.0/19','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:64.233.160.0/19','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:66.249.80.0/20','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:72.14.192.0/18','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:209.85.128.0/17','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:66.102.0.0/20','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:74.125.0.0/16','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:64.18.0.0/20','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:207.126.144.0/20','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:173.194.0.0/16','GOOGLE',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.54.190.0/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.54.190.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.54.190.128/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.54.190.192/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.116.0/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.111.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.116.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.111.128/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.34.0/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.34.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.34.128/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.34.192/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.90.0/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.90.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.90.128/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.55.90.192/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.54.51.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:65.54.61.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:207.46.66.0/28','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:157.55.0.192/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:157.55.1.128/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:157.55.2.0/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:157.55.2.64/26','MICROSOFT',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:69.63.179.25','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:69.63.178.128/25','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:69.63.184.0/25','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:66.220.144.128/25','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:66.220.155.0/24','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:69.171.232.128/25','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:66.220.157.0/25','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:69.171.244.0/24','FACEBOOK',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:199.16.156.0/22','TWITTER',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:199.59.148.0/22','TWITTER',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:8.25.194.26/31','TWITTER',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:204.92.114.203','TWITTER',0);
INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES ('SenderIP:204.92.114.204/31','TWITTER',0);

2

Re: Popular Sites To Whitelist From Greylisting

Thanks for your sharing. Maybe it's a good idea to add them during iRedMail installation.

3

Re: Popular Sites To Whitelist From Greylisting

Added this SQL file in iRedMail, will be available in next release.

If someone want to contribute, please:

1) Create a new topic in this forum
2) Or, open an issue in https://bitbucket.org/zhb/iredmail/issues

I will commit them into iRedMail.

4

Re: Popular Sites To Whitelist From Greylisting

This is great, thanks for the post!

5

Re: Popular Sites To Whitelist From Greylisting

While I like the idea, the actual SQL Code is killing me to look at, why not just repeat the VALUES (...) clause?

INSERT INTO greylisting_whitelist (Source,Comment,Disabled) VALUES
('SenderIP:216.239.32.0/19','GOOGLE',0),
('SenderIP:64.233.160.0/19','GOOGLE',0)........

6

Re: Popular Sites To Whitelist From Greylisting

*) If there's a duplicate value, the whole SQL command will abort, other values cannot be inserted.
*) The way you mention should work during iRedMail installation. but a running iRedMail server may have one/some of them in SQL database, then you can import this file.

7 (edited by camel1cz 2015-02-26 23:41:47)

Re: Popular Sites To Whitelist From Greylisting

Hi,

in general I like the idea of sharing whitelists, but on this approach I don't like the fact, it's not updated constantly. So entries could become outdated (either new IPs missing or old IPs left in the list).

It would be nice to have some online and up2date version of whitelists. Either DNS zones similar to DNSBL or at leased a mechanism how to update whitelists via cron (inserts and also deletes).

Just idea: Maybe it could be good to share stable members of autowhitelists across trusted iRM installations and update all iRM servers with it? So choosen servers would have R/W access to this database and all others will just read from it.

8

Re: Popular Sites To Whitelist From Greylisting

I could mock up some centralised thing in php and host it, giving trusted members access to edit it, along with a script to run on CRON to keep it up to date, anyone inserted in that?

9

Re: Popular Sites To Whitelist From Greylisting

that would be great 7t3chguy

10 (edited by camel1cz 2015-02-27 02:02:57)

Re: Popular Sites To Whitelist From Greylisting

Yes, I would like that and I can put my system into testing...

Btw. what do you thing, guys, about sharing the auto whitelists?
I see in my graylisting whitelist about 20 entries with last seen in this month and added months ago... aren't such entries candidates for permanent whitelist?

I very much like the idea it would update by itself...

11

Re: Popular Sites To Whitelist From Greylisting

7t3chguy - I would be very keen to support this idea.. Manual entry input into a site, CRONd update to servers. Bit of research will go a long way. This sounds like something that has been attempted lots before, collaboration is difficult...

camel1cz - Sharing AWL sounds problematic , its always possible people have different thresholds etc but still the option of sharing sounds nice.

12

Re: Popular Sites To Whitelist From Greylisting

I'm thinking of having different 'branches' of settings, so each opted in user can choose to have Automatically Updating Whitelist, but with different content, I'd be more interesting in writing the system than filling it with actual good decent data. I'd need someone I can rely on to do the other side of it every now and then.

Edit, would you guys be happy with this being written in PHP? PHP is installed on all iRedMail installations whether via mod_php or fpm, it'd just query my Centralized server and then update the MySQL database accordingly. One thing I need to check, if you're running PgSQL is the cluebringer db in PgSQL or MySQL?

13

Re: Popular Sites To Whitelist From Greylisting

7t3chguy wrote:

One thing I need to check, if you're running PgSQL is the cluebringer db in PgSQL or MySQL?

I run iRM on PgSQL backend and everything is in PgSQL, also the cluebringer DB is in separate PgSQL database.

Just an idea - is't such a list already available somewhere? Maybe we can just integrate with existing service?

14

Re: Popular Sites To Whitelist From Greylisting

If you can find such a service, then I'll write a *client* for it

15

Re: Popular Sites To Whitelist From Greylisting

I like the idea, actually, it's been discussed privately long time ago, but not publicly (that's the mistake).

The idea discussed in this thread sounds like DNSBL, or some kind of central Postfix policy server. I believe we all agree fighting spam is not a one-time job, we have to continue working on white/blacklist/... to get better effect.

I'm in if you guys have time and technical skills to build this "service". I will contribute if i have required skills/time/...

16

Re: Popular Sites To Whitelist From Greylisting

Another idea is, we need a way to share white/black/greylisting/... on each iRedMail server (with admin's permission, of course, and no sensitive info) to a central database, then we can analyze these data to find some way/data to fight spam.

Anyway, what i really want is, get as many admins as possible to contribute, then benefit as many admins as possible.

17

Re: Popular Sites To Whitelist From Greylisting

Hi Zhang,

I cannot agree more!

From my point of view, I would do following:
- create central server with all policy lists used in iRM: for graylisting autowhitelist and autoblacklist and for both graylisting and checkhelo whitelist and blacklist (6 lists in total for now).
- every admin of iRM should have following options for each of above lists:
a) not use the shared list and maintain it locally (current functionality) and probably should be default option,
b) use the shared list but not to contribute (download only, no data sent to the central server),
c) contribute only (only sends out the data to the central server while local list remains w/o change),
d) both contribute local data and also synchronize with central list.
- update should be done once per day and scheduled via cron to random time to spread the load,

On central server:
- data from any host should be accepted,
- data to any host should be provided,
- structure of central list tables should be similar to the local + reference to the table "origin" which should have at least source IP and flag if trusted,
- only data from trusted  origins should be published,

Note to auto lists from graylisting:
- central server needs to track also the thresholds used when gathering data,
- client should tell the central server its configurated thresholds while requsting the list - server should provide only data with the same or stronger threshold setup.
- on local mailserver we have to make sure synchronization will not interfere with local expiration routines.

For all entries in local storage should be set a source - either local (default value) or remote (data came from the central server)

Qustions:
- should we do any kind of authentication agains the central server? Probably generated hash should be OK if any.
- is https communication secure enough (man in the middle attack etc.)
- how to deal with duplicitely submited data (entry comes from more servers),
- what language should be used on the clients? (something already there, so PHP, python or perl?)
- how to deal with the expiration of entries? On central server we should have clear algorithm how we expire entries in all lists,
- how to deal with wrong entries? You know, the delisting requests, etc.

I can contribute if you're interested.

18

Re: Popular Sites To Whitelist From Greylisting

I can see one flaw with this, what if someone whitelists their own server, as a spammer, they then have access to spam anyone whom using these Auto Sync W/BLists

19 (edited by camel1cz 2015-02-28 02:53:24)

Re: Popular Sites To Whitelist From Greylisting

7t3chguy wrote:

I can see one flaw with this, what if someone whitelists their own server, as a spammer, they then have access to spam anyone whom using these Auto Sync W/BLists

We have to keep the trusted sources really trustful. So it cannot happen spammer will be able to submit IPs which will get into the public lists.

My opinion: even really few trusted servers with significant traffic and good admin can make life of the whole iRM community much more easier.

20

Re: Popular Sites To Whitelist From Greylisting

camel1cz wrote:
7t3chguy wrote:

I can see one flaw with this, what if someone whitelists their own server, as a spammer, they then have access to spam anyone whom using these Auto Sync W/BLists

We have to keep the trusted sources really trustful. So it cannot happen spammer will be able to submit IPs which will get into the public lists.

My opinion: even really few trusted servers with significant traffic and good admin can make life of the whole iRM community much more easier.

I was just referring to Zhang when he said:

ZhangHuangbin wrote:

Another idea is, we need a way to share white/black/greylisting/... on each iRedMail server (with admin's permission, of course, and no sensitive info) to a central database, then we can analyze these data to find some way/data to fight spam.
Anyway, what i really want is, get as many admins as possible to contribute, then benefit as many admins as possible.

21

Re: Popular Sites To Whitelist From Greylisting

I can write some PHP to handle this, if Zhang would be up for it, wouldn't it best be fit into iRedAdmin or iRedAPD? You were saying you wanted to get rid of Cluebringer and fit Greylisting into iRedAPD, you could add this at the same time.

22

Re: Popular Sites To Whitelist From Greylisting

I see... spreading the contribution too much could indeed generate troubles...

Questions are:
- is it possible to find out from outside, that a server is running iRM?
- would it be possible to get list of servers (their IPs/hostnames) running iRM from the discussed central database? (from outside of course)

If the answer to both questions is NO, I don't see any possible motivation for spammers to infiltrate iRM community.

Of course I'm not saying we shouldn't deal with this possibility.

7t3chguy wrote:

I can write some PHP to handle this, if Zhang would be up for it, wouldn't it best be fit into iRedAdmin or iRedAPD? You were saying you wanted to get rid of Cluebringer and fit Greylisting into iRedAPD, you could add this at the same time.

It's not my business, but we talk about a lot of work even to get current features and stability... dunno.
Of course I do see both the potential in iRedAPD and the duplicity/overhead using both tools.

23

Re: Popular Sites To Whitelist From Greylisting

camel1cz wrote:

- create central server with all policy lists used in iRM: for graylisting autowhitelist and autoblacklist and for both graylisting and checkhelo whitelist and blacklist (6 lists in total for now).

Agree. We can even query other DNSBL services to improve the effect.

camel1cz wrote:

- every admin of iRM should have following options for each of above lists:
a) not use the shared list and maintain it locally (current functionality) and probably should be default option,
b) use the shared list but not to contribute (download only, no data sent to the central server),
c) contribute only (only sends out the data to the central server while local list remains w/o change),
d) both contribute local data and also synchronize with central list.
- update should be done once per day and scheduled via cron to random time to spread the load,

Personally, i think we can discuss deep technical details later, just keep improving it on client-side (each iRedMail server).

camel1cz wrote:

On central server:
- data from any host should be accepted,
- data to any host should be provided,
- structure of central list tables should be similar to the local + reference to the table "origin" which should have at least source IP and flag if trusted,
- only data from trusted  origins should be published,

I have a better idea:

*) we create a web interface to allow mail server admins to register an account
*) admins register account and list the IP address(es) of their mail servers there.
*) we do some simple verification to allow contribution from registered mail servers, to make sure it's not spammer.
*) allow requests from registered mail servers (not from servers not registered in web interface)
*) need a way to analyze contributed data, and submit to grey/white/blacklisting and return OK/REJECT/DISCARD/... when mail servers request to verify client IP/sender/...
*) allow registered mail admins to submit grey/white/blacklisting IP addresses, stored in a separate place. we should verify them before submitting them to database in production use.
*) Of course we should allow registered accounts to judge data submitted by other admins, don't be a one-man show.
*) Allow others to request to remove IP address from blacklists.
*) Allow mail admins to manage their own data on this web interface, this data doesn't share to others, but admin has the option to share it. This way, we don't need a way to contribute local data to central server, because they just manage data on central server.

camel1cz wrote:

Note to auto lists from graylisting:
- central server needs to track also the thresholds used when gathering data,
- client should tell the central server its configurated thresholds while requsting the list - server should provide only data with the same or stronger threshold setup.
- on local mailserver we have to make sure synchronization will not interfere with local expiration routines.

I think this way is better (mentioned above):

*) Allow mail admins to manage their own data on this web interface, this data doesn't share to others, but admin has the option to share it. This way, we don't need a way to contribute local data to central server, because they just manage data on central server..

camel1cz wrote:

- should we do any kind of authentication agains the central server? Probably generated hash should be OK if any.

My idea is mentioned above: admin registers an account on central server, list their IP addresses there, then done.
Central server should drop request from unlisted IP addresses.

camel1cz wrote:

- is https communication secure enough (man in the middle attack etc.)

Do we have any other option? (A question, not a rhetorical question, don't get me wrong.)

- how to deal with duplicitely submited data (entry comes from more servers),

The more times it shows up, the more possible it's a spam/whitelist. right?

camel1cz wrote:

- what language should be used on the clients? (something already there, so PHP, python or perl?)

The best solution might be: it doesn't require any software installed, like DNSBL service.
If we must have a agent like iRedAPD, i hope it's easy to learn and understand, Python is a good choice. (of course i don't mean we HAVE to use Python.)

camel1cz wrote:

- how to deal with the expiration of entries? On central server we should have clear algorithm how we expire entries in all lists,

I think we can discuss this in the future.

camel1cz wrote:

My opinion: even really few trusted servers with significant traffic and good admin can make life of the whole iRM community much more easier.

Agree.

24

Re: Popular Sites To Whitelist From Greylisting

7t3chguy wrote:

I can write some PHP to handle this, if Zhang would be up for it, wouldn't it best be fit into iRedAdmin or iRedAPD? You were saying you wanted to get rid of Cluebringer and fit Greylisting into iRedAPD, you could add this at the same time.

I do want to get rid of Cluebringer and implement greylisting in iRedAPD. Yes, this is a good chance.

25

Re: Popular Sites To Whitelist From Greylisting

I think we're discussing a possible SAAS solution:

*) Client (mail server admin) manage grey/white/blacklist and more with the web interface in central server (it must be a fail-over cluster), just like a Cluebringer server running in this central server (with more features and user-friendly web ui).

*) Client controls whether to share the data on this central server (not publicly share to everyone else on internet), so that we can use the data to offer a service like DNSBL to help reduce spam.

interesting?