1

Topic: Disable spam virus scanning for outgoing mails

==== Required information ====
- iRedMail version:
- Store mail accounts in which backend (LDAP/MySQL/PGSQL):
- Web server (Apache or Nginx):
- Linux/BSD distribution name and version:
- Related log if you're reporting an issue:
======== Required information ====
- iRedMail version:  iRedMail-0.9.0
- Store mail accounts in which backend (LDAP/MySQL/PGSQL):  MySQL
- Web server (Apache or Nginx):
- Linux/BSD distribution name and version: Ubuntu 14.04
- Related log if you're reporting an issue:
====
Hi,

I have problems trying to disable spam virus scanning for outgoing mails. It happens that by enabling the option for MYUSERS basically shutdown all spam/virus checking in my server:

I've edited /etc/amavis/conf.d/50-user and added the bypass options under $policy_bank

...
$policy_bank{'MYUSERS'} = {
    [... DEFAULT CONFIGURATION HERE...]

    # don't perform spam/virus/header check.
    bypass_spam_checks_maps => [1],
    bypass_virus_checks_maps => [1],
    bypass_header_checks_maps => [1],

}
...

To my surprise, I started noticing a lot of emails coming through which even grandma would recognize as spam and having the "Always insert X-Spam-* headers" enabled in iRedAdmin-Pro and not seeing anything related to SPAM checks in emails headers told me that something had gone really wrong.

Can anyone give me a hint as of to where start looking to make sure that disabling spam virus scanning for outgoing mails does not completely disables external domains email checks?

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: Disable spam virus scanning for outgoing mails

Can you show me your mail.log when these e-mails come in? I need to see what Policy Banks they trigger, you're probably missing the new configuration I've devised for AmavisD-New which fixes this

3

Re: Disable spam virus scanning for outgoing mails

Try to list all your mail domain names in @local_domains_maps in /etc/amavis/conf.d/50-user. For example:

@local_domains_maps = (['domain1.com', 'domain2.com']);

4

Re: Disable spam virus scanning for outgoing mails

I was going to do the dynamic version of that ZhangHuangbin

5

Re: Disable spam virus scanning for outgoing mails

ZhangHuangbin wrote:

Try to list all your mail domain names in @local_domains_maps in /etc/amavis/conf.d/50-user. For example:

@local_domains_maps = (['domain1.com', 'domain2.com']);

Zhang, it would be really counterproductive having to add a domain manually every time. Currently there are 25 domains which could be tedious. sad

6

Re: Disable spam virus scanning for outgoing mails

My method is dynamic and will make all current and future domains work. It is near the bottom of this thread : http://www.iredmail.org/forum/topic3809 … sdnew.html

7 (edited by rafaelr 2015-03-26 19:29:04)

Re: Disable spam virus scanning for outgoing mails

Edited: Seems we posted at the same time. I will look at that link you provided above and let you know how it goes. Thanks!

8

Re: Disable spam virus scanning for outgoing mails

7t3chguy wrote:

My method is dynamic and will make all current and future domains work. It is near the bottom of this thread : http://www.iredmail.org/forum/topic3809 … sdnew.html

Dear 7t3chguy,

as mentioned in my reply in this link, your solution is not ideal. still need to modify '$sql_select_policy' setting in Amavisd to make it return per-recipient policy (records in `amavisd.users` and `amavisd.policy`). Still no good idea how to improve default $sql_select_policy setting.

9

Re: Disable spam virus scanning for outgoing mails

In such case, I guess it is more productive to leave it alone at least for now (else manually add the domains then).

10

Re: Disable spam virus scanning for outgoing mails

I can change it to per recipient easily, basically, change the view to only include users which should be treated by the myusers policy back, the query will check user and domain already

11

Re: Disable spam virus scanning for outgoing mails

Zhang, could you explain what the per-recipient part would do? The other problem is Amavisd makes the query for both the sender and the recipient, with no way of differentiating between the two, so it'd be per sender/recipient, not just per recipient.

12

Re: Disable spam virus scanning for outgoing mails

7t3chguy wrote:

could you explain what the per-recipient part would do?

No problem, and remember, we're talking about per-recipient policy lookup, not per-sender.

*) When you have '@lookup_sql_dsn = ' enabled in Amavisd config file (/etc/amavisd/amavisd.conf on RHEL/CentOS, or /etc/amavis/conf.d/50-user on Debian/Ubuntu, ...), you have per-recipient policy lookup enabled.

*) Below is SQL command defined in /usr/sbin/amavisd (depends on the linux distribution you're running, the path or file name may be different, e.g. /usr/sbin/amavisd-new, or /usr/local/usr/sbin/amavisd, etc), used to query per-recipient policy:

  # The SQL select clause to fetch per-recipient policy settings.
  # The %k will be replaced by a comma-separated list of query addresses
  # for a recipient (e.g. a full address, domain only, catchall), %a will be
  # replaced by an exact recipient address (same as the first entry in %k,
  # suitable for pattern matching), %l by a full unmodified localpart, %u by
  # a lowercased username (a localpart without extension), %e by lowercased
  # addr extension (which includes a delimiter), and %d for lowercased domain.
  # Use ORDER if there is a chance that multiple records will match - the
  # first match wins (i.e. the first returned record). If field names are
  # not unique (e.g. 'id'), the later field overwrites the earlier in a hash
  # returned by lookup, which is why we use 'users.*, policy.*, users.id',
  # i.e. the id is repeated at the end.
  # This is a legacy variable for upwards compatibility, now only referenced
  # by the program through a %sql_clause entry 'sel_policy' - newer config
  # files may assign directly to $sql_clause{'sel_policy'} if preferred.

  $sql_select_policy =                                                               
    'SELECT users.*, policy.*, users.id'.                                            
    ' FROM users LEFT JOIN policy ON users.policy_id=policy.id'.                                                              
    ' WHERE users.email IN (%k) ORDER BY users.priority DESC';

Please do read comment lines above.

Amavisd stores per-recipient policy in SQL table `amavisd.policy` (iRedMail uses database 'amavisd', so we use this database name for example here), according to above SQL command, we can know Amavisd will query 2 tables to get per-recipient policy: users, policy.

What data does Amavisd require in these 2 tables?

1) table 'amavisd.policy': define some policy rules. for example, with 'policy.bypass_spam_checks=Y', Amavisd will bypass spam checking for recipient who link to this policy. with 'policy.bypass_virus_checks=Y', Amavisd will bypass virus checking for recipient who link to this policy.

2) table 'amavisd.users': If you want to define some per-recipient policy for a LOCAL user, you must insert info about this local user in this table, THEN link to a policy (defined in 'amavisd.policy' to this user). The most important columns are:

- users.id: a unique id
- users.email: full email address, or '@domain.com' for per-domain account, '@.' for a catch-all account
- users.policy_id: link to policy record defined in table 'amavisd.policy'.

So, if you want to query virtual mail domains, please do return policy data in your new SQL query.

Hope it helps a little.

13

Re: Disable spam virus scanning for outgoing mails

The only thing I'm confused by, is when I had my MySQL Query Log on, it showed AmavisD querying the Query I wrote with both the Sender and the Recipient Email,Domain,Catchall

14

Re: Disable spam virus scanning for outgoing mails

@Zhang, when I try to untick "Prefix text [SPAM] to the subject of spam" in iRedAdmin Pro, it redirects me to ?msg=UPDATED, but when I go back onto the Spam Policy page, its still ticked.

Edit:
When its ticked and I hit SAVE, it removes the policy row, when its unticked and I hit save, it creates a policy row with Y in the spam_modifies_subj. But most importantly, nothing ever gets created in amavis.users so the MySQL Query never actually returns anything. I'm assuming what you meant to do is to not use policy groups, but have a separate policy per user? You'd need to modify the MySQL Query for that but it doesn't seem you do that.

and why is amavis.users.email of type VarBinary(255)?

15

Re: Disable spam virus scanning for outgoing mails

Alright, the problem is all the other lookups rely on the users table, which should be converted into a View. Id column [Autoincrement and PK] should be added to vmail.mailbox. Policy_id can be inherited from id, priority for all user level overrides will be the same. Fullname and email will be inherited from mailbox table.

Right now, will Adding an AI+PK ID Field to mailbox break anything?

There are still a few small details I'm unsure about, but it should have per-recipient and per-recipient-domain functionality. The Query I'll write will return one row, if no per-recipient or per-domain it'll just return a placeholder row so that Amavis knows its a local domain and to treat it as such [if E-Mail is in domains/alias_domains]

Wouldn't a catchall (@.) record trigger all domains to be assumed to be local?

Edit:
Just realised that the View doesn't even need Alias domain or User Aliases, so will be much much simpler.

16

Re: Disable spam virus scanning for outgoing mails

Alright, after a lot of playing, the only way I could get the MySQL Working would be a really ugly Union Select which had a Select with 40+ Null Columns. The easiest way would be if iRedAdmin upon the creation of a domain, would add it to the AmavisD DB, same for User, into the Policy table, [@exampledomain.com / user@exampleuser.com]
(better if the user is only added if his settings are non-default, but domain should be added each time)

This shouldn't be too hard for you to implement?

The harder way to implement it would be to create a GUI within iRedAdmin-Pro to create Policy Sets, then on each User and Domain you could attach that User/Domain to a specific Policy Set. This would give you Full Control with less repeated data. Domains would still have to be added dynamically upon being created [Even in OSE]

17

Re: Disable spam virus scanning for outgoing mails

7t3chguy wrote:

@Zhang, when I try to untick "Prefix text [SPAM] to the subject of spam" in iRedAdmin Pro, it redirects me to ?msg=UPDATED, but when I go back onto the Spam Policy page, its still ticked.

A bug, fixed days ago.

7t3chguy wrote:

When its ticked and I hit SAVE, it removes the policy row, when its unticked and I hit save, it creates a policy row with Y in the spam_modifies_subj. But most importantly, nothing ever gets created in amavis.users so the MySQL Query never actually returns anything. I'm assuming what you meant to do is to not use policy groups, but have a separate policy per user? You'd need to modify the MySQL Query for that but it doesn't seem you do that.

Could you please show me the SQL statements?

7t3chguy wrote:

and why is amavis.users.email of type VarBinary(255)?

It's defined by Amavisd, not iRedMail.
Reference: http://www.ijs.si/software/amavisd/README.sql-mysql.txt

7t3chguy wrote:

This shouldn't be too hard for you to implement?

Why save duplicate mail domains and users in Amavisd database? I don't like this idea.

18

Re: Disable spam virus scanning for outgoing mails

I think I've solved it. One MySQL View.
iRedAdmin would just have to create a new row in the Policy Table with the policy_name as the e-mail address, or @domain of the rules it wants to change.
(This seems to be how iRedAdmin-Pro is currently handling it, which right now does nothing since your Policies don't have a relevant User Table Entry, this View and modified Select would just fix that)

Edit:
Does Amavisd need me to return an ID? I can just run the Join on the email instead of ID if it doesn't need an ID.
If it needs an ID, then for this to work, I'd suggest giving vmail.mailbox an id column [AI PK]

19

Re: Disable spam virus scanning for outgoing mails

The only question right now, do you want Per-Domain or Per-Recipient to have higher priority? Like which one will override each, Amavis only accepts one row from DB per query.

20

Re: Disable spam virus scanning for outgoing mails

7t3chguy wrote:

I think I've solved it. One MySQL View.

Would you mind sharing the SQL command used to create VIEW?

7t3chguy wrote:

do you want Per-Domain or Per-Recipient to have higher priority?

Per-recipient policy should have higher priority than per-domain policy. catch-all policy (@.) has the lowest priority.

7t3chguy wrote:

This seems to be how iRedAdmin-Pro is currently handling it, which right now does nothing since your Policies don't have a relevant User Table Entry

Excuse me, what do you mean "policies don't have a revlvant user table entry"?

*) Policies are independent.
*) Relation between 'users' and 'policy' table is defined in 'users.policy_id' column. To define some policy for one recipient, you need to

1) create proper policy in 'policy' table;
2) add this recipient info in table 'users', and set the id (value of column 'policy.id') of the policy in 'users.policy' for this recipient.

Let me know if it's not explained clearly.

7t3chguy wrote:

Does Amavisd need me to return an ID?

Yes. With default sql statement, it returns 'users.id'.

7t3chguy wrote:

If it needs an ID, then for this to work, I'd suggest giving vmail.mailbox an id column [AI PK]

I still don't understand why you need an id column in `vmail.mailbox`.

If there's no policy defined for a recipient, just return NULL data of all columns.

21

Re: Disable spam virus scanning for outgoing mails

ZhangHuangbin wrote:

Excuse me, what do you mean "policies don't have a revlvant user table entry"?
*) Policies are independent.
*) Relation between 'users' and 'policy' table is defined in 'users.policy_id' column. To define some policy for one recipient, you need to
1) create proper policy in 'policy' table;
2) add this recipient info in table 'users', and set the id (value of column 'policy.id') of the policy in 'users.policy' for this recipient.
Let me know if it's not explained clearly.

iRedAdmin creates an entry within Policy, but it doesn't create a user entry, so the below SQL Query returns nothing even though there is an entry in Policy for my User made by iRedAdmin-Pro.

SELECT users.*, policy.*, users.id FROM users LEFT JOIN policy ON users.policy_id=policy.id WHERE users.email IN ('postmaster@webdevguru.co.uk') ORDER BY users.priority DESC
ZhangHuangbin wrote:

I still don't understand why you need an id column in `vmail.mailbox`.
If there's no policy defined for a recipient, just return NULL data of all columns.

What ID should I return then? Users are never tied to an ID and does it have to be unique, since there's only going to be one return row why does the ID matter to AmavisD?

ZhangHuangbin wrote:

Per-recipient policy should have higher priority than per-domain policy. catch-all policy (@.) has the lowest priority.

How would catch-all policy work, since if anything is returned then AmavisD understands this to be a Local User, and in each query it includes the catch-all. So that means that each AmavisD pass would be treated as if it was for a local user.



Current MySQL View:

CREATE 
    ALGORITHM = UNDEFINED 
    DEFINER = `root`@`localhost` 
    SQL SECURITY DEFINER
VIEW `users` AS
    SELECT 
        0 AS `id`,
        CONCAT('@', `vmail`.`domain`.`domain`) AS `email`
    FROM
        `vmail`.`domain` 
    UNION SELECT 
        1 AS `id`,
        `vmail`.`mailbox`.`username` AS `email`
    FROM
        `vmail`.`mailbox`

As you can see, the IDs are static, as they have no value they can be preset to;
MySQL Select Code:

SELECT 
    (IF( LEFT(users.email , 1) = '@', 0, 1)) AS priority,
    users.id,
    users.email,
    policy.*
FROM
    amavisd.users
        LEFT JOIN
    policy ON users.email = policy.policy_name
    WHERE users.email IN ('postmaster@webdevguru.co.uk', '@webdevguru.co.uk', '@.')
ORDER BY priority DESC;

for use in AmavisD, ('postmaster@webdevguru.co.uk', '@webdevguru.co.uk', '@.') needs to be replaced with (%k)

22

Re: Disable spam virus scanning for outgoing mails

7t3chguy wrote:

iRedAdmin creates an entry within Policy, but it doesn't create a user entry

Are you sure?
Currently, per-recipient policies can be defined in user profile page (per-user), domain profile page (per-domain), or under menu 'System -> Anti Spam -> Global spam policy' (global policy, lowest priority).

When you define a per-recipient policy, iRedAdmin-Pro will insert this account (user, domain, or catch-all account (@.)) to table 'users'. If not, it's a bug in iRedAdmin-Pro, i will try to reproduce this issue later.

7t3chguy wrote:

What ID should I return then? Users are never tied to an ID and does it have to be unique, since there's only going to be one return row why does the ID matter to AmavisD?

Amavisd expects a unique id for recipient, and it should be defined in 'amavisd.users' table, because other sql tables (e.g. amavisd.wblist) will use this id. Remember, if you defined a policy for one user, table 'users' should contain info of this user (e.g. full email address, a unique id number).

7t3chguy wrote:

How would catch-all policy work, since if anything is returned then AmavisD understands this to be a Local User, and in each query it includes the catch-all. So that means that each AmavisD pass would be treated as if it was for a local user.

That's what your VIEW should do. Must check whether recipient (full email address) or domain is local, if yes, return data (if no policy for this recipient, return NULL as value of all SQL columns (policy.* columns)).

23

Re: Disable spam virus scanning for outgoing mails

ZhangHuangbin wrote:

Are you sure?
Currently, per-recipient policies can be defined in user profile page (per-user), domain profile page (per-domain), or under menu 'System -> Anti Spam -> Global spam policy' (global policy, lowest priority).
When you define a per-recipient policy, iRedAdmin-Pro will insert this account (user, domain, or catch-all account (@.)) to table 'users'. If not, it's a bug in iRedAdmin-Pro, i will try to reproduce this issue later.

For me it just creates the amavisd.policy entry, nothing in amavisd.user.

ZhangHuangbin wrote:

Amavisd expects a unique id for recipient, and it should be defined in 'amavisd.users' table, because other sql tables (e.g. amavisd.wblist) will use this id. Remember, if you defined a policy for one user, table 'users' should contain info of this user (e.g. full email address, a unique id number).

As you should know after the trouble with my RCM Plugin, its impossible for a MySQL View to just come up with a Unique ID for each user that won't change over time, the only way would be create an ID in vmail.mailbox and reference that.

ZhangHuangbin wrote:

That's what your VIEW should do. Must check whether recipient (full email address) or domain is local, if yes, return data (if no policy for this recipient, return NULL as value of all SQL columns (policy.* columns)).

My VIEW does that, it'll list all local users and all local domains, the only thing that's not completely done right now is the SELECT Statement doesn't yet fall back to @domain if the user@domain.com entry is an empty result set. From reading, I've noticed that AmavisD respects a local column, so I could generate a local column and fill it with Y except for @., that way you could create a policy with policy_name of @. and whatever is set there will apply to all unmatched queries.

24

Re: Disable spam virus scanning for outgoing mails

@Zhang, could you please join me at irc.esper.net#iredmail

25

Re: Disable spam virus scanning for outgoing mails

7t3chguy wrote:

@Zhang, could you please join me at irc.esper.net#iredmail

I'm on freenode.net, #iredmail.

UPDATE: Cannot connect to irc.esper.net, can we move to freenode.net instead?