1 (edited by reno31 2016-03-08 22:33:45)

Topic: [HOW-TO]DSPAM and SPAM Training

##########################################################################################
#########################                  DSPAM is Deprecated                 #########################
#########################################################################################

Below Topic id for history, if you need to train your spam :
See this :  http://www.iredmail.org/forum/topic8169 … assin.html



==== Required information ====
- iRedMail version: 0.8.6
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): Mysql
- Linux/BSD distribution name and version: Debian 7.4
- Related log if you're reporting an issue: none
====

I All, First off all thanks for iredmail, it's great and quite easy to use and manage.
But, As non spam lover I want to have DSPAM as complement for spam filtering.
So Here Is a HowTo that worked for me, with debian and mysql backend.

Compilation

aptitude install libmysqlclient-dev
cd ~
wget <DSPAM TAR.GZ ADDRESS>
tar xvzf dspam-3.10.2.tar.gz
cd dspam-3.10.2
./configure --prefix=/usr --disable-dependency-tracking --includedir=/usr/include --with-logdir=/var/log/dspam/ --with-dspam-home=/var/amavis/dspam --sysconfdir=/etc/ --enable-domain-scale --without-delivery-agent --with-mysql-includes=/usr/include/mysql --with-storage-driver=mysql_drv --enable-virtual-users --enable-preferences-extension --enable-daemon --enable-debug
make
make install

Alternatives apt-get install dspam
Base de données

 mysql -u root -p
create database dspam;
GRANT SELECT, INSERT, UPDATE, DELETE ON dspam.* TO 'dspam'@'localhost' IDENTIFIED BY '******';
 FLUSH PRIVILEGES;
quit
cd ~
wget /spip/doc/Procmail/dspam/mysql_objects-4.1.sql
mysql -u root -p dspam < mysql_objects-4.1.sql

Configuration

cd /etc/
mv dspam.conf dspam.conf-orig
wget /spip/doc/Procmail/dspam.conf
sed -i 's/\/var\/amavis\/dspam/\/var\/lib\/amavis\/dspam/g' dspam.conf
sed -i 's/\*\*\*\*\*\*/toto/g' dspam.conf
cd /etc/init.d/
wget /spip/doc/Procmail/init.d/dspam
chmod +x dspam
insserv -v /etc/init.d/dspam

Permissions

chown amavis: /usr/bin/dspam*
chown amavis: /etc/dspam.conf
chmod 750 /usr/bin/dspam*
chmod 640 /etc/dspam.conf
chown amavis: /var/log/dspam
chown -R amavis: /var/lib/amavis/dspam

AmAviS

Ajouter ceci à /etc/amavis/conf.d/50-user

$sa_tag_level_deflt = 10
$dspam = 'dspam';
@spam_scanners = (
    ['SpamAssassin', 'Amavis::SpamControl::SpamAssassin'],
    ['DSPAM',  'Amavis::SpamControl::ExtProg', $dspam,
               [ qw(--client --stdout --process --deliver=innocent,spam
                    --user), $daemon_user ],
                score_factor => 0.39,
    ],
  );

Parametre DSPAM

 dspam_admin change preference default "dailyQuarantineSummary" "off"
   dspam_admin change preference default "enableBNR" "on"
   dspam_admin change preference default "enableWhitelist" "on"
   dspam_admin change preference default "fallbackDomain" "off"
   dspam_admin change preference default "ignoreGroups" "off"
   dspam_admin change preference default "ignoreRBLLookups" "off"
   dspam_admin change preference default "makeCorpus" "off"
   dspam_admin change preference default "optIn" "off"
   dspam_admin change preference default "optOut" "on"
   dspam_admin change preference default "optOutClamAV" "on"
   dspam_admin change preference default "processorBias" "on"
   dspam_admin change preference default "showFactors" "off"
   dspam_admin change preference default "signatureLocation" "headers"
   dspam_admin change preference default "spamAction" "tag"
   dspam_admin change preference default "spamSubject" "[SPAM]"
   dspam_admin change preference default "statisticalSedation" "6"
   dspam_admin change preference default "storeFragments" "off"
   dspam_admin change preference default "tagNonspam" "off"
   dspam_admin change preference default "tagSpam" "off"
   dspam_admin change preference default "trainingMode" "TOE"
   dspam_admin change preference default "trainPristine" "off"
   dspam_admin change preference default "whitelistThreshold" "10"

Id off user can be found via :  id -u username

mysql -u root -p dspam
INSERT INTO `dspam_virtual_uids` (`uid`, `username`) VALUES
(108, 'amavis');

Activating Vuser

dspam_admin change preference default "optIn" "on"
dspam_admin change preference default "optOut" "off"

Demarrage

/etc/init.d/dspam start
We check : ps aux | grep dspam
Amavis restart
/etc/init.d/amavis restart
We check
cat /var/log/mail.info | grep DSPAM

Maintenance

cd /var/lib/amavis/
wget  /spip/doc/Procmail/dspam-purge-4.1.sql
wget  /spip/doc/Procmail/caremail/cron/dspam-purgesql.sh
sed -i 's/\*\*\*\*\*\*/toto/g' dspam-purgesql.sh

Dovecot Folders

You need to replace in /etc/dovecot/dovecot.conf this parts :

namespace {                                                                        
    type = private                                                                 
    separator = /                                                                  
    prefix =                                                                       
    #location defaults to mail_location.                                        
    inbox = yes
}

By this parts

namespace {
    type = private
    separator = /
    prefix =
    #location defaults to mail_location.
    inbox = yes
    mailbox Sent {
        auto = subscribe
        special_use = \Sent
    }
    mailbox Drafts {
        auto = subscribe
        special_use = \Drafts
    }
    mailbox Trash {
        auto = subscribe
        special_use = \Trash
    }
    mailbox Junk {
        auto = subscribe
        special_use = \Junk
    }
    mailbox Spam {
        auto = subscribe
        #special_use = \Spam
    }
    mailbox SpamFalse {
        auto = subscribe
        #special_use = \SpamFalse
    }
}

Next thing : You have to comment out all "autocreate =" and "autosubscribe =" lines, AND remove "autocreate" plugin name in "mail_plugins =" (both "protocol lda {}" and "protocol imap {}" blocks).
and restart Dovecot, on login All users will now have all folder listed above.

mail adresses

Create by iredAdmin, iredAdminPro, mysql, or postfix-admin to virtual users,spam@domain.tld and ham@domain.tld give them some space but they won't need to much, since message won't stay there.

Script

sa_trap.sh
This one, go in all mailboxes, copy email in a folder, than add all email found in HAM and SPAM directory to the 2 special adresses spam@domain.tld and ham@domain.tld

#!/bin/sh
mkdir -p /var/tmp/spamtrap/cur
mkdir  /var/tmp/spamtrap/new
chown -R vmail: /var/tmp/spamtrap
chown -R vmail: /var/tmp/hamtrap

doveadm search -u "*@*" mailbox Spam ALL |
 while read guser guid uid; do
doveadm fetch -u "*@*" text mailbox-guid $guid uid $uid > /var/tmp/spamtrap/cur/msgspam.$uid
## Want to not expunge quote following line
doveadm expunge -u "*@*" mailbox Spam mailbox-guid $guid uid $uid
done

doveadm search -u "*@*" mailbox SpamFalse ALL |
 while read guser guid uid; do
doveadm fetch -u "*@*" text mailbox-guid $guid uid $uid > /var/tmp/hamtrap/cur/msgham.$uid
doveadm expunge -u "*@*" mailbox SpamFalse mailbox-guid $guid uid $uid
done

chown -R vmail: /var/tmp/spamtrap
chown -R vmail: /var/tmp/hamtrap

doveadm -v import -u ham@domain.tld maildir:/var/tmp/hamtrap "" mailbox INBOX all
doveadm -v import -u spam@domain.tld maildir:/var/tmp/spamtrap "" mailbox INBOX all
rm /var/tmp/spamtrap/cur/msgspam*
rm /var/tmp/hamtrap/cur/msgham*

sa_learn.sh
This one give all messages as SPAM or HAM to spamassassin and DSPAM

#!/bin/bash
# Script par tonio AT starbridge DOT org. (28-07-2012)
DATE=`date '+%s'`

doveadm search -u "spam@domain.tld" mailbox INBOX ALL |
 while read guid uid; do
doveadm fetch -u "spam@domain.tld" text mailbox-guid $guid uid $uid > /home/spam/msgspam-$DATE.$uid
#Expuging , but kepping copy in backup.
doveadm expunge -u "spam@domain.tld" mailbox INBOX mailbox-guid $guid uid $uid
done

doveadm search -u "ham@domain.tld" mailbox INBOX ALL |
 while read guid uid; do
doveadm fetch -u "ham@domain.tld" text mailbox-guid $guid uid $uid > /home/ham/msgham-$DATE.$uid
doveadm expunge -u "ham@domain.tld" mailbox INBOX mailbox-guid $guid uid $uid
done

chown amavis: /home/spam/* > /dev/null 2>&1
chown amavis: /home/ham/* > /dev/null 2>&1
chmod 666 /home/spam/* > /dev/null 2>&1
chmod 666 /home/ham/* > /dev/null 2>&1
su -c "/usr/bin/sa-learn -D --username='vmail' --spam /home/spam/" amavis
su -c "/usr/bin/sa-learn -D --username='vmail' --ham /home/ham/" amavis

#########DSPAM training########################
echo "Dspam Training"
#SPAM
#on fait le retrain uniquement si le tag est sur Innocent
/usr/bin/find /home/spam/ -type f -exec grep -l "X-DSPAM-Result: Innocent" {} \; | while read foo1
do
su -c "/usr/bin/dspam --client --user amavis --class=spam --source=error < '${foo1}'" amavis
done
#HAM
/usr/bin/find /home/ham/ -type f -exec grep -l "X-DSPAM-Result: Spam" {} \; | while read foo3
do
su -c "/usr/bin/dspam --client --user amavis --class=innocent --source=error < '${foo3}'" amavis
done
#
#########End DSPAM training########################
mv /home/spam/* /home/spam-backup/ > /dev/null 2>&1
mv /home/ham/* /home/ham-backup/ > /dev/null 2>&1
chown vmail: /home/spam-backup/* > /dev/null 2>&1
chown vmail: /home/ham-backup/* > /dev/null 2>&1
chmod 600 /home/spam-backup/* > /dev/null 2>&1
chmod 600 /home/ham-backup/* > /dev/null 2>&1
su -c "/usr/bin/sa-learn --backup" amavis  > /var/vmail/bayesrules

Replace all domain.tld by yours.

Crontab

add this line after the mysql backup in crontab root user :

# SA learn from spamtrap every 4h
0 */4 * * * /bin/bash /var/vmail/sa_trap.sh
30 */4 * * * /bin/bash /var/vmail/sa_learn.sh

sources

/spip/spip.php?article12
Because off Link limitation in post, please replace all missing domains by w.w.w.starbridge.o.r.g

oldstuff wrote:

You need to train DSPAM  for your needs, and as simple way to do it, I want to make 2 folder in all users mailboxes.
I thought it was needed to add in /etc/dovecot/conf.d/15-mailboxes the 2 folder I want to add, like this :

  mailbox "Sent Messages" {
    special_use = \Sent
 }
  mailbox Spam {
    #special_use = \Spam
    auto=subscribe
  }
  mailbox SpamFalse {
    #special_use = \SpamFalse
    auto=subscribe
  }
 # If you have a virtual "All messages" mailbox:
  #mailbox virtual/All {
  #  special_use = \All
  #}

Unfortunately it don't seem to work this way, conf.d folder don't even appears to be linked in dovecot.conf
So i added them in the /etc/dovecot/dovecot.conf file with the autocreate, but as I goggled this solution it seem deprecated.
And, must off all It didn't create folder where mailbox already exists.

So where i can add folder to existing mailboxes ?

My goal is, that spam who pass the filter (and vice-versa), can be moved by users in the "Spam" folder, which is periodically scanned by a script ( /spip/doc/Procmail/caremail/cron/sa-trap-sdbox like this one, adjust to the needs) who take email off all user to put them in spam@ or ham@. Those spam and ham address are after treated by another script ( /spip/doc/Procmail/caremail/sa-dspam-learn-sdbox ) which train dspam and spamassassin.
Those script run in crontab every X hours suited to your needs.

Is there requirements that could not be fit for iredmail ? Because this solution is way better for those who didn't use the webmail, and especially when it comes to explain to the 50years old technological refractors secretary how to handle spam...

DSPAM because the volume of spam that spamassassin let go is really outstanding (but working).
Please pardon my english im trying my best.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2

Re: [HOW-TO]DSPAM and SPAM Training

Thanks for your sharing. But i'm not familiar with DSPAM and cannot help here, sorry.
BTW, still no plan to integrate it into iRedMail.

3 (edited by reno31 2014-04-18 00:13:43)

Re: [HOW-TO]DSPAM and SPAM Training

Thats not much the DSPAM parts, it was more for sharing, but more folder part of dovecot.

How to handle them correctly do i need to add them in plugin {autocreate=spam,...but it seem deprecated or can i add them in the conf.d/15-mailboxes which doesn't seem to be linked in dovecot.conf (there is no !include conf.d/* ).
And how to add folder to existing mail accounts ?

I'll post the rest of the configuration if i manage to do it.

Edit: Found kinda similar problem here : http://www.iredmail.org/forum/topic5856 … tting.html

Edit 2 : Nevermind, i just do the solution in the other topic and it worked great