On both servers UID/GIDs are same:

Primary mail server
[root@mail log]# id vmail
uid=500(vmail) gid=500(vmail) groups=500(vmail)
[root@mail log]# id iredapd
uid=502(iredapd) gid=502(iredapd) groups=502(iredapd)
[root@mail log]# id policyd
uid=503(policyd) gid=503(policyd) groups=503(policyd)

Secondary mail server
[root@mail1 ~]# id vmail
uid=500(vmail) gid=500(vmail) groups=500(vmail)
[root@mail1 ~]# id iredapd
uid=502(iredapd) gid=502(iredapd) groups=502(iredapd)
[root@mail1 ~]# id policyd
uid=503(policyd) gid=503(policyd) groups=503(policyd)

I got it solved after complete re-installation of OS and iRedMail on primary server. Now everything is working fine. I think there must be some issue on Primary server, however it was fresh install then also it was acting weird. After re-installing CentOS 6.3 and configuring GlusterFS and iRedMail, it is working fine now.

Thanks for your kind support.

- vmail:vmail -> 2000:2000
- iredadmin:iredadmin -> 2001:2001
- iredapd:iredapd -> 2002:2002

Code commit log: https://bitbucket.org/zhb/iredmail/comm … 81a77e0771

Both server's vmail UID and GID is diferent, do I have to change it to same as primary on my secondary server?

I think so.

Looks like we have to use the same UID/GID while creating vmail user/group. I will fix it later.

Secondary server output:

[root@mail1 ~]# id admroot
id: admroot: No such user
[root@mail1 ~]# id iredapd
uid=502(iredapd) gid=502(iredapd) groups=502(iredapd)
[root@mail1 ~]# id policyd
uid=503(policyd) gid=503(policyd) groups=503(policyd)

I had deleted admroot user on both servers earlier.

[root@mail tmp]# id -u vmail
502
[root@mail tmp]# id -g vmail
503
[root@mail tmp]# id -g postfix
89
[root@mail tmp]# id -u postfix
89

[root@mail tmp]# id vmail
uid=502(vmail) gid=503(vmail) groups=503(vmail)
[root@mail tmp]# id postfix
uid=89(postfix) gid=89(postfix) groups=89(postfix),12(mail)

here secondary server's UID/GID output:
[root@mail1 ~]# id -u vmail
500
[root@mail1 ~]# id -g vmail
500
[root@mail1 ~]# id -u postfix
89
[root@mail1 ~]# id -g postfix
89

[root@mail1 ~]# id vmail
uid=500(vmail) gid=500(vmail) groups=500(vmail)
[root@mail1 ~]# id postfix
uid=89(postfix) gid=89(postfix) groups=89(postfix),12(mail)

Both server's vmail UID and GID is diferent, do I have to change it to same as primary on my secondary server?

Since it was reset to different user/group, please check UID and GID of below user/group:

- On primary server: user/group -> admroot
- On secondary server: user -> iredapd, group -> policyd

You can check them with command "id":

# id admroot
# id iredapd
# id policyd

If this is the issue, please also update file owner of /var/vmail.

Here is the out of Primary server:

# 2.0.18: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.22.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
auth_default_realm = mydomain.com
auth_mechanisms = PLAIN LOGIN
dict {
  acl = mysql:/etc/dovecot/dovecot-share-folder.conf
  expire = db:/var/lib/dovecot/expire/expire.db
  quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
first_valid_uid = 502
last_valid_uid = 502
listen = *
lock_method = dotlock
log_path = /var/log/dovecot.log
mail_gid = 503
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_uid = 502
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave
mmap_disable = yes
namespace {
  inbox = yes
  location =
  prefix =
  separator = /
  type = private
}
namespace {
  list = children
  location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
  prefix = Shared/%%u/
  separator = /
  subscriptions = yes
  type = shared
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
plugin {
  acl = vfile
  acl_shared_dict = proxy::acl
  auth_socket_path = /var/run/dovecot/auth-master
  autocreate = INBOX
  autocreate2 = Sent
  autocreate3 = Trash
  autocreate4 = Drafts
  autocreate5 = Junk
  autosubscribe = INBOX
  autosubscribe2 = Sent
  autosubscribe3 = Trash
  autosubscribe4 = Drafts
  autosubscribe5 = Junk
  expire = Trash 7 Trash/* 7 Junk 30
  expire_dict = proxy::expire
  quota = dict:user::proxy::quotadict
  quota_rule = *:storage=1G
  quota_warning = storage=85%% quota-warning 85 %u
  quota_warning2 = storage=90%% quota-warning 90 %u
  quota_warning3 = storage=95%% quota-warning 95 %u
  sieve = /%Lh/sieve/dovecot.sieve
  sieve_dir = /%Lh/sieve
  sieve_global_dir = /mnt/glusterfs/sieve
  sieve_global_path = /mnt/glusterfs/sieve/dovecot.sieve
}
protocols = pop3 imap sieve
service auth {
  unix_listener /var/spool/postfix/dovecot-auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-master {
    group = vmail
    mode = 0666
    user = vmail
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0660
    user = vmail
  }
}
service dict {
  unix_listener dict {
    group = vmail
    mode = 0660
    user = vmail
  }
}
service quota-warning {
  executable = script /usr/local/bin/dovecot-quota-warning.sh
  unix_listener quota-warning {
    group = vmail
    mode = 0660
    user = vmail
  }
}
ssl = required
ssl_cert = </etc/pki/tls/certs/iRedMail_CA.pem
ssl_key = </etc/pki/tls/private/iRedMail.key
userdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
protocol lda {
  auth_socket_path = /var/run/dovecot/auth-master
  lda_mailbox_autocreate = yes
  log_path = /var/log/sieve.log
  mail_plugins = quota sieve autocreate
  postmaster_address = root
}
protocol imap {
  imap_client_workarounds = tb-extra-mailbox-sep
  mail_plugins = quota imap_quota autocreate
}
protocol pop3 {
  mail_plugins = quota
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
  pop3_uidl_format = %08Xu%08Xv
}

Here is output of secondary server:

# 2.0.18: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.22.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
auth_default_realm = mydomain.com
auth_mechanisms = PLAIN LOGIN
dict {
  acl = mysql:/etc/dovecot/dovecot-share-folder.conf
  expire = db:/var/lib/dovecot/expire/expire.db
  quotadict = mysql:/etc/dovecot/dovecot-used-quota.conf
}
last_valid_uid = 500
listen = *
lock_method = dotlock
log_path = /var/log/dovecot.log
mail_gid = 500
mail_location = maildir:/%Lh/Maildir/:INDEX=/%Lh/Maildir/
mail_uid = 500
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date ihave
mmap_disable = yes
namespace {
  inbox = yes
  location =
  prefix =
  separator = /
  type = private
}
namespace {
  list = children
  location = maildir:/%%Lh/Maildir/:INDEX=/%%Lh/Maildir/Shared/%%u
  prefix = Shared/%%u/
  separator = /
  subscriptions = yes
  type = shared
}
passdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
plugin {
  acl = vfile
  acl_shared_dict = proxy::acl
  auth_socket_path = /var/run/dovecot/auth-master
  autocreate = INBOX
  autocreate2 = Sent
  autocreate3 = Trash
  autocreate4 = Drafts
  autocreate5 = Junk
  autosubscribe = INBOX
  autosubscribe2 = Sent
  autosubscribe3 = Trash
  autosubscribe4 = Drafts
  autosubscribe5 = Junk
  expire = Trash 7 Trash/* 7 Junk 30
  expire_dict = proxy::expire
  quota = dict:user::proxy::quotadict
  quota_rule = *:storage=1G
  quota_warning = storage=85%% quota-warning 85 %u
  quota_warning2 = storage=90%% quota-warning 90 %u
  quota_warning3 = storage=95%% quota-warning 95 %u
  sieve = /%Lh/sieve/dovecot.sieve
  sieve_dir = /%Lh/sieve
  sieve_global_dir = /mnt/glusterfs/sieve
  sieve_global_path = /mnt/glusterfs/sieve/dovecot.sieve
}
protocols = pop3 imap sieve
service auth {
  unix_listener /var/spool/postfix/dovecot-auth {
    group = postfix
    mode = 0666
    user = postfix
  }
  unix_listener auth-master {
    group = vmail
    mode = 0666
    user = vmail
  }
  unix_listener auth-userdb {
    group = vmail
    mode = 0660
    user = vmail
  }
}
service dict {
  unix_listener dict {
    group = vmail
    mode = 0660
    user = vmail
  }
}
service quota-warning {
  executable = script /usr/local/bin/dovecot-quota-warning.sh
  unix_listener quota-warning {
    group = vmail
    mode = 0660
    user = vmail
  }
}
ssl = required
ssl_cert = </etc/pki/tls/certs/iRedMail_CA.pem
ssl_key = </etc/pki/tls/private/iRedMail.key
userdb {
  args = /etc/dovecot/dovecot-ldap.conf
  driver = ldap
}
protocol lda {
  auth_socket_path = /var/run/dovecot/auth-master
  lda_mailbox_autocreate = yes
  log_path = /var/log/sieve.log
  mail_plugins = quota sieve autocreate
  postmaster_address = root
}
protocol imap {
  imap_client_workarounds = tb-extra-mailbox-sep
  mail_plugins = quota imap_quota autocreate
}
protocol pop3 {
  mail_plugins = quota
  pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
  pop3_uidl_format = %08Xu%08Xv
}

I tried to set same UIDs and GIDs on secondary server but once the permissions gets reset automatically dovecot conf changes to 500 uid gid automatically.

Could you please show us output of command "dovecot -n" on both GlusterFS nodes?

Upon googleing, i tried:

auditctl -w /mnt/glusterfs -p war

and check tracing bye:
ausearch -f /mnt/glusterfs/

it shows:

----
time->Wed Feb 13 14:47:43 2013
type=PATH msg=audit(1360766863.958:94): item=0 name="/mnt/glusterfs/vmail1/mydomain.com/s/y/s/sys-admin-2013.02.12.22.35.42//Maildir/.Sent" inode=1048598 dev=00:11 mode=040755 ouid=500 ogid=500 rdev=00:00
type=CWD msg=audit(1360766863.958:94):  cwd="/mnt/glusterfs/vmail1/mydomain.com/s/y/s/sys-admin-2013.02.12.22.35.42"
type=SYSCALL msg=audit(1360766863.958:94): arch=c000003e syscall=83 success=no exit=-17 a0=6552f0 a1=1ed a2=ffffffff a3=fffffffc items=1 ppid=3478 pid=5131 auid=4294967295 uid=502 gid=503 euid=502 suid=502 fsuid=502 egid=503 sgid=503 fsgid=503 tty=(none) ses=4294967295 comm="imap" exe="/usr/libexec/dovecot/imap" key=(null)

This is the output when permission gets changed automatically... on glusterfs directory.

#
# File generated by iRedMail (2013.02.12.22.12.30):
#
# Version:  0.8.3
# Project:  http://www.iredmail.org/
#
# Community: http://www.iredmail.org/forum/
#

#1   5   *   *   *   /usr/sbin/dovecot --exec-mail ext
1   */1   *   *   *   perl /var/www/awstats/awstats.pl -config=web -update >/dev/null
1   */1   *   *   *   perl /var/www/awstats/awstats.pl -config=smtp -update >/dev/null
# iRedMail: Backup OpenLDAP data on 03:00 AM
0   3   *   *   *   /bin/bash /mnt/glusterfs/backup/backup_openldap.sh
# iRedMail: Backup MySQL databases on 03:30 AM
30   3   *   *   *   /bin/bash /mnt/glusterfs/backup/backup_mysql.sh