1 Topic by Dominique (edited by Dominique 2015-05-19 16:13:10)

  • Dominique
  • Member
  • Offline
  • Registered: 2014-01-09
  • Posts: 98

Topic: Cluebringer starting before MySQL is ready makes it fail forever.

==== Required information ====
- iRedMail version: 0.9.1
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL (MariaDB)
- Web server (Apache or Nginx): Nginx
- Linux/BSD distribution name and version: CentOS 7
- Related log if you're reporting an issue:
====

I did a fresh install of iRedMail 0.9.1 without tweaking anything.  The problem I'm experiencing is Cluebringer going and staying in bypass mode due to a database connection problem.  Only when I restart the Cluebringer service manually it starts working, so it seems like once started and not being able to connect to the database, Cluebringer will stay in bypass mode forever.

You would expect it to try reconnecting later, as a database connection problem could be temporary, but it doesn't. The only other solution I can think of for now is making the cluebringer service start later, depending on MySQL, but I can't seem to figure out how to do that. 

And even then it still wouldn't be a watertight solution, as every limit you impose could very well be ignored until next restart if a database connection arises... without any notification or attempt at recovery from Cluebringer's side.  I know you didn't create the program but I think it's a major flaw to consider when keeping on using it the way it functions now.


----
The logfile shows this:

[2015/05/19-09:11:00 - 1196] [CORE] INFO: 2015/05/19-09:11:00 CONNECT TCP Peer: "[127.0.0.1]:35096" Local: "[127.0.0.1]:10031"
[2015/05/19-09:11:00 - 1196] [CBPOLICYD] WARNING: Client in BYPASS mode due to DB connection failure!
[2015/05/19-09:11:00 - 1196] [CBPOLICYD] NOTICE: Client BYPASS timeout exceeded, reconnecting...
[2015/05/19-09:11:00 - 1100] [CORE] INFO: Starting "1" children
[2015/05/19-09:11:00 - 2672] [CORE] DEBUG: Child Preforked (2672)
[2015/05/19-09:11:00 - 2672] [CBPOLICYD] DEBUG: Starting up caching engine

So even when it says that the bypass timeout exceeded and will try reconnecting it doesn't.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

I will try to fix it, come back to you later.

3 Reply by Dominique

  • Dominique
  • Member
  • Offline
  • Registered: 2014-01-09
  • Posts: 98

Re: Cluebringer starting before MySQL is ready makes it fail forever.

I'm sure you're pretty busy at the moment... with all those new versions... just making sure you don't forget about this though.

Let me know if I can test anything else meanwhile, like reinstall from scratch, hoping is was a fluke... since I'm postponing my upgrade meanwhile anyway I wouldn't mind.

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

I'm working on it today, still didn't get it working with sytemd. Please give me some more time.
Sorry about this trouble.

5 Reply by mir

  • mir
  • Member
  • Offline
  • Registered: 2014-05-30
  • Posts: 170

Re: Cluebringer starting before MySQL is ready makes it fail forever.

ZhangHuangbin wrote:

I'm working on it today, still didn't get it working with sytemd. Please give me some more time.
Sorry about this trouble.

Have you seen this? https://wiki.archlinux.org/index.php/Sy … pendencies

6 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

Hi @mir, Thanks for sharing.

Here's the solution, works for me.

*) Create file /etc/systemd/system/cbpolicyd.service, full content:

[Unit]
Description=The Postfix Policyd Daemon
Wants=mariadb.service postgresql.service
After=mariadb.service postgresql.service

[Service]
Type=forking
ExecStart=/usr/sbin/cbpolicyd --config=/etc/policyd/cluebringer.conf

[Install]
WantedBy=multi-user.target

*) Systemd reload, restart Cluebringer:

# systemctl daemon-reload
# systemctl restart cbpolicyd
# systemctl enable cbpolicyd

*) Rename /etc/init.d/cbpolicyd. e.g.

# mv /etc/init.d/cbpolicyd /etc/init.d/cbpolicyd.bak

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

I'm repacking cluebringer RPM and preparing virtual machine to test it. Let me know whether or not it works for you.

8 Reply by mir

  • mir
  • Member
  • Offline
  • Registered: 2014-05-30
  • Posts: 170

Re: Cluebringer starting before MySQL is ready makes it fail forever.

ZhangHuangbin wrote:

md/system/cbpolicyd.service, full content:
[Unit]
Description=The Postfix Policyd Daemon
Wants=mariadb.service postgresql.service
After=mariadb.service postgresql.service

Shouldn't there also be a mysql.service? Debian Jessie defaults to install mysql.

9 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

Sorry, my mistake, forgot Debian Jessie support. It should be:

After=mariadb.service mysql.service postgresql.service

10 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

Oops, i forgot that Cluebringer is not available in Debian 8 (Jessie) and Ubuntu 15.04, so there's no need to list 'mysql.service' in 'After='.

We will develop throttling and greylisting plugins for iRedAPD to replace Cluebringer.

11 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

UPDATE:

If you didn't try above solution yet, please don't try it. I already packed new Cluebringer RPMs for CentOS 7, what you need to do is updating cluebringer with yum, then all is done.

Tested with iRedMail + PostgreSQL backend, works fine. Testing MySQL and LDAP backend right now.

12 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,770

Re: Cluebringer starting before MySQL is ready makes it fail forever.

UPDATE:

Verified new cluebringer RPM with MySQL/PGSQL/OpenLDAP backends on CentOS 7, all work as expected.
It's recommended to update cluebringer package to fix this issue:

# yum clean metadata
# yum update cluebringer

It will enable cbpolicyd by default, but i suggest you enable it manually to make sure it will be started after system starts up:

# systemctl enable cbpolicyd

Note: it will remove /etc/init.d/cbpolicyd, and install /usr/lib/systemd/system/cbpolicyd.service. You should manage service with 'systemctl' command. for example:

# systemctl restart cbpolicyd

13 Reply by Dominique

  • Dominique
  • Member
  • Offline
  • Registered: 2014-01-09
  • Posts: 98

Re: Cluebringer starting before MySQL is ready makes it fail forever.

I will run a few more tests, but it seems like updating cluebringer like you advised solved the problem.  It now seems like it's correctly connecting to the database every time an email is being sent... without failing at boot time.

Thanks a lot!