redis服務開機啟動過異常，造成haresources裏面的服務掛載失敗（非法關機造成的）
阿新 • • 發佈：2018-05-02
haresources裏面的服務掛載失敗1、剛開始ha日誌報umail_postgresql啟動不了，發現/etc/init.d/目錄下這個服務文件都沒有，成為臨時文件了。於是以為在/etc/ha.d/haresources文件中去掉這個服務就可以。
備註：/etc/ha.d/haresources原本中有kkmail_nginx kkmail_mysqld kkmail_app kkmail_postgresql服務。
2、去掉後並重啟了服務器，發現依然不行。ha日誌繼續提示app服務啟動錯誤。
3、想知道為什麽啟動不了，先開啟下日誌看看：
備註：/etc/ha.d/haresources原本中有kkmail_nginx kkmail_mysqld kkmail_app kkmail_postgresql服務。
2、去掉後並重啟了服務器，發現依然不行。ha日誌繼續提示app服務啟動錯誤。
[root@hlt1 app]# /etc/init.d/kkmail_app start authenticator: started rulefilter: started receiver: started dispatcher: started sizequerier: started proxy_monitor: start failure exit_code= 1
查看進程監控服務，發現有幾個進程啟動不起來。
[root@hlt1 app]# /usr/local/kkmail/app/engine/bin/supervisorctl -c /usr/local/kkmail/app/conf/supervisord.ini authenticator STOPPED Not started dispatcher STOPPED Not started ldap_monitor FATAL Exited too quickly (process log may have details) operation RUNNING pid 26746, uptime 0:00:12 proxy_monitor STOPPED Not started receiver STOPPED Not started rulefilter STOPPED Not started search_manager BACKOFF Exited too quickly (process log may have details) service_listener FATAL Exited too quickly (process log may have details) sizequerier STOPPED Not started smtp_monitor FATAL Exited too quickly (process log may have details) task_monitor FATAL Exited too quickly (process log may have details) kkmail_clamd STARTING kkmail> restart all kkmail_clamd: stopped operation: stopped sizequerier: started authenticator: started rulefilter: started receiver: started dispatcher: started ldap_monitor: ERROR (abnormal termination) proxy_monitor: ERROR (abnormal termination) service_listener: ERROR (abnormal termination) operation: started smtp_monitor: ERROR (abnormal termination) task_monitor: ERROR (abnormal termination) search_manager: ERROR (abnormal termination) kkmail_clamd: started kkmail> exit
備註：
我剛開始好像打不開，報錯如下：
Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.
For help, use /usr/local/kkmail/app/engine/bin/supervisord -h
然後我ps -ef |grep supervisord，然後kill -9 id後，運行這個命令好了/usr/local/kkmail/app/engine/bin/supervisord -c /usr/local/kkmail/app/conf/supervisord.ini
3、想知道為什麽啟動不了，先開啟下日誌看看：
[root@hlt1 store]# cat /usr/local/kkmail/app/conf/supervisord.conf.d/proxy_monitor.ini ; proxy_monitor ; [program:proxy_monitor] command=/usr/local/kkmail/app/exec/proxy_monitor autostart=false autorestart=true startsecs=0 user=kkmail stdout_logfile=NONE stdout_logfile=/usr/local/kkmail/app/log/proxy_monitor.log #把這行開啟 ;stdout_logfile_maxbytes=50MB ;stdout_logfile_backups=5 stderr_logfile=NONE stderr_logfile=/usr/local/kkmail/app/log/proxy_monitor.err #把這行開啟 ;stderr_logfile_maxbytes=50MB ;stderr_logfile_backups=5
[root@hlt1 store]# cat /usr/local/kkmail/app/log/proxy_monitor.log
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: Traceback (most recent call last):
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/service/python/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: result = self._run(*self.args, **self.kwargs)
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/src/proxy_monitor.py", line 49, in check_open_subprocess
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/src/lib/ModProxyRule.py", line 47, in is_proxy_open
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/src/lib/ModProxyRule.py", line 15, in get_proxy_cache
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/engine/lib/python2.7/site-packages/redis/client.py", line 1231, in ttl
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: return self.execute_command('TTL', name)
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/engine/lib/python2.7/site-packages/redis/client.py", line 673, in execute_command
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: connection.send_command(*args)
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/engine/lib/python2.7/site-packages/redis/connection.py", line 610, in send_command
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: self.send_packed_command(self.pack_command(*args))
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/engine/lib/python2.7/site-packages/redis/connection.py", line 585, in send_packed_command
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: self.connect()
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: File "/usr/local/kkmail/app/engine/lib/python2.7/site-packages/redis/connection.py", line 489, in connect
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: raise ConnectionError(self._error_message(e))
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: ConnectionError: Error 111 connecting to unix socket: /usr/local/kkmail/data/redis/redis.sock. Connection refused.
2018-05-01 19:10:47 Proxy_Monitor[ERROR]: <Greenlet at 0x7fba5cc4de10: check_open_subprocess> failed with ConnectionError
4、看上面報錯好像是redis沒啟動成功造成的，然後依次把mysql、app重啟下，重啟後再到supervisor重啟所有進程。
[root@hlt1 supervisord.conf.d]# /usr/local/kkmail/app/engine/bin/supervisorctl -c /usr/local/kkmail/app/conf/supervisord.ini authenticator RUNNING pid 35317, uptime 0:00:15 dispatcher RUNNING pid 35431, uptime 0:00:14 ldap_monitor RUNNING pid 35295, uptime 0:00:15 operation RUNNING pid 35299, uptime 0:00:15 proxy_monitor RUNNING pid 35503, uptime 0:00:13 receiver RUNNING pid 35404, uptime 0:00:14 rulefilter RUNNING pid 35375, uptime 0:00:15 search_manager RUNNING pid 35298, uptime 0:00:15 service_listener RUNNING pid 35297, uptime 0:00:15 sizequerier RUNNING pid 35474, uptime 0:00:14 smtp_monitor RUNNING pid 35301, uptime 0:00:15 task_monitor STARTING kkmail_clamd STARTING kkmail> restart all ldap_monitor: stopped proxy_monitor: stopped kkmail_clamd: stopped service_listener: stopped sizequerier: stopped authenticator: stopped rulefilter: stopped search_manager: stopped receiver: stopped operation: stopped smtp_monitor: stopped task_monitor: stopped dispatcher: stopped sizequerier: started authenticator: started rulefilter: started receiver: started dispatcher: started ldap_monitor: started proxy_monitor: started service_listener: started search_manager: started operation: started smtp_monitor: started kkmail_clamd: started task_monitor: started
5、雖然解決了這些問題，但是重啟服務器發現redis服務依然不能開機啟動，還有apache服務。redis啟動不起來就造成app啟動不了，app啟動不起來就造成掛載失敗。
後來我把/etc/ha.d/haresources文件只留kkmail_nginx服務後，重啟服務器就可以掛載了。然後手動啟動app和mysqld。
6、兩臺服務器都升級到最新版本，並且停止ha服務來回測試OK。
redis服務開機啟動過異常，造成haresources裏面的服務掛載失敗（非法關機造成的）