問題

neutron designate日誌中發現錯誤"Got lower serial for", 並且建立zone時永遠停留在PENDING狀態.

解決

minidns中看到下列日誌:

var/log/designate/designate-mdns.log.2.gz:2018-10-23 23:27:36.016 94713 INFO designate.mdns.handler [req-26d8910d-61d5-4fd6-bc6b-df7acaf36c12 - - - - -] NotFound, refusing. Question was xxx.openstack-au-east-2.oc.xxx.com. IN SOA 
var/log/designate/designate-mdns.log.2.gz:2018-10-23 23:27:36.024 94713 WARNING designate.mdns.handler [req-fa5518dd-9506-44f8-a2ee-ee7d79ffaa3c - - - - -] ZoneNotFound while handling axfr request. Question was xxx.openstack-au-east-2.oc.xxx.com. IN AXFR: ZoneNotFound: Could not find Zone 
 

根據程式碼[1], 查詢DB時報ZoneNotFound從而導致無法構建SOA無法構建axfr response, 從而導致minidns master與bind9 slave無法做zone transfer, 這樣導致bind9 slave的serial number也無法更新, 最終在minidns notify時看到"Got lower serial for"

在發生ZoneNotFound的附近有下列日誌:

[req-b48612b0-ed3e-46d9-8510-6634282ef0a2 - - - - -] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') [SQL: u'SELECT 1']
 

於是, 用下列測試程式重現了上述DB error,

import time
from sqlalchemy.engine import create_engine
url = 'mysql+pymysql://[email protected]:3306/mysql'
engine = create_engine(url, pool_recycle=4).connect()
query = 'SELECT NOW();'
while True:
    print('Q1', engine.execute(query).fetchall())
    engine.execute('SET wait_timeout=2')
    time.sleep(3)
    print('Q2', engine.execute(query).fetchall())
 

或者使用oslo.db測試程式:

mysql -u root -p -e "SET GLOBAL wait_timeout=5, slow_query_log=on, long_query_time=0.0;"

$ cat test.py
import sys
import time
from oslo_config import cfg
from oslo_db import options as db_options
from oslo_db.sqlalchemy import session as db_session
from sqlalchemy.sql.expression import select
_facade = db_session.EngineFacade("mysql://[email protected]/test")
x = _facade.get_session()
print(x.scalar(select([23])))
time.sleep(5)
print(x.scalar(select([23])))

連線線的過期時間(pool_recycle, 連線池裡的連線空閒一段時間後自動釋放, oslo中預設是3600)不能大於伺服器端的wait_timeout時間(這裡是2, 預設是8小時). 所以應該設定wait_timeout大於3600, 或者haproxy中的配置大於3600.
wait_timeout是mysql的一個設定，主要是用來斷開不使用的資料庫連線。當連線空閒的時間達到wait_timeout設定的最大值時，mysql會主動切斷這個連線，以供別的客戶端連線資料庫。這個值一般是28800，也就是8小時。在mysql中可以通過： show variables like “%timeout%”;獲取。
另外，當資料庫主動切斷連線的時候，mysql客戶端並不知道這個連線已經被切斷，所以程式並不知道其已經無效了，如果mysql客戶端再不支援ReConnect，雙重的問題疊加在一起就會導致連線池返回無效連線的可能.

其他

mysqldump --single-transaction -u root -p designate --skip-extended-insert > /tmp/designate-$(date +%s).sql

Reference