lepus部署及mysql和redis的監控配置詳情請見前兩篇文章

mysql錯誤解決

mysql監控錯誤資訊

2017-08-02 09:37:20 [WARNING] check mysql 192.168.1.222:10121 failure: sleep 3 seconds and check again.
./include/functions.py:45: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)

./include/functions.py:47: Warning: Out of range value for column 'pid' at row 1
  curs.execute(sql,param) 
./include/functions.py:45: Warning: Out of range value for column 'connections' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)
 

http://blog.csdn.net/wlzjsj/article/details/76572805

原始碼閱讀

mysql監控的原始碼是check_mysql.py 貼出根據出錯資訊的部分程式碼

def mysql_exec(sql,param):
    try:
        conn=MySQLdb.connect(host=host,user=user,passwd=passwd,port=int(port),connect_timeout=5,charset='utf8')
        conn.select_db(dbname)
        curs = conn.cursor()
        if param <> '':
            curs.execute(sql,param)
        else:
            #print "sql:" + str(sql)
            #print "parm:" + str(param)
            curs.execute(sql)
        conn.commit()
        curs.close()
        conn.close()
    except Exception,e:
       print "mysql execute: " + str(e)
       #import traceback
       #traceback.print_exc()
 

注：#註釋的是我的除錯程式碼 打出錯誤資訊後，

./include/functions.py:47: Warning: Out of range value for column 'pid' at row 1
  curs.execute(sql,param)

sql:insert into mysql_processlist(server_id,host,port,tags,pid,p_user,p_host,p_db,command,time,status,info) values(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
parm:(282L, u'192.xx.xx.xx', u'3306', u'46_3306', 4091675221L, u'common', u'192.xx.xx.xx:8647', u'dbname', u'Query', 2L, u'updating', u"DELETE FROM xxx WHERE EndDateTime<'2017-08-01T00:00:00' LIMIT 2500")
 

可見第一個錯誤pid的超出範圍是mysql_processlist表的pid欄位出的問題，我們看下這個表結構：

mysql> desc mysql_processlist;
+-------------+--------------+------+-----+-------------------+----------------+
| Field       | Type         | Null | Key | Default           | Extra          |
+-------------+--------------+------+-----+-------------------+----------------+
| id          | int(11)      | NO   | PRI | NULL              | auto_increment |
| server_id   | smallint(4)  | YES  | MUL | NULL              |                |
| host        | varchar(30)  | NO   |     | NULL              |                |
| port        | varchar(10)  | NO   |     | NULL              |                |
| tags        | varchar(50)  | NO   | MUL |                   |                |
| pid         | int(10)      | YES  |     | NULL              |                |
| p_user      | varchar(50)  | YES  |     | NULL              |                |
| p_host      | varchar(50)  | YES  |     | NULL              |                |
| p_db        | varchar(30)  | YES  |     | NULL              |                |
| command     | varchar(30)  | YES  |     | NULL              |                |
| time        | varchar(200) | NO   |     | 0                 |                |
| status      | varchar(50)  | YES  |     | NULL              |                |
| info        | text         | YES  |     | NULL              |                |
| create_time | timestamp    | YES  | MUL | CURRENT_TIMESTAMP |                |
+-------------+--------------+------+-----+-------------------+----------------+
14 rows in set (0.01 sec)

http://blog.csdn.net/wlzjsj/article/details/76572805
pid是int（10），我們知道int的長度是從 -2^31 (-2,147,483,648) 到 2^31 - 1 (2,147,483,647) 的整型資料（所有數字）。儲存大小為 4 個位元組。

我們儲存的位數已經超過了最大儲存位，這裡參考下mysql自帶的information_schema.PROCESSLIST表，可以看出ID欄位是以biginnt(4)儲存的，bigint範圍是從 -2^63 (-9223372036854775808) 到 2^63-1 (9223372036854775807) 的整型資料（所有數字）。儲存大小為 8 個位元組。

我們需要把pid修改為bigint(4).

解決

mysql> alter table mysql_processlist modify pid bigint(4);

那接下來看其它錯誤，有了第一個思路，接下來基本可以判斷都是表結構的問題了，程式碼裡分別做了param為空和不為空的2種判斷進行sql插入，所以需要修改2個地方，param不為空時除錯結果：

sql:insert into mysql_status(server_id,host,port,tags,connect,role,uptime,version,max_connections,max_connect_errors,open_files_limit,table_open_cache,max_tmp_tables,max_heap_table_size,max_allowed_packet,open_files,open_tables,threads_connected,threads_running,threads_waits,threads_created,threads_cached,connections,aborted_clients,aborted_connects,connections_persecond,bytes_received_persecond,bytes_sent_persecond,com_select_persecond,com_insert_persecond,com_update_persecond,com_delete_persecond,com_commit_persecond,com_rollback_persecond,questions_persecond,queries_persecond,transaction_persecond,created_tmp_tables_persecond,created_tmp_disk_tables_persecond,created_tmp_files_persecond,table_locks_immediate_persecond,table_locks_waited_persecond,key_buffer_size,sort_buffer_size,join_buffer_size,key_blocks_not_flushed,key_blocks_unused,key_blocks_used,key_read_requests_persecond,key_reads_persecond,key_write_requests_persecond,key_writes_persecond,innodb_version,innodb_buffer_pool_instances,innodb_buffer_pool_size,innodb_doublewrite,innodb_file_per_table,innodb_flush_log_at_trx_commit,innodb_flush_method,innodb_force_recovery,innodb_io_capacity,innodb_read_io_threads,innodb_write_io_threads,innodb_buffer_pool_pages_total,innodb_buffer_pool_pages_data,innodb_buffer_pool_pages_dirty,innodb_buffer_pool_pages_flushed,innodb_buffer_pool_pages_free,innodb_buffer_pool_pages_misc,innodb_page_size,innodb_pages_created,innodb_pages_read,innodb_pages_written,innodb_row_lock_current_waits,innodb_buffer_pool_pages_flushed_persecond,innodb_buffer_pool_read_requests_persecond,innodb_buffer_pool_reads_persecond,innodb_buffer_pool_write_requests_persecond,innodb_rows_read_persecond,innodb_rows_inserted_persecond,innodb_rows_updated_persecond,innodb_rows_deleted_persecond,query_cache_hitrate,thread_cache_hitrate,key_buffer_read_rate,key_buffer_write_rate,key_blocks_used_rate,created_tmp_disk_tables_rate,connections_usage_rate,open_files_usage_rate,open_tables_usage_rate) values(%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s);
parm:(278L, u'192.xx.xx.xx', u'3306', u'2x_3306', 1, 'master', u'8788268', u'5.5.28-log', u'2768', u'1000000', u'16384', u'1024', u'32', u'16777216', u'16777216', u'374', u'1024', u'260', u'3', 1L, u'7367813', u'122', u'3389257780', u'19583460', u'147622', 400, 158, 1388, 560, 17, 22, 0, 0, 0, 1633, 1633, 0, 4, 1, 0, 613, 0, u'536870912', u'8388608', u'4194304', u'0', u'0', u'428684', 0, 0, 0, 0, u'1.1.8', u'1', u'17179869184', u'ON', u'OFF', u'2', u'', u'0', u'200', u'4', u'4', 1048576, 1001590, 58340, 122667533, 92, 46894, 16384, 5966915, 10902064, 122667533, 0, 28, 558362, 0, 416, 279904, 19, 16, 0, '     0.00', '     1.00', '     1.00', '     0.62', '     1.00', '     0.02', '     0.09', '     0.02', '     1.00')


./include/functions.py:47: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)
./include/functions.py:47: Warning: Out of range value for column 'connections' at row 1
  curs.execute(sql,param)

解決：

mysql> alter table mysql_status modify max_connect_errors int(10);
mysql> alter table mysql_status modify connections  bigint(4);

param為空時除錯結果：

sql:insert into mysql_status_history SELECT *,LEFT(REPLACE(REPLACE(REPLACE(create_time,'-',''),' ',''),':',''),12) from mysql_status
param:
--------
./include/functions.py:47: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)
./include/functions.py:47: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)
./include/functions.py:47: Warning: Out of range value for column 'connections' at row 1
  curs.execute(sql,param)
./include/functions.py:47: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)
./include/functions.py:47: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)
./include/functions.py:47: Warning: Out of range value for column 'max_connect_errors' at row 1
  curs.execute(sql,param)

手動執行sql：

mysql> insert into mysql_status_history SELECT *,LEFT(REPLACE(REPLACE(REPLACE(create_time,'-',''),' ',''),':',''),12) from mysql_status
    -> ;
Query OK, 10 rows affected, 12 warnings (0.00 sec)
Records: 10  Duplicates: 0  Warnings: 12

mysql> show warnings;
+---------+------+--------------------------------------------------------------+
| Level   | Code | Message                                                      |
+---------+------+--------------------------------------------------------------+
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 1  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 2  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 3  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 4  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 5  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 6  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 7  |
| Warning | 1264 | Out of range value for column 'connections' at row 7         |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 8  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 9  |
| Warning | 1264 | Out of range value for column 'max_connect_errors' at row 10 |
| Warning | 1264 | Out of range value for column 'connections' at row 10        |
+---------+------+--------------------------------------------------------------+
12 rows in set (0.00 sec)

解決：

mysql> alter table mysql_status_history modify max_connect_errors int(10);
Query OK, 51708 rows affected (4.82 sec)
Records: 51708  Duplicates: 0  Warnings: 0

mysql> alter table mysql_status_history modify  connections  bigint(4);
Query OK, 51708 rows affected (3.74 sec)
Records: 51708  Duplicates: 0  Warnings: 0

http://blog.csdn.net/wlzjsj/article/details/76572805

Redis錯誤資訊

redis的錯誤資訊很相似，基本不用怎麼判斷了。這裡原始碼的除錯就不貼出了：

[email protected]:/usr/local/lepus# python check_redis.py
2017-08-02 11:00:58 [INFO] check redis controller started.
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'keyspace_hits' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
./include/functions.py:45: Warning: Out of range value for column 'rdb_changes_since_last_save' at row 1
  curs.execute(sql,param)
2017-08-02 11:00:59 [INFO] check redis controller finished.

解決

mysql> alter table redis_status modify rdb_changes_since_last_save int(10);
Query OK, 15 rows affected (0.01 sec)
Records: 15  Duplicates: 0  Warnings: 0

mysql> alter table redis_status modify keyspace_hits bigint(4);
Query OK, 15 rows affected (0.02 sec)
Records: 15  Duplicates: 0  Warnings: 0
mysql> alter table redis_status_history modify keyspace_hits bigint(4);
Query OK, 187 rows affected (0.02 sec)
Records: 187  Duplicates: 0  Warnings: 0

mysql> alter table redis_status_history modify rdb_changes_since_last_save int(10);
Query OK, 187 rows affected (0.02 sec)
Records: 187  Duplicates: 0  Warnings: 0

http://blog.csdn.net/wlzjsj/article/details/76572805

剩餘問題：

2017-08-02 11:28:36 [WARNING] check mysql 192.xx.x.xx:xx failure: -1 error totally whack
2017-08-02 11:28:36 [WARNING] check mysql 192.xx.xx.xx:xx failure: sleep 3 seconds and check again.
2017-08-02 11:28:36 [WARNING] check mysql 192.xx.xx.xx:xx failure: -1 error totally whack
2017-08-02 11:28:36 [WARNING] check mysql 192.xx.1.xx:xxx failure: sleep 3 seconds and check again.
2017-08-02 11:28:36 [WARNING] check mysql 192.x8.1.xxx:3306 failure: -1 error totally whack
2017-08-02 11:28:36 [WARNING] check mysql 192.1xx.1.xx:3xx failure: sleep 3 seconds and check again.