MongoDB的mongos實例因無法分配mlock內存掛掉
阿新 • • 發佈：2017-11-16
大致 cannot 方法 class 列表 from read 版本 mongos
問題版本
mongodb-v3.4.4
問題描述
mongos兩天死了兩次，死前遺言只有日誌：
2017-11-01T11:25:27.135+0800 F - [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to mlock: Cannot allocate memory 2017-11-01T11:25:27.140+0800 I - [NetworkInterfaceASIO-TaskExecutorPool-1-0] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246 2017-11-01T11:25:27.140+0800 I - [NetworkInterfaceASIO-TaskExecutorPool-1-0] ***aborting after fassert() failure 2017-11-01T11:25:27.182+0800 I NETWORK [thread2] connection accepted from 100.97.227.0:37132 #282914 (1156 connections now open) 2017-11-01T11:25:27.191+0800 F - [NetworkInterfaceASIO-TaskExecutorPool-1-0] Got signal: 6 (Aborted). 0x55db5fab1241 0x55db5fab0339 0x55db5fab081d 0x7f848d578390 0x7f848d1d3428 0x7f848d1d502a 0x55db5f2e90e7 0x55db5f3452cd 0x55db5f407733 0x55db5f4039ce 0x55db5f40538c 0x55db5f3da9c7 0x55db5f3fc16f 0x55db5f3fedaf 0x55db5f7dba02 0x55db5f7dc1be 0x55db5f7ddf64 0x55db5f7a69d4 0x55db5f7b9498 0x55db5f7b9ae2 0x55db5fb8edd4 0x55db5fb8efd1 0x55db5f7cf91f 0x55db5fe8e120 0x7f848d56e6ba 0x7f848d2a482d
關鍵點：Failed to mlock: Cannot allocate memory和Got signal: 6 (Aborted).
排查
mongodb的jira上有兩個issue跟這個一模一樣：
- SERVER-29086
- SERVER-28997
分析
如SERVER-28997
SaslSCRAMSHA1ClientConversations have a SCRAMSecrets which they ‘ll pull out of the cache. SCRAMSecrets allocate secure storage in their default constructor, so they may be populated. Instead, SaslSCRAMSHA1ClientConversation and the cache should store shared_ptrs to SCRAMSecret.
大致意思就是說SaslSCRAMSHA1ClientConversation這玩意兒每次初始化對象都會申請一片安全的存儲，這個會填充memlock區域，正確的做法應該是共享這片區域。
處理
按照SERVER-29086的意思，可以調整memlock大小，來緩解這個問題。
按照SERVER-28997的意思，這是3.4.4版本的bug，所以可以升級到bug-fixed版本3.4.6，這是最終的解決方法。
註意：要多多關註各種issue，jira，郵件列表。
MongoDB的mongos實例因無法分配mlock內存掛掉