memcache原理1.5.8——記憶體分配與淘汰
阿新 • • 發佈：2019-01-05
本文主要參考：
好，下面上貨。
首先需要了解一下memcache是如何進行記憶體管理的。
記憶體分配首先通過命令列的-m引數給資料預留記憶體。然後記憶體會按照預設每頁1M大小分配給需要的slab class。然後這1M記憶體根據需要配切分成指定大小的chunks。然後看一下啟動memcache時的一些log比如上面的例子中，如果要儲存的資料大小是50bytes，那麼會儲存到slab1中，如果是90bytes，會儲存到slab2中。memcache會找到對應的合適大小的slab class，然後儲存到其中。$ ./memcached -vv slab class 1: chunk size 80 perslab 13107 slab class 2: chunk size 104 perslab 10082 slab class 3: chunk size 136 perslab 7710 slab class 4: chunk size 176 perslab 5957 slab class 5: chunk size 224 perslab 4681 slab class 6: chunk size 280 perslab 3744 slab class 7: chunk size 352 perslab 2978 slab class 8: chunk size 440 perslab 2383 slab class 9: chunk size 552 perslab 1899 slab class 10: chunk size 696 perslab 1506 [...etc...]
這裡需要強調的是，在老的1.4.25版本前，memcache是懶過期（沒有過期自動清理機制），同時也沒有page回收再分配機制，也就是我們常說的，一旦memcache儲存的格局定下來就不會發生變化了。
但是，在新版本中已經有了這兩個的支援，下面我詳細介紹一下新版memcache的記憶體分配機制和淘汰機制。
總體的過程是這樣的，首先lru_crawler會隔指定時間進行查詢是否有過期item。然後如果有過期會進行刪除處理（這裡和歷史版本1.4.X中的懶過期有區別，過期後會直接刪除掉）。下圖是回收memcache的page到global_page_pool中。
下圖是memcache的預設設定：
需要注意的是，啟動的命令需要新增
/usr/local/memcached/bin/memcached -d -m 4 -u root -p 13212 -o slab_reassign,slab_automove=1下面看一下官網的一些解釋：
slab重新分配
Slabs Reassign
--------------
NOTE: This command is subject to change as of this writing.
The slabs reassign command is used to redistribute memory once a running
instance has hit its limit. It might be desirable to have memory laid out
differently than was automatically assigned after the server started.
slabs reassign <source class> <dest class>\r\n
- <source class> is an id number for the slab class to steal a page from
A source class id of -1 means "pick from any valid class"
- <dest class> is an id number for the slab class to move a page to
The response line could be one of:
- "OK" to indicate the page has been scheduled to move
- "BUSY [message]" to indicate a page is already being processed, try again
later.
- "BADCLASS [message]" a bad class id was specified
- "NOSPARE [message]" source class has no spare pages
- "NOTFULL [message]" dest class must be full to move new pages to it
- "UNSAFE [message]" source class cannot move a page right now
- "SAME [message]" must specify different source/dest ids.
slab自動分配
Slabs Automove
--------------
NOTE: This command is subject to change as of this writing.
The slabs automove command enables a background thread which decides on its
own when to move memory between slab classes. Its implementation and options
will likely be in flux for several versions. See the wiki/mailing list for
more details.
The automover can be enabled or disabled at runtime with this command.
slabs automove <0|1>
- 0|1|2 is the indicator on whether to enable the slabs automover or not.
The response should always be "OK\r\n"
- <0> means to set the thread on standby
- <1> means to return pages to a global pool when there are more than 2 pages
worth of free chunks in a slab class. Pages are then re-assigned back into
other classes as-needed.
- <2> is a highly aggressive mode which causes pages to be moved every time
there is an eviction. It is not recommended to run for very long in this
mode unless your access patterns are very well understood.LRU_Crawler
-----------
NOTE: This command (and related commands) are subject to change as of this
writing.
The LRU Crawler is an optional background thread which will walk from the tail
toward the head of requested slab classes, actively freeing memory for expired
items. This is useful if you have a mix of items with both long and short
TTL's, but aren't accessed very often. This system is not required for normal
usage, and can add small amounts of latency and increase CPU usage.
lru_crawler <enable|disable>
- Enable or disable the LRU Crawler background thread.
The response line could be one of:
- "OK" to indicate the crawler has been started or stopped.
- "ERROR [message]" something went wrong while enabling or disabling.
lru_crawler sleep <microseconds>
- The number of microseconds to sleep in between each item checked for
expiration. Smaller numbers will obviously impact the system more.
A value of "0" disables the sleep, "1000000" (one second) is the max.
The response line could be one of:
- "OK"
- "CLIENT_ERROR [message]" indicating a format or bounds issue.
lru_crawler tocrawl <32u>
- The maximum number of items to inspect in a slab class per run request. This
allows you to avoid scanning all of very large slabs when it is unlikely to
find items to expire.
The response line could be one of:
- "OK"
- "CLIENT_ERROR [message]" indicating a format or bound issue.
lru_crawler crawl <classid,classid,classid|all>
- Takes a single, or a list of, numeric classids (ie: 1,3,10). This instructs
the crawler to start at the tail of each of these classids and run to the
head. The crawler cannot be stopped or restarted until it completes the
previous request.
The special keyword "all" instructs it to crawl all slabs with items in
them.
The response line could be one of:
- "OK" to indicate successful launch.
- "BUSY [message]" to indicate the crawler is already processing a request.
- "BADCLASS [message]" to indicate an invalid class was specified.
lru_crawler metadump <classid,classid,classid|all>
- Similar in function to the above "lru_crawler crawl" command, this function
outputs one line for every valid item found in the matching slab classes.
Similar to "cachedump", but does not lock the cache and can return all
items, not just 1MB worth.
Lines are in "key=value key2=value2" format, with value being URI encoded
(ie: %20 for a space).
The exact keys available are subject to change, but will include at least:
"key", "exp" (expiration time), "la", (last access time), "cas",
"fetch" (if item has been fetched before).
The response line could be one of:
- "OK" to indicate successful launch.
- "BUSY [message]" to indicate the crawler is already processing a request.
- "BADCLASS [message]" to indicate an invalid class was specified.