嗯，之前摘录了一篇文章，今天再补一篇英语的，需要的自己谷歌翻译，以下原文完整转载自：Memcached and unexpected evictions
Memcached and unexpected evictions
A few months ago we had to troubleshoot a Memcached cluster that was evicting a lot of objects (all of the same size) very frequently. Objects that were supposed to last for 3 days (using TTL) would last only a few minutes forcing the service using the cache to re-send the original -and supposedly cached request- to an external service overloading it and causing rate limiting errors and other issues.
How does it work
When Memcached starts up, it reserves a space of memory in the operating system and then will create the slabs where it will persist data. A slab is basically a “category” defined by a size. Objects of a certain size will get assigned to a page that belongs in a respective slab. Slabs allow classifying pages for objects of similar sizes
A slab may contain none, one or multiple pages, each page contains only items of the same size (the size defined in the slab category), and they will keep only items of that size until the server gets restarted or flushed. Pages do not get reassigned to other slabs. This means, that once a slab is created for a specific size, the space allotted for this slab is taken until the server is restarted or flushed.
To better understand this, let’s take a look at example: the data written has a size and it gets categorized based on the size per slab. So here are some slabs definitions:
name, max sizeslab 1, 100bytesslab 2, 200bytesslab 3, 300bytesA page’s default size is 1MB (default max size of an object in memcached BTW), so for example with the slabs above, a page of slab 1 will be able to hold 10240 items of that size (1MB / 100 bytes). A slab 2 page will be able to hold 5120 and so on. Each page will have chunks of data, and each chunk is reserved with the size assigned by the slab.
When you put a new object, say of size 140 bytes, it will get assigned to a page in the slab 2 because slab1_max_size < 140 < slab2_max_size.
Once a page is full in a slab, if there’s enough memory left (from the reserved when starting up) then memcached will create a new page inside that slab.
Notice how inserting a 140 bytes item in slab 2 makes you automatically lose 60 bytes of space that you won’t get back until the item is replaced with a larger item when it is expired or evicted because the chunk size reserved for items in pages in that slab is 200 bytes. This is part of the drawbacks of the approach used by memcached for memory management as you might lose some memory. Now, losing 60 bytes might not be the end of the world, but say you have the following slabs (again fictional slabs)
name | sizeslab 41 | 750000 bytesslab 42 | 1000000 bytes (1 MB)Then say you persist an object of size 752 KB, that will get assigned to slab42 and will take one entire page (remember pages are 1MB). You would have lost 248 KB. Now, that’s a bigger loss and imagine if by mistake you deploy a program that writes 10K items of similar size to that slab. You notice the error a few minutes later, but it’s too late. Your memcached server has already created 10K pages for slab 42 (10 GB), and you’ve “lost” 2.48 GB of memory (248KB * 10K pages) and your server will keep those 10K pages in memory until is restarted or flushed -even if the items in those pages have expired, reducing the chances of creating new pages for a slab that is writing real data as you have 10GB available for that.
With that said it is important to notice that evictions don’t happen globally but instead based on slabs. If there’s enough space to create new pages in a slab, that would happen, but memcached will start evicting items.
Back to our problematic evictions…
The cause of our problem was that at some point we stored a lot of items in very specific slabs of different sizes, some of these slabs reaching more than 12000 pages (12 GB assigned to just one slab) Even if that data that was not longer accessed and won’t get reassigned to another slab without flushing/restarting the server.
So the other slabs (specifically the one where we were trying to persist our data) would run out of pages because all the pages have been already assigned to other slabs and then Memcached would start evicting items in the slab and recycle pages to make space for the new data and since we were inserting new items very frequently, the reutilization of pages would cause these evictions.
But aren’t you wasting a lot of space…
One point worth mentioning is that the memory model used by memcached while it might be a bit inefficient as you tend to lose some memory space here and there but it is also very robust and safe. First, the server will never run out of memory -it will reuse pages, and that is definitely a huge gain. Also, it doesn’t require a lot of monitoring for out of memory errors, you just have to monitor your slab creation.
Another good thing about this model is that you can actually tune the service for your needs according to the size of the data you are persisting and you could also modify the size of your pages. All this, allows you to adjust the server for your needs while keeping some of the benefits that this model offers you. More info here: https://dom.as/2008/12/25/memcached-for-small-objects/
How to monitor your slabs
You just need to be aware of the kind of data you are storing there and check your slabs periodically by using the command ‘stats slabs’. This command will return you something like:
…
STAT 15:chunk_size 2320
STAT 15:chunks_per_page 451
STAT 15:total_pages 1213
STAT 15:total_chunks 547063
STAT 15:used_chunks 546908
STAT 15:free_chunks 1
STAT 15:free_chunks_end 154
STAT 15:mem_requested 1600678512
STAT 15:get_hits 122277227
STAT 15:cmd_set 116294868
STAT 15:delete_hits 0
STAT 15:incr_hits 0
STAT 15:decr_hits 0
STAT 15:cas_hits 0
STAT 15:cas_badval 0
…The main take aways from the results above are:
- total_pages: shows you how many pages have been created for this slab. Remember 1 page equals 1 MB so, keep that in mind when you want to know how many MB are you using for a certain slab
- chunk_size: size of each item inside a page in this category
- chunks_per_page: how many items per page will get assigned here. This is helpful for debugging, as you can estimate how quickly you are going to create new pages based on the rate of items you write to the server.
TL;DR
Objects that are supposed to remain in Memcached for extended periods are evicted very frequently Check your slabs, maybe other slabs are taking all the space and to add new items Memcached has to evict in order to reuse those pages.
Some good resources to learn more:
https://www.adayinthelifeof.nl/2011/02/06/memcache-internals/
https://work.tinou.com/2011/04/memcached-for-dummies.html
https://www.couyon.net/blog/using-redis-as-a-lru-cache-dont-do-it
如您从本文得到了有价值的信息或帮助，请考虑扫描文末二维码捐赠和鼓励。
如本文对您有用，捐赠和留言 将是对我最好的支持~（捐赠可转为站内积分）
如愿意，请向朋友推荐本站，谢谢。
尊重他人劳动成果。转载请务必附上原文链接，我将感激不尽。
留言