Memstore-Local Allocation Buffer

下面的内容全部来自 HBase: The Definitive Guide 中的性能优化一节，目的就是对自己的阅读进行一个简单的总结。

主要介绍什么是MSLAB？ 为什么提出这个东西？ 为什么能够带来性能上的提升？ 是否有什么副作用？ 是否广泛应用在production场景中？

Version 0.90 of HBase introduced an advanced mechanism to mitigate the issue of heap fragmentation due to too much churn on the memstore instances of a region server: the memstore-local allocation buffers, or MSLAB in short.

The above explained how tenured KeyValue instances, once they are flushed to disk, cause holes in the old generation heap. Once there is no more space large enough available for a new allocation caused by the fragmentation the JRE falls back to the stop the world garbage collector, which rewrites the entire heap space compacting it to the remaining active objects.

The key to reduce these compacting collections is to reduce fragmentation, and the MSLABs were built to help with that. The key idea behind them is that only objects of exactly the same size should be allocated from the heap. Once these objects tenure and eventually get collected they leave holes in the heap of a specific size. Allocating new objects subsequently of the exact same size will always reuse these holes: there is no promotion error and therefore no stop-the-world compacting collection is required.

The MSLABs are buffers of fixed sizes containing KeyValue instances of varying sizes. Whenever a buffer cannot complete fit a newly added KeyValue then it is considered full and a new buffer is created, once again of the given fixed size.

The feature is by default enabled in version 0.92, and disabled in version 0.90 of HBase. You can use the hbase.hregion.memstore.mslab.enabled configuration property to override it either way. It is recommended to thoroughly test your setup with this new feature, as it might delay the inevitable only longer - which is a good thing - and therefore you still have to deal with long garbage collection pauses. If you are still experiences these pauses you could plan to restart the servers every few days, or weeks, before the pause happens.

Note

As of this writing this feature is not yet widely tested in long running production environments. Due diligence is advised.

The size of each allocated, fixed-sized buffer is controlled by the hbase.hregion.memstore.mslab.chunksize property. The default is 2MB and is a sensible starting point. Based on your KeyValue instances you may have to adjust this value: if you store larger cells, for example, 100 kilobytes in size, then you need to increase the MSLAB size to fit more than just a few cells.

There is also an upper boundary of what is stored in the buffers. It is set by the hbase.hregion.memstore.mslab.max.allocation property and defaults to 256KB. Any cell that is larger will be directly allocated in the Java heap. If you are storing a lot of KeyValue instances that are larger then this upper limit, you will run into fragmentation related pauses earlier.

The MSLABs come not without cost: they are more wasteful in regards to heap usage as you will most likely not fill every buffer to the last byte. The remaining unused capacity of the buffer is wasted. Once again, you need to find a balance between this extra space needed and a significantly increased time before your servers are stopped by the compacting collector.

Finally, because the buffers require an additional byte array copy operation they are also slightly slower, compared to directly using the KeyValue instances. Measure the impact on your workload and see if it has no adverse effect.