Hbase实战 2.2.7 合并：HBase日常工作



2.2.7 合并：HBase日常工作

The Delete command doesn’t delete the value immediately. Instead, it marks the
record for deletion. That is, a new “tombstone” record is written for that value, marking
it as deleted. The tombstone is used to indicate that the deleted value should no
longer be included in Get or Scan results. Because HFiles are immutable, it’s not until
a major compaction runs that these tombstone records are reconciled and space is truly
recovered from deleted records.

删除命令并不立即删除数据。它只是标记该记录已经被删除（标记保存在哪呢？）。也就是说，关于该数据的一个tombstone记录被写入，标记该记录被删除。该tombstone用做指示该记录不应该保存在Get、Scan的结果中。因为HFile是不变的，所以，直到大合并执行时，tombstone记录才会被同步，删除数据的空间才会被回收。

Compactions come in two flavors: minor and major. Both types result in a consolidation
of the data persisted in HFiles. A minor compaction folds HFiles together,
creating a larger HFile from multiple smaller HFiles, as shown in figure 2.3. Restricting
the number of HFiles is important for read performance, because all of them
must be referenced to read a complete row. During the compaction, HBase reads the
content of the existing HFiles, writing records into a new one. Then, it swaps in the
new HFile as the current active one and deletes the old ones that formed the new
one.2 HBase decides which HFiles to compact based on their number and relative
sizes. Minor compactions are designed to be minimally detrimental to HBase performance,
so there is an upper limit on the number of HFiles involved. All of these settings
are configurable.

合并有两种：大合并和小合并。两种合并的结果都是保存在HFile中的数据被合并。小合并把多个小的HFile结合在一起，形成一个较大的HFile。限制HFile的个数对提高读性能非常重要，因为读取一个完整行时，这些HFile都会被引用到（这个意思是要从所有的HFile的block中找到所有符合条件的block吧，HFile越多，查找越耗时？）。合并过程中，HBase读取现存HFile，把记录写入新的HFile。然后进行新旧HFile的切换并删除旧的HFile。HBase决定合并哪些HFile，取决于他们的号码和大小。小合并设计出发点是轻微影响HBase的性能，因此所涉及的HFile的数量有上限。这些设置都是可以配置的。

When a compaction operates over all HFiles in a column family in a given region, it’s
called a major compaction. Upon completion of a major compaction, all HFiles in the
column family are merged into a single file. Major compactions can also be triggered for the entire table (or a particular region) manually from the shell.

合并一个列簇的一个region的所有HFile的时候，这个合并类型就是大合并。当一个大合并结束的时候，一个列簇的所有HFile被合并成一个单独的文件（根据前面一句话，大合并应该不会跨region？）。通过shell，大合并可以手动触发，触发对象可以是整个表，也可以是一个特殊的region。

This is a relatively expensive operation and isn’t done often. Minor compactions, on the other hand, are relatively lightweight and happen more frequently.

大合并相对比较耗费资源，不能经常做。另一方面，小合并相对轻量级，可以经常进行。

Major compactions are the only chance HBase has to clean up deleted records. Resolving a delete requires removing both the deleted record and the deletion marker. There’s no guarantee that both the record and marker are in the same HFile. A major compaction is the only time when
HBase is guaranteed to have access to both of these entries at the same time.
The compaction process is described in greater detail, along with incremental
illustrations, in a post on the NGDATA blog.

大合并是HBase清除已删除记录的唯一机会。处理删除记录包括删除记录本身和删除标记。已删除的记录和删除标记不一定在同一个HFile里面（如果删除标记不保存在特殊的地方，那么查询某条记录岂不是要搜索所有的HFile吗？）。大合并是唯一时机，HBase同时访问这两个信息。合并过程在NGDATA blog（www.ngdata.com/site/blog/74-ng.html）里有详细描述，还有插图。