At a high level, doc values are essentially a serialized column-store. As we discussed in the last section, column-stores excel at certain operations because the data is naturally laid out in a fashion that is amenable to those queries.
But they also excel at compressing data, particularly numbers. This is important for both saving space on disk and for faster access. Modern CPU’s are many orders of magnitude faster than disk drives (although the gap is narrowing quickly with upcoming NVMe drives). That means it is often advantageous to minimize the amount of data that must be read from disk, even if it requires extra CPU cycles to decompress.
To see how it can help compression, take this set of doc values for a numeric field:
Doc Terms
-----------------------------------------------------------------
Doc_1 | 100
Doc_2 | 1000
Doc_3 | 1500
Doc_4 | 1200
Doc_5 | 300
Doc_6 | 1900
Doc_7 | 4200
-----------------------------------------------------------------
The column-stride layout means we have a contiguous block of numbers:[100,1000,1500,1200,300,1900,4200].
xxx
Doc values use several tricks like this. In order, the following compression schemes are checked:
If all values are identical (or missing), set a flag and record the value
If there are fewer than 256 values, a simple table encoding is used
If there are > 256 values, check to see if there is a common divisor
If there is no common divisor, encode everything as an offset from the smallest value
You’ll note that these compression schemes are not "traditional" general purpose compression like DEFLATE or LZ4. Because the structure of column-stores are rigid and well-defined, we can achieve higher compression by using specialized schemes rather than the more general compression algorithms like LZ4.
You may be thinking "Well that’s great for numbers, but what about strings?" Strings are encoded similarly, with the help of an ordinal table. The strings are de-duplicated and sorted into a table, assigned an ID, and then those ID’s are used as numeric doc values. Which means strings enjoy many of the same compression benefits that numerics do.
The ordinal table itself has some compression tricks, such as using fixed, variable or prefix-encoded strings.
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
