eXtremeDB列数据库（FE金融版本）compression 压缩

eXtremeDB implementation with RLE (run lengthencoding) compression.

RLE is the compression algorithm that we use to compress sequences. No, other fields can't becompressed on the database level. However since the eXtremeDB exposesprogramming API, the application is free to compress any data before writinginto the database (through _put API) and consequently uncompress it afterreading it back (through the _get API), which could used by the app and customize the compression algorithm in order to lessen the data compression rate to less than  5%  and even lower.

On sequences with many duplicate elementsRLE exhibits  better performance and reduces memory requirements.Naturally if x={1,1,1,1,1,1,1,1,1,1}, y={2,2,2,2,2,2,2,2,2,2} tocalculate x+y its necessary to do just one addition instead of 10 as whenthe raw sequence is stored. Another strong point is better use of stack. Thenon-compressed implementation the number of elements in the tile is predefinedand is the same for all data types -- we keep 100 integers or a hundreddoubles. This is necessary to avoid extra verifications and load on demandwhile calculating binary expressions (operators). With the compression enabledwe have to do it anyway, so the tile can be loaded with a 200 integers and 100integers.

 

The downside of using RLE is that theimplementation requires more complicated logic, which negatively affectsperformance while calculating expressions and unavoidable extra space requiredto keep the counters (currently a byte for each element). Thus if the sequencecontains predominately unique elements, the non-compressed version isbetter. 

 

Besides, there is no change of code from compression to noncompression data, what the developer need to do is that the applications need to link in libmcoseqrle.ainstead of the libmcoseq.a.