Good answer to "Riak secondary indices" performance impact

##A very good answer to help understand levelDB and secondary indices. Copied from stackoverflow.

#Question

1 down vote favorite

What is the performance impact of including secondary indexes (2i) in an object written to Riak? Let's consider two scenarios, both with a large number of objects within one bucket. Each object has a secondary index, let's call it example_bin (but it could be an integer index as well): Almost each value of example_bin is a different value. A query for one specific value of the index returns only one, or just a few objects. Such index could be an e-mail address or the registration time (as a unix timestamp). There is only several possible values of example_int index. Therefore, the query for a specific index value returns huge number of objects. Such index could represent a category of users, such as 'administrators' or 'customers'. What is the performance impact when updating these objects? I understand the index needs to be checked every time the object is updated. Can either of the examples above comprise a time- or resource-consuming task for Riak?

#Answer

1 down vote favorite

What is the performance impact of including secondary indexes (2i) in an object written to Riak? Let's consider two scenarios, both with a large number of objects within one bucket. Each object has a secondary index, let's call it example_bin (but it could be an integer index as well): Almost each value of example_bin is a different value. A query for one specific value of the index returns only one, or just a few objects. Such index could be an e-mail address or the registration time (as a unix timestamp). There is only several possible values of example_int index. Therefore, the query for a specific index value returns huge number of objects. Such index could represent a category of users, such as 'administrators' or 'customers'. What is the performance impact when updating these objects? I understand the index needs to be checked every time the object is updated. Can either of the examples above comprise a time- or resource-consuming task for Riak?

The performance impact of updating an object in Riak when using LevelDB as the backend should not be affected by how many entries there are in each index. It could however be affected by how much total data is stored in each partition, and how much data has been written since the key was last updated, and how many different index entries are specified for that individual object.

How LevelDB stores data

When a value is written to LevelDB, it is added to a .log file in the top level. When the .log file reaches a certain size (I think 1Mb, don't recall if that size is configurable or not), the file is cut off and a new one started. These files are not sorted. When there several files at the top level (level 0), a compaction is triggered. The compaction will combine one or more of the top level files, sort the keys they contain, and merge these sorted lists with the appropriate .sst files from Level 1. A manifest file is created for each of the sorted levels that indicates the range of keys stored in each .sst file. This process is also repeated as necessary for lower levels, with each level being able to store roughly 10 times the amount of data of the previous level.

When a new value is written for a key that already exists, it is simply written to the top level, masking any previously written values in lower levels. The previous values will be replaced as normal compactions move the new value down to lower levels.

How this affects reading

When a key is requested, LevelDB starts at level 0 and checks each file there to see if it contains the key. If not it moves down to level 1 and checks the file that the manifest indicates would contain the key. This is repeated for successively lower levels until the key is found or lowest level has been reached. Thus the most recently written value of the key is returned. As the total amount of data stored in each LevelDB backend increases, the number of levels used, the number of files that must be searched, and therefore the time to read the oldest data increases.

How indexes are implemented in LevelDB backends

When storing a value in a LevelDB backend, the raw key used by the backend the is sext encoding of {o,Bucket,Key}. If any index entries are specified in the object, for each index, an additional key is stored that is the sext encoding of {i,Bucket,IndexName,IndexValue,Key}.

In order to remove any obsolete index entries when an value is updated, a GET must be performed before each PUT or DELETE, the index specs of the previous object compared to the index specs of the object being stored, and any obsolete {i,...} keys removed and any new ones added.

Index Queries

Since LevelDB stores data in a sorted manner, an index query is implemented as a fold starting at the key {i,Bucket,IndexName,FirstValue,<<>>} through {i,Bucket,IndexName,LastValue,<<255,255,255,255>>} (the binary <<255,... is a theoretical value indicating the last possible key in the sort order). The manifest in each sorted level is consulted, so only data files that contain a portion of the range being folded need to be opened.

Summary

Almost each value of example_bin is a different value. A query for one specific value of the index returns only one, or just a few objects. Such index could be an e-mail address or the registration time (as a unix timestamp).

The index entries needed to query and individual value are likely to fall entirely within a single file in each sorted level, so this query will will need to open and search all of the files in Level 0, and 1 file in each lower level. The number of levels present, which is dependent on the amount of data stored, will be the largest determining factor.

There is only several possible values of example_int index. Therefore, the query for a specific index value returns huge number of objects. Such index could represent a category of users, such as 'administrators' or 'customers'.

The larger range of index entries to query an individual value makes it more likely that more than one file will need to be consulted for each sorted level. This query would take longer that the previous one because of the larger number of index entries included in the fold, and possibly the larger number of files that need to be opened.

What is the performance impact when updating these objects? I understand the index needs to be checked every time the object is updated. Can either of the examples above comprise a time- or resource-consuming task for Riak?

The time needed here will depend on that required to retrieve any old object, and the number of changes to the index entries. The entire index is never considered as a whole during this process, only the entries on the previous object, and the entries on the new object. So the performance would not be impacted by the number of entries in any index, but by the number of indexes for which this object has or had an entry.

##博客仅作个人记录##

1 down vote favorite

What is the performance impact of including secondary indexes (2i) in an object written to Riak? Let's consider two scenarios, both with a large number of objects within one bucket. Each object has a secondary index, let's call it example_bin (but it could be an integer index as well): Almost each value of example_bin is a different value. A query for one specific value of the index returns only one, or just a few objects. Such index could be an e-mail address or the registration time (as a unix timestamp). There is only several possible values of example_int index. Therefore, the query for a specific index value returns huge number of objects. Such index could represent a category of users, such as 'administrators' or 'customers'. What is the performance impact when updating these objects? I understand the index needs to be checked every time the object is updated. Can either of the examples above comprise a time- or resource-consuming task for Riak?