Special encoding of small aggregate data types
特别编码小集合的数据类型hash-max-zipmap-entries 64 (hash-max-ziplist-entries for Redis >= 2.6)
hash-max-zipmap-value 512 (hash-max-ziplist-value for Redis >= 2.6)
list-max-ziplist-entries 512
list-max-ziplist-value 64
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
set-max-intset-entries 512
If a specially encoded value will overflow the configured max size, Redis will automatically convert it into normal encoding. This operation is very fast for small values, but if you change the setting in order to use specially encoded values for much larger aggregate types the suggestion is to run some benchmark and test to check the conversion time.
如果这个特殊的编码的值大于配置的值,Redis将自动转换成普通的编码。配置小值的情况下操作是非常快的,但是如果你想加大配置,你最好反复测试。
Using 32 bit instances使用32为redis实例
Bit and byte level operations(bit和byte级别的操作)
Redis 2.2 introduced new bit and byte level operations: GETRANGE, SETRANGE, GETBIT and SETBIT. Using this commands you can treat the Redis string type as a random access array. For instance if you have an application where users are identified by an unique progressive integer number, you can use a bitmap in order to save information about sex of users, setting the bit for females and clearing it for males, or the other way around. With 100 millions of users this data will take just 12 megabyte of RAM in a Redis instance. You can do the same using GETRANGE and SETRANGE in order to store one byte of information for user. This is just an example but it is actually possible to model a number of problems in very little space with this new primitives.
Use hashes when possible在可能的时候使用hash
Small hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible. For instance if you have objects representing users in a web application, instead of using different keys for name, surname, email, password, use a single hash with all the required fields.
小hash编码消耗的空间很小,因此如果可能的话应该尝试使用它。比如,在web应用中user,使用不同的key表示name,seruname,email,password,这些可以使用hash中的fields代替。
If you want to know more about this, read the next section.
如果想知道更多关于hash的使用,继续往下看。
Using hashes to abstract a very memory efficient plain key-value store on top of Redis 在redis中使用hash,在key-value存储中是领先的
I understand the title of this section is a bit scaring, but I'm going to explain in details what this is about.
Basically it is possible to model a plain key-value store using Redis where values can just be just strings, that is not just more memory efficient than Redis plain keys but also much more memory efficient than memcached.
当使用redis,values仅仅只是string类型,那么redis的效率比memcached高,但是redis要消耗更多的内存。
Let's start with some fact: a few keys use a lot more memory than a single key containing a hash with a few fields. How is this possible? We use a trick. In theory in order to guarantee that we perform lookups in constant time (also known as O(1) in big O notation) there is the need to use a data structure with a constant time complexity in the average case, like a hash table.
But many times hashes contain just a few fields. When hashes are small we can instead just encode them in an O(N)data structure, like a linear array with length-prefixed key value pairs. Since we do this only when N is small, the amortized time for HGET and HSET commands is still O(1): the hash will be converted into a real hash table as soon as the number of elements it contains will grow too much (you can configure the limit in redis.conf).
很多时候hash只包含几个字段。当hash比较小时,我们可以只编码一个O(N)的数据结构,像一个有 长度和前缀key-value对的线性数组。因为现在的N非常小,所有HGET和HSET命令消耗也只是O(1):当这个hash不断地增加新元素那么它将被转换成一个真正的hash table(转换的限制可以在redis.conf中配置)
This does not work well just from the point of view of time complexity, but also from the point of view of constant times, since a linear array of key value pairs happens to play very well with the CPU cache (it has a better cache locality than a hash table).
从时间复杂度来说这并不是很好,但是从恒定时间的观点上来看,key- value对的线性数组很好地利用了CPU 的缓存(它的缓存位置比hash好)。
However since hash fields and values are not (always) represented as full featured Redis objects, hash fields can't have an associated time to live (expire) like a real key, and can only contain a string. But we are okay with this, this was anyway the intention when the hash data type API was designed (we trust simplicity more than features, so nested data structures are not allowed, as expires of single fields are not allowed).
然而hash field和value不能表示所有的Redis 对象,hash field 不能像真正的key一样关联时间,并且只能包含string类型。但是这也是好的,因为这就是hash数据类型被设计的意图(我们坚信简单的特点,因此在hash中,嵌套不被允许,超时也不被允许)。
So hashes are memory efficient. This is very useful when using hashes to represent objects or to model other problems when there are group of related fields. But what about if we have a plain key value business?
因此hash是有内存效率的。在我们需要对表示一些有相关联字段的对象或者问题模式的时候非常有用。但是,如何只有一个 key-value的 业务会怎么样呢?
Imagine we want to use Redis as a cache for many small objects, that can be JSON encoded objects, small HTML fragments, simple key -> boolean values and so forth. Basically anything is a string -> string map with small keys and values.
Now let's assume the objects we want to cache are numbered, like:
现在假定我们要去缓存这些对象,像:
- object:102393
- object:1234
- object:5
- a Key named object:12
- a Field named 34
HSET object:12 34 somevalue
As you can see every hash will end containing 100 fields, that is an optimal compromise between CPU and memory saved.
require 'rubygems'
require 'redis'
UseOptimization = true
def hash_get_key_field(key)
s = key.split(":")
if s[1].length > 2
{:key => s[0]+":"+s[1][0..-3], :field => s[1][-2..-1]}
else
{:key => s[0]+":", :field => s[1]}
end
end
def hash_set(r,key,value)
kf = hash_get_key_field(key)
r.hset(kf[:key],kf[:field],value)
end
def hash_get(r,key,value)
kf = hash_get_key_field(key)
r.hget(kf[:key],kf[:field],value)
end
r = Redis.new
(0..100000).each{|id|
key = "object:#{id}"
if UseOptimization
hash_set(r,key,"val")
else
r.set(key,"val")
end
}
This is the result against a 64 bit instance of Redis 2.2:
- UseOptimization set to true: 1.7 MB of used memory
- UseOptimization set to false; 11 MB of used memory
hash-max-zipmap-entries 256(元素小于256使用zipmap存储)
Also remember to set the following field accordingly to the maximum size of your keys and values:
hash-max-zipmap-value 1024(key和value的长度小于2014使用zipmapcc)
Every time a hash will exceed the number of elements or element size specified it will be converted into a real hash table, and the memory saving will be lost.
Memory allocation内存配置
maxmemory
setting enables (however there are small extra allocations possible).
maxmemory
设置最大内存(可能会有额外的消耗).
- Redis will not always free up (return) memory to the OS when keys are removed. This is not something special about Redis, but it is how most malloc() implementations work. For example if you fill an instance with 5GB worth of data, and then remove the equivalent of 2GB of data, the Resident Set Size (also known as the RSS, which is the number of memory pages consumed by the process) will probably still be around 5GB, even if Redis will claim that the user memory is around 3GB. This happens because the underlying allocator can't easily release the memory. For example often most of the removed keys were allocated in the same pages as the other keys that still exist.
- The previous point means that you need to provision memory based on your peak memory usage. If your workload from time to time requires 10GB, even if most of the times 5GB could do, you need to provision for 10GB.
- However allocators are smart and are able to reuse free chunks of memory, so after you freed 2GB of your 5GB data set, when you start adding more keys again, you'll see the RSS (Resident Set Size) to stay steady and don't grow more, as you add up to 2GB of additional keys. The allocator is basically trying to reuse the 2GB of memory previously (logically) freed.
- Because of all this, the fragmentation ratio is not reliable when you had a memory usage that at peak is much larger than the currently used memory. The fragmentation is calculated as the amount of memory currently in use (as the sum of all the allocations performed by Redis) divided by the physical memory actually used (the RSS value). Because the RSS reflects the peak memory, when the (virtually) used memory is low since a lot of keys / values were freed, but the RSS is high, the ratio
mem_used / RSS
will be very high.
maxmemory
is not set Redis will keep allocating memory as it finds fit and thus it can (gradually) eat up all your free memory. Therefore it is generally advisable to configure some limit. You may also want to set maxmemory-policy
tonoeviction
(which is not the default value in some older versions of Redis).maxmemory
Redis就会一直分配能找到的内存并且逐步地吃光所有的空闲内存。因此一般都要明确地配置限制。你也可以设置maxmemory-policy为noeviction
(在redis以前的一些版本不是默认值).
It makes Redis return an out of memory error for write commands if and when it reaches the limit - which in turn may result in errors in the application but will not render the whole machine dead because of memory starvation.