内存使用统计


# Server

redis_version:3.0.7


# Memory

used_memory:7270580648

 - used_memory: Total number of bytes allocated by Redis using its allocator (either standard libc, jemalloc, or an alternative allocator such as tcmalloc)


used_memory_rss:7979900928

 - used_memory_rss: Number of bytes that Redis allocated as seen by the operating system (a.k.a resident set size). This is the number reported by tools such as top(1) and ps(1)

 

used_memory_peak:8110394944

 - used_memory_peak: Peak memory consumed by Redis (in bytes)


used_memory_lua:36864

 - used_memory_lua: Number of bytes used by the Lua engine

 

mem_fragmentation_ratio:1.10

 - mem_fragmentation_ratio: Ratio between used_memory_rss and used_memory

 

mem_allocator:jemalloc-3.6.0

 - mem_allocator: Memory allocator, chosen at compile time



内存消耗划分


Redis进程自身内存，对象内存，缓冲内存，和内存碎片。其中一个空的Redis进程消耗的内存可忽略不计，主要看后三者。


1.对象内存

其是Redis内存占用最大的一块，存储着应用的所有数据。Redis中的数据采用key-value形式，创建键值对时，至少创建两个类型对象，key对象和value对象。对象内存消耗可简单理解为sizeof(keys)+sizeof(values)。键对象都是字符串。value对象较复杂，主要包括5种基本数据类型：字符串，列表，哈希，集合，有序集合。其它的数据类型都是在这5种数据结构上实现的。


使用Redis时，应控制过长的key，和控制value的规模，避免响应延时和内存溢出。



2.缓冲内存

主要包括：客户端缓冲，复制积压缓冲区，和AOF缓冲区。


客户端缓冲指的是所有接入Redis服务的TCP连接的输入输出缓冲。输入缓冲无法控制，最大空间为1G，超过将断开。


输出缓冲通过client-output-buffer-limit控制，看下redis.conf中对这个参数的解释：


# The client output buffer limits can be used to force disconnection of clients

# that are not reading data from the server fast enough for some reason (a

# common reason is that a Pub/Sub client can't consume messages as fast as the

# publisher can produce them).


# The limit can be set differently for the three different classes of clients:

# normal -> normal clients including MONITOR clients

# slave  -> slave clients

# pubsub -> clients subscribed to at least one pubsub channel or pattern


# The syntax of every client-output-buffer-limit directive is the following:

# client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>


# A client is immediately disconnected once the hard limit is reached, or if

# the soft limit is reached and remains reached for the specified number of

# seconds (continuously).

# So for instance if the hard limit is 32 megabytes and the soft limit is

# 16 megabytes / 10 seconds, the client will get disconnected immediately

# if the size of the output buffers reach 32 megabytes, but will also get

# disconnected if the client reaches 16 megabytes and continuously overcomes

# the limit for 10 seconds.


# By default normal clients are not limited because they don't receive data

# without asking (in a push way), but just after a request, so only

# asynchronous clients may create a scenario where data is requested faster

# than it can read.


# Instead there is a default limit for pubsub and slave clients, since

# subscribers and slaves receive data in a push fashion.


# Both the hard or the soft limit can be disabled by setting them to zero.

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60


可见客户端可分为3类：普通客户端，从客户端，和订阅客户端。Redis对普通客户端默认没有限制，一般普通客户端的内存消耗可忽略不计。但配置中提到了MONITOR命令，即当使用有大量数据输出的命令，且数据无法及时推送给客户端时，这部分内存消耗就不能忽略了。使用CLIENT LIST，输出中omem代表这部分内存。


id=39 addr=192.168.4.124:53000 fd=44 name= age=346151 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get


 - omem: The amount of memory used by the client for the output buffer.


Redis对从客户端和订阅客户端默认进行了限制，主要是Redis推送数据给该类客户端，防止连接缓慢或消费缓慢，造成缓冲区溢出。



对于复制积压缓冲去，Redis配置提供了repl-backlog-size控制，对于该缓冲区主节点只申请一次，从节点共享。设置较大的此缓冲区，可以有效避免全量复制（默认才1mb）。


# Set the replication backlog size. The backlog is a buffer that accumulates

# slave data when slaves are disconnected for some time, so that when a slave

# wants to reconnect again, often a full resync is not needed, but a partial

# resync is enough, just passing the portion of data the slave missed while

# disconnected.


# The bigger the replication backlog, the longer the time the slave can be

# disconnected and later be able to perform a partial resynchronization.

# The backlog is only allocated once there is at least a slave connected.


# repl-backlog-size 1mb



AOF缓冲区，这部分空间用于Redis重写期间保存最近的写入命令，大小用户无法控制，取决于AOF重写时间和写入命令量，通常很小。



3.内存碎片

Redis默认使用jemalloc内存分配器，可选的还有glibc，tcmalloc。内存分配器是为了更好的管理和重复利用内存，内存分配策略一般采用固定大小的内存块进行。如保存5KB的对象时，jemalloc可能会采用8KB的块存储，而剩下的3KB空间就不好再分配给其它对象，变成了内存碎片。


jemalloc针对内存碎片问题专门做了优化，一般不会存在过度碎片化的问题，正常的碎片率在1.03左右，但如下场景容易出现内存碎片的问题：

 - 频繁更新操作，对已存在的键执行APPEND，SETRANGE等更新操作。

 - 大量删除过期键，键对象删除后，释放的空间无法得到充分利用，导致碎片上升。


内存碎片问题常见的解决方法：

 - 数据对齐：数据尽量采用固定长度的字符串，但这视具体业务而定。

 - 安全重启：借助高可用架构Sentinel或Cluster，主从切换，重启。



还有一部分内存使用，子进程内存消耗，主要指Redis执行AOF或RDB重写时，fork子进程的内存消耗。看下RDB重写的过程，Redis利用了Linux的copy-on-write特性，优化了内存的使用。


 - Redis forks. We now have a child and a parent process.

 - The child starts to write the dataset to a temporary RDB file.

 - When the child is done writing the new RDB file, it replaces the old one.


对于copy-on-write，可参考链接https://linux.die.net/man/2/fork，从此也可知道Redis fork子进程并不需要消耗1倍的父进程内存，实际消耗根据期间写入命令量决定。


另外此处要设置两个内核参数，从Redis的启动日志中可以看到该提示：


# WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.


设置vm.overcommit_memory = 1，允许内核可以分配所有的物理内存，防止Redis执行fork时，因系统剩余内存不足而失败。


# WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.


设置echo never > /sys/kernel/mm/transparent_hugepage/enabled，关闭THP，防止降低fork的速度，和copy-on-write期间内存过度消耗。