2016/11/4
NoSql入门
当今主流的技术
1.单机MySql
2.Memcached(缓冲) + Mysql + 垂直拆分
3.Mysql主从读写分离
4.分表分库+水平拆分+mysql集群
5.Mysql的扩展性瓶颈
6.nosql
NoSql入门
这些类型的数据储存不需要固定的储存，无需多余操作就可以横向扩展。
NoSql数据模型简介：
BSON是一种类json的一种二进制形式的存储格式，简称Binary JSON，它和JSON一样，支持内嵌的文档对象和数据对象
聚合模型
1.KV键值对 2.BSon 3.列族 4.图形
四大分类 1.KV键值：典型介绍 Redis Oracle DOB
2.文档型数据库（bson格式比较多）应用；CouchDB,MongoDB
MongoDB是一个基于分布式文件存储的数据库，由C++语言编写，为web应用提供可扩展的高性能数据储存解决方案。
3.列存储数据库 分布式文件系统 Hbase Cassandra 分布式文件系统
4.图关系数据库 社交网络，推荐系统，专注与构造关系图谱 Neo4J InfoGrid
在分布式数据库中CAP原理 CAP+BASE
1.传统ACIO分别是什么 A(Atomicity) 原子性 C(Consistency) 一致性 I(Isolation)独立性 D(Durability) 持久性
2.CAP consistency(强一致性) Availablibity(可用性) Partition tolerance(分区容错性)
3.CAP的3进2
CAP理论的核心是：一个分布式系统不可能同时很好的满足一致性，可用性和分区容错性这三个需求，最多只能同时
较好的满足两个。
因此，根据CAP原理将NoSql数据库分成了满足CA原则，满足CP原则和满足AP原则的三大类。
CA 单点集群：满足一致性，可用性，通常在可扩展性上不太强大。 Oracle
CP 满足一致性 分区容忍的系统通常性能不是特别高。 Redis Mongdb
AP 满足可用性 分区域容忍性的系统，通常可能对一致性要求低一些。大多数网站架构的选择
4.Base
基本可用 软状态 最终一致
Redis :REmote DIctionary Server(远程字典服务器)
特点1.Redis支持数据持久化，可以将内存中的数据保持在磁盘中，重启的时候可以再次加重进行使用
2.Redis 不仅仅支持简单的key-value类型的数据，同时还提供list,set,zset,hash等数据结构的储存
3.Redis支持数据的备份，即master-slave模式的数据备份
Redis安装
1.创建一个redis的文件目录
创建一个redis的文件目录 mkdir -p /usr/local/redis
2.编译源码包（指定安装的目录）
make PREFIX=/usr/local/redis install
3.创建一个redis的配置文件
mkdir -p /usr/local/redis/etc/
4.拷贝配置文件到
/usr/local/redis/etc/
cp redis.conf /usr/local/redis/etc/
5.通过配置文件启动redis # redis 的默认端口 6379
./redis-server /usr/local/redis/etc/redis.conf
6.测试redis是否成功
% cd src
% ./redis-cli -p 6379
redis> ping
PONG
redis> set foo bar
OK
redis> get foo
"bar"
Redis 是单进程的 默认安装的16个库
使用select 7 切换库
dbsize 查看当前库的key的个数
keys * 查询当前所以的key
flushdb 删除当前库Key
flushALL 删除所有库的Key
Redis 索引库从0开始
Redis支持的数据类型
key 键
keys * 查询存在的key
exists key 的名字 判定某个key是否存在
move key 2 把当前key转移库
expire key 秒数 为给定的key设定过期时间
ttl key 查看还有多少秒过期 -1 永不过期 -2 已过期
type key 查看key的类型
String 字符串
可以理解为何Memcached 相同的类型，一个key对应一个value
是二进制安全的，意思是String 可以包含任何数据，比如jpg图片或者序列化的对象
最多支持512M
单键单value
set/get/del/append/strlen 设置
Incr/decr/incrby/decrby 数字加减
getrange区域读取/setrange 区域设置
setex(set expire) 键秒值
setnx(set if not exist) 如果不存在设置值
mset/mget/msetnx 设置多个值 要成功都成功
List 列表
是简单的字符串列表，按照插入顺序排列，你可以添加一个元素到列表的头部或者尾部，底层实际上是个列表
单键多vlaue
lpush/rpush/lrange LRANGE list 0 -1 查询列表
lpop/rpop 出一个
lindex 按照索引下表获取元素 （从上到下）
lrem list 个数 key 删除几个value
ltrim list start end 开始start 结束end 截取后重新复制给list
rpoplpush 源列表 目的列表
lset key index value 设置列表下标的值
linser key before/after 值1 值2
Set 集合
是string类型的无序集合，它通过HashTable实现。
sadd set01 1 1 2 2 3 3 添加集合
SMEMBERS 获取集合
SISMEMBER key member 判断 member 元素是否集合 key 的成员。
scard 获取集合里面的元素个数
SREM key member [member ...] 移除集合 key 中的一个或多个 member 元素，不存在的 member 元素会被忽略。
srandmember key m （随机出m个数）
spop key 随机出栈
smove key1 key2 将key1的值给key2
数据集合
差集 sdiff
交集 sinter
并级 sunion
Hash 哈希 （json 套json相当于）
Hash是一个键值对的集合
Hash是一个String类型的field和value的映射表，hash特别适合用户储存对象
类型与java语言的 Map<String,Object>
hset/hget/hmset 多个设置/hmget /hgetall 获取所有/hdel 删除
hlen
hexists
hkeys /hvals 获取key 获取value
hincriby/hincrbyfloat 递增
hsetnx 不存在的话赋值
Zset（sorted set） 有序集合
和set一样也是string类型元素的集合，且不允许重复的成员
不同的是每个元素都会关联一个double类型的分数
redis正是通过分数来为集合中的成员进行从小到大的排序的，zset的成员是唯一的，但分数却可以重复。
在set基础上，加了一个score的值
之前set是k1,v1 v2 v3
现在zset是k1 score1 v1 score2 v2
zadd/zrange 添加 zadd k1 score1 v1 score2 v2 -withscores
zrangebyscore key 开始score结束score
zrem key 删除value的score
Redis配置文件
daemonize no Redis默认不是以守护进程方式运行的，可以通过该配置修改，使用yes启用守护进程
tcp-backlog 511 是一个连接的队列
# TCP listen() backlog.
#
# In high requests-per-second environments you need an high backlog in order
# to avoid slow clients connections issues. Note that the Linux kernel
# will silently truncate it to the value of /proc/sys/net/core/somaxconn so
# make sure to raise both the value of somaxconn and tcp_max_syn_backlog
# in order to get the desired effect.
timeout 0
# Close the connection after a client is idle for N seconds (0 to disable)
loglevel notice
日志级别
SNAPSHOTTING
Redis的快照
SECURITY
config get requirepass 获取Redis的密码
config set requirepass 设置Redis的密码
auth 登录密码 登录Redis
LIMITS
maxclients 10000
maxmemory <bytes> 最大的缓存
maxmemory-policy 过期配置
# volatile-lru -> remove the key with an expire set using an LRU algorithm lru最近最少使用 LRU algorithm 最近最少算法
# allkeys-lru -> remove any key according to the LRU algorithm
# volatile-random -> remove a random key with an expire set
# allkeys-random -> remove a random key, any key
# volatile-ttl -> remove the key with the nearest expire time (minor TTL) ttl值最小的key
# noeviction -> don't expire at all, just return an error on write operations 永不过期
maxmemory-samples 5 设置样本的数量 redis默认会检查的样例测试
以下配置说明来自与 https://my.oschina.net/wfire/blog/301147
# redis 配置文件示例
# 当你需要为某个配置项指定内存大小的时候，必须要带上单位，
# 通常的格式就是 1k 5gb 4m 等酱紫：
#
# 1k => 1000 bytes
# 1kb => 1024 bytes
# 1m => 1000000 bytes
# 1mb => 1024*1024 bytes
# 1g => 1000000000 bytes
# 1gb => 1024*1024*1024 bytes
#
# 单位是不区分大小写的，你写 1K 5GB 4M 也行
################################## INCLUDES ###################################
# 假如说你有一个可用于所有的 redis server 的标准配置模板，
# 但针对某些 server 又需要一些个性化的设置，
# 你可以使用 include 来包含一些其他的配置文件，这对你来说是非常有用的。
#
# 但是要注意哦，include 是不能被 config rewrite 命令改写的
# 由于 redis 总是以最后的加工线作为一个配置指令值，所以你最好是把 include 放在这个文件的最前面，
# 以避免在运行时覆盖配置的改变，相反，你就把它放在后面（外国人真啰嗦）。
#
# include /path/to/local.conf
# include /path/to/other.conf
################################ 常用 #####################################
# 默认情况下 redis 不是作为守护进程运行的，如果你想让它在后台运行，你就把它改成 yes。
# 当redis作为守护进程运行的时候，它会写一个 pid 到 /var/run/redis.pid 文件里面。
daemonize no
# 当redis作为守护进程运行的时候，它会把 pid 默认写到 /var/run/redis.pid 文件里面，
# 但是你可以在这里自己制定它的文件位置。
pidfile /var/run/redis.pid
# 监听端口号，默认为 6379，如果你设为 0 ，redis 将不在 socket 上监听任何客户端连接。
port 6379
# TCP 监听的最大容纳数量
#
# 在高并发的环境下，你需要把这个值调高以避免客户端连接缓慢的问题。
# Linux 内核会一声不响的把这个值缩小成 /proc/sys/net/core/somaxconn 对应的值，
# 所以你要修改这两个值才能达到你的预期。
tcp-backlog 511
# 默认情况下，redis 在 server 上所有有效的网络接口上监听客户端连接。
# 你如果只想让它在一个网络接口上监听，那你就绑定一个IP或者多个IP。
#
# 示例，多个IP用空格隔开:
#
# bind 192.168.1.100 10.0.0.1
# bind 127.0.0.1
# 指定 unix socket 的路径。
#
# unixsocket /tmp/redis.sock
# unixsocketperm 755
# 指定在一个 client 空闲多少秒之后关闭连接（0 就是不管它）
timeout 0
# tcp 心跳包。
#
# 如果设置为非零，则在与客户端缺乏通讯的时候使用 SO_KEEPALIVE 发送 tcp acks 给客户端。
# 这个之所有有用，主要由两个原因：
#
# 1) 防止死的 peers
# 2) Take the connection alive from the point of view of network
# equipment in the middle.
#
# On Linux, the specified value (in seconds) is the period used to send ACKs.
# Note that to close the connection the double of the time is needed.
# On other kernels the period depends on the kernel configuration.
#
# A reasonable value for this option is 60 seconds.
# 推荐一个合理的值就是60秒
tcp-keepalive 0
# 定义日志级别。
# 可以是下面的这些值：
# debug (适用于开发或测试阶段)
# verbose (many rarely useful info, but not a mess like the debug level)
# notice (适用于生产环境)
# warning (仅仅一些重要的消息被记录)
loglevel notice
# 指定日志文件的位置
logfile ""
# 要想把日志记录到系统日志，就把它改成 yes，
# 也可以可选择性的更新其他的syslog 参数以达到你的要求
# syslog-enabled no
# 设置 syslog 的 identity。
# syslog-ident redis
# 设置 syslog 的 facility，必须是 USER 或者是 LOCAL0-LOCAL7 之间的值。
# syslog-facility local0
# 设置数据库的数目。
# 默认数据库是 DB 0，你可以在每个连接上使用 select <dbid> 命令选择一个不同的数据库，
# 但是 dbid 必须是一个介于 0 到 databasees - 1 之间的值
databases 16
################################ 快照 ################################
#
# 存 DB 到磁盘：
#
# 格式：save <间隔时间（秒）> <写入次数>
#
# 根据给定的时间间隔和写入次数将数据保存到磁盘
#
# 下面的例子的意思是：
# 900 秒后如果至少有 1 个 key 的值变化，则保存
# 300 秒后如果至少有 10 个 key 的值变化，则保存
# 60 秒后如果至少有 10000 个 key 的值变化，则保存
#
# 注意：你可以注释掉所有的 save 行来停用保存功能。
# 也可以直接一个空字符串来实现停用：
# save ""
save 900 1
save 300 10
save 60 10000
# 默认情况下，如果 redis 最后一次的后台保存失败，redis 将停止接受写操作，
# 这样以一种强硬的方式让用户知道数据不能正确的持久化到磁盘，
# 否则就会没人注意到灾难的发生。
#
# 如果后台保存进程重新启动工作了，redis 也将自动的允许写操作。
#
# 然而你要是安装了靠谱的监控，你可能不希望 redis 这样做，那你就改成 no 好了。
stop-writes-on-bgsave-error yes
# 是否在 dump .rdb 数据库的时候使用 LZF 压缩字符串
# 默认都设为 yes
# 如果你希望保存子进程节省点 cpu ，你就设置它为 no ，
# 不过这个数据集可能就会比较大
rdbcompression yes
# 是否校验rdb文件
rdbchecksum yes
# 设置 dump 的文件位置
dbfilename dump.rdb
# 工作目录
# 例如上面的 dbfilename 只指定了文件名，
# 但是它会写入到这个目录下。这个配置项一定是个目录，而不能是文件名。
dir ./
################################# 主从复制 #################################
# 主从复制。使用 slaveof 来让一个 redis 实例成为另一个reids 实例的副本。
# 注意这个只需要在 slave 上配置。
#
# slaveof <masterip> <masterport>
# 如果 master 需要密码认证，就在这里设置
# masterauth <master-password>
# 当一个 slave 与 master 失去联系，或者复制正在进行的时候，
# slave 可能会有两种表现：
#
# 1) 如果为 yes ，slave 仍然会应答客户端请求，但返回的数据可能是过时，
# 或者数据可能是空的在第一次同步的时候
#
# 2) 如果为 no ，在你执行除了 info he salveof 之外的其他命令时，
# slave 都将返回一个 "SYNC with master in progress" 的错误，
#
slave-serve-stale-data yes
# 你可以配置一个 slave 实体是否接受写入操作。
# 通过写入操作来存储一些短暂的数据对于一个 slave 实例来说可能是有用的，
# 因为相对从 master 重新同步数而言，据数据写入到 slave 会更容易被删除。
# 但是如果客户端因为一个错误的配置写入，也可能会导致一些问题。
#
# 从 redis 2.6 版起，默认 slaves 都是只读的。
#
# Note: read only slaves are not designed to be exposed to untrusted clients
# on the internet. It's just a protection layer against misuse of the instance.
# Still a read only slave exports by default all the administrative commands
# such as CONFIG, DEBUG, and so forth. To a limited extent you can improve
# security of read only slaves using 'rename-command' to shadow all the
# administrative / dangerous commands.
# 注意：只读的 slaves 没有被设计成在 internet 上暴露给不受信任的客户端。
# 它仅仅是一个针对误用实例的一个保护层。
slave-read-only yes
# Slaves 在一个预定义的时间间隔内发送 ping 命令到 server 。
# 你可以改变这个时间间隔。默认为 10 秒。
#
# repl-ping-slave-period 10
# The following option sets the replication timeout for:
# 设置主从复制过期时间
#
# 1) Bulk transfer I/O during SYNC, from the point of view of slave.
# 2) Master timeout from the point of view of slaves (data, pings).
# 3) Slave timeout from the point of view of masters (REPLCONF ACK pings).
#
# It is important to make sure that this value is greater than the value
# specified for repl-ping-slave-period otherwise a timeout will be detected
# every time there is low traffic between the master and the slave.
# 这个值一定要比 repl-ping-slave-period 大
#
# repl-timeout 60
# Disable TCP_NODELAY on the slave socket after SYNC?
#
# If you select "yes" Redis will use a smaller number of TCP packets and
# less bandwidth to send data to slaves. But this can add a delay for
# the data to appear on the slave side, up to 40 milliseconds with
# Linux kernels using a default configuration.
#
# If you select "no" the delay for data to appear on the slave side will
# be reduced but more bandwidth will be used for replication.
#
# By default we optimize for low latency, but in very high traffic conditions
# or when the master and slaves are many hops away, turning this to "yes" may
# be a good idea.
repl-disable-tcp-nodelay no
# 设置主从复制容量大小。这个 backlog 是一个用来在 slaves 被断开连接时
# 存放 slave 数据的 buffer，所以当一个 slave 想要重新连接，通常不希望全部重新同步，
# 只是部分同步就够了，仅仅传递 slave 在断开连接时丢失的这部分数据。
#
# The biggest the replication backlog, the longer the time the slave can be
# disconnected and later be able to perform a partial resynchronization.
# 这个值越大，salve 可以断开连接的时间就越长。
#
# The backlog is only allocated once there is at least a slave connected.
#
# repl-backlog-size 1mb
# After a master has no longer connected slaves for some time, the backlog
# will be freed. The following option configures the amount of seconds that
# need to elapse, starting from the time the last slave disconnected, for
# the backlog buffer to be freed.
# 在某些时候，master 不再连接 slaves，backlog 将被释放。
#
# A value of 0 means to never release the backlog.
# 如果设置为 0 ，意味着绝不释放 backlog 。
#
# repl-backlog-ttl 3600
# 当 master 不能正常工作的时候，Redis Sentinel 会从 slaves 中选出一个新的 master，
# 这个值越小，就越会被优先选中，但是如果是 0 ， 那是意味着这个 slave 不可能被选中。
#
# 默认优先级为 100。
slave-priority 100
# It is possible for a master to stop accepting writes if there are less than
# N slaves connected, having a lag less or equal than M seconds.
#
# The N slaves need to be in "online" state.
#
# The lag in seconds, that must be <= the specified value, is calculated from
# the last ping received from the slave, that is usually sent every second.
#
# This option does not GUARANTEES that N replicas will accept the write, but
# will limit the window of exposure for lost writes in case not enough slaves
# are available, to the specified number of seconds.
#
# For example to require at least 3 slaves with a lag <= 10 seconds use:
#
# min-slaves-to-write 3
# min-slaves-max-lag 10
#
# Setting one or the other to 0 disables the feature.
#
# By default min-slaves-to-write is set to 0 (feature disabled) and
# min-slaves-max-lag is set to 10.
################################## 安全 ###################################
# Require clients to issue AUTH <PASSWORD> before processing any other
# commands. This might be useful in environments in which you do not trust
# others with access to the host running redis-server.
#
# This should stay commented out for backward compatibility and because most
# people do not need auth (e.g. they run their own servers).
#
# Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#
# 设置认证密码
# requirepass foobared
# Command renaming.
#
# It is possible to change the name of dangerous commands in a shared
# environment. For instance the CONFIG command may be renamed into something
# hard to guess so that it will still be available for internal-use tools
# but not available for general clients.
#
# Example:
#
# rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
#
# It is also possible to completely kill a command by renaming it into
# an empty string:
#
# rename-command CONFIG ""
#
# Please note that changing the name of commands that are logged into the
# AOF file or transmitted to slaves may cause problems.
################################### 限制 ####################################
# Set the max number of connected clients at the same time. By default
# this limit is set to 10000 clients, however if the Redis server is not
# able to configure the process file limit to allow for the specified limit
# the max number of allowed clients is set to the current file limit
# minus 32 (as Redis reserves a few file descriptors for internal uses).
#
# 一旦达到最大限制，redis 将关闭所有的新连接
# 并发送一个‘max number of clients reached’的错误。
#
# maxclients 10000
# 如果你设置了这个值，当缓存的数据容量达到这个值， redis 将根据你选择的
# eviction 策略来移除一些 keys。
#
# 如果 redis 不能根据策略移除 keys ，或者是策略被设置为 ‘noeviction’，
# redis 将开始响应错误给命令，如 set，lpush 等等，
# 并继续响应只读的命令，如 get
#
# This option is usually useful when using Redis as an LRU cache, or to set
# a hard memory limit for an instance (using the 'noeviction' policy).
#
# WARNING: If you have slaves attached to an instance with maxmemory on,
# the size of the output buffers needed to feed the slaves are subtracted
# from the used memory count, so that network problems / resyncs will
# not trigger a loop where keys are evicted, and in turn the output
# buffer of slaves is full with DELs of keys evicted triggering the deletion
# of more keys, and so forth until the database is completely emptied.
#
# In short... if you have slaves attached it is suggested that you set a lower
# limit for maxmemory so that there is some free RAM on the system for slave
# output buffers (but this is not needed if the policy is 'noeviction').
#
# 最大使用内存
# maxmemory <bytes>
# 最大内存策略，你有 5 个选择。
#
# volatile-lru -> remove the key with an expire set using an LRU algorithm
# volatile-lru -> 使用 LRU 算法移除包含过期设置的 key 。
# allkeys-lru -> remove any key accordingly to the LRU algorithm
# allkeys-lru -> 根据 LRU 算法移除所有的 key 。
# volatile-random -> remove a random key with an expire set
# allkeys-random -> remove a random key, any key
# volatile-ttl -> remove the key with the nearest expire time (minor TTL)
# noeviction -> don't expire at all, just return an error on write operations
# noeviction -> 不让任何 key 过期，只是给写入操作返回一个错误
#
# Note: with any of the above policies, Redis will return an error on write
# operations, when there are not suitable keys for eviction.
#
# At the date of writing this commands are: set setnx setex append
# incr decr rpush lpush rpushx lpushx linsert lset rpoplpush sadd
# sinter sinterstore sunion sunionstore sdiff sdiffstore zadd zincrby
# zunionstore zinterstore hset hsetnx hmset hincrby incrby decrby
# getset mset msetnx exec sort
#
# The default is:
#
# maxmemory-policy noeviction
# LRU and minimal TTL algorithms are not precise algorithms but approximated
# algorithms (in order to save memory), so you can tune it for speed or
# accuracy. For default Redis will check five keys and pick the one that was
# used less recently, you can change the sample size using the following
# configuration directive.
#
# The default of 5 produces good enough results. 10 Approximates very closely
# true LRU but costs a bit more CPU. 3 is very fast but not very accurate.
#
# maxmemory-samples 5
############################## APPEND ONLY MODE ###############################
# By default Redis asynchronously dumps the dataset on disk. This mode is
# good enough in many applications, but an issue with the Redis process or
# a power outage may result into a few minutes of writes lost (depending on
# the configured save points).
#
# The Append Only File is an alternative persistence mode that provides
# much better durability. For instance using the default data fsync policy
# (see later in the config file) Redis can lose just one second of writes in a
# dramatic event like a server power outage, or a single write if something
# wrong with the Redis process itself happens, but the operating system is
# still running correctly.
#
# AOF and RDB persistence can be enabled at the same time without problems.
# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.
#
# Please check http://redis.io/topics/persistence for more information.
appendonly no
# The name of the append only file (default: "appendonly.aof")
appendfilename "appendonly.aof"
# The fsync() call tells the Operating System to actually write data on disk
# instead to wait for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
#
# Redis supports three different modes:
#
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log . Slow, Safest.
# everysec: fsync only one time every second. Compromise.
#
# The default is "everysec", as that's usually the right compromise between
# speed and data safety. It's up to you to understand if you can relax this to
# "no" that will let the operating system flush the output buffer when
# it wants, for better performances (but if you can live with the idea of
# some data loss consider the default persistence mode that's snapshotting),
# or on the contrary, use "always" that's very slow but a bit safer than
# everysec.
#
# More details please check the following article:
# http://antirez.com/post/redis-persistence-demystified.html
#
# If unsure, use "everysec".
# appendfsync always
appendfsync everysec
# appendfsync no
# When the AOF fsync policy is set to always or everysec, and a background
# saving process (a background save or AOF log background rewriting) is
# performing a lot of I/O against the disk, in some Linux configurations
# Redis may block too long on the fsync() call. Note that there is no fix for
# this currently, as even performing fsync in a different thread will block
# our synchronous write(2) call.
#
# In order to mitigate this problem it's possible to use the following option
# that will prevent fsync() from being called in the main process while a
# BGSAVE or BGREWRITEAOF is in progress.
#
# This means that while another child is saving, the durability of Redis is
# the same as "appendfsync none". In practical terms, this means that it is
# possible to lose up to 30 seconds of log in the worst scenario (with the
# default Linux settings).
#
# If you have latency problems turn this to "yes". Otherwise leave it as
# "no" that is the safest pick from the point of view of durability.
no-appendfsync-on-rewrite no
# Automatic rewrite of the append only file.
# Redis is able to automatically rewrite the log file implicitly calling
# BGREWRITEAOF when the AOF log size grows by the specified percentage.
#
# This is how it works: Redis remembers the size of the AOF file after the
# latest rewrite (if no rewrite has happened since the restart, the size of
# the AOF at startup is used).
#
# This base size is compared to the current size. If the current size is
# bigger than the specified percentage, the rewrite is triggered. Also
# you need to specify a minimal size for the AOF file to be rewritten, this
# is useful to avoid rewriting the AOF file even if the percentage increase
# is reached but it is still pretty small.
#
# Specify a percentage of zero in order to disable the automatic AOF
# rewrite feature.
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
################################ LUA SCRIPTING ###############################
# Max execution time of a Lua script in milliseconds.
#
# If the maximum execution time is reached Redis will log that a script is
# still in execution after the maximum allowed time and will start to
# reply to queries with an error.
#
# When a long running script exceed the maximum execution time only the
# SCRIPT KILL and SHUTDOWN NOSAVE commands are available. The first can be
# used to stop a script that did not yet called write commands. The second
# is the only way to shut down the server in the case a write commands was
# already issue by the script but the user don't want to wait for the natural
# termination of the script.
#
# Set it to 0 or a negative value for unlimited execution without warnings.
lua-time-limit 5000
################################ REDIS 集群 ###############################
#
# 启用或停用集群
# cluster-enabled yes
# Every cluster node has a cluster configuration file. This file is not
# intended to be edited by hand. It is created and updated by Redis nodes.
# Every Redis Cluster node requires a different cluster configuration file.
# Make sure that instances running in the same system does not have
# overlapping cluster configuration file names.
#
# cluster-config-file nodes-6379.conf
# Cluster node timeout is the amount of milliseconds a node must be unreachable
# for it to be considered in failure state.
# Most other internal time limits are multiple of the node timeout.
#
# cluster-node-timeout 15000
# A slave of a failing master will avoid to start a failover if its data
# looks too old.
#
# There is no simple way for a slave to actually have a exact measure of
# its "data age", so the following two checks are performed:
#
# 1) If there are multiple slaves able to failover, they exchange messages
# in order to try to give an advantage to the slave with the best
# replication offset (more data from the master processed).
# Slaves will try to get their rank by offset, and apply to the start
# of the failover a delay proportional to their rank.
#
# 2) Every single slave computes the time of the last interaction with
# its master. This can be the last ping or command received (if the master
# is still in the "connected" state), or the time that elapsed since the
# disconnection with the master (if the replication link is currently down).
# If the last interaction is too old, the slave will not try to failover
# at all.
#
# The point "2" can be tuned by user. Specifically a slave will not perform
# the failover if, since the last interaction with the master, the time
# elapsed is greater than:
#
# (node-timeout * slave-validity-factor) + repl-ping-slave-period
#
# So for example if node-timeout is 30 seconds, and the slave-validity-factor
# is 10, and assuming a default repl-ping-slave-period of 10 seconds, the
# slave will not try to failover if it was not able to talk with the master
# for longer than 310 seconds.
#
# A large slave-validity-factor may allow slaves with too old data to failover
# a master, while a too small value may prevent the cluster from being able to
# elect a slave at all.
#
# For maximum availability, it is possible to set the slave-validity-factor
# to a value of 0, which means, that slaves will always try to failover the
# master regardless of the last time they interacted with the master.
# (However they'll always try to apply a delay proportional to their
# offset rank).
#
# Zero is the only value able to guarantee that when all the partitions heal
# the cluster will always be able to continue.
#
# cluster-slave-validity-factor 10
# Cluster slaves are able to migrate to orphaned masters, that are masters
# that are left without working slaves. This improves the cluster ability
# to resist to failures as otherwise an orphaned master can't be failed over
# in case of failure if it has no working slaves.
#
# Slaves migrate to orphaned masters only if there are still at least a
# given number of other working slaves for their old master. This number
# is the "migration barrier". A migration barrier of 1 means that a slave
# will migrate only if there is at least 1 other working slave for its master
# and so forth. It usually reflects the number of slaves you want for every
# master in your cluster.
#
# Default is 1 (slaves migrate only if their masters remain with at least
# one slave). To disable migration just set it to a very large value.
# A value of 0 can be set but is useful only for debugging and dangerous
# in production.
#
# cluster-migration-barrier 1
# In order to setup your cluster make sure to read the documentation
# available at http://redis.io web site.
################################## SLOW LOG ###################################
# The Redis Slow Log is a system to log queries that exceeded a specified
# execution time. The execution time does not include the I/O operations
# like talking with the client, sending the reply and so forth,
# but just the time needed to actually execute the command (this is the only
# stage of command execution where the thread is blocked and can not serve
# other requests in the meantime).
#
# You can configure the slow log with two parameters: one tells Redis
# what is the execution time, in microseconds, to exceed in order for the
# command to get logged, and the other parameter is the length of the
# slow log. When a new command is logged the oldest one is removed from the
# queue of logged commands.
# The following time is expressed in microseconds, so 1000000 is equivalent
# to one second. Note that a negative number disables the slow log, while
# a value of zero forces the logging of every command.
slowlog-log-slower-than 10000
# There is no limit to this length. Just be aware that it will consume memory.
# You can reclaim memory used by the slow log with SLOWLOG RESET.
slowlog-max-len 128
############################# Event notification ##############################
# Redis can notify Pub/Sub clients about events happening in the key space.
# This feature is documented at http://redis.io/topics/keyspace-events
#
# For instance if keyspace events notification is enabled, and a client
# performs a DEL operation on key "foo" stored in the Database 0, two
# messages will be published via Pub/Sub:
#
# PUBLISH __keyspace@0__:foo del
# PUBLISH __keyevent@0__:del foo
#
# It is possible to select the events that Redis will notify among a set
# of classes. Every class is identified by a single character:
#
# K Keyspace events, published with __keyspace@<db>__ prefix.
# E Keyevent events, published with __keyevent@<db>__ prefix.
# g Generic commands (non-type specific) like DEL, EXPIRE, RENAME, ...
# $ String commands
# l List commands
# s Set commands
# h Hash commands
# z Sorted set commands
# x Expired events (events generated every time a key expires)
# e Evicted events (events generated when a key is evicted for maxmemory)
# A Alias for g$lshzxe, so that the "AKE" string means all the events.
#
# The "notify-keyspace-events" takes as argument a string that is composed
# by zero or multiple characters. The empty string means that notifications
# are disabled at all.
#
# Example: to enable list and generic events, from the point of view of the
# event name, use:
#
# notify-keyspace-events Elg
#
# Example 2: to get the stream of the expired keys subscribing to channel
# name __keyevent@0__:expired use:
#
# notify-keyspace-events Ex
#
# By default all notifications are disabled because most users don't need
# this feature and the feature has some overhead. Note that if you don't
# specify at least one of K or E, no events will be delivered.
notify-keyspace-events ""
############################### ADVANCED CONFIG ###############################
# Hashes are encoded using a memory efficient data structure when they have a
# small number of entries, and the biggest entry does not exceed a given
# threshold. These thresholds can be configured using the following directives.
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
# Similarly to hashes, small lists are also encoded in a special way in order
# to save a lot of space. The special representation is only used when
# you are under the following limits:
list-max-ziplist-entries 512
list-max-ziplist-value 64
# Sets have a special encoding in just one case: when a set is composed
# of just strings that happens to be integers in radix 10 in the range
# of 64 bit signed integers.
# The following configuration setting sets the limit in the size of the
# set in order to use this special memory saving encoding.
set-max-intset-entries 512
# Similarly to hashes and lists, sorted sets are also specially encoded in
# order to save a lot of space. This encoding is only used when the length and
# elements of a sorted set are below the following limits:
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
# HyperLogLog sparse representation bytes limit. The limit includes the
# 16 bytes header. When an HyperLogLog using the sparse representation crosses
# this limit, it is converted into the dense representation.
#
# A value greater than 16000 is totally useless, since at that point the
# dense representation is more memory efficient.
#
# The suggested value is ~ 3000 in order to have the benefits of
# the space efficient encoding without slowing down too much PFADD,
# which is O(N) with the sparse encoding. The value can be raised to
# ~ 10000 when CPU is not a concern, but space is, and the data set is
# composed of many HyperLogLogs with cardinality in the 0 - 15000 range.
hll-sparse-max-bytes 3000
# Active rehashing uses 1 millisecond every 100 milliseconds of CPU time in
# order to help rehashing the main Redis hash table (the one mapping top-level
# keys to values). The hash table implementation Redis uses (see dict.c)
# performs a lazy rehashing: the more operation you run into a hash table
# that is rehashing, the more rehashing "steps" are performed, so if the
# server is idle the rehashing is never complete and some more memory is used
# by the hash table.
#
# The default is to use this millisecond 10 times every second in order to
# active rehashing the main dictionaries, freeing memory when possible.
#
# If unsure:
# use "activerehashing no" if you have hard latency requirements and it is
# not a good thing in your environment that Redis can reply form time to time
# to queries with 2 milliseconds delay.
#
# use "activerehashing yes" if you don't have such hard requirements but
# want to free memory asap when possible.
activerehashing yes
# The client output buffer limits can be used to force disconnection of clients
# that are not reading data from the server fast enough for some reason (a
# common reason is that a Pub/Sub client can't consume messages as fast as the
# publisher can produce them).
#
# The limit can be set differently for the three different classes of clients:
#
# normal -> normal clients
# slave -> slave clients and MONITOR clients
# pubsub -> clients subscribed to at least one pubsub channel or pattern
#
# The syntax of every client-output-buffer-limit directive is the following:
#
# client-output-buffer-limit <class> <hard limit> <soft limit> <soft seconds>
#
# A client is immediately disconnected once the hard limit is reached, or if
# the soft limit is reached and remains reached for the specified number of
# seconds (continuously).
# So for instance if the hard limit is 32 megabytes and the soft limit is
# 16 megabytes / 10 seconds, the client will get disconnected immediately
# if the size of the output buffers reach 32 megabytes, but will also get
# disconnected if the client reaches 16 megabytes and continuously overcomes
# the limit for 10 seconds.
#
# By default normal clients are not limited because they don't receive data
# without asking (in a push way), but just after a request, so only
# asynchronous clients may create a scenario where data is requested faster
# than it can read.
#
# Instead there is a default limit for pubsub and slave clients, since
# subscribers and slaves receive data in a push fashion.
#
# Both the hard or the soft limit can be disabled by setting them to zero.
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
# Redis calls an internal function to perform many background tasks, like
# closing connections of clients in timeout, purging expired keys that are
# never requested, and so forth.
#
# Not all tasks are performed with the same frequency, but Redis checks for
# tasks to perform accordingly to the specified "hz" value.
#
# By default "hz" is set to 10. Raising the value will use more CPU when
# Redis is idle, but at the same time will make Redis more responsive when
# there are many keys expiring at the same time, and timeouts may be
# handled with more precision.
#
# The range is between 1 and 500, however a value over 100 is usually not
# a good idea. Most users should use the default of 10 and raise this up to
# 100 only in environments where very low latency is required.
hz 10
# When a child rewrites the AOF file, if the following option is enabled
# the file will be fsync-ed every 32 MB of data generated. This is useful
# in order to commit the file to the disk more incrementally and avoid
# big latency spikes.
aof-rewrite-incremental-fsync yes
持久化之RDB(Redis DataBase)
概述
在指定的时间间隔内将内存中的数据集快照写入到磁盘，也就是Snapshot快照，它恢复时是将快照文件直接读到内存里，
Redis会单独创建（fork）一个子进程进行持久化，会先将数据写入到一个临时文件中，待持久化过程都结束了，再用这个临时文件替换
上次持久化好的文件，整个过程中，主进程是不进行任何IO操作，这就保证了极高的性能，如果需要进行大规模的数据恢复且对于数据
恢复的完整性不是非常敏感，那RDB方式比AOF方式更加的高效，RDB的缺点是最后一次持久化的数据可能丢失。
Fork
Fork的作用是复制一个与当前进程一样的进程，新进程的所有数据（变量、环境）都和原进程一致，但是是一个全新的进程，
并作为源进程的子进程
保存文件
dump.rdb
配置位置 Snapshot /usr/local/redis/etc/redis.conf
save 900 1 15分钟修改1次 或
save 300 10 5分钟修改10次 或
save 60 10000 1分钟修改了10000次
stop-writes-on-bgsave-error yes 当出现错误时停止
rdbcompression yes LZF 压缩算法
rdbchecksum yes 压缩后校验
dbfilename dump.rdb 压缩文件名称
dir ./ 备份目录
如何触发RDB快照 执行save命令 快速保存快照 bgsave 后台异步备份
如何恢复 备份 dump.rdb
优势
劣势
如何停止 修改配置文件 save ""
小总结
持久化之AOF(Append Only File) APPEND ONLY MODE
概述
以日志的形式来记录每个写操作，将Redis执行过的所有写指令记录下来（读操作不记录）
只许追加文件但不可以修改文件，redis启动之后会读取该文件重新构建数据，换计划，redis
重启的话就根据日志文件的内容将写指令从前到后执行一次以完成数据的恢复工作。
aof保存的是appendonly.aof文件
配置位置 APPEND ONLY MODE /usr/local/redis/etc/redis.conf
appendonly no 默认是关闭的
appendfilename "appendonly.aof" 文件名称
aof 和 RDB 同时存在的话 先加载的aof
如果aof启动失败后 可以使用./redis-check-aof --fix appendonly.aof 命令进行恢复
appendfsync everysec
always 没事发生数据变更，都会记录到磁盘
everysec 异步操作 每秒记录，如果一秒内宕机，有数据丢失
No
no-appendfsync-on-rewrite no
重写时是否可以运用appendfsync，用默认no即可。保证数据的安全性。
重写基础
使用命令 bgrewriteaof 当AOF文件的大小超过设定的范围值时，Redis就会启动AOF文件的内容压缩
重写原理
AOF文件持续增长而过大时，会fork出一条新进程来将文件重写（也是先写临时文件最后在rename）
遍历新进程的内存中数据，每条记录有一条的Set语句，重写aof文件的操作，并没有读取旧的aof文件
而是将整个内存中的数据库内存用命令的方式重写一个写的aof文件。
触发机制
Redis 会记录上次重写时的AOF大小，默认配置是当AOF文件大小是上次rewrite后大小的一倍且文件大小大于64M时触发
auto-aof-rewrite-percentage 100 设置重写的基准值 100% 大于1倍
auto-aof-rewrite-min-size 64mb 设置重写的基准值 64M
优势
每秒同步 appendfsync everysec 性能差但是数据完整性较好
劣势
相同数据集的数据而言aof文件要远大于rdb文件，恢复速度慢于rdb
aof运行效率要慢于rdb,每秒同步策略效率较好，不同步效率和rdb相同
Reids事务
可以一次执行多个命令，本质是一组命令的集合，一个事务中的所有命令都会序列化，按顺序地串行化执行执行而不会被其它命令插入，不许加塞
DISCARD
取消事务，放弃执行事务块内的所有命令。
EXEC
执行所有事务块内的命令。
MULTI
标记一个事务块的开始。
事务块内的多条命令会按照先后顺序被放进一个队列当中，最后由 EXEC 命令原子性(atomic)地执行。
UNWATCH
取消 WATCH 命令对所有 key 的监视
WATCH key [key ...] 监控 WATCH的指令类似于乐观锁
监视一个(或多个) key ，如果在事务执行之前这个(或这些) key 被其他命令所改动，那么事务将被打断。
悲观锁/乐观锁/CAS(Check and set)
Reids事务先监控然后在开启一个事务
Reids的发布和订阅机制
进程间的一种消息通信模式，发送者（pub）发送消息，订阅者（sub）接收消息。
Reids主从复制
读写分离 容灾恢复
配置
a.一主二从机
1.拷贝多个redis.conf文件
2.开启daemonize yes
3.Pid文件名字
4.指定端口
5.log文件名字
6.Dump.rdb名字
7.info replication 查询信息
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
8.命令slaveof IP 端口 备份机 （保存主机的数据） 临时生效
如果想每次断开后都可以连接主机的话，需要修改从机的配置文件 redis.conf
b.星火相传（去中心化）
1.中途变更转向：会清除之前的数据，重新建立拷贝最新的
2.slaveof 新IP 新端口
c.反客为主
当主机挂掉后，从机可以当主机
salaveno no one
新主机配置 把从机变为主机
d.哨兵模式
自动监控，自动把丛集转为主机
1.新建一个配置文件
主机配置文件
sentinel.conf
sentinel monitor 被监控的主机名 IP 端口 票数
2.启动哨兵配置文件
redis-sentinel /etc/usr/local/redis/etc/sentinel.conf
缺点
有一定的系统延迟 写操作在主机上，从机需要同步数据
扫一扫
714
评论
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
查看更多评论
添加红包
