redis和数据库的同步+varnish(CDN缓存系统)
redis和数据库的同步+varnishCDN缓存系统
问题出现
CDN：企业自觉维护，高额费用
r/w
client(app)-----------------mysql(ssd->io)
- 读写
-
-
- redis
客户端访问应用，应用访问代理
通过redis作缓冲(即缓存)，拆分读r到redis
优先从redis读出，没有的数据从数据库读出，并且需要由app写入redis。若更新数据库而不更新redis，读的仍然是redis缓存的数据
“数据库与redis如何同步更新？”
redis服务
redis持久化
“关注度”自增问题依赖数据库——————>数据库压力过大
server1:
systemctl start redis
redis-cli
127.0.0.1:6379> INCR 1
(integer) 1
127.0.0.1:6379> INCR 1
(integer) 2
127.0.0.1:6379> INCR 1
(integer) 3
127.0.0.1:6379> INCR 1
(integer) 4
127.0.0.1:6379> INCR 1
(integer) 5
127.0.0.1:6379> INCR 1
(integer) 6
127.0.0.1:6379> INCR 1
(integer) 7
127.0.0.1:6379> INCR 1
(integer) 8
127.0.0.1:6379>
ls
apache-tomcat-7.0.37.tar.gz
CMakeLists.txt
codis-release3.2.zip
dump.db
vim /etc/redis/redis.conf
appendonly yes
##启用AOF持久化方式
##默认写入该日志
appendfsync everysec
##每秒钟强制写入磁盘一次
##完全由操作系统来控制，每秒
##每执行一次刷新一次，每一次更新都能记录下来
systemctl restart redis
cd /var/lib/redis/
ls
appendonly.aof dump.rdb
redis-cli
127.0.0.1:6379> INCR 1
(integer) 1
127.0.0.1:6379> INCR 1
(integer) 2
127.0.0.1:6379> INCR 1
(integer) 3
127.0.0.1:6379> INCR 1
(integer) 4
127.0.0.1:6379> INCR 1
(integer) 5
127.0.0.1:6379> INCR 1
(integer) 6
127.0.0.1:6379> INCR 1
(integer) 7
127.0.0.1:6379> INCR 1
(integer) 8
127.0.0.1:6379>
cat appendonly.aof
ls
appendonly.aof dump.rdb
ll
total 8
-rw------- 1 root root 191 Apr 16 18:39 appendonly.aof
-rw------- 1 root root 113 Apr 16 18:38 dump.rdb
redis-cli
127.0.0.1:6379> info
#Replication
role:master
RDB与AOF方式的优缺点
rdb：快照存储(默认)更利于备份与恢复；
AOF:数据一致性，以追加的方式保存所有写操作；
rdb方式：
优点：1.紧凑2.主从复制，快速
缺点：1.保存数据没有AOF详尽2.经常fork子进程保存数据集到硬盘上，数据较大,fork的过程非常耗时 硬关机对操作系统伤害很大
实验环境准备
同步方式一
app—r/w—>redis<----func---->mysql
func：构建函数，为过滤器
同步方式二
阿里：模拟主从复制
binlog
mysql(trigger触发器触发udf用户自定义函数)
-->update 更新数据库
-----------> canal(slave)
app (java开发)
复制二进制日志
-->
redis------------> javaclient更新redis
server1:redis
server2、3:lnmp架构，rpm包
server4:数据库
server2:
ls
rpm -qa|grep php
php-5.4.16-46.el7.x86_64
php-cli-5.4.16-46.el7.x86_64
php-common-5.4.16-46.el7.x86_64
##安装php相关的包
##解决所有的依赖
cd rhel7/
ls
yum install php-* libgearman-1.1.12-18.el7.x86_64.rpm libgearman-devel-1.1.12-18.el7.x86_64.rpm libevent-devel-2.0.21-4.el7.x86_64.rpm -y
##启动php，默认五个
##php5000端口到位
server1:
cd /usr/local/
ls
cd nginx/
ls
cd ..
rsync -a nginx server2:/usr/local/
##nginx用于发布
##同步nginx到server2
##不加斜杠目录一起同步
##做软连接
server2:
cd /usr/local/nginx/
ls
cd sbin/
ls
nginx nginx.old
pwd
/usr/local/nginx/sbin
cd
vim .bash_profile
##编辑环境变量
PATH=$PATH:$HOME/bin:/usr/local/mysql/bin:/usr/local/nginx/sbin
source .bash_profile
which nginx
/usr/local/nginx/sbin/nginx
cd /usr/local/nginx/
cd conf/
ls
cp nginx.conf.default nginx.conf
cp: overwrite ‘nginx.conf’? y
vim nginx.conf
nginx -t
nginx
cd ..
cd html/
ls
location ~ \.php$ {
root html;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
include fastcgi.conf;
}
172.25.111.2/index.php
真机：
scp test.php test.sql server2:/usr/local/nginx/html/
##下载脚本，往数据库写东西
sever2：
cd html/
ls
server4:
yum install mariadb-server.x86_64 -y
##安装数据库服务端
systemctl start mariadb
server2:
scp test.sql server4:
server4:
ls
MHA-7 mysql-router-community-8.0.21-1.el7.x86_64.rpm test.sql
mysql < test.sql
##将test.sql传给server4
mysql
show databases;
##检查数据
use test
show tables;
select * from test;
##数据查询先从redis远程连接，再连接数据库取数据
##授予权限给test.*授予用户，从远端，密码为westos
server1:
netstat -antlp
server2:
vim test.php
$redis->connect('172.25.111.1',6379) or die ("could net connect redis server");
$connect = mysql_connect('172.25.111.4','redis','westos');
server4:
mysql
grant all on test.* to redis@'%' identified by 'westos';
use test
select * from test;
update test set name='westos' where id=1;
##第一次访问redis，无，从数据库下载
##数据库写入redis
##并未访问数据库
测试
http://172.25.111.2/test.php
http://172.25.111.2/test.php(再次刷新)
mysql（无密码登陆）
##作数据库更新，redis并未更新
##除非redis数据过期，才会重载数据
##无法做到数据库和redis数据同步
redis和数据库之间的同步
scp lib_mysqludf_json-master.zip server4:/root/
##json模块
server4：
yum install -y unzip gcc
unzip lib_mysqludf_json-master.zip
yum install -y mariadb-devel.x86_64
cd lib_mysqludf_json-master
gcc $(mysql_config --cflags) -shared -fPIC -o lib_mysqludf_json.so lib_mysqludf_json.c
ll lib_mysqludf_json.so
##数据库对.so文件进行调用
mysql
show global variables like 'plugin_dir';
cp lib_mysqludf_json.so /usr/lib64/mysql/plugin/
mysql
MariaDB [(none)]> CREATE FUNCTION json_object RETURNS STRING SONAME
-> 'lib_mysqludf_json.so';
select * from mysql.func;
真机：
scp gearman-mysql-udf-0.6.tar.gz server4:/root/
scp libevent-devel-2.0.21-4.el7.x86_64.rpm libgearman-1.1.12-18.el7.x86_64.rpm libgearman-devel-1.1.12-18.el7.x86_64.rpm server4:/root/
server4:
tar zxf gearman-mysql-udf-0.6.tar.gz
##这个插件是用来管理调用Geaman的分布式的队列
cd gearman-mysql-udf-0.6/
ls
./configure --libdir=/usr/lib64/mysql/plugin/
##安装缺少的东西
##编译出的库放到什么位置
make
make install
mysql
CREATE FUNCTION gman_do_background RETURNS STRING SONAME 'libgearman_mysql_udf.so';
CREATE FUNCTION gman_servers_set RETURNS STRING SONAME 'libgearman_mysql_udf.so';
##注册UDF函数
##定义了3个用户自定义函数
select * from mysql.func;
SELECT gman_servers_set('172.25.111.1:4730');
scp gearmand-1.1.12-18.el7.x86_64.rpm server2:/root/
server2:
yum install -y gearmand-1.1.12-18.el7.x86_64.rpm
systemctl start gearmand
##server2安装后台服务
netstat -antlp
netstat -antlp | grep :4730
server4:
SELECT gman_servers_set('172.25.111.2:4730');
cd
vim test.sql
#CREATE TABLE `test` (`id` int(7) NOT NULL AUTO_INCREMENT, `name` char(8) DEFAULT NULL, PRIMARY KEY (`id`)) ENGINE=InnoDB DEFAULT CHARSET=utf8;
#INSERT INTO `test` VALUES (1,'test1'),(2,'test2'),(3,'test3'),(4,'test4'),(5,'test5'),(6,'test6'),(7,'test7'),(8,'test8'),(9,'test9');
##打开触发器
mysql < test.sql
mysql
show TRIGGERS FROM test;
##通过触发器将关系型数据发往后端4730
##将job发往给其余worker执行
scp worker.php server2:/root/
server2:
php -m
vim worker.php
$redis = new Redis();
$redis->connect('172.25.111.1', 6379);
##数据库连php,连redis，连后端
##将数据同步给redis
which php
/usr/bin/php
nohup php worker.php &
[1] 18558
nohup: ignoring input and appending output to ‘nohup.out’
##php/c/java/python/golang ---->redis/gearman
##将php代码打入后台
ps ax
测试
server4:
use test
##进入数据库
select * from test;
##查询数据表
update test set name='redhat' where id=1;
##update数据
##用机制直接更新了数据
server1:
redis-cli
127.0.0.1:6379> get 1
"redhat"
server4:
update test set name='redhat';
http://172.25.111.2/test.php
update test set name='westos';
http://172.25.111.2/test.php
CDN(内容分发网络)——varnish
Varnish Proxy + Cache HTTP Server(Apache,Nginx,IIS,…)
client->|DNS->CDN->|proxy->web server
server4:
systemctl stop mariadb
server2:
systemctl stop gearmand
scp jemalloc-3.6.0-1.el7.x86_64.rpm jemalloc-devel-3.6.0-1.el7.x86_64.rpm varnish-4.0.5-1.el7.x86_64.rpm varnish-libs-4.0.5-1.el7.x86_64.rpm server1:/root/
server1:varnish(CDN缓存系统)
systemctl stop redis.service
yum install -y jemalloc-* varnish-*
##安装时有依赖性
cd /etc/varnish/
ls
default.vcl secret varnish.params
##.vcl主配置文件，reload时会编译成二进制程序
vim default.vcl
##所有加载参数在其中
##监听端口改为80
ll
systemctl status varnish
##使用内存作缓存
##开启varnish
vim /usr/lib/systemd/system/varnish.service
ls
default.vcl secret varnish.params
vim varnish.params
VARNISH_LISTEN_PORT=80
VARNISH_STORAGE="malloc,256M"
##默认把8080作为后端，改为检测原站
##访问1会帮你到本地取
##默认120s后重来
netstat -antlp
systemctl start varnish
systemctl status varnish
netstat -antlp
vim default.vcl
backend default {
.host = "172.25.111.1";
.port = "80";
}
systemctl reload varnish
vim default.vcl
##varnish进来时，都是由该函数进行处理
##该信息返回给客户端，成功与失败返回的端口不一样
if (obj.hits > 0) {
set resp.http.X-Cache = "HIT from westos cache";
}
else {
set resp.http.X-Cache = "MISS from westos cache";
}
return (deliver);
##观察从哪儿命中
##全栈，重定向
测试
curl -I 172.25.111.1
扩展功能
1.添加一个默认后端
2个后端
varnish而言www一条缓存
直接域名也是一条缓存
为了提高缓存利用率，做一条缓存，发往后端
web1—>server2
web2—>server3
server2:
rsync -a nginx server3:/usr/local/
server3:
cd /usr/local/
ls
vim .bash_profile
source .bash_profile
PATH=$PATH:$HOME/bin:/usr/local/mysql/bin:/usr/local/nginx/sbin
which nginx
cd /usr/local/nginx/conf/
ls
vim nginx.conf
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi.conf;
# }
vim default.vcl
systemctl reload varnish
backend web1 {
.host = "172.25.111.2";
.port = "80";
}
backend web2 {
.host = "172.25.111.3";
.port = "80";
}
if (req.http.host ~ "^(www.)?westos.org") {
set req.http.host = "www.westos.org";
set req.backend_hint = web1;
#return(pass);
}
elseif (req.http.host ~ "^bbs.westos.org") {
set req.backend_hint = web2;
#return(pass);
}
else {
return(synth(404,"Not in cache"));
}
测试
curl -I www.westos.org
curl -I bbs.westos.org
##命中
##调用到了后端
server3:
cd conf/
vim nginx.conf
server {
listen 80;
server_name bbs.westos.org;
location / {
root /bbs;
index index.html;
}
}
nginx -s reload
mkdir /bbs
cd /
cd bbs/
ls
echo bbs.westos.org > index.html
server2:
cd
mkdir /www
echo www.westos.org > /www/index.html
cd /usr/local/nginx/conf/
ls
vim nginx.conf
nginx -s reload
server {
listen 80;
server_name bbs.westos.org;
location / {
root /www;
index index.html;
}
}
##添加虚拟主机
##CDN当作代理
测试
真机：
curl bbs.westos.org
curl www.westos.org
##跟IP进不来
vim /www/index.html
www.westos.org
www.westos.org
nginx -s reload
测试
curl www.westos.org
www.westos.org
www.westos.org
##访问不同域名获得页面不一样
##解析的域名都在CDN，转发到正确后端
2.模拟负载均衡
通过nginx虚拟主机来完成
主机：bbs,www
server3:
cd
mkdir /www
echo www.westos.org > /www/index.html
cd /usr/local/nginx/conf/
vim nginx.conf
nginx -s reload
server {
listen 80;
server_name www.westos.org;
location / {
root /www;
index index.html;
}
}
server1：
vim default.vcl
systemctl reload varnish
import directors;
##导入语句块到初始化
##负载均衡名称
##web_cluster集群为后端
sub vcl_init {
new web_cluster = directors.round_robin();
web_cluster.add_backend(web1);
web_cluster.add_backend(web2);
}
sub vcl_recv {
if (req.http.host ~ "^(www.)?westos.org") {
set req.http.host = "www.westos.org";
set req.backend_hint = web_cluster.backend();
return(pass);
}
##不等待不找缓存，直接到后端去找
##快速看到均衡效果，##每次不缓存
}
测试
curl www.westos.org
www.westos.org #server3提供
curl www.westos.org
www.westos.org #server2提供
www.westos.org
##varnish可以到server2、3的虚拟主机上
##不断负载均衡
3.设置健康检查
问题模拟
server2:
nginx -s stop
真机：
curl www.westos.org
curl www.westos.org
vim default.vcl
systemctl reload varnish
probe backend_healthcheck {
.url = "/index.html";
.window = 3;
.threshold = 2;
.interval = 3s;
}
backend web1 {
.host = "172.25.111.2";
.port = "80";
.probe = backend_healthcheck;
}
##4后版本需要定义健康检测
##不会再调度不健康的后端
backend web2 {
.host = "172.25.111.3";
.port = "80";
.probe = backend_healthcheck;
}
真机：
curl www.westos.org
www.westos.org
curl www.westos.org ##只跳转到server3
www.westos.org
curl www.westos.org
www.westos.org
curl www.westos.org
www.westos.org
##自动添加建康后端
4.nginx代理配置
如何让nginx拿到客户端信息？
所有客户端先访问CDN服务器，拿不到原地址，该方式为发往后端的请求添加首部
##定义域名
##访问2时，调度到后端的3上
##CDN代理后端应用
1009
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
