逻辑图 实现nginx节点自动化加入容器IP代理
1、部署三台Consu集群然后通过Consul Template实时监测Leader的变化。
2、Leader如果变化就触发渲染模板动作,执行相应命令。
3、 Node1节点创建Nginx随机分配端口,Registrator监听到事件后会将宿主级IP与暴露的端口注册到本地的Consul client。
4、Consul Clinet 再将这个请求转发到 Consul agent service中。
环境说明:
Linux系统版本:Linux Centos 7.4 x64
Consul应用版本:consul_1.4.0
Template应用版本:consul-template_0.20.0
Docker应用版本:18.09.0
Consul管理节点:192.168.11.3
Docker工作节点:192.168.11.8
Docker工作节点:192.168.11.31
主机 | IP | 服务 | 备注 |
Consul Server | 192.168.11.3 | Consul Server、consul-template、Nginx | |
Docker Node1 | 192.168.11.8 | Consul Client、Docker、registrator、Nginx(应用) |
|
Docker Node2 | 192.168.11.31 | Consul Client、Docker、registrator、Nginx(应用) |
管理节点安装 Consul 服务端安装:
官网地址:https://www.consul.io/downloads
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install consul
server模式初始化集群
consul agent \
-server \
-bootstrap \
-ui \
-data-dir=/var/lib/consul-data \
-bind=192.168.11.3 \
-client=0.0.0.0 \
-node=test
==> Starting Consul agent...
Version: '1.10.1'
Node ID: '1b25ca3a-a0f2-4b3b-0244-450eec0c808a'
Node name: 'test'
Datacenter: 'dc1' (Segment: '<all>')
Server: true (Bootstrap: true)
Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Cluster Addr: 192.168.11.3 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false, Auto-Encrypt-TLS: false
..........
..........
..........
2021-07-21T14:34:05.131+0800 [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=udp
2021-07-21T14:34:05.132+0800 [INFO] agent: Started DNS server: address=0.0.0.0:8600 network=tcp
2021-07-21T14:34:05.132+0800 [INFO] agent: Starting server: address=[::]:8500 network=tcp protocol=http
2021-07-21T14:34:05.132+0800 [WARN] agent: DEPRECATED Backwards compatibility with pre-1.9 metrics enabled. These metrics will be removed in a future version of Consul. Set `telemetry { disable_compat_1.9 = true }` to disable them.
2021-07-21T14:34:05.132+0800 [INFO] agent: started state syncer
2021-07-21T14:34:05.132+0800 [INFO] agent: Consul agent running!
参数详解:
# 使用命令
consul agent \
# 设置server模式
-server \
# 自选举为自己Leader
-bootstrap \
# 启用内置web页面
-ui \
# 集群数据保存位置
-data-dir=/var/lib/consul-data \
# 集群之间通讯绑定地址
-bind=192.168.11.3 \
# 客户端访问地址
-client=0.0.0.0 \
# 指定管理节点名称默认主机名,必须是唯一的
-node=test
查看Consul节点启动信息
[root@consul/]# netstat -tunlp|grep consul
tcp 0 0 192.168.11.3:8300 0.0.0.0:* LISTEN 7599/consul
tcp 0 0 192.168.11.3:8301 0.0.0.0:* LISTEN 7599/consul
tcp 0 0 192.168.11.3:8302 0.0.0.0:* LISTEN 7599/consul
tcp6 0 0 :::8500 :::* LISTEN 7599/consul
tcp6 0 0 :::8600 :::* LISTEN 7599/consul
udp 0 0 192.168.11.3:8301 0.0.0.0:* 7599/consul
udp 0 0 192.168.11.3:8302 0.0.0.0:* 7599/consul
udp6 0 0 :::8600 :::* 7599/consul
Consul 常用命令
consul info # 查看详情
consul members # 查看集群信息
consul leaderIP # 查看leaderIP地址
consul catalog services #查看已注册的服务信息
#通过HTTP API获取集群信息
curl 127.0.0.1:8500/v1/status/peers # 集群server成员
curl 127.0.0.1:8500/v1/status/leader # 集群Raft leader
curl 127.0.0.1:8500/v1/catalog/services # 注册的所有服务
curl 127.0.0.1:8500/v1/catalog/services/nginx # 服务信息
curl 127.0.0.1:8500/v1/catalog/nodes # 集群节点详细信息
Consul API管理
类型 描述
kv 键值存储
agent agent管理
catalog nodes和services管理
health 健康检查
session session管理
acl ACL管理
event 用户事件
status Consul系统状态
Client 节点安装 Consul(这里直接docker容器的方式运行)
docker安装:
sudo yum remove docker \
docker-client \
docker-client-latest \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-engine
sudo yum install -y yum-utils
sudo yum-config-manager \
--add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install docker-ce docker-ce-cli containerd.io
节点 node1
docker run -d \
--name=consul \
-p 8301:8301 \
-p 8301:8301/udp \
-p 8500:8500 \
-p 8600:8600 \
-p 8600:8600/udp \
--restart=always \
progrium/consul \
-join 192.168.11.3 -advertise 192.168.11.8 -client 0.0.0.0 -node=node01
节点 node2
docker run -d \
> --name=consul \
> -p 8301:8301 \
> -p 8301:8301/udp \
> -p 8500:8500 \
> -p 8600:8600 \
> -p 8600:8600/udp \
> --restart=always \
> progrium/consul \
> -join 192.168.11.3 -advertise 192.168.11.31 -client 0.0.0.0 -node=node02
查看Consul集群;
[root@consul/]# consul members
Node Address Status Type Build Protocol DC Segment
test 192.168.11.3:8301 alive server 1.10.1 2 dc1 <all>
node01 192.168.11.8:8301 alive client 0.5.2 2 dc1 <default>
node02 192.168.11.31:8301 alive client 0.5.2 2 dc1 <default>
在管理节点上通过web界面查看
节点管理:
测试注册服务示例:
curl -X PUT -d \
'{"id": "jetty","name": "service_name","address": "192.168.11.31","port": 8080,"tags": ["cmdb1"],"checks":
[{"http": "http://192.168.11.31:8080/","interval": "5s"}]}' \
http://127.0.0.1:8500/v1/agent/service/register
[root@consul-server/]# curl -X PUT -d \
> '{"id": "jetty","name": "service_name","address": "192.168.11.31","port": 8080,"tags": ["cmdb1"],"checks":
> [{"http": "http://192.168.11.31:8080/","interval": "5s"}]}' \
> http://127.0.0.1:8500/v1/agent/service/register
2021-07-21T15:57:50.413+0800 [WARN] agent: Service name will not be discoverable via DNS due to invalid characters. Valid characters include all alpha-numerics and dashes.: service=service_name
2021-07-21T15:57:50.504+0800 [INFO] agent: Synced service: service=jetty
工作节点:安装Registrator与template
Registrator简介
consul-template:一个守护程序,用于实时查询consul集群数据,并更新文件系统上的任意数量的指定模板,生成配置文件,更新完成后可以选择运行任何命令。
gliderlabs/registrator:检查容器运行状态自动注册到服务中心。registrator支持服务注册有consul、etcd、SkyDNS2。
工作节点1节点2:下载解压部署
consul-template 下载地址:https://releases.hashicorp.com/consul-template/
[root@consul tools]# wget https://releases.hashicorp.com/consul-template/0.20.0/consul-template_0.20.0_linux_386.zip
unzip consul-template_0.20.0_linux_386.zip # 解压压缩包
mv consul-template /usr/bin # 将应用移动到bin目录下
创建Registrator项目节点1+节点2;
docker run -d \
--name=registrator \
-v /var/run/docker.sock:/tmp/docker.sock \
--restart=always \
gliderlabs/registrator:latest \
consul://192.168.11.8:8500
docker run -d \
--name=registrator \
-v /var/run/docker.sock:/tmp/docker.sock \
--restart=always \
gliderlabs/registrator:latest \
consul://192.168.11.31:8500
参数详解:
# docker run -d \
# 指定容器名称
--name=registrator \
# 给容器挂载存储卷,挂载到容器的某个目录
-v /var/run/docker.sock:/tmp/docker.sock \
# 添加重启策略
--restart=always \
# 指定镜像
gliderlabs/registrator:latest \
# 指定docker本地IP,连接注册到consul client。
consul://192.168.1.77:8500
注:它可以将之前创建的容器注册。
大家可以看到之前注册过的容器服务,已经可以看到了;
测试Nginx集群节点自动注册进Consul
在两个节点上部署Nginx容器,随机分配个端口;
docker run -d -P nginx
节点1
[root@consul /]# docker ps -a|grep nginx
776af0c7aed2 nginx "/docker-entrypoint.…" 6 minutes ago Up 6 minutes 0.0.0.0:49153->80/tcp, :::49153->80/tcp romantic_jemison
节点2
[centos@DN-N1 ~]$ sudo docker ps -a|grep nginx
de8c9ad819c1 nginx "/docker-entrypoint.…" 7 minutes ago Up 7 minutes 0.0.0.0:32768->80/tcp blissful_johnson
在Consul server节点web界面查看服务状态;
管理节点部署Nginx
yum -y install nginx
创建 nginx.ctmpl
vim /opt/consul/nginx.ctmpl
upstream http_backend {
ip_hash;
# nginx 为服务名称
{{range service "nginx"}}
# 服务名 服务IP:端口号
server {{ .Address }}:{{ .Port }};
# 结束
{{ end }}
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://http_backend;
}
}
管理节点:配置template自动部署
consul-template \
-consul-addr 192.168.11.3:8500 \
-template "/opt/consul/nginx.ctmpl:/etc/nginx/conf.d/default2.conf:/usr/sbin/nginx -s reload" \
-log-level=info
[root@consul /]# consul-template \
> -consul-addr 192.168.11.3:8500 \
> -template "/opt/consul/nginx.ctmpl:/etc/nginx/conf.d/default2.conf:/usr/sbin/nginx -s reload" \
> -log-level=info
2021/07/22 07:07:39.592988 [INFO] consul-template v0.20.0 (b709612c)
2021/07/22 07:07:39.593847 [INFO] (runner) creating new runner (dry: false, once: false)
2021/07/22 07:07:39.595662 [INFO] (runner) creating watcher
2021/07/22 07:07:39.596048 [INFO] (runner) starting
管理节点:查看Nginx的状态和测试结果;
[root@consul~]# netstat -tunlp|grep nginx
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 15071/nginx: master
# 生成的defaulte2.conf , 自动获取的server地址;
[root@consul~]# cat /etc/nginx/conf.d/default2.conf
upstream http_backend {
ip_hash;
server 192.168.11.8:49153;
server 192.168.11.31:32768;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://http_backend;
}
}
测试新增几台Nginx容器,验证服务的自动发现;
在查看Nginx的server配置文档;
[root@consul ~]# cat /etc/nginx/conf.d/default2.conf
upstream http_backend {
ip_hash;
server 192.168.11.8:49154;
server 192.168.11.8:49157;
server 192.168.11.8:49155;
server 192.168.11.8:49156;
server 192.168.11.8:49153;
server 192.168.11.31:32768;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://http_backend;
}
}
测试删除一个节点上的Nginx容器,consul会同步删除;