单机Nomad+Consul+consul-template+Nginx反向代理

单机Nomad+Consul+consul-template+Nginx反向代理

虚拟机：Ubuntu18.04

IP：192.168.10.10

测试过程：在虚拟机中，Nomad起一个job容器，consul通过check脚本检查这个容器，并将其IP和端口自动更新到consul-template中，输出一个文件。最后利用Nginx反向代理进行域名访问容器。

准备工作：修改docker用户权限

1 Nomad与Consul连接

1.1 Nomad

创建/home/.../nomad.d/nomad.hcl：

datacenter = "dc1"
data_dir = "/home/.../nomad/data"

server {
  enabled = true
  bootstrap_expect = 1

  server_join {
    retry_join = ["192.168.10.10:4648"]
  }
}

client {
  enabled = true
  servers = ["192.168.10.10:4647"]
}

执行nomad agent -config /home/.../nomad.d启动nomad agent。

1.2 Consul

创建/home/.../consul.d/consul.hcl：

{
  "datacenter": "dc1",
  "data_dir": "/home/.../consul/data",
  "log_level": "INFO",
  "node_name": "consul_server1",
  "server": true,
  "bind_addr": "192.168.10.10",
  "client_addr": "0.0.0.0",
  "bootstrap_expect":1,
  "ui": true,
  "enable_script_checks": true,
}

创建/home/.../consul.d/http-echo.hcl：

service {
  name = "http-echo"
  port = 5678
  check {
    args = ["/home/.../consul/check.sh"]
    interval = "3s"
  }
}

创建/home/.../consul/check.sh文件作为consul检测http-echo的脚本，注意chmod给权限：

#!/bin/bash
http=`docker ps | grep "http-echo"`

  if [ -n "$http" ];#非空
  then
    exit 0
  else
    exit 2

fi

exit 0表示检测成功，1表示警告，其他表示失败。

执行consul agent -config-dir /home/.../consul.d启动consul agent。

1.3 Nomad Job(http-echo容器)

创建http-echo.nomad以备之后用：

job "http-echo" {
  datacenters = ["dc1"]
  group "example" {
    count = 1
    network {
      port "http" {
        static = "5678"
      }
    }
    task "server" {
      driver = "docker"
      config {
        image = "hashicorp/http-echo"
        ports = ["http"]
        args = [
          "-listen",
          ":5678",
          "-text",
          "hello world",
        ]
      }
    }
  }
}

2 consul-template

源码：https://github.com/hashicorp/consul-template

二进制文件：https://releases.hashicorp.com/consul-template/

参考：configuration-file，Templating Language

给权限：

sudo cp ./consul-template /usr/bin/consul-template
sudo chmod 777 /usr/bin/consul-template

创建consul-template-config.hcl template配置文件：

consul {
  address = "192.168.10.10:8500"

  retry {
    enabled  = true
    attempts = 12
    backoff  = "1s"
  }
}
template {
  source      = "/home/.../consul/http-echo.ctmpl"
  destination = "/home/.../consul/ip_port.txt"
  perms       = 0600
  command     = "/home/.../consul/template.sh"
}

source表示模版文件，destination表示输出文件，perms表示文件访问权限，command表示每次变化执行的命令。

创建模版文件/home/.../consul/http-echo.ctmpl：

{{range service "http-echo"}}server  {{.Address}}:{{.Port}};{{end}}

创建空文件/home/.../consul/ip_port.txt和/home/.../consul/ng.conf，或者perms改成0644允许自动创建。

/home/.../consul/template.sh脚本如下，注意chmod给权限：

#!/bin/bash
txt=`cat /home/.../consul/ip_port.txt`

if [[  $txt =~ "server" ]];
  then
    echo "upstream http_backend {" | cat > /home/.../consul/ng.conf

    cat /home/.../consul/ip_port.txt >> /home/.../consul/ng.conf

    echo "     
}

server {
   listen 80;
   server_name test.com;

   location / {
        proxy_pass http://http_backend;
        proxy_redirect off;
        proxy_set_header Host \$host;
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP \$remote_addr;
    }
}"  | cat >> /home/.../consul/ng.conf

    service nginx reload
  else
    > /home/.../consul/ng.conf
fi

因为如果upstream中没有server，service nginx reload会报错，所以需要判断一下输出文件是否有内容，没有的话就把nginx反向代理的配置文件设为空。

sudo consul-template -config=consul-template-config.hcl执行consul-template。

其实只有service nginx reload需要用到sudo权限，可以把这行改成echo "密码" | sudo -S service nginx reload。

3 Nginx反向代理

安装Nginx：https://blog.csdn.net/weixin_43739110/article/details/121079232

在/etc/nginx/nginx.conf中加：

...
http{
  ...
  include /home/.../consul/ng.conf;
  ...
}
...

service nginx restart重启Nginx。

nomad job run http-echo.nomad启动之前创建的Nomad job容器。

浏览器访问192.168.10.10:8500可以看到：

运行容器后，会看到/home/.../consul/ip_port.txt中的内容变为：

server  192.168.10.10:5678;

/home/.../consul/ng.conf中的内容变为：

upstream http_backend {
server  192.168.10.10:5678;
     
}

server {
   listen 80;
   server_name test.com;

   location / {
        proxy_pass http://http_backend;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

/etc/hosts文件中添加：

192.168.10.10  test.com

浏览器登陆test.com可以看到：

nomad job stop http-echo关闭Nomad job，会看到/home/.../consul/ip_port.txt和/home/.../consul/ng.conf中的内容消失。

浏览器登陆test.com会502错误。

4 Consul DNS

参考：https://www.consul.io/docs/discovery/dns

修改/etc/systemd/resolved.conf：

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=127.0.0.1
#FallbackDNS=
Domains=~consul
#LLMNR=no
#MulticastDNS=no
DNSSEC=no
#Cache=yes
#DNSStubListener=no

将本机53端口映射到8600端口：

sudo iptables --table nat --append OUTPUT --destination localhost --protocol udp --match udp --dport 53 --jump REDIRECT --to-ports 8600
sudo iptables --table nat --append OUTPUT --destination localhost --protocol tcp --match tcp --dport 53 --jump REDIRECT --to-ports 8600

systemctl restart systemd-resolved重启systemd-resolved。

访问结点的域名格式：

<node>.node[.datacenter].<domain>

访问服务的域名格式：

[tag.]<service>.service[.datacenter].<domain>

比如，将/home/.../consul/ng.conf中的内容变为：

upstream http_backend {
server  192.168.10.10:5678;
     
}

server {
   listen 80;
   server_name consul_server1.node.consul;

   location / {
        proxy_pass http://http_backend;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

这样不需要在/etc/hosts中添加映射，就可以在浏览器直接访问consul_server1.node.consul，会出现http-echo容器的hello world。