consul + consul-template + tls 安装指南

实验环境：
Consul v0.8.1
consul-template v0.19.0

consul参考：
https://github.com/hashicorp/consul-template
https://github.com/hashicorp/consul

证书制作参考：
https://github.com/cloudflare/cfssl
https://coreos.com/os/docs/latest/generate-self-signed-certificates.html

使用cfssl 创建 consul 证书：
各个节点必须同步时钟：ntpdate cn.ntp.org.cn

使用cfssl制作证书以及consul安装过程：


（一）安装cfssl （参考：https://github.com/cloudflare/cfssl）

$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ chmod +x cfssl_linux-amd64
$ sudo mv cfssl_linux-amd64 /root/local/bin/cfssl


$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ chmod +x cfssljson_linux-amd64
$ sudo mv cfssljson_linux-amd64 /root/local/bin/cfssljson


$ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
$ chmod +x cfssl-certinfo_linux-amd64
$ sudo mv cfssl-certinfo_linux-amd64 /root/local/bin/cfssl-certinfo


$ export PATH=/root/local/bin:$PATH
$ mkdir ssl
$ cd ssl
$ cfssl print-defaults config > config.json
$ cfssl print-defaults csr > csr.json

（二）创建 CA (Certificate Authority)
创建 CA 配置文件：

$ cat ca-config.json
{
  "signing": {
    "default": {
      "expiry": "8760h"
    },
    "profiles": {
      "consul": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "8760h"
      }
    }
  }
}

ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
server auth：表示 client 可以用该 CA 对 server 提供的证书进行验证；

client auth：表示 server 可以用该 CA 对 client 提供的证书进行验证；

创建 CA 证书签名请求：

$ cat ca-csr.json
{
  "CN": "*.example.com",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "consul-group",
      "OU": "System"
    }
  ]
}

"CN"：Common Name，浏览器使用该字段验证网站是否合法；
"O"：Organization，所属的组 (Group)；
生成 CA 证书和私钥：

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca
$ ls ca*
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

（三）创建consul 集群 TLS 秘钥和证书
创建 consul 证书签名请求：

$ cat > consul-csr.json <<EOF
{
  "CN": "*.example.com",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "consul-group",
      "OU": "System"
    }
  ]
}
EOF

hosts 字段指定授权使用该证书的 consul 节点 IP,不填即任何节点都可以用；

生成 consul 证书和私钥：

$ cfssl gencert -ca=./ca.pem \
  -ca-key=./ca-key.pem \
  -config=./ca-config.json \
  -profile=consul consul-csr.json | cfssljson -bare consul
$ ls consul*
consul.csr  consul-csr.json  consul-key.pem  consul.pem

（四）配置consul 集群 TLS ：

(一般 一个client，3个server）
(
bootstrap : "bootstrap": true, "server": true,
server : "bootstrap": false, "server": true,
client : "bootstrap": false, "server": false,
)

把证书分发到各个节点：

[root@etcd-client ssl]#scp ./consul*.pem ./ca.pem root@192.168.5.7:/etc/consul.d/ssl
root@192.168.5.7's password:
consul-key.pem                                100% 1675     1.6KB/s   00:00    
consul.pem                                    100% 1424     1.4KB/s   00:00    
ca.pem                                        100% 1391     1.4KB/s   00:00    

配置consul证书(配置文件只有一台bootstrap,当3台节点加入集群后，bootstrap改为false,让集群自动选举leader)：

[root@etcd-host0 consul.d]# cat /etc/consul.d/config.json
{
"bootstrap": true,
"server": true,
"client_addr": "192.168.5.7",
"bind_addr": "192.168.5.7",
"datacenter": "hyd",
"ui": true,
"data_dir": "/var/consul",
"encrypt": "QTW1Rcdes9eyQFnUng/84g==",
"log_level": "DEBUG",
"enable_syslog": true,


"ports": {
  "https": 8501
},


"key_file": "/etc/consul.d/ssl/consul-key.pem",
"cert_file": "/etc/consul.d/ssl/consul.pem",
"ca_file": "/etc/consul.d/ssl/ca.pem",


"verify_outgoing": true,
"verify_incoming": true


}

启动集群：

[root@etcd-host0 consul.d]# cat /etc/systemd/system/consul.service
[Unit]
Description=consul agent
Requires=network-online.target
After=network-online.target


[Service]
EnvironmentFile=-/etc/sysconfig/consul
Environment=GOMAXPROCS=2
Restart=on-failure
ExecStart=/usr/local/bin/consul agent -config-file=/etc/consul.d/config.json -rejoin
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM


[Install]
WantedBy=multi-user.target

其他节点加入集群：

[root@k8s-client ssl]# consul join -client-cert=./consul.pem -client-key=./consul-key.pem -ca-file=./ca.pem -http-addr=https://h1.example.com:8501 192.168.5.8
[root@k8s-client ssl]# consul join -client-cert=./consul.pem -client-key=./consul-key.pem -ca-file=./ca.pem -http-addr=https://h1.example.com:8501 192.168.5.86

查看节点：

[root@etcd-client ssl]# consul members -client-cert=./consul.pem -client-key=./consul-key.pem -ca-file=./ca.pem -http-addr=https://h1.example.com:8501
Node        Address            Status  Type    Build  Protocol  DC
etcd-host0  192.168.5.7:8301   alive   server  0.8.5  2         hyd
etcd-host1  192.168.5.8:8301   alive   server  0.8.5  2         hyd
etcd-host2  192.168.5.86:8301  alive   client  0.8.5  2         hyd

（五）一般不使用服务的证书访问，再生成一份客户端的证书（可以设置客户端证书的过期时间短些,这里就没有使用ca-config.json,而新建一个证书配置文件ca-config-admin.json,把过期时间设置为60小时）：

[root@etcd-client ssl]# cat admin-csr.json
{
  "CN": "*.example.com",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

[root@etcd-node1 ssl]# cat ./ca-config-admin.json
{
  "signing": {
    "default": {
      "expiry": "60h"
    },
    "profiles": {
      "consul": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "60h"
      }
    }
  }
}

生成客户端证书：

cfssl gencert -ca=./ca.pem \
  -ca-key=./ca-key.pem \
  -config=./ca-config-admin.json \
  -profile=consul admin-csr.json | cfssljson -bare admin

验证证书：

[root@etcd-client ssl]# cfssl-certinfo -cert ./admin.pem
[root@etcd-client ssl]# openssl x509  -noout -text -in  admin.pem

[root@k8s-node1 ssl]# consul members -client-cert=./admin.pem -client-key=./admin-key.pem -ca-file=./ca.pem -http-addr=https://h1.example.com:8501
Node        Address            Status  Type    Build  Protocol  DC
etcd-host0  192.168.5.7:8301   alive   server  0.8.5  2         hyd
etcd-host1  192.168.5.8:8301   alive   server  0.8.5  2         hyd
etcd-host2  192.168.5.86:8301  alive   client  0.8.5  2         hyd

查看master节点：

[root@etcd-client ssl]# curl -k https://h1.example.com:8501/v1/status/leader?pretty --key ./admin-key.pem --cert ./admin.pem
"192.168.5.7:8300"

consul-template 客户端连接consul：

[root@etcd-client consul]# cat template-config.json
consul {
address = "h1.example.com:8501"  
ssl {
  enabled = true
  verify = false
  ca_cert = "/root/consul/ssl/ca.pem"
  cert = "/root/consul/ssl/admin.pem"
  key = "/root/consul/ssl/admin-key.pem"
  }
}

log_level = "debug"  

template {  
source = "./tmpltest.ctmpl"  
destination = "./result"  
}

[root@k8s-node1 consul]# consul-template -config=./template-config.json -once

技巧：
1. 为了使证书更安全，最好生成证书时带上IP，这样该证书只能在指定的IP上使用（就是上面提到的hosts）。
2.外部访问可以生成一个1小时的证书，一小时后证书失效。
附录：
https://www.openssl.org/docs/manmaster/man5/x509v3_config.html#Extended-Key-Usage
extended Key Usage (扩展密钥的使用说明：)
 Value                  Meaning
 -----                  -------
 serverAuth             SSL/TLS Web Server Authentication.
 clientAuth             SSL/TLS Web Client Authentication.

bugs：

1.
./consul-template -consul-addr 192.168.5.7:8501 -template "./tmpltest.ctmpl:./result" -once -consul-ssl -consul-ssl-cert=./admin.pem -consul-ssl-key=./admin-key.pem -consul-ssl-ca-cert=./ca.pem -log-level=debug


2017/07/06 17:20:45.786509 [WARN] (view) kv.block(com/test/name): Get https://192.168.5.7:8501/v1/kv/com/test/name?stale=&wait=60000ms: x509: certificate signed by unknown authority (retry attempt 4 after "2s")
出现该问题原因：
consul-template 版本问题，升级 consul-template v0.18.5 (9902dd5) 到 consul-template v0.19.0 (33b34b3) 问题解决。
参考： https://github.com/hashicorp/consul-template/issues/965

2.
zyh@local:~/consul/ssl$ consul members -ca-file=./ca.cert -client-cert=./consul.cert -client-key=./consul.key -http-addr=https://192.168.0.9:8080
Error retrieving members: Get https://192.168.0.9:8080/v1/agent/members: x509: cannot validate certificate for 192.168.0.9 because it doesn't contain any IP SANs
出现该问题的原因：使用IP访问，改为域名访问就好了。
在输出证书的时候，你填写了域名，请使用该域名访问：

3.
[root@k8s-node1 ssl]# consul members -client-cert=./consul.pem -client-key=./consul-key.pem -ca-file=./ca.pem -http-addr=https://h1.example.com:8501
Error retrieving members: Get https://h1.example.com:8501/v1/agent/members: remote error: tls: bad certificate
各个节点和客户端的时钟不同步，请先同步时钟，问题就解决了。（ntpdate cn.ntp.org.cn）

（六）其他

openssl 制作证书方法： 

参考 https://www.digitalocean.com/community/tutorials/how-to-secure-consul-with-tls-encryption-on-ubuntu-14-04