ingress-nginx-controller的HTTPS双向认证配置

运维有道

已于 2024-01-07 12:03:07 修改

阅读量1.8k

点赞数 31

分类专栏： kubernetes 文章标签： kubernetes https nginx

于 2024-01-07 11:52:07 首次发布

本文链接：https://blog.csdn.net/ityoudao/article/details/135437680

版权

kubernetes 专栏收录该内容

3 篇文章 0 订阅

订阅专栏

原来我们使用云厂商的应用程序网关配置HTTPS双向认证，但后来客户提出要求使用证书吊销列表CRL。遗憾的是，云厂商的应用程序网关并不支持通过CRL进行吊销检查，这使得我们不得不将HTTPS双向认证迁移到Kubernetes集群中的ingress-nginx-controller。接下来，我们将详细描述在Kubernetes环境中生成服务端和客户端证书，并配置Ingress进行HTTPS双向认证的步骤。此外，我们还会探讨如何配置客户端证书的证书吊销列表，以满足客户的需求。

生成服务端和客户端证书并配置Ingress

生成服务端用的CA证书和密钥、服务端证书和密钥

mkdir ingress-mtls && ingress-mtls

# Generate the CA Key and Certificate
# 这里openssl生成的证书是pem格式而非der格式，指定证书格式可以使用`-outform arg   output format - DER or PEM`参数
openssl req -x509 -sha256 -newkey rsa:4096 -keyout server-ca-key.pem -out server-ca-cert.pem -days 36525 -nodes -subj '/CN=Aispeech Server Cert Authority'

# Generate the Server Key, and Certificate and Sign with the CA Certificate
# 使用X509v3 Subject Alternative Name
echo subjectAltName = DNS:mtls.dev.ityoudao.cn > server-extfile.cnf
openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn'
openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -extfile server-extfile.cnf -set_serial 01 -out server-cert.pem
# 创建完整的证书链文件
cat server-cert.pem server-ca-cert.pem > server-cert-chain.pem

# ll server*
-rw-r--r--. 1 root root 1846 1月   5 16:39 server-ca-cert.pem
-rw-r--r--. 1 root root 3268 1月   5 16:39 server-ca-key.pem
-rw-r--r--. 1 root root 1704 1月   5 16:39 server-cert.pem
-rw-r--r--. 1 root root 1598 1月   5 16:39 server.csr
-rw-r--r--. 1 root root 3272 1月   5 16:39 server-key.pem

生成客户端用的CA证书和密钥、客户端证书和密钥

# Generate the CA Key and Certificate
openssl req -x509 -sha256 -newkey rsa:4096 -keyout client-ca-key.pem -out client-ca-cert.pem -days 36525 -nodes -subj '/CN=Aispeech Client Cert Authority'

# Generate the Client Key, and Certificate and Sign with the CA Certificate
openssl req -new -newkey rsa:4096 -keyout client-key.pem -out client.csr -nodes -subj '/CN=Client001'
openssl x509 -req -sha256 -days 365 -in client.csr -CA client-ca-cert.pem -CAkey client-ca-key.pem -set_serial 01 -out client-cert.pem

# ll client*
-rw-r--r--. 1 root root 1846 1月   5 15:52 client-ca-cert.pem
-rw-r--r--. 1 root root 3272 1月   5 15:52 client-ca-key.pem
-rw-r--r--. 1 root root 1688 1月   5 15:52 client-cert.pem
-rw-r--r--. 1 root root 1586 1月   5 15:52 client.csr
-rw-r--r--. 1 root root 3268 1月   5 15:52 client-key.pem

创建保存服务端证书和密钥的Secret

kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert-chain.pem --key ./server-key.pem

创建保存客户端CA证书的Secret

kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem

创建HTTPS双向认证的Ingress

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mtls-test-ingress
  namespace: odcp
  annotations:
    # Enable client certificate authentication
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    # Create the secret containing the trusted ca certificates
    nginx.ingress.kubernetes.io/auth-tls-secret: "odcp/mtls-client-ca-secret"
    # Specify the verification depth in the client certificates chain
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
    # Specify an error page to be redirected to verification errors
    #nginx.ingress.kubernetes.io/auth-tls-error-page: "https://www.ityoudao.cn/"
    # Specify if certificates are passed to upstream server
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "false"
spec:
  ingressClassName: nginx
  rules:
  - host: mtls.dev.ityoudao.cn
    http:
      paths:
      - backend:
          service:
            name: nginx-test
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - mtls.dev.ityoudao.cn
    secretName: mtls-server-secret
EOF

Ingress的客户端证书的相关配置可以参考官网User Guide - Annotations - Client Certificate Authentication

使用curl命令测试HTTPS双向认证

不带客户端证书和密钥，返回400 Bad Request错误

# curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem
HTTP/1.1 400 Bad Request
Date: Fri, 05 Jan 2024 08:58:47 GMT
Content-Type: text/html
Content-Length: 230
Connection: close
Cache-Control: no-cache

带客户端证书和密钥，返回200 OK正常

# curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem --cert ./client-cert.pem --key ./client-key.pem
HTTP/1.1 200 OK
Date: Fri, 05 Jan 2024 08:59:12 GMT
Content-Type: text/html
Content-Length: 2161
Connection: keep-alive
Last-Modified: Tue, 19 Sep 2023 06:28:05 GMT
ETag: "65093f75-871"
Accept-Ranges: bytes
Cache-Control: no-cache

使用带--cacert server-ca-cert.pem参数的curl命令，没有报服务端证书错误，说明服务端证书和密钥配置正确！不带客户端证书和密钥、带客户端证书和密钥的测试结果均符合预期，说明客户端证书和密钥配置正确！

遇到的问题和解决办法

ingress-nginx-controller报x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn错误

# kubectl logs -n ingress-nginx ingress-nginx-controller-default-7d94c6957d-l6wwr --tail=100 | grep mtls
W0105 08:11:38.291923       7 controller.go:1347] Unexpected error validating SSL certificate "odcp/mtls-server-secret" for server "mtls.dev.ityoudao.cn": x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn
W0105 08:11:38.291937       7 controller.go:1353] SSL certificate "odcp/mtls-server-secret" does not contain a Common Name or Subject Alternative Name for server "mtls.dev.ityoudao.cn": x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn

后来发现是因为，误用服务端CA证书和私钥文件创建了mtls-server-secret，因此CN和SAN校验都不通过，导致ingress-nginx-controller报x509: certificate is not valid for any names, but wanted to match mtls.dev.ityoudao.cn错误，更换为正确的服务端证书和密钥即可解决问题！

kubectl delete secret -n odcp mtls-server-secret
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-ca-cert.pem --key ./server-ca-key.pem

curl命令报curl: (60) Peer's certificate has an invalid signature.错误

# curl https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem
curl: (60) Peer's certificate has an invalid signature.
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

对端证书签名无效，也就是说服务端证书无效，这里的原因是证书链不完整，使用完整的证书链文件即可解决问题：

# 错误的做法，证书链不完整
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert.pem --key ./server-key.pem

# 正确的做法，完整的证书链
cat server-cert.pem server-ca-cert.pem > server-cert-chain.pem
kubectl create secret tls -n odcp mtls-server-secret --cert ./server-cert-chain.pem --key ./server-key.pem

ingress-nginx-controller报x509: certificate relies on legacy Common Name field, use SANs instead错误

I0105 08:43:27.668845       7 store.go:584] "Secret was added and it is used in ingress annotations. Parsing" secret="odcp/mtls-server-secret"
I0105 08:43:27.669756       7 backend_ssl.go:65] "Adding secret to local store" name="odcp/mtls-server-secret"
W0105 08:43:27.673743       7 controller.go:1347] Unexpected error validating SSL certificate "odcp/mtls-server-secret" for server "mtls.dev.ityoudao.cn": x509: certificate relies on legacy Common Name field, use SANs instead

在SUSE官网看到一句说明：“While in the past it was sufficient to have the server name as “Common Name(CN)” within the “Subject” of a certificate and additionally within “Subject Alternative Name”, newer implementations may just check for “Subject Alternative Name”.”，翻译为中文意思是：过去，仅在证书的“主题”中设置“通用名称(CN)”，然后在“主题备用名称”中进行附加就足够了，但较新的实现可能只会检查“主题备用名称”。
这里是因为服务端证书只使用了Subject，没有使用X509v3 Subject Alternative Name，而ingress-nginx-controller需要检查主题备用名称SAN，导致报x509: certificate relies on legacy Common Name field, use SANs instead错误，在服务端证书使用SAN即可解决问题：

# 错误的做法，没有使用X509v3 Subject Alternative Name
openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn'
openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -set_serial 01 -out server-cert.pem

# 正确的做法，使用了X509v3 Subject Alternative Name
echo subjectAltName = DNS:mtls.dev.ityoudao.cn > server-extfile.cnf
openssl req -new -newkey rsa:4096 -keyout server-key.pem -out server.csr -nodes -subj '/CN=mtls.dev.ityoudao.cn'
openssl x509 -req -sha256 -days 365 -in server.csr -CA server-ca-cert.pem -CAkey server-ca-key.pem -extfile server-extfile.cnf -set_serial 01 -out server-cert.pem

配置客户端证书的证书吊销列表

假设需要被吊销的证书文件为client-cert.pem，使用openssl ca -revoke命令吊销证书

touch /etc/pki/CA/index.txt
openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem

如果openssl ca -revoke命令unable to load CA private key错误：

# openssl ca -revoke client-cert.pem
Using configuration from /etc/pki/tls/openssl.cnf
Error opening CA private key /etc/pki/CA/private/cakey.pem
140215560439696:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/private/cakey.pem','r')
140215560439696:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:
unable to load CA private key

# 解决办法：将客户端CA证书和密钥文件拷贝到`/etc/pki/CA/`目录下，或者直接用`-cert`和`-keyfile`参数指定客户端CA证书和密钥文件
# openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem

如果openssl ca -revoke命令unable to open '/etc/pki/CA/index.txt'错误：

# openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/index.txt: No such file or directory
unable to open '/etc/pki/CA/index.txt'
139702194755472:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/index.txt','r')
139702194755472:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:

# 解决办法：创建一个空的`/etc/pki/CA/index.txt`文件即可
# touch /etc/pki/CA/index.txt
# openssl ca -revoke client-cert.pem -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
Adding Entry with serial number 01 to DB for /CN=Client001
Revoking Certificate 01.
Data Base Updated
# cat /etc/pki/CA/index.txt
R 250104075252Z 240105091344Z 01  unknown /CN=Client001
# cat /etc/pki/CA/index.txt.attr
unique_subject = yes
# cat /etc/pki/CA/index.txt.old

使用openssl ca -gencrl命令生成证书吊销列表文件

echo 01 > /etc/pki/CA/crlnumber
openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem

如果openssl ca -gencrl命令error while loading CRL number错误：

# openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/crlnumber: No such file or directory
error while loading CRL number
140466499000208:error:02001002:system library:fopen:No such file or directory:bss_file.c:402:fopen('/etc/pki/CA/crlnumber','r')
140466499000208:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:404:

# 解决办法：创建`/etc/pki/CA/crlnumber`文件即可
# echo 01 > /etc/pki/CA/crlnumber
# openssl ca -gencrl -out client-ca.crl -cert client-ca-cert.pem -keyfile client-ca-key.pem
Using configuration from /etc/pki/tls/openssl.cnf
# ll client-ca.crl
-rw-r--r--. 1 root root 954 1月   5 17:27 client-ca.crl
# cat /etc/pki/CA/crlnumber
02
# cat /etc/pki/CA/crlnumber.old
01

使用openssl crl命令查看证书吊销列表文件

openssl crl -text -noout -in client-ca.crl

# 结果如下：
# openssl crl -text -noout -in client-ca.crl
Certificate Revocation List (CRL):
        Version 2 (0x1)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: /CN=Aispeech Client Cert Authority
        Last Update: Jan  5 09:27:46 2024 GMT
        Next Update: Feb  4 09:27:46 2024 GMT
        CRL extensions:
            X509v3 CRL Number:
                1
Revoked Certificates:
    Serial Number: 01
        Revocation Date: Jan  5 09:13:44 2024 GMT
    Signature Algorithm: sha256WithRSAEncryption
         86:b3:a3:1e:31:75:bb:0f:f3:32:af:2b:57:e6:e2:ca:fb:94:
         9b:c1:0b:15:39:af:9e:9a:d6:25:9c:3a:69:e0:29:dd:d6:b6:
         55:79:2f:24:c3:7f:40:38:b6:77:6e:08:03:bb:23:0c:9c:5f:
         db:0d:18:4d:c6:d6:01:74:6f:d5:ba:bb:bc:e3:95:96:91:eb:
         e1:2b:27:3c:62:33:a8:eb:ed:4a:26:5a:a3:27:cd:f8:02:73:
         1c:64:c0:60:95:02:ea:b3:63:ba:d4:86:65:83:c7:cf:fb:7e:
         9a:71:7e:4f:9d:b2:50:0b:eb:fb:c5:f4:c2:41:8f:57:50:27:
         88:a2:19:02:e5:84:6a:f1:30:ea:71:db:23:8a:04:f0:11:75:
         c5:8c:6d:04:36:99:6d:42:c6:c0:8d:00:0e:3d:b7:dd:0a:28:
         ed:e2:5e:7c:f4:56:41:35:d3:73:3c:2f:96:86:92:1e:d7:5a:
         d2:e5:dc:7c:a1:7b:a9:10:ef:5a:7e:30:1d:86:6f:97:c5:58:
         b8:3b:ee:75:cf:35:ad:7d:fb:47:ac:a8:7a:81:65:56:4d:2e:
         15:60:c0:c9:39:ca:b0:cb:21:a1:48:33:11:51:ee:3e:d5:c2:
         9b:ba:c5:fc:ac:04:99:87:2c:2b:56:fe:06:62:76:f8:31:df:
         dd:b8:a1:f4:bc:d6:87:18:79:00:2d:a6:15:cd:c3:88:80:48:
         2a:59:70:8b:06:1d:08:11:39:b5:35:7d:58:c3:3b:27:b5:89:
         5f:18:fb:5e:4b:48:4c:04:6a:20:08:96:ad:3d:65:23:da:ad:
         0c:74:d6:fc:2a:79:8c:41:42:3b:bd:c5:d0:cb:28:3c:f5:68:
         9c:e1:d4:7d:28:c1:3b:20:36:90:d0:71:97:3d:54:78:0f:49:
         6e:a2:f4:56:7b:4e:64:03:a6:18:6e:8d:d9:a8:28:96:1f:94:
         20:35:1d:6d:2c:f4:56:65:4c:0a:07:2c:c8:5f:44:6f:6c:53:
         e4:13:d8:56:0c:07:79:7a:0d:cb:a6:e1:de:7e:f0:12:aa:9d:
         f9:5b:59:6b:61:18:21:96:96:75:4c:3f:b0:ab:73:27:d7:41:
         ac:50:ea:99:56:13:0f:b9:df:4c:6b:0a:e4:5e:df:56:76:9e:
         3a:ce:ab:41:1f:2b:96:bc:9f:77:96:0b:c6:fa:a9:7c:ae:94:
         a9:c7:f2:68:a0:b2:07:82:1e:74:48:b7:68:f7:da:b0:0d:c4:
         5c:08:8a:0e:86:14:70:89:6f:25:9b:63:1d:9b:b4:87:28:0e:
         15:23:0f:05:51:c9:9e:b7:57:be:06:b8:74:9d:4f:79:d3:49:
         91:16:a3:59:d6:54:64:9a

创建保存客户端CA证书和证书吊销列表的Secret

kubectl delete secret -n odcp mtls-client-ca-secret
kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem --from-file=ca.crl=./client-ca.crl

再次使用curl命令测试客户端证书

# curl -I https://mtls.dev.ityoudao.cn --cacert server-ca-cert.pem --cert ./client-cert.pem --key ./client-key.pem
HTTP/1.1 400 Bad Request
Date: Fri, 05 Jan 2024 09:30:37 GMT
Content-Type: text/html
Content-Length: 208
Connection: close
Cache-Control: no-cache

被吊销的客户端证书和密钥，访问服务报400 Bad Request错误，结果符合预期，说明客户端证书吊销成功！

更新证书吊销列表

如果证书吊销列表发生变更，可以使用如下命令更新证书吊销列表

# 更新证书吊销列表client-ca.crl
kubectl create secret generic -n odcp mtls-client-ca-secret --from-file=ca.crt=./client-ca-cert.pem --from-file=ca.crl=./client-ca.crl --dry-run="client" -o yaml | kubectl apply -f -

Secret更新后，ingress-nginx-controller会自动reload，reload可能会影响长连接服务：

# ingress-nginx-controller的reload日志
I0105 09:49:31.929155       7 store.go:615] "secret was updated and it is used in ingress annotations. Parsing" secret="odcp/mtls-client-ca-secret"
I0105 09:49:31.929841       7 backend_ssl.go:57] "Updating secret in local store" name="odcp/mtls-client-ca-secret"
I0105 09:49:31.946486       7 controller.go:166] "Configuration changes detected, backend reload required"
I0105 09:49:33.286314       7 controller.go:183] "Backend successfully reloaded"
I0105 09:49:33.288423       7 event.go:285] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-default-7d94c6957d-l6wwr", UID:"afac1300-97ad-4304-8ff0-95357f4e3144", APIVersion:"v1", ResourceVersion:"783734248", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration

在本文中，我们深入探讨了在Kubernetes环境中配置ingress-nginx-controller的HTTPS双向认证的过程。文章详细介绍了生成服务端和客户端证书的步骤，以及配置Ingress实现HTTPS双向认证的过程。同时，我们解决了在部署过程中遇到的一系列问题，包括证书配置错误、证书链不完整、以及ingress-nginx-controller报错等。最后，我们还介绍了如何配置客户端证书的证书吊销列表，以满足客户对安全性的额外需求。通过本文，希望大家可以获得在Kubernetes环境中进行HTTPS双向认证的全面指南，帮助应对实际部署中可能遇到的各种挑战。

运维有道

关注

31
点赞
踩
31

收藏

觉得还不错? 一键收藏
1
评论
ingress-nginx-controller的HTTPS双向认证配置

我们将详细描述在Kubernetes环境中生成服务端和客户端证书，并配置Ingress进行HTTPS双向认证的步骤。此外，我们还会探讨如何配置客户端证书的证书吊销列表，以满足客户的需求。
复制链接

扫一扫