【K8S 二】搭建Docker Registry私有仓库（自签发证书+登录认证）（K8S和非K8S环境下）

目录

生成证书（更新：2022-08-02）

单SAN(Subject Alternative Name)的场景

多SAN(Subject Alternative Name)场景

创建openssl配置文件cert-info.conf 

 创建证书

 查看生成的证书

docker配置证书，以访问registry

将生成的私有证书追加到系统的证书管理文件

 生成认证文件

 定义K8S下启动配置文件

应用启动配置创建Docker Registry

 CURL验证仓库

Docker验证

K8S应用该私有仓库

非K8S环境下启动Docker Registry

---

生成证书（更新：2022-08-02）

Go 1.15 版本开始废弃 CommonName，因此推荐使用 SAN 证书。所以下面的证书创建方式就不能用了，否则会报错：

Error response from daemon: Get "https://192.168.186.96:8443/v2/": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0

# 创建CA证书
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 36500  -subj "/CN=192.168.186.96"  -key ca.key  -out ca.crt

# 创建私钥
openssl genrsa -out registry.key 4096
openssl req  -new -sha512 -subj "/CN=192.168.186.96"  -key registry.key  -out registry.csr

# v3.ext文件
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth 
subjectAltName = @alt_names
[alt_names]
DNS.1=registry-in-96
IP.1=192.168.186.96
EOF
# 创建证书
openssl x509 -req -sha512 -days 36500 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in registry.csr -out registry.crt

ll
total 28
-rw-r--r--. 1 root root 1805 Aug  2 11:54 ca.crt
-rw-r--r--. 1 root root 3243 Aug  2 11:54 ca.key
-rw-r--r--. 1 root root   17 Aug  2 11:56 ca.srl
-rw-r--r--. 1 root root 1850 Aug  2 11:56 registry.crt
-rw-r--r--. 1 root root 1590 Aug  2 11:55 registry.csr
-rw-r--r--. 1 root root 3243 Aug  2 11:55 registry.key
-rw-r--r--. 1 root root  253 Aug  2 11:55 v3.ext

我们对比一下上面的证书创建过程以及下面的创建命令，会发现：下面仅仅只是上面CA证书和秘钥的创建而已。

单SAN(Subject Alternative Name)的场景

openssl req -subj '/C=CN/ST=BJ/L=BeiJing/OU=IAM5.0/CN=rhel-8-11.8' \
-newkey rsa:4096 -nodes -sha256 -keyout rhel-8-11.8.key \
-x509 -days 365 -out rhel-8-11.8.crt

多SAN(Subject Alternative Name)场景

创建openssl配置文件cert-info.conf 

[ req ]
default_bits        = 4096
default_md          = sha256
default_keyfile     = server-key.pem
distinguished_name  = subject
req_extensions      = extensions
x509_extensions     = extensions
string_mask         = utf8only

[ subject ]
countryName                 = Country Name (2 letter code)
countryName_default         = CN

stateOrProvinceName         = State or Province Name (full name)
stateOrProvinceName_default = BJ

localityName                = Locality Name (eg, city)
localityName_default        = BeiJing

organizationName            = Organization Name (eg, company)
organizationName_default    = Ultrapower.com.cn

commonName                  = Common Name (e.g. server FQDN or YOUR name)
commonName_default          = rhel-8-11.8

emailAddress                = Email Address
emailAddress_default        = 

[ extensions ]

subjectKeyIdentifier        = hash
authorityKeyIdentifier      = keyid,issuer

basicConstraints            = CA:TRUE
#basicConstraints            = CA:FALSE
#keyUsage                    = nonRepudiation, digitalSignature, keyEncipherment
#extendedKeyUsage            = serverAuth
subjectAltName              = @alternate_names
nsComment                   = "OpenSSL Generated Certificate"

[ alternate_names ]

DNS.1       = rhel-8-11.8
IP.1        = 192.168.11.8

 创建证书

openssl req -new -newkey rsa:4096 -nodes \
-keyout rhel-8-11.8.key -x509 -days 365 -out rhel-8-11.8.crt \
-config cert-info.conf 

 查看生成的证书

openssl x509 -in rhel-8-11.8.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            ……
        Signature Algorithm: ……
        Issuer: C = CN, ST = BeiJing, L = Chaoyang, OU = IAM5.0, CN = rhel-8-11.8
        Validity
            Not Before: Nov 19 04:49:44 2020 GMT
            Not After : Nov 19 04:49:44 2021 GMT
        Subject: C = CN, ST = BeiJing, L = Chaoyang, OU = IAM5.0, CN = rhel-8-11.8
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
……

docker配置证书，以访问registry

mkdir -p /etc/docker/certs.d/192.168.186.96:8443
cp registry.crt /etc/docker/certs.d/192.168.186.96:8443

上述可以解决docker访问，否则报错：

Login did not succeed, error: Error response from daemon: Get "https://192.168.186.96:8443/v2/": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "192.168.186.96")

下面的方式已经不能使用。

将生成的私有证书追加到系统的证书管理文件

如果不追加到ca-bundle.crt，则docker login、pull、push都会出错。使用curl -u admin  -X GET https://192.168.186.96:8443/v2/_catalog验证，错误打印如下：

Enter host password for user 'admin':

curl: (6) Could not resolve host:  -x; Unknown error

或
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

cat registry.crt >> /etc/pki/tls/certs/ca-bundle.crt 

 生成认证文件

yum -y install httpd-tools
htpasswd -Bbn 用户名 密码 > /home/docker/auth/htpasswd

 定义K8S下启动配置文件

apiVersion: apps/v1
kind: Deployment 
metadata:
  name: docker-registry
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: docker-registry
  replicas: 1
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      nodeName: rhel-8-11.8
      containers:
      - name: docker-registry
        image: registry:2.7.1
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8443
        env:
        - name: REGISTRY_HTTP_ADDR
          value: "0.0.0.0:8443" 
        - name: REGISTRY_HTTP_TLS_CERTIFICATE
          value: "/certs/rhel-8-11.8.crt"
        - name: REGISTRY_HTTP_TLS_KEY
          value: "/certs/rhel-8-11.8.key"
        - name: REGISTRY_AUTH
          value: "htpasswd"
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: "/auth/htpasswd"
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: "Registry Realm"
        volumeMounts:
        - name: reg-data
          mountPath: /var/lib/registry
        - name: reg-auth
          mountPath: /auth
        - name: reg-certs
          mountPath: /certs
      volumes:
      - name: reg-data
        hostPath:
          path: /home/docker/images
      - name: reg-auth
        hostPath:
          path: /home/docker/auth
      - name: reg-certs
        hostPath:
          path: /home/docker/certs.d
---
apiVersion: v1
kind: Service
metadata:
  name: docker-registry
  namespace: kube-system
  labels:
    name: docker-registry
spec:
  type: NodePort
  ports:
  - port: 8443
    targetPort: 8443
    nodePort: 443
    protocol: TCP
  selector:
    app: docker-registry

应用启动配置创建Docker Registry

kubectl apply -f ~/docker-registry.yaml
kubectl get all  -n kube-system|grep registry|column -t

pod/docker-registry-5f4b67459b-cb8lr        1/1       Running        0       40s           
service/docker-registry                     NodePort  10.254.52.131  <none>  8443:443/TCP  40s
deployment.apps/docker-registry             1/1       1              1       40s           
replicaset.apps/docker-registry-5f4b67459b  1         1              1       40s 

 CURL验证仓库

curl -u admin -X GET https://rhel-8-11.8/v2/_catalog
或
curl -u 用户名:密码 -X GET https://rhel-8-11.8/v2/_catalog

Enter host password for user 'admin':
{"repositories":["pause-amd64"]}

Docker验证

# docker login rhel-8-11.8

# docker tag mysql:5.6.43 rhel-8-11.8/mysql:5.6.43
# docker push rhel-8-11.8/tomcat-app:v1

# docker tag kubeguide/tomcat-app:v1 rhel-8-11.8/tomcat-app:v1
# docker push rhel-8-11.8/tomcat-app:v1

如果成功则说明仓库搭建成功了。

K8S应用该私有仓库

创建Secret

# kubectl create secret docker-registry rhel-8-11.8.reg.key --docker-server=rhel-8-11.8 --docker-username=admin --docker-password=admin_passwd --docker-email=none

查看Secret

# kubectl get secret rhel-8-11.8.reg.key -oyaml

应用Secret

编辑pod启动配置文件# vi mysql-rc.yaml 
apiVersion: v1
kind: ReplicationController
metadata:
  name: mysql
spec:
  replicas: 1
  selector:
    app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: rhel-8-11.8/mysql:5.6.43
        ports:
        - containerPort: 3306
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "123456"
      imagePullSecrets:
      - name: rhel-8-11.8.reg.key

应用mysql-rc.yaml，并验证是否生效！

# kubectl get po
NAME          READY   STATUS    RESTARTS   AGE
mysql-drkpc   1/1     Running   0          27s

非K8S环境下启动Docker Registry

只需提前生成证书和秘钥文件、认证文件等。上面启动命令没有带证书，下面的带了证书。

docker run -d -e REGISTRY_AUTH="htpasswd" -e REGISTRY_AUTH_HTPASSWD_PATH="/auth/htpasswd" \
              -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
              -v /opt/registry/images:/var/lib/registry \
              -v /opt/registry/auth:/auth \
              -p 5000:5000 192.168.186.96/library/registry

docker run -d -e REGISTRY_HTTP_ADDR="0.0.0.0:8443" \
              -e REGISTRY_HTTP_TLS_CERTIFICATE="/certs/registry.crt" \
              -e REGISTRY_HTTP_TLS_KEY="/certs/registry.key" \
              -e REGISTRY_AUTH="htpasswd" -e REGISTRY_AUTH_HTPASSWD_PATH="/auth/htpasswd" \
              -e REGISTRY_AUTH_HTPASSWD_REALM="Registry Realm" \
              -v /opt/registry/images:/var/lib/registry \
              -v /opt/registry/auth:/auth \
              -v /opt/registry/certs:/certs \
              -p 8443:8443 192.168.186.96/library/registry