docker 镜像下载加速(安装 kubernetes 必备)

panbuhei

已于 2022-04-02 13:41:07 修改

阅读量7.5k

点赞数

分类专栏：容器文章标签： docker kubernetes

于 2022-04-02 10:48:27 首次发布

本文链接：https://blog.csdn.net/qq_43584691/article/details/123903647

版权

容器专栏收录该内容

4 篇文章 2 订阅

订阅专栏

大佬的文章

https://blog.k8s.li/kubespray-tips.html
https://fuckcloudnative.io/posts/docker-registry-proxy/

docker registry 可以通过设置 remoteurl 参数将其作为远端仓库的缓存仓库，这样当你通过这个私有仓库的地址拉取镜像时，regiistry 会先将镜像缓存到本地存储，然后再提供给拉取的客户端。

我们可以选择通过 制作镜像 、环境变量的方式进行配置，详细的配置参数可参考官方文档。

1 前期准备

我在准备阶段白扔了几两银子，还多花了一些时间；这里介绍经过爬坑之后，个人感觉最划算的准备方式，至于我的爬坑经历就不赘述了。

一个个人域名：现在各大公有云厂商基本都可以买到域名，而且有很多便宜的域名。我是在新网买的(130元5年)，新网的优点是各种认证、备案通过的比较快；缺点是不提供免费的 SSL 证书。这样的话，就需要在别的地方(比如，阿里云、腾讯云)创建免费证书，然后在新网添加必要的解析。如果觉得麻烦的，可以直接在 “会魔法的服务器” 所在的公有云厂商(比如,阿里云、腾讯云)注册域名。
一个会魔法的服务器：我选择的是腾讯云的轻量应用服务器(中国香港区)，虽然资源配置较低，并且流量有限，但是足够个人或中小型企业使用了。具体配置信息见下图：

PS：我在阿里云和新网都注册了一个域名，阿里云的好几天认证都没通过；新网的几个小时就可以使用了。

PS：我的 SSL 证书使用的是阿里云的免费证书，当然腾讯云也有；免费证书只支持单域名；所以，每个三级域名都需要申请证书。

2 制作通用镜像

为了能够支持缓存 docker.io、gcr.io、k8s.gcr.io、quay.io 和 ghcr.io 等常见的公共镜像仓库，我们需要对 registry 的配置文件进行定制。Dockerfile 如下：

FROM registry:2.6
LABEL maintainer="registry-proxy Docker Maintainers https://fuckcloudnative.io"
ENV PROXY_REMOTE_URL="" \
    DELETE_ENABLED=""
COPY entrypoint.sh /entrypoint.sh

其中，entrypoint.sh 用来将环境变量传入配置文件：

#!/bin/sh

set -e

CONFIG_YML=/etc/docker/registry/config.yml

if [ -n "$PROXY_REMOTE_URL" -a `grep -c "$PROXY_REMOTE_URL" $CONFIG_YML` -eq 0 ]; then
    echo "proxy:" >> $CONFIG_YML
    echo "  remoteurl: $PROXY_REMOTE_URL" >> $CONFIG_YML
    echo "  username: $PROXY_USERNAME" >> $CONFIG_YML
    echo "  password: $PROXY_PASSWORD" >> $CONFIG_YML
    echo "------ Enabled proxy to remote: $PROXY_REMOTE_URL ------"
elif [ $DELETE_ENABLED = true -a `grep -c "delete:" $CONFIG_YML` -eq 0 ]; then
    sed -i '/rootdirectory/a\  delete:' $CONFIG_YML
    sed -i '/delete/a\    enabled: true' $CONFIG_YML
    echo "------ Enabled local storage delete -----"
fi

sed -i "/headers/a\    Access-Control-Allow-Origin: ['*']" $CONFIG_YML
sed -i "/headers/a\    Access-Control-Allow-Methods: ['HEAD', 'GET', 'OPTIONS', 'DELETE']" $CONFIG_YML
sed -i "/headers/a\    Access-Control-Expose-Headers: ['Docker-Content-Digest']" $CONFIG_YML

case "$1" in
    *.yaml|*.yml) set -- registry serve "$@" ;;
    serve|garbage-collect|help|-*) set -- registry "$@" ;;
esac

exec "$@"

3 运行 registry-proxy，并为其添加认证功能

为了防止他人使用，可以为 registry 添加认证功能。建议通过环境变量的方式进行配置；因为在用户密码变化的时候，不用重新构建镜像。

3.1 首先，生成用户密码文件：

$ mkdir -pv /opt/auth
$ htpasswd -Bbn admin ****** > /opt/auth/htpasswd 
$ htpasswd -Bbn  panbuhei ****** >> /opt/auth/htpasswd

3.2 然后，部署 registry-proxy。我这里选择使用 docker-compose 来部署，代码如下：

$ mkdir -pv /opt/docker-compose/registry-proxy/
$ cat << "EOF" > /opt/docker-compose/registry-proxy/docker-compose.yml
version: '3'
services:
  k8s-gcr-registry:
    image: wupanfeng035/registry-proxy:v1.0
    container_name: k8s-gcr-registry
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /var/lib/registry:/var/lib/registry
      - /opt/auth/htpasswd:/opt/auth/htpasswd
    ports:
      - 127.0.0.1:5001:5000
    environment:
      - PROXY_REMOTE_URL=https://k8s.gcr.io
      - REGISTRY_AUTH_HTPASSWD_REALM=basic-auth
      - REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswd

  gcr-registry:
    image: wupanfeng035/registry-proxy:v1.0
    container_name: gcr-registry
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /var/lib/registry:/var/lib/registry
      - /opt/auth/htpasswd:/opt/auth/htpasswd
    ports:
      - 127.0.0.1:5002:5000
    environment:
      - PROXY_REMOTE_URL=https://gcr.io
      - REGISTRY_AUTH_HTPASSWD_REALM=basic-auth
      - REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswd

  hub-registry:
    image: wupanfeng035/registry-proxy:v1.0
    container_name: hub-registry
    restart: always
      - /etc/localtime:/etc/localtime
      - /var/lib/registry:/var/lib/registry
      - /opt/auth/htpasswd:/opt/auth/htpasswd
    ports:
      - 127.0.0.1:5003:5000
    environment:
      - PROXY_REMOTE_URL=https://registry-1.docker.io
      ### 需要下载 dockerhub 的私有仓库时，请配置用户密码
      #- PROXY_USERNAME=test001
      #- PROXY_PASSWORD=********
      - REGISTRY_AUTH_HTPASSWD_REALM=basic-auth
      - REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswd

  quay-registry:
    image: wupanfeng035/registry-proxy:v1.0
    container_name: quay-registry
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /var/lib/registry:/var/lib/registry
      - /opt/auth/htpasswd:/opt/auth/htpasswd
    ports:
      - 127.0.0.1:5004:5000
    environment:
      - PROXY_REMOTE_URL=https://quay.io
      - REGISTRY_AUTH_HTPASSWD_REALM=basic-auth
      - REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswd

  ghcr-registry:
    image: wupanfeng035/registry-proxy:v1.0
    container_name: ghcr-registry
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /var/lib/registry:/var/lib/registry
      - /opt/auth/htpasswd:/opt/auth/htpasswd
    ports:
      - 127.0.0.1:5005:5000
    environment:
      - PROXY_REMOTE_URL=https://ghcr.io
      - REGISTRY_AUTH_HTPASSWD_REALM=basic-auth
      - REGISTRY_AUTH_HTPASSWD_PATH=/opt/auth/htpasswd
EOF

3.3 部署 registry-proxy

$ cd /opt/docker-compose/registry-proxy/
$ docker-compose up -d

4 发布 registry-proxy

由于需要缓存多个公共仓库，并且都需通过 443 端口发布；但是 443 端口只有一个。所以，需要根据域名来转发请求到不同的 registry-proxy 服务。我这里选择使用简单且熟悉的 nginx 实现，配置文件如下所示：

server {
    listen       80;
    listen       443 ssl;
    server_name  k8s-gcr.panbuhei.online;

    ssl_certificate /usr/local/nginx/conf/cert/k8s-gcr.panbuhei.online.pem;
    ssl_certificate_key /usr/local/nginx/conf/cert/k8s-gcr.panbuhei.online.key;
    ssl_prefer_server_ciphers on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 30;

    if ($request_method !~* GET|HEAD) {
         return 403;
    }
    location / {
        proxy_pass   http://localhost:5001;
    }
}

server {
    listen       80;
    listen       443 ssl;
    server_name  gcr.panbuhei.online;

    ssl_certificate /usr/local/nginx/conf/cert/gcr.panbuhei.online.pem;
    ssl_certificate_key /usr/local/nginx/conf/cert/gcr.panbuhei.online.key;
    ssl_prefer_server_ciphers on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 30;

    if ($request_method !~* GET|HEAD) {
         return 403;
    }
    location / {
        proxy_pass   http://localhost:5002;
    }
}

server {
    listen       80;
    listen       443 ssl;
    server_name  hub.panbuhei.online;

    ssl_certificate /usr/local/nginx/conf/cert/hub.panbuhei.online.pem;
    ssl_certificate_key /usr/local/nginx/conf/cert/hub.panbuhei.online.key;
    ssl_prefer_server_ciphers on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 30;

    if ($request_method !~* GET|HEAD) {
         return 403;
    }
    location / {
        proxy_pass   http://localhost:5003;
    }
}
server {
    listen       80;
    listen       443 ssl;
    server_name  quay.panbuhei.online;

    ssl_certificate /usr/local/nginx/conf/cert/quay.panbuhei.online.pem;
    ssl_certificate_key /usr/local/nginx/conf/cert/quay.panbuhei.online.key;
    ssl_prefer_server_ciphers on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 30;

    if ($request_method !~* GET|HEAD) {
         return 403;
    }
    location / {
        proxy_pass   http://localhost:5004;
    }
}

server {
    listen       80;
    listen       443 ssl;
    server_name  ghcr.panbuhei.online;

    ssl_certificate /usr/local/nginx/conf/cert/ghcr.panbuhei.online.pem;
    ssl_certificate_key /usr/local/nginx/conf/cert/ghcr.panbuhei.online.key;
    ssl_prefer_server_ciphers on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 30;

    if ($request_method !~* GET|HEAD) {
         return 403;
    }
    location / {
        proxy_pass   http://localhost:5005;
    }
}

5 验证

root@ubuntu20:~# docker pull k8s-gcr.panbuhei.online/kube-controller-manager:v1.23.5
Error response from daemon: Head "https://k8s-gcr.panbuhei.online/v2/kube-controller-manager/manifests/v1.23.5": no basic auth credentials

### 登陆
root@ubuntu20:~# docker login k8s-gcr.panbuhei.online
Username: panbuhei
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

### 下载
root@ubuntu20:~# docker pull k8s-gcr.panbuhei.online/kube-apiserver:v1.23.5
v1.23.5: Pulling from kube-apiserver
2df365faf0e3: Already exists 
8c99db1114c6: Already exists 
b6a9a43f03b3: Pull complete 
Digest: sha256:ddf5bf7196eb534271f9e5d403f4da19838d5610bb5ca191001bde5f32b5492e
Status: Downloaded newer image for k8s-gcr.panbuhei.online/kube-apiserver:v1.23.5
k8s-gcr.panbuhei.online/kube-apiserver:v1.23.5

6 清理 registry-proxy 缓存

由于资源有限，所以，需要定期删除缓存到本地磁盘的部分镜像。方法也比较简单，单独再部署一个 registry，共用其他 registry-proxy 的存储，并启用 delete 功能，然后再通过 API 或者 WebUI 进行删除。这里介绍两个 Docker Registry WebUI 工具：

docker-registry-web：由 JAVA 编写，镜像比较大，并且耗费内存资源。
docker-registry-ui：底层通过轻量级的 nginx 发布。

为什么 docker-registry-web 比较耗费资源，还选择使用它呢？主要原因是：我想通过 nginx 代理它，这样就可以在系统的 nginx 上对其做一些限制操作，比如配置 TLS、限制访问 IP 等。经过测试发现 docker-registry-ui 只能通过 “ip:port” 的方式访问，不能再通过系统 nginx 代理。所以，最后选择了 docker-registry-web。

5.1 docker-registry-web 的 docker-compose 代码示例：

$ mkdir -pv /opt/docker-compose/clean-registry/
$ cat << "EOF" > /opt/docker-compose/clean-registry/docker-compose.yml
version: '3.2'
services:
  registry-local:
    image: registry:latest
    container_name: registry-local
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
      - /var/lib/registry:/var/lib/registry
    ports:
      - 127.0.0.1:5000:5000
    environment:
      - REGISTRY_DELETE_ENABLED=true

  registry-web:
    image: hyper/docker-registry-web
    container_name: registry-web
    links: 
      - registry-local
    restart: always
    volumes:
      - /etc/localtime:/etc/localtime
    ports:
      - 127.0.0.1:8080:8080
    deploy:
      resources:
        limits:
           cpus: '1'
           memory: 1G
        reservations:
           memory: 512M
    environment:
      - JAVA_OPTS=-Xmx1024m -Xms512m -Xss256k
      - REGISTRY_URL=http://registry-local:5000/v2
      - REGISTRY_NAME=Panbuhei Registry-proxy
      - REGISTRY_READONLY=false
EOF

5.2 部署

$ cd /opt/docker-compose/clean-registry/

### 由于做了资源限制, 并且没有使用 swarm，所以要加上 --compatibility 参数
$ docker-compose --compatibility up -d

5.3 nginx 发布代码：

server {
    listen       80;
    server_name  clean.panbuhei.online;
    location / {
       return 301 https://$host$request_uri;
    }

}

server {
    listen       443 ssl;
    server_name  clean.panbuhei.online;

    ssl_certificate /usr/local/nginx/conf/cert/clean.panbuhei.online.pem;
    ssl_certificate_key /usr/local/nginx/conf/cert/clean.panbuhei.online.key;
    ssl_prefer_server_ciphers on;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4";

    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 30;

    if ($request_method !~* GET|HEAD) {
         return 403;
    }
    location / {
        proxy_pass   http://localhost:8080;
    }
}

5.4 访问验证
在这里插入图片描述

这里可能会有疑问：为什么，当我删除图像的所有标签时，图像仍然在 UI 中？因为这是 docker registry 的限制，垃圾收集器(garbage-collect)不会删除空 images。如果要删除空 images，则需要删除 registry 中的文件夹。(见 garbage-collect)

下是为一个删除空 images 的脚本代码：

#!/bin/sh
# remove_nullImageDir.sh
REGISTRY="127.0.0.1:5000"
REGISTRY_NAME=registry-local
repositories=$(curl -s http://${REGISTRY}/v2/_catalog | grep -o '"[^"]*"' | tr -d '"')

# docker exec $REGISTRY_NAME registry garbage-collect /etc/docker/registry/config.yml

for i in $repositories; do
  [ "$i" = "repositories" ] && continue
  curl -s http://${REGISTRY}/v2/${i}/tags/list | egrep '"tags":null|NAME_UNKNOWN';
  if [ $? -eq 0 ]; then
    docker exec -it $REGISTRY_NAME rm -rf /var/lib/registry/docker/registry/v2/repositories/$i
    echo "delete empty repository $i"
  fi
done