OpenShift 4 - 配置OpenShift可使用的外部Image Registry和Mirror Registry

《OpenShift 4.x HOL教程汇总》

概念

OpenShift可访问的Image Registy包括其自身内部的Image Registry和其外部的Image Registry，而外部的Image Registry又可分为运行在远端的Image Registry，以及为了加速远端访问而运行在本地（或近地的）Mirror Image Registry。Mirror Image Registry除了可以用来加速访问外部Image外，我们在离线安装OpenShift的时候也会用到它。
以上OpenShift用到的Image Registry配置是以对象的方式存放在Etcd中。当OpenShift的Machine Config Operator发现这些对象变化后会自动将其推送到每个Node并写入 /etc/containers/registries.conf 文件，OpenShift然后会重启对应的Node使配置生效。

配置OpenShift的Image Registry

OpenShift 4把它能访问到的外部Registry的配置信息是保存在“image.config.openshift.io/cluster”对象中。当OpenShift的Machine Config Operator（MCO）发现image.config.openshift.io配置变化后，会将最新配置对送到所有Node并写入 /etc/containers/registries.conf 文件，然后重启对应的Node使配置生效。

OpenShift使用一个名为cluster的image.config.openshift.io对象管理其能访问到的Image Registry目标和策略。

$ oc get image.config.openshift.io
NAME      AGE
cluster   12d

查看OpenShift可以访问到Registry的配置，其中“External Registry Hostnames”和“Internal Registry Hostname”定义了外部和内部Registry。

$ oc describe image.config.openshift.io cluster
Name:         cluster
Namespace:
Labels:       <none>
Annotations:  release.openshift.io/create-only: true
API Version:  config.openshift.io/v1
Kind:         Image
Metadata:
  Creation Timestamp:  2020-07-18T09:54:27Z
  Generation:          1
  Resource Version:    25273
  Self Link:           /apis/config.openshift.io/v1/images/cluster
  UID:                 c94ebed4-684a-4c98-86aa-658a8685ef79
Spec:
Status:
  External Registry Hostnames:
    default-route-openshift-image-registry.apps.cluster-beijing-959a.beijing-959a.example.opentlc.com
  Internal Registry Hostname:  image-registry.openshift-image-registry.svc:5000
Events:                        <none>

上面只是Registry的配置信息，其中OpenShift内部的Registry是通过以下的Cluster Operator运行的。

$ oc get co image-registry
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
image-registry   4.5.4     True        False         False      2d2h

如果要修改OpenShift可访问的Registry目标，可先编辑image.config.openshift.io/cluster对象：

$ oc edit image.config.openshift.io/cluster

增加“registrySources”部分内容。

spec:
  registrySources:
    insecureRegistries:
    - ocr.example.com
    allowedRegistries:
    - quay.io
    - ocr.example.com

保存后OpenShift会自动通过MCO将最新的配置同步到各个节点。可以从Node的Events查看节点的重启过程。

如果需要查看一个Node的Registry配置，可先通过debug进入该Node，然后查看/etc/containers/registries.conf（该文件是podman可以访问的registry的配置），确认最新的Registry配置已经加进去了。（如果使用非完整名称的Image，podman会去由“unqualified-search-registries”定义的Registry查找该镜像）

$ oc debug node/<NODE-NAME>
sh-4.2# chroot /host
sh-4.2# cat /etc/containers/registries.conf
unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']
[[registry]]
  prefix = ""
  location = "ocr.example.com"
  insecure = true

[[registry]]
  prefix = ""
  location = "quay.io"
  allowed = true

配置OpenShift的Mirror Image Registry

在一个受限的网络中（或访问远程Image Registry的网速比较慢），可以为OpenShift使用的外部Image Registry建一个Mirror，以加速OpenShift访问远程Registry上Image。Registry Mirror可以是运行在自己本地（或区域）的Docker Distribution、Quay等提供的Registry服务。

Image Registry Mirror的配置信息是保存在OpenShift集群范围的ImageContentSourcePolicy对象。当OpenShift的Machine Config Operator（MCO）发现ImageContentSourcePolicy配置变化后，会将最新配置对送到所有Node并写入 /etc/containers/registries.conf 文件，然后重启对应的Node使配置生效。

注意：本文只对说明如何将Mirror Image Registry的配置加到OpenShift中，而部署一套Mirror Image Registry可参见《OpenShift 4 - 部署Mirror Registry》。

1. 创建如下内容的registryrepomirror.yaml文件，其中为“registry.access.redhat.com/ubi8/ubi-minimal”镜像提供了2个Mirror，一个是“ example.io/example/ubi-minimal”，另一个是“example.com/example/ubi-minimal”，OpenShift会按照顺序访问Mirror的。

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: ubi8repo
spec:
  repositoryDigestMirrors:
  - mirrors:
    - example.io/example/ubi-minimal
    source: registry.access.redhat.com/ubi8/ubi-minimal
  - mirrors:
    - example.com/example/ubi-minimal
    source: registry.access.redhat.com/ubi8/ubi-minimal

2. 创建ImageContentSourcePolicy对象。

$ oc create -f registryrepomirror.yaml

3. 查看Node的状态，确认Node会重启以使得配置生效。

$ oc get node
NAME                                              STATUS                       ROLES    AGE  VERSION
ip-10-0-131-190.ap-southeast-1.compute.internal   Ready                        master   13d   v1.18.3+012b3ec
ip-10-0-141-25.ap-southeast-1.compute.internal    Readyy,SchedulingDisabled    worker   13d   v1.18.3+012b3ec
ip-10-0-165-133.ap-southeast-1.compute.internal   Ready                        master   13d   v1.18.3+012b3ec
ip-10-0-174-85.ap-southeast-1.compute.internal    Ready                        worker   13d   v1.18.3+012b3ec
ip-10-0-216-155.ap-southeast-1.compute.internal   Ready                        master   13d   v1.18.3+012b3ec

4. 确认配置已经更新，有2个registry.mirror的配置。

$ oc debug node <NODE-NAME>
sh-4.2# chroot /host
sh-4.4# cat /etc/containers/registries.conf
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/ubi8/ubi-minimal"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "example.com/example/ubi-minimal"

  [[registry.mirror]]
    location = "example.io/example/ubi-minimal"

[[registry]]
  prefix = ""
  location = "ocr.example.com"
  insecure = true

[[registry]]
  prefix = ""
  location = "untrusted.com"
  blocked = true

5. 在Node内部发起对远程镜像“registry.access.redhat.com/ubi8/ubi-minimal”的访问，通过日志确认系统是从Mirror中获得该Image的。

sh-4.2# podman pull --log-level=debug registry.access.redhat.com/ubi8/ubi-minimal

参考

1. https://computingforgeeks.com/how-to-setup-red-hat-quay-registry-on-centos-rhel/
2. https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.7/html-single/images/index#installation-restricted-network-samples_samples-operator-alt-registry
3. https://access.redhat.com/documentation/zh-cn/openshift_container_platform/4.7/html-single/images/index#images-configuration-parameters_image-configuration
4. https://computingforgeeks.com/allow-insecure-registries-in-openshift-okd-4-cluster/
5. https://rhthsa.github.io/openshift-demo/build-with-oc.html
6. https://docs.openshift.com/container-platform/4.7/post_installation_configuration/preparing-for-users.html#images-configuration-shortname_post-install-preparing-for-users
7. https://www.redhat.com/sysadmin/manage-container-registries