OpenShift 4 - 如何用Machine Config Operator修改集群节点CoreOS的配置

8 篇文章 0 订阅
2 篇文章 0 订阅

OpenShift 4.x HOL教程汇总
说明:本文已经在OpenShift 4.6环境中验证

OpenShift 4 和 RHCOS

不像OpenShift 3和底层RHEL的那种松散关系(安装和维护都是相对独立的),OpenShift 4 和RHCOS(Red Hat CoreOS)操作系统是紧密依赖关系,它们是一起安装、一起运行。
为了获得Immutable Architecture(不可变架构)的优势,RedHat官方不建议直接进入RHCOS操作系统进行操作系统配置修改,而建议通过OpenShift 4的Operator机制修改和操作RHCOS的配置。无论是运行在节点操作系统上的kubelet以及CRI-O用到的Registry配置、还是Kernel、NetworkManager或系统时间同步,甚至连RHCOS升级都是通过Operator完成的。因此,从某个角度上可以把看作是OpenShift 4用Operator控制的一种Kubernetes资源。

MachineConfig相关概念

MachineConfigOperator 及相关对象

OpenShift 4 是使用系统级的Machine Config Operator(MCO)相关对象实现集群对节点的RHCOS管理。MCO及其相关组件都运行在openshift-machine-config-operator项目中,可以执行以下命令查看该项目下的主要对象。

$ oc get all -n openshift-machine-config-operator
NAME                                             READY   STATUS    RESTARTS   AGE
pod/etcd-quorum-guard-7666889cd6-2hch9           1/1     Running   0          20h
pod/etcd-quorum-guard-7666889cd6-c6m8k           1/1     Running   0          20h
pod/etcd-quorum-guard-7666889cd6-ht7bb           1/1     Running   0          20h
pod/machine-config-controller-7574fd5777-zssfz   1/1     Running   0          20h
pod/machine-config-daemon-2bfzg                  2/2     Running   0          21h
pod/machine-config-daemon-2g5kw                  2/2     Running   0          21h
pod/machine-config-daemon-dw2r9                  2/2     Running   0          21h
pod/machine-config-daemon-gg5vd                  2/2     Running   0          21h
pod/machine-config-daemon-w6cg2                  2/2     Running   0          21h
pod/machine-config-operator-6477b6c7c8-vcxrt     1/1     Running   0          20h
pod/machine-config-server-5bxmf                  1/1     Running   0          20h
pod/machine-config-server-8h78p                  1/1     Running   0          20h
pod/machine-config-server-lqxql                  1/1     Running   0          20h
 
NAME                            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/machine-config-daemon   ClusterIP   172.30.9.67   <none>        9001/TCP   9d
 
NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE
daemonset.apps/machine-config-daemon   5         5         5       5            5           kubernetes.io/os=linux            9d
daemonset.apps/machine-config-server   3         3         3       3            3           node-role.kubernetes.io/master=   9d
 
NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/etcd-quorum-guard           3/3     3            3           9d
deployment.apps/machine-config-controller   1/1     1            1           9d
deployment.apps/machine-config-operator     1/1     1            1           9d
 
NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/etcd-quorum-guard-5fc4989788           0         0         0       9d
replicaset.apps/etcd-quorum-guard-7666889cd6           3         3         3       20h
replicaset.apps/machine-config-controller-67b8b87bb8   0         0         0       9d
replicaset.apps/machine-config-controller-7574fd5777   1         1         1       21h
replicaset.apps/machine-config-operator-5b87d8f4c5     0         0         0       9d
replicaset.apps/machine-config-operator-6477b6c7c8     1         1         1       21h

在MCO中包括以下三个子组件,MCS、MCC和MCD:
在这里插入图片描述

  1. MCS(MachineConfigServer):为新加入集群的新Machine提供Ignition配置。在openshift-machine-config-operator项目中它是以DaemonSet的形式运行在。可以看到MCS会在每个Master节点上都运行一份,新加入节点是通过haproxy访问MCS的。
    在这里插入图片描述
  2. MCC(MachineConfigController):当使用MachineConfig定义了配置,MCC负责协调所有的Machines进行升级。它在openshift-machine-config-operator项目是以Deployment的方式存在的。
    1)Template Controller:负责从内部模板中为machine的预定义角色(缺省是master, worker)生成MachineConfigs。
    2)Render Controller:负责发现MachineConfigPoold的MachineConfigs,并生成静态MachineConfig定义文件(例如rendered-master-XXXX和rendered-worker-XXXX)。
    3)Update Controller:负责与每台machine上运行的MCD程协调,将machine升级到MachineConfig定义的。
    4)KubeletConfigController :负责将获得的KubeletConfig CRD配置对象实施到节点的kubelet运行环境中。
  3. MCD(MachineConfigDaemon):负责在升级过程中对Machine实施新的MachineConfig。它在openshift-machine-config-operator项目中MCD是以DaemonSet运行在集群中的所有节点上。另外MCD还负责通过OSTree对RHCOS进行滚动升级。

Machine Config Operator是通过基于CRD的对象来定义以上这些和machineconfig相关的对象的。我们可以查看在OpenShift内部定义的与machineconfig相关的CRD对象:

$ oc get crd -n openshift-config | grep machineconfig
containerruntimeconfigs.machineconfiguration.openshift.io   2020-09-09T10:17:19Z
controllerconfigs.machineconfiguration.openshift.io         2020-09-09T10:24:12Z
kubeletconfigs.machineconfiguration.openshift.io            2020-09-09T10:17:20Z
machineconfigpools.machineconfiguration.openshift.io        2020-09-09T10:17:22Z
machineconfigs.machineconfiguration.openshift.io            2020-09-09T10:17:21Z

最后可通过查看openshift-machine-config-operator项目中的对象了解Machine Config Operator维护的MachineConfigServer、MachineConfigController和MachineConfigDaemon服务,可以看到MachineConfigController和MachineConfigDaemon都是以daemonset运行的。

 oc get all -o wide -n openshift-machine-config-operator
NAME                                             READY   STATUS    RESTARTS   AGE     IP                NODE                          NOMINATED NODE   READINESS GATES
pod/machine-config-controller-7d9bcdf859-stbmm   1/1     Running   0          3h19m   10.128.0.51       master-0.ocp4-1.example.com   <none>           <none>
pod/machine-config-daemon-blqc7                  2/2     Running   0          5d17h   192.168.203.181   master-0.ocp4-1.example.com   <none>           <none>
pod/machine-config-daemon-dcdr5                  2/2     Running   0          5d16h   192.168.203.184   worker-0.ocp4-1.example.com   <none>           <none>
pod/machine-config-daemon-n9vcz                  2/2     Running   0          5d17h   192.168.203.182   master-1.ocp4-1.example.com   <none>           <none>
pod/machine-config-daemon-pq4nz                  2/2     Running   0          5d16h   192.168.203.185   worker-1.ocp4-1.example.com   <none>           <none>
pod/machine-config-daemon-pxs9d                  2/2     Running   0          5d17h   192.168.203.183   master-2.ocp4-1.example.com   <none>           <none>
pod/machine-config-operator-699d8cf454-llrdl     1/1     Running   0          3h19m   10.128.0.65       master-0.ocp4-1.example.com   <none>           <none>
pod/machine-config-server-4x82g                  1/1     Running   0          5d17h   192.168.203.182   master-1.ocp4-1.example.com   <none>           <none>
pod/machine-config-server-dns26                  1/1     Running   0          5d17h   192.168.203.183   master-2.ocp4-1.example.com   <none>           <none>
pod/machine-config-server-rlmcv                  1/1     Running   0          5d17h   192.168.203.181   master-0.ocp4-1.example.com   <none>           <none>

NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE     SELECTOR
service/machine-config-daemon   ClusterIP   172.30.21.191   <none>        9001/TCP   5d17h   k8s-app=machine-config-daemon

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                     AGE     CONTAINERS                          IMAGES                                                                                                                                                                                                                                          SELECTOR
daemonset.apps/machine-config-daemon   5         5         5       5            5           kubernetes.io/os=linux            5d17h   machine-config-daemon,oauth-proxy   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c2db2a95359d4a69fe2c7ccc05dfd9fac0df30f588cc34ebf5e6aefb393828f,quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84264dd5793949e5fb6429952cdcbec78091576651dd07f0b943f22194b27b51   k8s-app=machine-config-daemon
daemonset.apps/machine-config-server   3         3         3       3            3           node-role.kubernetes.io/master=   5d17h   machine-config-server               quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c2db2a95359d4a69fe2c7ccc05dfd9fac0df30f588cc34ebf5e6aefb393828f                                                                                                                          k8s-app=machine-config-server

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS                  IMAGES                                                                                                                   SELECTOR
deployment.apps/machine-config-controller   1/1     1            1           5d17h   machine-config-controller   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c2db2a95359d4a69fe2c7ccc05dfd9fac0df30f588cc34ebf5e6aefb393828f   k8s-app=machine-config-controller
deployment.apps/machine-config-operator     1/1     1            1           5d17h   machine-config-operator     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c2db2a95359d4a69fe2c7ccc05dfd9fac0df30f588cc34ebf5e6aefb393828f   k8s-app=machine-config-operator

NAME                                                   DESIRED   CURRENT   READY   AGE     CONTAINERS                  IMAGES                                                                                                                   SELECTOR
replicaset.apps/machine-config-controller-7d9bcdf859   1         1         1       5d17h   machine-config-controller   quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c2db2a95359d4a69fe2c7ccc05dfd9fac0df30f588cc34ebf5e6aefb393828f   k8s-app=machine-config-controller,pod-template-hash=7d9bcdf859
replicaset.apps/machine-config-operator-699d8cf454     1         1         1       5d17h   machine-config-operator     quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7c2db2a95359d4a69fe2c7ccc05dfd9fac0df30f588cc34ebf5e6aefb393828f   k8s-app=machine-config-operator,pod-template-hash=699d8cf454

MachineConfigPool

在OpenShift 4中,Node和MachineConfig是多对多关系,即一个Node可以使用多个MachineConfig,同时一个MachineConfig可以属于多个Node。一个Node使用哪个MachineConfig配置是MachineConfigPool决定的,即MachineConfigPool将MachineConfig和Node关联起来。
在这里插入图片描述

  1. 查看machineconfig
$ oc get machineconfig
NAME                                                        GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                                   99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
00-worker                                                   99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
01-master-container-runtime                                 99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
01-master-kubelet                                           99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
01-worker-container-runtime                                 99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
01-worker-kubelet                                           99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
99-master-58a1f119-d339-4e40-b218-ca459783104e-registries   99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
99-master-ssh                                                                                          3.1.0             3d21h
99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries   99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
99-worker-ssh                                                                                          3.1.0             3d21h
rendered-master-68ea973a611b4e8a11a6492328c661bd            99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f            99eb744f5094224edb60d88ca85d607ab151ebdf   3.1.0             3d21h
  1. 查看node
$ oc get node
NAME                                              STATUS   ROLES    AGE     VERSION
ip-10-0-139-50.ap-southeast-1.compute.internal    Ready    master   3d21h   v1.19.0+d59ce34
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready    worker   3d21h   v1.19.0+d59ce34
ip-10-0-172-203.ap-southeast-1.compute.internal   Ready    master   3d21h   v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready    worker   3d21h   v1.19.0+d59ce34
ip-10-0-205-123.ap-southeast-1.compute.internal   Ready    master   3d21h   v1.19.0+d59ce34
  1. 查看machineconfigpool。
$ oc get machineconfigpool
NAME     CONFIG                                              UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-68ea973a611b4e8a11a6492328c661bd   True      False      False      3              3                   3                     0                      3d21h
worker   rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f   True      False      False      2              2                   2                     0                      3d21h
$ oc describe MachineConfigPool worker
...
Spec:
  Configuration:
    Name:  rendered-worker-a733a9c295cb7b15cf7af2e7251e9a3f
    Source:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         00-worker
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-container-runtime
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-kubelet
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-ssh
  Machine Config Selector:
    Match Labels:
      machineconfiguration.openshift.io/role:  worker
  Node Selector:
    Match Labels:
      node-role.kubernetes.io/worker:
...
  1. 对于名为worker的MachineConfigPool,查看被RenderController加工后所包含到的相关配置文件。
$ RENDERED_WORKER_MACHINE_CONFIG=$(oc get machineconfigpool worker --template={{.spec.configuration.name}})
$ oc describe machineconfigs ${RENDERED_WORKER_MACHINE_CONFIG} | grep Path:
        Path:        /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
        Path:        /etc/tmpfiles.d/cleanup-cni.conf
        Path:        /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
        Path:        /etc/containers/storage.conf
        Path:        /etc/NetworkManager/dispatcher.d/90-long-hostname
        Path:        /etc/modules-load.d/iptables.conf
        Path:        /etc/kubernetes/kubelet-ca.crt
        Path:        /etc/systemd/system.conf.d/kubelet-cgroups.conf
        Path:        /etc/NetworkManager/conf.d/sdn.conf
        Path:        /var/lib/kubelet/config.json
        Path:        /etc/kubernetes/ca.crt
        Path:        /etc/sysctl.d/forward.conf
        Path:        /etc/sysctl.d/inotify.conf
        Path:        /usr/local/sbin/set-valid-hostname.sh
        Path:        /etc/kubernetes/kubelet-plugins/volume/exec/.dummy
        Path:        /etc/containers/registries.conf
        Path:        /etc/crio/crio.conf.d/00-default
        Path:        /etc/containers/policy.json
        Path:        /etc/kubernetes/cloud.conf
        Path:        /etc/kubernetes/kubelet.conf

通过MachineConfig修改Worker节点CRI-O使用的registries.conf配置

  1. 在Terminal -1执行命令进入一个Worker Node,然后查看“/etc/containers/registries.conf”文件内容,确认文件里的内容。
$ oc debug node/<WORKER_NODE_NAME>
sh-4.4# chroot /host
sh-4.4# cat /etc/containers/registries.conf
unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']
  1. 在Terminal-2创建内容如下的“50-worker-container-registries.yaml文件”。
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: 50-worker-container-registries
spec:
  config:
    ignition:
      version: 3.2.0
    storage:
      files:
      - contents:
          source: data:,%5Bregistries.search%5D%0Aregistries%20%3D%20%5B'registry.access.redhat.com'%2C%20'docker.io'%2C%20'quay.io'%5D%0A%0A%5Bregistries.insecure%5D%0Aregistries%20%3D%20%5B%5D%0A%0A%5Bregistries.block%5D%0Aregistries%20%3D%20%5B%5D%0A
        filesystem: root
        mode: 420
        path: /etc/containers/registries.conf
  1. 在Terminal-2中执行命令,创建MachineConfig。
$ oc create -f 50-worker-container-registries.yaml
  1. 在Terminal-2中查看worker节点的状态,通过状态(SchedulingDisabled)确认2个worker节点逐一进行了配置更新。
$ oc get nodes -l node-role.kubernetes.io/worker -w
NAME                                              STATUS                     ROLES    AGE   VERSION
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.19.0+d59ce34
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-155-228.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready,SchedulingDisabled   worker   8d    v1.19.0+d59ce34
ip-10-0-190-239.ap-southeast-1.compute.internal   Ready                      worker   8d    v1.19.0+d59ce34
  1. 在Terminal-1中查看Worker节点的“/etc/containers/registries.conf”f配置已经被修改。
sh-4.4#  cat /etc/containers/registries.conf
[registries.search]
registries = ['registry.access.redhat.com', 'docker.io', 'quay.io']
 
[registries.insecure]
registries = []
 
[registries.block]
registries = []
sh-4.4# exit
exit
sh-4.4# exit
exit
  1. 再次查看machineconfigpool,确认worker使用的rendered后的配置已经变了。
$ oc get machineconfigpool
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-68ea973a611b4e8a11a6492328c661bd   True      False      False      3              3                   3                     0                      9d
worker   rendered-worker-90e84f625715399f91f5ffff66019273   True      False      False      2              2                   2                     0                      9d
  1. 再次查看名为worker的machineconfigpool的配置信息,其中包括前面新加的名为“50-worker-container-registries”的MachineConfig。
$ oc describe MachineConfigPool worker
。。。
Spec:
  Configuration:
    Name:  rendered-worker-90e84f625715399f91f5ffff66019273
    Source:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         00-worker
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-container-runtime
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-worker-kubelet
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         50-worker-container-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-d20cffe7-8ad8-42fe-9a9d-9dc35f256f45-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-worker-ssh
  Machine Config Selector:
    Match Labels:
      machineconfiguration.openshift.io/role:  worker
  Node Selector:
    Match Labels:
      node-role.kubernetes.io/worker:
...
  1. 再次查看名为worker的machineconfigs中对应的配置文件,确认最后一行已经有对“registries.conf”文件的配置了。
$ oc describe machineconfigs ${RENDERED_WORKER_MACHINE_CONFIG} | grep Path:
        Path:        /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
        Path:        /etc/tmpfiles.d/cleanup-cni.conf
        Path:        /etc/kubernetes/static-pod-resources/configmaps/cloud-config/ca-bundle.pem
        Path:        /etc/containers/storage.conf
        Path:        /etc/NetworkManager/dispatcher.d/90-long-hostname
        Path:        /etc/modules-load.d/iptables.conf
        Path:        /etc/kubernetes/kubelet-ca.crt
        Path:        /etc/systemd/system.conf.d/kubelet-cgroups.conf
        Path:        /etc/NetworkManager/conf.d/sdn.conf
        Path:        /var/lib/kubelet/config.json
        Path:        /etc/kubernetes/ca.crt
        Path:        /etc/sysctl.d/forward.conf
        Path:        /etc/sysctl.d/inotify.conf
        Path:        /usr/local/sbin/set-valid-hostname.sh
        Path:        /etc/kubernetes/kubelet-plugins/volume/exec/.dummy
        Path:        /etc/containers/registries.conf
        Path:        /etc/crio/crio.conf.d/00-default
        Path:        /etc/containers/policy.json
        Path:        /etc/kubernetes/cloud.conf
        Path:        /etc/kubernetes/kubelet.conf
        Path:        /etc/containers/registries.conf

参考

https://github.com/openshift/machine-config-operator
https://github.com/coreos/ignition/blob/master/doc/configuration-v3_0.md
https://www.redhat.com/en/blog/openshift-container-platform-4-how-does-machine-config-pool-work
http://jstakun.blogspot.com/2019/06/managing-cluster-nodes-configuration-in.html

  • 4
    点赞
  • 3
    收藏
    觉得还不错? 一键收藏
  • 1
    评论

“相关推荐”对你有帮助么?

  • 非常没帮助
  • 没帮助
  • 一般
  • 有帮助
  • 非常有帮助
提交
评论 1
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值