8.28笔记

一、主控

1.安装etcd数据库

 
  1. [root@node1 ~]# yum -y install etcd

  2. 已加载插件:fastestmirror

  3. Determining fastest mirrors

  4. epel/x86_64/metalink

  5. | 5.1 kB

  6. 00:00:00

  7. * base: mirrors.aliyun.com

  8. * epel: repo.jing.rocks

  9. * extras: mirrors.aliyun.com

  10. * updates: mirrors.aliyun.com

  11. base

  12. | 3.6 kB

  13. 00:00:00

  14. extras

  15. | 2.9 kB

  16. 00:00:00

  17. updates

  18. | 2.9 kB

  19. 00:00:00

  20. 没有可⽤软件包 fannel。

  21. 正在解决依赖关系

  22. --> 正在检查事务

  23. ---> 软件包 etcd.x86_64.0.3.3.11-2.el7.centos 将

  24. 被 安装

  25. --> 解决依赖关系完成

  26. 依赖关系解决

  27. ==================================================

  28. ==================================================

  29. ============

  30. Package 架构 版本

  31. ⼤⼩

  32. ==================================================

  33. ==================================================

  34. ============

  35. 正在安装:

  36. etcd x86_64

  37. 3.3.11-2.el7.centos extras

  38. 10 M

  39. 事务概要

  40. ==================================================

  41. ==================================================

  42. ============

  43. 安装 1 软件包

  44. 总下载量:10 M

  45. 安装⼤⼩:45 M

  46. Downloading packages:

  47. etcd-3.3.11-2.el7.centos.x86_64.rpm

  48. | 10 MB

  49. 00:00:12

  50. Running transaction check

  51. Running transaction test

  52. Transaction test succeeded

  53. Running transaction

  54. 正在安装 : etcd-3.3.11-2.el7.centos.x86_64

  55. 1/1

  56. 验证中 : etcd-3.3.11-2.el7.centos.x86_64

  57. 1/1

  58. 已安装:

  59. etcd.x86_64 0:3.3.11-2.el7.centos

  60. 完毕!

2.安装flanne

 
  1. [root@node1 ~]# yum -y install flannel

  2. 已加载插件:fastestmirror

  3. Loading mirror speeds from cached hostfile

  4. * base: mirrors.aliyun.com

  5. * epel: repo.jing.rocks

  6. * extras: mirrors.aliyun.com

  7. * updates: mirrors.aliyun.com

  8. 正在解决依赖关系

  9. --> 正在检查事务

  10. ---> 软件包 flannel.x86_64.0.0.7.1-4.el7 将被 安

  11. --> 解决依赖关系完成

  12. 依赖关系解决

  13. ==================================================

  14. ==================================================

  15. ============

  16. Package 架构

  17. 版本 源

  18. ⼤⼩

  19. ==================================================

  20. ==================================================

  21. ============

  22. 正在安装:

  23. flannel x86_64

  24. 0.7.1-4.el7 extras

  25. 7.5 M

  26. 事务概要

  27. ==================================================

  28. ==================================================

  29. ============

  30. 安装 1 软件包

  31. 总下载量:7.5 M

  32. 安装⼤⼩:41 M

  33. Downloading packages:

  34. flannel-0.7.1-4.el7.x86_64.rpm

  35. | 7.5 MB

  36. 00:00:45

  37. Running transaction check

  38. Running transaction test

  39. Transaction test succeeded

  40. Running transaction

  41. 正在安装 : flannel-0.7.1-4.el7.x86_64

  42. 1/1

  43. 验证中 : flannel-0.7.1-4.el7.x86_64

  44. 1/1

  45. 已安装:

  46. flannel.x86_64 0:0.7.1-4.el7

  47. 完毕!

3.修改etcd数据库配置

 
  1. [root@node1 ~]# vim /etc/etcd/etcd.conf

  2. [root@node1 ~]# cat /etc/etcd/etcd.conf

  3. #[Member]

  4. #ETCD_CORS=""

  5. ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

  6. #ETCD_WAL_DIR=""

  7. #ETCD_LISTEN_PEER_URLS="http://localhost:2380"

  8. #

  9. #-------------------------------------------------

  10. -------------------

  11. #ETCD_LISTEN_CLIENT_URLS="http://localhost:2379"

  12. ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379,http:

  13. //0.0.0.0:4001"

  14. #-------------------------------------------------

  15. ------------------

  16. #ETCD_MAX_SNAPSHOTS="5"

  17. #ETCD_MAX_WALS="5"

  18. ETCD_NAME="default"

  19. #ETCD_SNAPSHOT_COUNT="100000"

  20. #ETCD_HEARTBEAT_INTERVAL="100"

  21. #ETCD_ELECTION_TIMEOUT="1000"

  22. #ETCD_QUOTA_BACKEND_BYTES="0"

  23. #ETCD_MAX_REQUEST_BYTES="1572864"

  24. #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"

  25. #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"

  26. #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"

  27. #

  28. #[Clustering]

  29. #ETCD_INITIAL_ADVERTISE_PEER_URLS="http://localhos

  30. t:2380"

  31. #

  32. #

  33. #-------------------------------------------------

  34. -----

  35. #ETCD_ADVERTISE_CLIENT_URLS="http://localhost:2379

  36. "

  37. ETCD_ADVERTISE_CLIENT_URLS="http://10.1.1.10:2379,

  38. http://10.1.1.10:4001"

  39. #-------------------------------------------------

  40. -----

  41. #ETCD_DISCOVERY=""

  42. #ETCD_DISCOVERY_FALLBACK="proxy"

  43. #ETCD_DISCOVERY_PROXY=""

  44. #ETCD_DISCOVERY_SRV=""

  45. #ETCD_INITIAL_CLUSTER="default=http://localhost:23

  46. 80"

  47. #ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  48. #ETCD_INITIAL_CLUSTER_STATE="new"

  49. #ETCD_STRICT_RECONFIG_CHECK="true"

  50. #ETCD_ENABLE_V2="true"

  51. #

  52. #[Proxy]

  53. #ETCD_PROXY="off"

  54. #ETCD_PROXY_FAILURE_WAIT="5000"

  55. #ETCD_PROXY_REFRESH_INTERVAL="30000"

  56. #ETCD_PROXY_DIAL_TIMEOUT="1000"

  57. #ETCD_PROXY_WRITE_TIMEOUT="5000"

  58. #ETCD_PROXY_READ_TIMEOUT="0"

  59. #

  60. #[Security]

  61. #ETCD_CERT_FILE=""

  62. #ETCD_KEY_FILE=""

  63. #ETCD_CLIENT_CERT_AUTH="false"

  64. #ETCD_TRUSTED_CA_FILE=""

  65. #ETCD_AUTO_TLS="false"

  66. #ETCD_PEER_CERT_FILE=""

  67. #ETCD_PEER_KEY_FILE=""

  68. #ETCD_PEER_CLIENT_CERT_AUTH="false"

  69. #ETCD_PEER_TRUSTED_CA_FILE=""

  70. #ETCD_PEER_AUTO_TLS="false"

  71. #

  72. #[Logging]

  73. #ETCD_DEBUG="false"

  74. #ETCD_LOG_PACKAGE_LEVELS=""

  75. #ETCD_LOG_OUTPUT="default"

  76. #

  77. #[Unsafe]

  78. #ETCD_FORCE_NEW_CLUSTER="false"

  79. #

  80. #[Version]

  81. #ETCD_VERSION="false"

  82. #ETCD_AUTO_COMPACTION_RETENTION="0"

  83. #

  84. #[Profiling]

  85. #ETCD_ENABLE_PPROF="false"

  86. #ETCD_METRICS="basic"

  87. #

  88. #[Auth]

  89. #ETCD_AUTH_TOKEN="simple"

4.启动数据库

[root@node1 ~]# systemctl start etcd.service

5.设置开机启动

 
  1. [root@node1 ~]# systemctl enable etcd.service

  2. Created symlink from /etc/systemd/system/multiuser.target.wants/etcd.service to

  3. /usr/lib/systemd/system/etcd.service.

  4. [root@node1 ~]# ss -anput|grep 2370

  5. tcp ESTAB 0 0 127.0.0.1:34666

  6. 127.0.0.1:2370 users:

  7. (("etcd",pid=1959,fd=11))

  8. tcp LISTEN 0 128 [::]:2370

  9. [::]:* users:

  10. (("etcd",pid=1959,fd=6))

  11. tcp ESTAB 0 0

  12. [::ffff:127.0.0.1]:2370

  13. [::ffff:127.0.0.1]:34666 users:

  14. (("etcd",pid=1959,fd=15))

6.测试端⼝

 
  1. [root@node1 ~]# netstat -lnput|grep 4001

  2. tcp6 0 0 :::4001 :::*

  3. LISTEN 1959/etcd

  4. [root@node1 ~]# systemctl restart etcd.service

  5. [root@node1 ~]# netstat -lnput|grep 2379

  6. tcp6 0 0 :::2379 :::*

  7. LISTEN 2028/etcd

7.测试数据库功能

 
  1. [root@node1 ~]# etcdctl set testdir/testkey0 1000

  2. 1000

  3. [root@node1 ~]# etcdctl get testdir/testkey0

  4. 1000

8.测试集群健康

 
  1. [root@node1 ~]# etcdctl -C http://10.1.1.10:4001

  2. cluster-health

  3. member 8e9e05c52164694d is healthy: got healthy

  4. result from http://10.1.1.10:2379

  5. cluster is healthy

  6. [root@node1 ~]# etcdctl -C http://10.1.1.10:2379

  7. cluster-health

  8. member 8e9e05c52164694d is healthy: got healthy

  9. result from http://10.1.1.10:2379

  10. cluster is healthy

9.修改flannel配置⽂件

 
  1. [root@node1 ~]# vim /etc/sysconfig/flanneld

  2. [root@node1 ~]# cat /etc/sysconfig/flanneld

  3. # Flanneld configuration options

  4. # etcd url location. Point this to the server

  5. where etcd runs

  6. # ------------------------------------------------

  7. -------

  8. FLANNEL_ETCD_ENDPOINTS="http://10.1.1.10:2379"

  9. #-------------------------------------------------

  10. -------

  11. # etcd config key. This is the configuration key

  12. that flannel queries

  13. # For address range assignment

  14. FLANNEL_ETCD_PREFIX="/atomic.io/network"

  15. # Any additional options that you want to pass

  16. #FLANNEL_OPTIONS=""

10.向数据库中存⼊⽹段信息

 
  1. [root@node1 ~]# etcdctl mk

  2. /atomic.io/network/config '{ "Network" :

  3. "172.20.0.0/16" }'

  4. { "Network" : "172.20.0.0/16" }

  5. [root@node1 ~]# etcdctl get

  6. /atomic.io/network/config

  7. { "Network" : "172.20.0.0/16" }

11.启动并设置开机启动flanneld

 
  1. [root@node1 ~]# systemctl start flanneld.service

  2. [root@node1 ~]# systemctl enable flanneld.service

  3. Created symlink from /etc/systemd/system/multiuser.target.wants/flanneld.service to

  4. /usr/lib/systemd/system/flanneld.service.

  5. Created symlink from

  6. /etc/systemd/system/docker.service.wants/flanneld.

  7. service to

  8. /usr/lib/systemd/system/flanneld.service.

12.查看ip地址

 
  1. [root@node1 ~]# ip a s

  2. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc

  3. noqueue state UNKNOWN group default qlen 1000

  4. link/loopback 00:00:00:00:00:00 brd

  5. 00:00:00:00:00:00

  6. inet 127.0.0.1/8 scope host lo

  7. valid_lft forever preferred_lft forever

  8. inet6 ::1/128 scope host

  9. valid_lft forever preferred_lft forever

  10. 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu

  11. 1500 qdisc pfifo_fast state UP group default qlen

  12. 1000

  13. link/ether 00:0c:29:94:72:1b brd

  14. ff:ff:ff:ff:ff:ff

  15. inet 10.1.1.10/24 brd 10.1.1.255 scope global

  16. ens33

  17. valid_lft forever preferred_lft forever

  18. inet6 fe80::20c:29ff:fe94:721b/64 scope link

  19. valid_lft forever preferred_lft forever

  20. 3: flannel0:

  21. <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472

  22. qdisc pfifo_fast state UNKNOWN group default qlen

  23. 500

  24. link/none

  25. inet 172.20.20.0/16 scope global flannel0

  26. valid_lft forever preferred_lft forever

  27. inet6 fe80::1fae:d70e:eb38:5ec9/64 scope link

  28. flags 800

  29. valid_lft forever preferred_lft forever

13.安装docker

 
  1. [root@node1 ~]# source docker.sh

  2. overlay

  3. br_netfilter

  4. net.bridge.bridge-nf-call-iptables = 1

  5. net.bridge.bridge-nf-call-ip6tables = 1

  6. net.ipv4.ip_forward = 1

  7. * Applying /usr/lib/sysctl.d/00-system.conf ...

  8. net.bridge.bridge-nf-call-ip6tables = 0

  9. net.bridge.bridge-nf-call-iptables = 0

  10. net.bridge.bridge-nf-call-arptables = 0

  11. * Applying /usr/lib/sysctl.d/10-default-yamascope.conf ...

  12. kernel.yama.ptrace_scope = 0

  13. * Applying /usr/lib/sysctl.d/50-default.conf ...

  14. kernel.sysrq = 16

  15. kernel.core_uses_pid = 1

  16. kernel.kptr_restrict = 1

  17. net.ipv4.conf.default.rp_filter = 1

  18. net.ipv4.conf.all.rp_filter = 1

  19. net.ipv4.conf.default.accept_source_route = 0

  20. net.ipv4.conf.all.accept_source_route = 0

  21. net.ipv4.conf.default.promote_secondaries = 1

  22. net.ipv4.conf.all.promote_secondaries = 1

  23. fs.protected_hardlinks = 1

  24. fs.protected_symlinks = 1

  25. * Applying /etc/sysctl.d/99-sysctl.conf ...

  26. * Applying /etc/sysctl.d/k8s.conf ...

  27. net.bridge.bridge-nf-call-iptables = 1

  28. net.bridge.bridge-nf-call-ip6tables = 1

  29. net.ipv4.ip_forward = 1

  30. * Applying /etc/sysctl.conf ...

  31. 已加载插件:fastestmirror

  32. Loading mirror speeds from cached hostfile

  33. * base: mirrors.aliyun.com

  34. * epel: repo.jing.rocks

  35. * extras: mirrors.aliyun.com

  36. * updates: mirrors.aliyun.com

  37. 软件包 device-mapper-persistent-data-0.8.5-

  38. 3.el7_9.2.x86_64 已安装并且是最新版本

  39. 软件包 7:lvm2-2.02.187-6.el7_9.5.x86_64 已安装并

  40. 且是最新版本

  41. 正在解决依赖关系

  42. --> 正在检查事务

  43. ---> 软件包 yum-utils.noarch.0.1.1.31-54.el7_8

  44. 将被 安装

  45. --> 正在处理依赖关系 python-kitchen,它被软件包

  46. yum-utils-1.1.31-54.el7_8.noarch 需要

  47. --> 正在处理依赖关系 libxml2-python,它被软件包

  48. yum-utils-1.1.31-54.el7_8.noarch 需要

  49. --> 正在检查事务

  50. ---> 软件包 libxml2-python.x86_64.0.2.9.1-

  51. 6.el7_9.6 将被 安装

  52. ---> 软件包 python-kitchen.noarch.0.1.1.1-5.el7

  53. 将被 安装

  54. --> 正在处理依赖关系 python-chardet,它被软件包

  55. python-kitchen-1.1.1-5.el7.noarch 需要

  56. --> 正在检查事务

  57. ---> 软件包 python-chardet.noarch.0.2.2.1-3.el7

  58. 将被 安装

  59. --> 解决依赖关系完成

  60. 依赖关系解决

  61. ==================================================

  62. ==================================================

  63. ============

  64. Package 架构

  65. 版本 源

  66. ⼤⼩

  67. ==================================================

  68. ==================================================

  69. ============

  70. 正在安装:

  71. yum-utils noarch

  72. 1.1.31-54.el7_8 base

  73. 122 k

  74. 为依赖⽽安装:

  75. libxml2-python x86_64

  76. 2.9.1-6.el7_9.6 updates

  77. 247 k

  78. python-chardet noarch

  79. 2.2.1-3.el7 base

  80. 227 k

  81. python-kitchen noarch

  82. 1.1.1-5.el7 base

  83. 267 k

  84. 事务概要

  85. ==================================================

  86. ==================================================

  87. ============

  88. 安装 1 软件包 (+3 依赖软件包)

  89. 总下载量:863 k

  90. 安装⼤⼩:4.3 M

  91. Downloading packages:

  92. (1/4): yum-utils-1.1.31-54.el7_8.noarch.rpm

  93. | 122 kB

  94. 00:00:00

  95. libxml2-python-2.9.1-6.el7_9.6 FAILED

  96. http://mirrors.cloud.aliyuncs.com/centos/7/updates

  97. /x86_64/Packages/libxml2-python-2.9.1-

  98. 6.el7_9.6.x86_64.rpm: [Errno 14] curl#6 - "Could

  99. not resolve host: mirrors.cloud.aliyuncs.com;

  100. Unknown error"

  101. 正在尝试其它镜像。

  102. python-chardet-2.2.1-3.el7.noa FAILED

  103. http://mirrors.cloud.aliyuncs.com/centos/7/os/x86_

  104. 64/Packages/python-chardet-2.2.1-3.el7.noarch.rpm:

  105. [Errno 14] curl#6 - "Could not resolve host:

  106. mirrors.cloud.aliyuncs.com; Unknown error"

  107. 正在尝试其它镜像。

  108. (2/4): libxml2-python-2.9.1-6.el7_9.6.x86_64.rpm

  109. | 247 kB

  110. 00:00:01

  111. (3/4): python-chardet-2.2.1-3.el7.noarch.rpm

  112. | 227 kB

  113. 00:00:07

  114. python-kitchen-1.1.1-5.el7.noa FAILED

  115. http://mirrors.aliyuncs.com/centos/7/os/x86_64/Pac

  116. kages/python-kitchen-1.1.1-5.el7.noarch.rpm:

  117. [Errno 14] curl#7 - "Failed connect to

  118. mirrors.aliyuncs.com:80; Connection refused"

  119. 正在尝试其它镜像。

  120. (4/4): python-kitchen-1.1.1-5.el7.noarch.rpm

  121. | 267 kB

  122. 00:00:01

  123. --------------------------------------------------

  124. --------------------------------------------------

  125. ------------

  126. 总计

  127. 38 kB/s | 863 kB

  128. 00:00:22

  129. Running transaction check

  130. Running transaction test

  131. Transaction test succeeded

  132. Running transaction

  133. 正在安装 : python-chardet-2.2.1-3.el7.noarch

  134. 1/4

  135. 正在安装 : python-kitchen-1.1.1-5.el7.noarch

  136. 2/4

  137. 正在安装 : libxml2-python-2.9.1-

  138. 6.el7_9.6.x86_64

  139. 3/4

  140. 正在安装 : yum-utils-1.1.31-54.el7_8.noarch

  141. 4/4

  142. 验证中 : python-kitchen-1.1.1-5.el7.noarch

  143. 1/4

  144. 验证中 : yum-utils-1.1.31-54.el7_8.noarch

  145. 2/4

  146. 验证中 : libxml2-python-2.9.1-

  147. 6.el7_9.6.x86_64

  148. 3/4

  149. 验证中 : python-chardet-2.2.1-3.el7.noarch

  150. 4/4

  151. 已安装:

  152. yum-utils.noarch 0:1.1.31-54.el7_8

  153. 作为依赖被安装:

  154. libxml2-python.x86_64 0:2.9.1-6.el7_9.6

  155. python-chardet.noarch 0:2.2.1-3.el7

  156. python-kitchen.noarch 0:1.1.1-5.el7

  157. 完毕!

  158. 已加载插件:fastestmirror

  159. adding repo from:

  160. https://mirrors.aliyun.com/dockerce/linux/centos/docker-ce.repo

  161. grabbing file https://mirrors.aliyun.com/dockerce/linux/centos/docker-ce.repo to

  162. /etc/yum.repos.d/docker-ce.repo

  163. repo saved to /etc/yum.repos.d/docker-ce.repo

  164. 已加载插件:fastestmirror

  165. Loading mirror speeds from cached hostfile

  166. * base: mirrors.aliyun.com

  167. * epel: repo.jing.rocks

  168. * extras: mirrors.aliyun.com

  169. * updates: mirrors.aliyun.com

  170. docker-ce-stable

  171. | 3.5 kB

  172. 00:00:00

  173. (1/2): docker-ce-stable/7/x86_64/updateinfo

  174. | 55 B

  175. 00:00:10

  176. (2/2): docker-ce-stable/7/x86_64/primary_db

  177. | 152 kB

  178. 00:00:11

  179. 正在解决依赖关系

  180. --> 正在检查事务

  181. ---> 软件包 containerd.io.x86_64.0.1.6.33-

  182. 3.1.el7 将被 安装

  183. --> 正在处理依赖关系 container-selinux >= 2:2.74,

  184. 它被软件包 containerd.io-1.6.33-3.1.el7.x86_64 需要

  185. ---> 软件包 docker-buildxplugin.x86_64.0.0.14.1-1.el7 将被 安装

  186. ---> 软件包 docker-ce.x86_64.3.26.1.4-1.el7 将被

  187. 安装

  188. --> 正在处理依赖关系 docker-ce-rootless-extras,它

  189. 被软件包 3:docker-ce-26.1.4-1.el7.x86_64 需要

  190. --> 正在处理依赖关系 libcgroup,它被软件包

  191. 3:docker-ce-26.1.4-1.el7.x86_64 需要

  192. ---> 软件包 docker-ce-cli.x86_64.1.26.1.4-1.el7

  193. 将被 安装

  194. ---> 软件包 docker-composeplugin.x86_64.0.2.27.1-1.el7 将被 安装

  195. --> 正在检查事务

  196. ---> 软件包 container-selinux.noarch.2.2.119.2-

  197. 1.911c772.el7_8 将被 安装

  198. --> 正在处理依赖关系 policycoreutils-python,它被软

  199. 件包 2:container-selinux-2.119.2-

  200. 1.911c772.el7_8.noarch 需要

  201. ---> 软件包 docker-ce-rootlessextras.x86_64.0.26.1.4-1.el7 将被 安装

  202. --> 正在处理依赖关系 fuse-overlayfs >= 0.7,它被软

  203. 件包 docker-ce-rootless-extras-26.1.4-1.el7.x86_64

  204. 需要

  205. --> 正在处理依赖关系 slirp4netns >= 0.4,它被软件包

  206. docker-ce-rootless-extras-26.1.4-1.el7.x86_64 需要

  207. ---> 软件包 libcgroup.x86_64.0.0.41-21.el7 将被

  208. 安装

  209. --> 正在检查事务

  210. ---> 软件包 fuse-overlayfs.x86_64.0.0.7.2-

  211. 6.el7_8 将被 安装

  212. --> 正在处理依赖关系 libfuse3.so.3(FUSE_3.2)

  213. (64bit),它被软件包 fuse-overlayfs-0.7.2-

  214. 6.el7_8.x86_64 需要

  215. --> 正在处理依赖关系 libfuse3.so.3(FUSE_3.0)

  216. (64bit),它被软件包 fuse-overlayfs-0.7.2-

  217. 6.el7_8.x86_64 需要

  218. --> 正在处理依赖关系 libfuse3.so.3()(64bit),它被软

  219. 件包 fuse-overlayfs-0.7.2-6.el7_8.x86_64 需要

  220. ---> 软件包 policycoreutils-python.x86_64.0.2.5-

  221. 34.el7 将被 安装

  222. --> 正在处理依赖关系 setools-libs >= 3.3.8-4,它被

  223. 软件包 policycoreutils-python-2.5-34.el7.x86_64 需

  224. --> 正在处理依赖关系 libsemanage-python >= 2.5-

  225. 14,它被软件包 policycoreutils-python-2.5-

  226. 34.el7.x86_64 需要

  227. --> 正在处理依赖关系 audit-libs-python >= 2.1.3-

  228. 4,它被软件包 policycoreutils-python-2.5-

  229. 34.el7.x86_64 需要

  230. --> 正在处理依赖关系 python-IPy,它被软件包

  231. policycoreutils-python-2.5-34.el7.x86_64 需要

  232. --> 正在处理依赖关系 libqpol.so.1(VERS_1.4)

  233. (64bit),它被软件包 policycoreutils-python-2.5-

  234. 34.el7.x86_64 需要

  235. --> 正在处理依赖关系 libqpol.so.1(VERS_1.2)

  236. (64bit),它被软件包 policycoreutils-python-2.5-

  237. 34.el7.x86_64 需要

  238. --> 正在处理依赖关系 libapol.so.4(VERS_4.0)

  239. (64bit),它被软件包 policycoreutils-python-2.5-

  240. 34.el7.x86_64 需要

  241. --> 正在处理依赖关系 checkpolicy,它被软件包

  242. policycoreutils-python-2.5-34.el7.x86_64 需要

  243. --> 正在处理依赖关系 libqpol.so.1()(64bit),它被软

  244. 件包 policycoreutils-python-2.5-34.el7.x86_64 需要

  245. --> 正在处理依赖关系 libapol.so.4()(64bit),它被软

  246. 件包 policycoreutils-python-2.5-34.el7.x86_64 需要

  247. ---> 软件包 slirp4netns.x86_64.0.0.4.3-4.el7_8

  248. 将被 安装

  249. --> 正在检查事务

  250. ---> 软件包 audit-libs-python.x86_64.0.2.8.5-

  251. 4.el7 将被 安装

  252. ---> 软件包 checkpolicy.x86_64.0.2.5-8.el7 将被

  253. 安装

  254. ---> 软件包 fuse3-libs.x86_64.0.3.6.1-4.el7 将被

  255. 安装

  256. ---> 软件包 libsemanage-python.x86_64.0.2.5-

  257. 14.el7 将被 安装

  258. ---> 软件包 python-IPy.noarch.0.0.75-6.el7 将被

  259. 安装

  260. ---> 软件包 setools-libs.x86_64.0.3.3.8-4.el7 将

  261. 被 安装

  262. --> 解决依赖关系完成

  263. 依赖关系解决

  264. ==================================================

  265. ==================================================

  266. ============

  267. Package 架构 版

  268. 本 源

  269. ⼤⼩

  270. ==================================================

  271. ==================================================

  272. ============

  273. 正在安装:

  274. containerd.io x86_64

  275. 1.6.33-3.1.el7 docker-cestable 35 M

  276. docker-buildx-plugin x86_64

  277. 0.14.1-1.el7 docker-cestable 14 M

  278. docker-ce x86_64

  279. 3:26.1.4-1.el7 docker-cestable 27 M

  280. docker-ce-cli x86_64

  281. 1:26.1.4-1.el7 docker-cestable 15 M

  282. docker-compose-plugin x86_64

  283. 2.27.1-1.el7 docker-cestable 13 M

  284. 为依赖⽽安装:

  285. audit-libs-python x86_64

  286. 2.8.5-4.el7 base

  287. 76 k

  288. checkpolicy x86_64

  289. 2.5-8.el7 base

  290. 295 k

  291. container-selinux noarch

  292. 2:2.119.2-1.911c772.el7_8 extras

  293. 40 k

  294. docker-ce-rootless-extras x86_64

  295. 26.1.4-1.el7 docker-cestable 9.4 M

  296. fuse-overlayfs x86_64

  297. 0.7.2-6.el7_8 extras

  298. 54 k

  299. fuse3-libs x86_64

  300. 3.6.1-4.el7 extras

  301. 82 k

  302. libcgroup x86_64

  303. 0.41-21.el7 base

  304. 66 k

  305. libsemanage-python x86_64

  306. 2.5-14.el7 base

  307. 113 k

  308. policycoreutils-python x86_64

  309. 2.5-34.el7 base

  310. 457 k

  311. python-IPy noarch

  312. 0.75-6.el7 base

  313. 32 k

  314. setools-libs x86_64

  315. 3.3.8-4.el7 base

  316. 620 k

  317. slirp4netns x86_64

  318. 0.4.3-4.el7_8 extras

  319. 81 k

  320. 事务概要

  321. ==================================================

  322. ==================================================

  323. ============

  324. 安装 5 软件包 (+12 依赖软件包)

  325. 总下载量:116 M

  326. 安装⼤⼩:407 M

  327. Downloading packages:

  328. (1/17): container-selinux-2.119.2-

  329. 1.911c772.el7_8.noarch.rpm

  330. | 40 kB 00:00:00

  331. (2/17): audit-libs-python-2.8.5-4.el7.x86_64.rpm

  332. | 76 kB

  333. 00:00:02

  334. (3/17): checkpolicy-2.5-8.el7.x86_64.rpm

  335. | 295 kB

  336. 00:00:03

  337. warning: /var/cache/yum/x86_64/7/docker-cestable/packages/docker-buildx-plugin-0.14.1-

  338. 1.el7.x86_64.rpm: Header V4 RSA/SHA512 Signature,

  339. key ID 621e9f35: NOKEY

  340. docker-buildx-plugin-0.14.1-1.el7.x86_64.rpm 的公钥

  341. 尚未安装

  342. (4/17): docker-buildx-plugin-0.14.1-

  343. 1.el7.x86_64.rpm

  344. | 14 MB 00:02:11

  345. (5/17): containerd.io-1.6.33-3.1.el7.x86_64.rpm

  346. | 35 MB

  347. 00:05:22

  348. (6/17): docker-ce-cli-26.1.4-1.el7.x86_64.rpm

  349. | 15 MB

  350. 00:00:35

  351. (7/17): docker-ce-26.1.4-1.el7.x86_64.rpm

  352. | 27 MB

  353. 00:03:50

  354. (8/17): fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm

  355. | 54 kB

  356. 00:00:03

  357. (9/17): libcgroup-0.41-21.el7.x86_64.rpm

  358. | 66 kB

  359. 00:00:03

  360. (10/17): fuse3-libs-3.6.1-4.el7.x86_64.rpm

  361. | 82 kB

  362. 00:00:03

  363. (11/17): libsemanage-python-2.5-14.el7.x86_64.rpm

  364. | 113 kB

  365. 00:00:02

  366. (12/17): python-IPy-0.75-6.el7.noarch.rpm

  367. | 32 kB

  368. 00:00:00

  369. (13/17): policycoreutils-python-2.5-

  370. 34.el7.x86_64.rpm

  371. | 457 kB 00:00:05

  372. (14/17): slirp4netns-0.4.3-4.el7_8.x86_64.rpm

  373. | 81 kB

  374. 00:00:02

  375. (15/17): setools-libs-3.3.8-4.el7.x86_64.rpm

  376. | 620 kB

  377. 00:00:02

  378. (16/17): docker-compose-plugin-2.27.1-

  379. 1.el7.x86_64.rpm

  380. | 13 MB 00:01:28

  381. (17/17): docker-ce-rootless-extras-26.1.4-

  382. 1.el7.x86_64.rpm |

  383. 9.4 MB 00:01:33

  384. --------------------------------------------------

  385. --------------------------------------------------

  386. ------------

  387. 总计

  388. 262 kB/s | 116 MB

  389. 00:07:31

  390. 从 https://mirrors.aliyun.com/dockerce/linux/centos/gpg 检索密钥

  391. 导⼊ GPG key 0x621E9F35:

  392. ⽤户ID : "Docker Release (CE rpm)

  393. <docker@docker.com>"

  394. 指纹 : 060a 61c5 1b55 8a7f 742b 77aa c52f

  395. eb6b 621e 9f35

  396. 来⾃ : https://mirrors.aliyun.com/dockerce/linux/centos/gpg

  397. Running transaction check

  398. Running transaction test

  399. Transaction test succeeded

  400. Running transaction

  401. 正在安装 : libcgroup-0.41-21.el7.x86_64

  402. 1/17

  403. 正在安装 : docker-buildx-plugin-0.14.1-

  404. 1.el7.x86_64

  405. 2/17

  406. 正在安装 : setools-libs-3.3.8-4.el7.x86_64

  407. 3/17

  408. 正在安装 : audit-libs-python-2.8.5-

  409. 4.el7.x86_64

  410. 4/17

  411. 正在安装 : slirp4netns-0.4.3-4.el7_8.x86_64

  412. 5/17

  413. 正在安装 : libsemanage-python-2.5-

  414. 14.el7.x86_64

  415. 6/17

  416. 正在安装 : python-IPy-0.75-6.el7.noarch

  417. 7/17

  418. 正在安装 : fuse3-libs-3.6.1-4.el7.x86_64

  419. 8/17

  420. 正在安装 : fuse-overlayfs-0.7.2-6.el7_8.x86_64

  421. 9/17

  422. 正在安装 : checkpolicy-2.5-8.el7.x86_64

  423. 10/17

  424. 正在安装 : policycoreutils-python-2.5-

  425. 34.el7.x86_64

  426. 11/17

  427. 正在安装 : 2:container-selinux-2.119.2-

  428. 1.911c772.el7_8.noarch

  429. 12/17

  430. 正在安装 : containerd.io-1.6.33-3.1.el7.x86_64

  431. 13/17

  432. 正在安装 : docker-compose-plugin-2.27.1-

  433. 1.el7.x86_64

  434. 14/17

  435. 正在安装 : 1:docker-ce-cli-26.1.4-1.el7.x86_64

  436. 15/17

  437. 正在安装 : docker-ce-rootless-extras-26.1.4-

  438. 1.el7.x86_64

  439. 16/17

  440. 正在安装 : 3:docker-ce-26.1.4-1.el7.x86_64

  441. 17/17

  442. 验证中 : docker-compose-plugin-2.27.1-

  443. 1.el7.x86_64

  444. 1/17

  445. 验证中 : checkpolicy-2.5-8.el7.x86_64

  446. 2/17

  447. 验证中 : fuse3-libs-3.6.1-4.el7.x86_64

  448. 3/17

  449. 验证中 : python-IPy-0.75-6.el7.noarch

  450. 4/17

  451. 验证中 : fuse-overlayfs-0.7.2-6.el7_8.x86_64

  452. 5/17

  453. 验证中 : libsemanage-python-2.5-

  454. 14.el7.x86_64

  455. 6/17

  456. 验证中 : slirp4netns-0.4.3-4.el7_8.x86_64

  457. 7/17

  458. 验证中 : 2:container-selinux-2.119.2-

  459. 1.911c772.el7_8.noarch

  460. 8/17

  461. 验证中 : containerd.io-1.6.33-3.1.el7.x86_64

  462. 9/17

  463. 验证中 : 3:docker-ce-26.1.4-1.el7.x86_64

  464. 10/17

  465. 验证中 : 1:docker-ce-cli-26.1.4-1.el7.x86_64

  466. 11/17

  467. 验证中 : policycoreutils-python-2.5-

  468. 34.el7.x86_64

  469. 12/17

  470. 验证中 : docker-ce-rootless-extras-26.1.4-

  471. 1.el7.x86_64

  472. 13/17

  473. 验证中 : audit-libs-python-2.8.5-

  474. 4.el7.x86_64

  475. 14/17

  476. 验证中 : setools-libs-3.3.8-4.el7.x86_64

  477. 15/17

  478. 验证中 : docker-buildx-plugin-0.14.1-

  479. 1.el7.x86_64

  480. 16/17

  481. 验证中 : libcgroup-0.41-21.el7.x86_64

  482. 17/17

  483. 已安装:

  484. containerd.io.x86_64 0:1.6.33-3.1.el7

  485. docker-buildx-plugin.x86_64 0:0.14.1-1.el7

  486. docker-ce.x86_64 3:26.1.4-1.el7

  487. docker-ce-cli.x86_64 1:26.1.4-1.el7

  488. docker-compose-plugin.x86_64 0:2.27.1-1.el7

  489. 作为依赖被安装:

  490. audit-libs-python.x86_64 0:2.8.5-4.el7

  491. checkpolicy.x86_64 0:2.5-8.el7

  492. container-selinux.noarch 2:2.119.2-

  493. 1.911c772.el7_8 docker-ce-rootlessextras.x86_64 0:26.1.4-1.el7

  494. fuse-overlayfs.x86_64 0:0.7.2-6.el7_8

  495. fuse3-libs.x86_64 0:3.6.1-4.el7

  496. libcgroup.x86_64 0:0.41-21.el7

  497. libsemanage-python.x86_64 0:2.5-14.el7

  498. policycoreutils-python.x86_64 0:2.5-34.el7

  499. python-IPy.noarch 0:0.75-6.el7

  500. setools-libs.x86_64 0:3.3.8-4.el7

  501. slirp4netns.x86_64 0:0.4.3-4.el7_8

  502. 完毕!

14.docker服务没有开启之前查看ip

 
  1. [root@node1 ~]# ifconfig

  2. ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>

  3. mtu 1500

  4. inet 10.1.1.10 netmask 255.255.255.0

  5. broadcast 10.1.1.255

  6. inet6 fe80::20c:29ff:fe94:721b prefixlen

  7. 64 scopeid 0x20<link>

  8. ether 00:0c:29:94:72:1b txqueuelen 1000

  9. (Ethernet)

  10. RX packets 105369 bytes 146824987 (140.0

  11. MiB)

  12. RX errors 0 dropped 0 overruns 0 frame

  13. 0

  14. TX packets 39963 bytes 3021936 (2.8 MiB)

  15. TX errors 0 dropped 0 overruns 0 carrier

  16. 0 collisions 0

  17. flannel0:

  18. flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>

  19. mtu 1472

  20. inet 172.20.20.0 netmask 255.255.0.0

  21. destination 172.20.20.0

  22. inet6 fe80::1fae:d70e:eb38:5ec9 prefixlen

  23. 64 scopeid 0x20<link>

  24. unspec 00-00-00-00-00-00-00-00-00-00-00-

  25. 00-00-00-00-00 txqueuelen 500 (UNSPEC)

  26. RX packets 0 bytes 0 (0.0 B)

  27. RX errors 0 dropped 0 overruns 0 frame

  28. 0

  29. TX packets 3 bytes 144 (144.0 B)

  30. TX errors 0 dropped 0 overruns 0 carrier

  31. 0 collisions 0

  32. lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

  33. inet 127.0.0.1 netmask 255.0.0.0

  34. inet6 ::1 prefixlen 128 scopeid

  35. 0x10<host>

  36. loop txqueuelen 1000 (Local Loopback)

  37. RX packets 993 bytes 65146 (63.6 KiB)

  38. RX errors 0 dropped 0 overruns 0 frame

  39. 0

  40. TX packets 993 bytes 65146 (63.6 KiB)

  41. TX errors 0 dropped 0 overruns 0 carrier

  42. 0 collisions 0

15.启动docker服务后查看ip

 
  1. [root@node1 ~]# systemctl start docker.service

  2. [root@node1 ~]# ifconfig

  3. docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu

  4. 1500

  5. inet 172.17.0.1 netmask 255.255.0.0

  6. broadcast 172.17.255.255

  7. ether 02:42:ba:f0:5f:1e txqueuelen 0

  8. (Ethernet)

  9. RX packets 0 bytes 0 (0.0 B)

  10. RX errors 0 dropped 0 overruns 0 frame

  11. 0

  12. TX packets 0 bytes 0 (0.0 B)

  13. TX errors 0 dropped 0 overruns 0 carrier

  14. 0 collisions 0

  15. ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>

  16. mtu 1500

  17. inet 10.1.1.10 netmask 255.255.255.0

  18. broadcast 10.1.1.255

  19. inet6 fe80::20c:29ff:fe94:721b prefixlen

  20. 64 scopeid 0x20<link>

  21. ether 00:0c:29:94:72:1b txqueuelen 1000

  22. (Ethernet)

  23. RX packets 105431 bytes 146830165 (140.0

  24. MiB)

  25. RX errors 0 dropped 0 overruns 0 frame

  26. 0

  27. TX packets 40003 bytes 3028954 (2.8 MiB)

  28. TX errors 0 dropped 0 overruns 0 carrier

  29. 0 collisions 0

  30. flannel0:

  31. flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>

  32. mtu 1472

  33. inet 172.20.20.0 netmask 255.255.0.0

  34. destination 172.20.20.0

  35. inet6 fe80::1fae:d70e:eb38:5ec9 prefixlen

  36. 64 scopeid 0x20<link>

  37. unspec 00-00-00-00-00-00-00-00-00-00-00-

  38. 00-00-00-00-00 txqueuelen 500 (UNSPEC)

  39. RX packets 0 bytes 0 (0.0 B)

  40. RX errors 0 dropped 0 overruns 0 frame

  41. 0

  42. TX packets 3 bytes 144 (144.0 B)

  43. TX errors 0 dropped 0 overruns 0 carrier

  44. 0 collisions 0

  45. lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

  46. inet 127.0.0.1 netmask 255.0.0.0

  47. inet6 ::1 prefixlen 128 scopeid

  48. 0x10<host>

  49. loop txqueuelen 1000 (Local Loopback)

  50. RX packets 1001 bytes 65562 (64.0 KiB)

  51. RX errors 0 dropped 0 overruns 0 frame

  52. 0

  53. TX packets 1001 bytes 65562 (64.0 KiB)

  54. TX errors 0 dropped 0 overruns 0 carrier

  55. 0 collisions 0

16.查看flannel⼦⽹ip

 
  1. [root@node1 ~]# cat /run/flannel/subnet.env

  2. FLANNEL_NETWORK=172.20.0.0/16

  3. FLANNEL_SUBNET=172.20.20.1/24

  4. FLANNEL_MTU=1472

  5. FLANNEL_IPMASQ=false

17.从其他主机复制⼀份daemon.json⽂件

 
  1. [root@node1 ~]# scp

  2. root@10.1.1.50:/etc/docker/daemon.json

  3. /etc/docker/

  4. root@10.1.1.50's password:

  5. daemon.json

  6. 100% 385 111.9KB/s

  7. 00:00

18.修改添加桥ip和路由字节1472-1500

 
  1. [root@node1 ~]# vim /etc/docker/daemon.json

  2. [root@node1 ~]# cat /etc/docker/daemon.json

  3. {

  4. "insecure-registries":[

  5. "http://10.1.1.50"

  6. ],

  7. "registry-mirrors": [

  8. "https://do.nark.eu.org",

  9. "https://dc.j8.work",

  10. "https://docker.m.daocloud.io",

  11. "https://dockerproxy.com",

  12. "https://docker.mirrors.ustc.edu.cn",

  13. "https://docker.nju.edu.cn"

  14. ],

  15. "hosts": [

  16. "tcp://0.0.0.0:2375",

  17. "unix:///var/run/docker.sock"

  18. ],

  19. "bip": "172.20.20.1/24",

  20. "mtu": 1472

  21. }

19.修改docker配置⽂件

 
  1. [root@node1 ~]# vim

  2. /usr/lib/systemd/system/docker.service

20.加载配置,重启docker服务

 
  1. [root@node1 ~]# systemctl daemon-reload

  2. [root@node1 ~]# systemctl restart docker.service

21.查看ip地址

 
  1. [root@node1 ~]# ip a s

  2. 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc

  3. noqueue state UNKNOWN group default qlen 1000

  4. link/loopback 00:00:00:00:00:00 brd

  5. 00:00:00:00:00:00

  6. inet 127.0.0.1/8 scope host lo

  7. valid_lft forever preferred_lft forever

  8. inet6 ::1/128 scope host

  9. valid_lft forever preferred_lft forever

  10. 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu

  11. 1500 qdisc pfifo_fast state UP group default qlen

  12. 1000

  13. link/ether 00:0c:29:94:72:1b brd

  14. ff:ff:ff:ff:ff:ff

  15. inet 10.1.1.10/24 brd 10.1.1.255 scope global

  16. ens33

  17. valid_lft forever preferred_lft forever

  18. inet6 fe80::20c:29ff:fe94:721b/64 scope link

  19. valid_lft forever preferred_lft forever

  20. 3: flannel0:

  21. <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472

  22. qdisc pfifo_fast state UNKNOWN group default qlen

  23. 500

  24. link/none

  25. inet 172.20.20.0/16 scope global flannel0

  26. valid_lft forever preferred_lft forever

  27. inet6 fe80::1fae:d70e:eb38:5ec9/64 scope link

  28. flags 800

  29. valid_lft forever preferred_lft forever

  30. 4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP>

  31. mtu 1472 qdisc noqueue state DOWN group default

  32. link/ether 02:42:ba:f0:5f:1e brd

  33. ff:ff:ff:ff:ff:ff

  34. inet 172.20.20.1/24 brd 172.20.20.255 scope

  35. global docker0

  36. valid_lft forever preferred_lft forever

22.拉取⼀个镜像测试ip地址

 
  1. [root@node1 ~]# docker pull centos

  2. Using default tag: latest

  3. latest: Pulling from library/centos

  4. a1d0c7532777: Pull complete

  5. Digest:

  6. sha256:a27fd8080b517143cbbbab9dfb7c8571c40d67d534b

  7. bdee55bd6c473f432b177

  8. Status: Downloaded newer image for centos:latest

  9. docker.io/library/centos:latest

  10. [root@node1 ~]# docker run -it centos:latest

  11. /bin/bash

  12. [root@be00f4423f5b /]# [root@node1 ~]# docker

  13. images

  14. REPOSITORY TAG IMAGE ID CREATED

  15. SIZE

  16. centos latest 5d0da3dc9764 2 years ago

  17. 23

二、从控

1.安装flannel

yum -y install flannel

2.配置flanner 配置flannel要访问的etcd数据库所在的位置

 
  1. root@localhost ~]# cat /etc/sysconfig/flanneld

  2. # Flanneld configuration options

  3. # etcd url location. Point this to the server

  4. where etcd runs

  5. FLANNEL_ETCD_ENDPOINTS="http://192.168.71.10:2379"

  6. # etcd config key. This is the configuration key

  7. that flannel queries

  8. # For address range assignment

  9. FLANNEL_ETCD_PREFIX="/atomic.io/network"

  10. # Any additional options that you want to pass

  11. #FLANNEL_OPTIONS=""

3.启动flannel

systemctl star flanneld

4.查看flannel分配的ip⽹段

 
  1. [root@localhost ~]# cat /run/flannel/subnet.env

  2. FLANNEL_NETWORK=172.20.0.0/16

  3. FLANNEL_SUBNET=172.20.99.1/24

  4. FLANNEL_MTU=1472

  5. FLANNEL_IPMASQ=false

5.安装docker

6.将flannel分配⽹段写⼊到daemon.json

 
  1. [root@localhost ~]# cat /etc/docker/daemon.json

  2. {

  3. "registry-mirrors": [

  4. "https://do.nark.eu.org",

  5. "https://dc.j8.work",

  6. "https://docker.m.daocloud.io",

  7. "https://dockerproxy.com",

  8.  
  9. "https://docker.mirrors.ustc.edu.cn",

  10. "https://docker.nju.edu.cn"

  11. ],

  12. "hosts": [

  13. "tcp://0.0.0.0:2375",

  14. "unix:///var/run/docker.sock"

  15. ],

  16. "insecure-registries":[

  17. "http://192.168.71.50:5000"

  18. ],

  19. "bip" : "172.20.99.1/24",

  20. "mtu" : 1472

  21. }

  22. [root@localhost ~]#

7.重启docaker 如果不能重启,就修改⼀下远程管理

systemctl restart docker.service

8.拉取⼀个centos镜像

 
  1. docker pull centos

  2. docker run -it centos:latest /bin/bash

9.ping node1中容器的ip地址

ping 172.20.78.2

总结,⼯作原理

1.使⽤flanner为docker主机(宿主)分配⽹段

2.⽹段的信息以及ip的信息保存在etcd数据库中

3.当flanner开始运⾏的时候,会从etcd数据库中读取{"Network":"172.20.0.0/16"},随机为当前的主机添加⼀个flannel0⽹课172.20.78.0

4.配置docker的daemon⽂件,让docker0⽹卡变成和flannel的⽹段⼀致,之后docker下创建的容器的ip就在flannel的⽹段控制之内

  • 23
    点赞
  • 5
    收藏
    觉得还不错? 一键收藏
  • 0
    评论
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值