k8s教程01（k8s环境配置及私有仓库搭建）

最新推荐文章于 2024-05-03 09:57:59 发布

已无测试机

最新推荐文章于 2024-05-03 09:57:59 发布

阅读量5.8k

点赞数 4

分类专栏： kubernetes 文章标签： kubernetes docker 容器

本文链接：https://blog.csdn.net/DDJ_TEST/article/details/125100735

版权

kubernetes 专栏收录该内容

8 篇文章 0 订阅

订阅专栏

Infrastructure as a Service（laas，基础架构作为服务）阿里云
platform as a service（Paas，平台作为服务）新浪云
Software as a Service（Saas，软件作为服务） Office 365

Apache MESOS资源管理器
docker Swarm

高可用集群副本数据最好是>= 3奇数个

资料地址：
https://gitee.com/WeiboGe2012/kubernetes-kubeadm-install
https://gitee.com/llqhz/ingress-nginx根据版本找资料

k8s架构

在这里插入图片描述
APISERVER：所有服务访问统一入口
CrontrollerManager：维持副本期望数目
Scheduler：负责介绍任务，选择合适的节点进行分配任务
ETCD：键值对数据库储存K8S集群所有重要信息(持久化)
Kubelet：直接跟容器引擎交互实现容器的生命周期管理
Kube-prox：负责写入规则至IPTABLES、 IPVS 实现服务映射访问的

其他插件说明
COREDNS：可以为集群中的SVC创建一一个域名IP的对应关系解析
DASHBOARD：给K8S 集群提供个B/S结构访间体系
INGRESS CONTROLLER：官方只能实现四层代理，INGRESS 可以实现七层代理
FEDERATION：提供一个可以跨集群中心多K8S统一管理功能
PROMETHEUS：提供K8S集群的监控能力
ELK：提供K8S集群日志统一分析介入平台

Pod概念

pod控制器类型

Repl icationController & ReplicaSet & Deployment

ReplicationController用来确保容器应用的副本数始终保持在用户定义的副本数，即如果有容器异常退出，会自动创建新的Pod来替代;而如果异常多出来的容器也会自动回收。在新版本的Kubernetes 中建议使用ReplicaSet 来取代Replicat ionControlle

ReplicaSet跟Repl icat ionController没有本质的不同，只是名字不一样，并且Repl icaSet支持集合式的selector

虽然ReplicaSet可以独立使用，但一般般还是建议使用Deployment 来自动管理ReplicaSet，这样就无需担心跟其他机制的不兼容问题(比如ReplicaSet 不支持rolling-update但Deployment 支持)

HPA (Hor izontalPodAutoScale)

Horizontal Pod Autoscaling 仅适用于Deployment和ReplicaSet ，在V1版本中仅支持根据Pod的CPU利用率扩所容，在vlalpha 版本中，支持根据内存和用户自定义的metric 扩缩容

StatefullSet

StatefulSet是为了解决有状态服务的问题(对应Deployments 和Repl icaSets是为无状态服务而设计)，其应用场景包括:

*稳定的持久化存储，即Pod重新调度后还是能访问到相同的持久化数据，基于PVC来实现
*稳定的网络标志，即Pod重新调度后其PodName 和HostName 不变，基于Headless Service(即没有Cluster IP的Service )来实现
*有序部署，有序扩展，即Pod是有顺序的，在部署或者扩展的时候要依据定义的顺序依次依次进行(即从0到N-1,在下一个Pod运行之前所有之前的Pod必须都是Running 和Ready状态)，基于init containers 来实现
*有序收缩，有序删除(即从N-1到0)

DaemonSet

DaemonSet确保全部(或者-些) Node上运行一个Pod的副本。当有Node加入集群时，也会为他们新增一个Pod。当有Node从集群移除时，这些Pod也会被回收。删除DaemonSet 将会删除它创建的所有Pod
使用DaemonSet 的一些典型用法:

*运行集群存储daemon, 例如在每个Node上运行 glusterd、 ceph。
*在每个Node上运行日志收集daemon,例如fluentd、 logstash。
*在每个Node上运行监控daemon, 例如Prometheus Node Exporter

Job，Cronjob

Job负责批处理任务，即仅执行一次的任务，它保证批处理任务的-一个或多个Pod成功结束

Cron Job管理基于时间的Job，即:

*在给定时间点只运行一次
*周期性地在给定时间点运行

服务发现

在这里插入图片描述

网络通信模式

Kubernetes的网络模型假定了所有Pod都在一-个可以直接连通的扁平的网络空间中，这在GCE (Google Compute Engine) 里面是现成的网络模型，Kubernetes 假定这个网络已经存在。而在私有云里搭建Kubernetes 集群，就不能假定这个网络已经存在了。我们需要自己实现这个网络假设，将不同节点上的Docker 容器之间的互相访问先打通，然后运行Kubernetes

同一个Pod内的多个容器之间：lo
各Pod之间的通讯: Overlay Network
Pod与Service 之间的通讯：各节点的Iptables 规则

网络解决方案kubernetes + flannel

Flannel是CoreOS 团队针对Kubernetes 设计的一个网络规划服务，简单来说，它的功能是让集群中的不同节点主机创建的Docker容器都具有全集群唯—的虚拟IP地址。而且它还能在这些IP地址之间建立-一个覆盖网络(Overlay Network) ，通过这个覆盖网络，将数据包原封不动地传递到目标容器内
在这里插入图片描述
ETCD之Flannel 提供说明:

存储管理Flannel 可分配的IP地址段资源
监控ETCD中每个Pod的实际地址，并在内存中建立维护Pod节点路由表

不同情况下网络通讯方式

1、同一个Pod内部通讯:同一个 Pod共享同一个网络命名空间，共享同一个Linux协议栈Pod1至Pod2
2、Pod1至Pod2

不在同一台主机，Pod的地址是与docker0在同一个网段的，但docker0网段与宿主机网卡是两个完全不同的IP网段，并且不同Node之间的通信能通过宿主机的物理网卡进行。将Pod的IP和所在Node的IP关联起来，通过这个关联让Pod可以互相访问!
Pod1与Pod2在同一.台机器，由Docker0 网桥直接转发请求至Pod2，不需要经过Flannel演示

3、Pod至Service 的网络:目前基于性能考虑，全部为iptables 维护和转发
4、Pod到外网: Pod向外网发送请求，查找路由表，转发数据包到宿主机的网卡，宿主网卡完成路由选择后，iptables执行Masquerade,把源IP更改为宿主网卡的IP, 然后向外网服务器发送请求
5、外网访问Pod: Service

组件通讯示意图

在这里插入图片描述

k8s集群安装

前期准备

在这里插入图片描述
1、安装k8s的节点必须是大于1核心的CPU
2、安装节点的网络信息192.168.192.0/24 master:131 node1:130 node2:129

四台CentOS7：一个master主服务器和两个node节点和一个harbor，都是：仅主机模式
在这里插入图片描述
一台Windows10：并安装koolshare
KoolCenter固件下载服务器：http://fw.koolcenter.com/

下载IMG写盘工具

创建虚拟机

在这里插入图片描述

看一下是哪一个网卡是仅主机运行

设置完成后，浏览器访问koolshare路由ip：192.168.1.1密码：koolshare

修改成与k8s集群一样的网段

再次登录使用的是新的路由ip：192.168.192.1
在这里插入图片描述
诊断访问国内网站正常

访问国外网站，点击【酷软】

下载koolss

直接搜索SSR节点或有SSR服务器直接填写上去

k8s集群安装

设置系统主机名以及hosts文件

hostnamectl set-hostname k8s-master01

hostnamectl set-hostname k8s-node01

hostnamectl set-hostname k8s-node02

设置ip地址
vi /etc/sysconfig/network-scripts/ifcfg-ens33
master主机

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="ea3b61ed-9232-4b69-b6c0-2f863969e750"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.192.131
NETMASK=255.255.255.0
GATEWAY=192.168.192.1
DNS1=192.168.192.1
DNS2=114.114.114.114

node01主机

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="ea3b61ed-9232-4b69-b6c0-2f863969e750"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.192.130
NETMASK=255.255.255.0
GATEWAY=192.168.192.1
DNS1=192.168.192.1
DNS2=114.114.114.114

node02主机

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="ea3b61ed-9232-4b69-b6c0-2f863969e750"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.192.129
NETMASK=255.255.255.0
GATEWAY=192.168.192.1
DNS1=192.168.192.1
DNS2=114.114.114.114

三台主机重启网络

service network restart

设置master主机的hosts文件，加入以下主机名
vi /etc/hosts

192.168.192.131 k8s-master01
192.168.192.130 k8s-node01
192.168.192.129 k8s-node02

将master主机的hosts文件拷贝到node01、node02主机上

[root@localhost ~]# scp /etc/hosts root@k8s-node01:/etc/hosts
The authenticity of host 'k8s-node01 (192.168.192.130)' can't be established.
ECDSA key fingerprint is SHA256:M5BalHyNXU5W49c5/9iZgC4Hl370O0Wr/c5S/FYFIvw.
ECDSA key fingerprint is MD5:28:23:b8:eb:af:d1:bd:bb:8c:77:e0:01:3c:62:7a:cb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'k8s-node01,192.168.192.130' (ECDSA) to the list of known hosts.
root@k8s-node01's password: 
hosts                                         100%  241   217.8KB/s   00:00    
[root@localhost ~]# scp /etc/hosts root@k8s-node02:/etc/hosts
The authenticity of host 'k8s-node02 (192.168.192.129)' can't be established.
ECDSA key fingerprint is SHA256:M5BalHyNXU5W49c5/9iZgC4Hl370O0Wr/c5S/FYFIvw.
ECDSA key fingerprint is MD5:28:23:b8:eb:af:d1:bd:bb:8c:77:e0:01:3c:62:7a:cb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'k8s-node02,192.168.192.129' (ECDSA) to the list of known hosts.
root@k8s-node02's password: 
hosts                                         100%  241   143.1KB/s   00:00

三台主机都安装依赖包

yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git

三台主机：设置防火墙为iptables并设置空规则

systemctl stop firewalld && systemctl disable firewalld

yum -y install iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save

三台主机：关闭selinux

swapoff -a && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config

三台主机：调整内核参数，对于K8S

cat > kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
net.ipv4.tcp_tw_recycle=0
vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它
vm.overcommit_memory=1 # 不检查物理内存是否够用
vm.panic_on_oom=0 # 开启 OOM
fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=1048576
fs.file-max=52706963
fs.nr_open=52706963
net.ipv6.conf.all.disable_ipv6=1
net.netfilter.nf_conntrack_max=2310720
EOF

cp kubernetes.conf /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf

三台主机：调整系统时区

# 设置系统时区为中国/上海
timedatectl set-timezone Asia/Shanghai
# 将当前的UTC时间写入硬件时钟
timedatectl set-local-rtc 0
# 重启依赖于系统时间的服务
systemctl restart rsyslog
systemctl restart crond

三台主机：关闭系统不需要的服务

systemctl stop postfix && systemctl disable postfix

三台主机：设置rsyslogd和systemd journald

mkdir /var/log/journal  # 持久化保存日志的目录

mkdir /etc/systemd/journald.conf.d

cat > /etc/systemd/journald.conf.d/99-prophet.conf <<EOF
[Journal]
# 持久化保存到磁盘
Storage=persistent

# 压缩历史日志
Compress=yes

SyncIntervalSec=5m
RateLimitInterval=30s
RateLimitBurst=1000

# 最大占用空间10G
SystemMaxUse=10G

# 但日志文件最大 200M
SystemMaxFileSeize=200M

# 日志保存时间2周
MaxRetentionSec=2week

# 不将日志转发到syslog
ForwardToSyslog=no
EOF

systemctl restart systemd-journald

三台主机：升级系统内核为5.4

CentOS 7.x系统自带的3.10.x内核存在一些Bugs，导致运行的Docker、Kubernetes 不稳定，例如: rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm

uname -r 命令查看内核信息

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
# 安装完成后检查/boot/grub2/grub.cfg 中对应内核menuentry 中是否包含initrd16 配置，如果没有，再安装一次!
yum --enablerepo=elrepo-kernel install -y kernel-lt

# 上面查询出内核版本后，设置开机从新内核启动
grub2-set-default "CentOS Linux (4.4.182-1.el7.elrepo.x86_64) 7 (Core)"
grub2-set-default "CentOS Linux (5.4.195-1.el7.elrepo.x86_64) 7 (Core)"

三台主机：kube-proxy开启ipvs的前置条件

# ==============================================================内核4.44版本
modprobe br_netfilter

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF

chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_Vs -e nf_conntrack_ipv4


# ==============================================================内核5.4版本
# 1.安装ipset和ipvsadm
yum install ipset ipvsadmin -y

# 2.添加需要加载的模块写入脚本文件
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

# 3.为脚本添加执行权限
chmod +x /etc/sysconfig/modules/ipvs.modules
# 4.执行脚本文件
cd /etc/sysconfig/modules/
./ipvs.modules
# 5.查看对应的模块是否加载成功
lsmod | grep -e -ip_vs -e nf_conntrack

三台主机：安装Docker软件

参考安装docker：https://blog.csdn.net/DDJ_TEST/article/details/114983746

# 设置存储库
# 安装yum-utils包（提供yum-config-manager 实用程序）并设置稳定存储库。
$ sudo yum install -y yum-utils
$ sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# 安装指定版本的 Docker Engine 和 containerd
$ sudo yum install -y docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io

启动docker

$ sudo systemctl start docker

通过运行hello-world 映像验证 Docker Engine 是否已正确安装。

$ sudo docker run hello-world

设置开机定时启动docker

$ sudo systemctl enable docker

关闭docker服务

$ sudo systemctl stop docker

重启docker服务

$ sudo systemctl restart  docker

验证

[root@k8s-master01 ~]# docker version
Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df9ba
 Built:             Wed Sep  4 16:51:21 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:22:32 2019
  OS/Arch:          linux/amd64
  Experimental:     false

# 创建/etc/docker目录
mkdir /etc/docker

# 配置daemon
cat > /etc/docker/daemon.json <<EOF
{
	"exec-opts": ["native.cgroupdriver=systemd"],
	"log-driver": "json-file",
	"log-opts": {
		"max-size": "100m"
	},
	"insecure-registries": ["https://hub.atguigu.com"]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# 重启docker服务
systemctl daemon-reload && systemctl restart docker && systemctl enable docker

三台主机：安装kubeadm（主从配置）

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum -y install kubeadm-1.15.1 kubectl-1.15.1 kubelet-1.15.1
systemctl enable kubelet.service

kubeadm-basic.images.tar.gz和harbor-offline-installer-v2.3.2.tgz 下载
上传到master节点上

tar -zxvf kubeadm-basic.images.tar.gz

vim load-images.sh将镜像文件上传到docker

#!/bin/bash

ls /root/kubeadm-basic.images > /tmp/image-list.txt

cd /root/kubeadm-basic.images

for i in $( cat /tmp/image-list.txt )
do
    docker load -i $i
done


rm -rf /tmp/image-list.txt

chmod a+x load-images.sh

./load-images.sh

将解压的文件复制到node01、node02

scp -r kubeadm-basic.images load-images.sh root@k8s-node01:/root

scp -r kubeadm-basic.images load-images.sh root@k8s-node02:/root

在node01、node02上执行load-images.sh文件

./load-images.sh

初始化主节点

# 生成模板文件
kubeadm config print init-defaults > kubeadm-config.yaml

vim kubeadm-config.yaml

apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.192.131      # 改成master主机的ip
  bindPort: 6443
nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: k8s-master01
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
  type: CoreDNS
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.15.1		# 改成安装的版本号
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"	# 添加默认podSubnet网段
  serviceSubnet: 10.96.0.0/12
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
    SupportIPVSProxyMode: true
mode: ipvs

kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log

遇到报错，找问题3、4

查看日志文件

[root@k8s-master01 ~]# vim kubeadm-init.log

[init] Using Kubernetes version: v1.15.1
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.192.131 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [192.168.192.131 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.192.131]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 23.004030 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
ecaa903ab475ec8d361a7a844feb3973b437a6e36981be7d949dccda63c15d00
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.192.131:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:3236bf910c84de4e1f5ad24b1b627771602d5bad03e7819aad18805c440fd8aa
~

执行上面的命令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

查看k8s节点

[root@k8s-master01 ~]# kubectl get node
NAME           STATUS     ROLES    AGE   VERSION
k8s-master01   NotReady   master   19m   v1.15.1

加入主节点以及其余工作节点

执行安装日志中的加入命令即可

部署网络

下面这条不用

kubectl apply -f https://github.com/WeiboGe2012/kube-flannel.yml/blob/master/kube-flannel.yml

执行下面的命令

[root@k8s-master01 ~]# mkdir -p install-k8s/core
[root@k8s-master01 ~]# mv kubeadm-init.log kubeadm-config.yaml install-k8s/core
[root@k8s-master01 ~]# cd install-k8s/
[root@k8s-master01 install-k8s]# mkdir -p plugin/flannel
[root@k8s-master01 install-k8s]# cd plugin/flannel
[root@k8s-master01 flannel]# wget https://github.com/WeiboGe2012/kube-flannel.yml/blob/master/kube-flannel.yml

[root@k8s-master01 flannel]# kubectl create -f kube-flannel.yml

[root@k8s-master01 flannel]# kubectl get pod -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-4kj2t               1/1     Running   0          92m
coredns-5c98db65d4-7zsr7               1/1     Running   0          92m
etcd-k8s-master01                      1/1     Running   0          91m
kube-apiserver-k8s-master01            1/1     Running   0          91m
kube-controller-manager-k8s-master01   1/1     Running   0          91m
kube-flannel-ds-amd64-g4gh9            1/1     Running   0          18m
kube-proxy-t7v46                       1/1     Running   0          92m
kube-scheduler-k8s-master01            1/1     Running   0          91m

[root@k8s-master01 flannel]# kubectl get node
NAME           STATUS   ROLES    AGE   VERSION
k8s-master01   Ready    master   93m   v1.15.1

[root@k8s-master01 flannel]# ifconfig
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.0.1  netmask 255.255.255.0  broadcast 10.244.0.255
        ether c6:13:60:e7:e8:21  txqueuelen 1000  (Ethernet)
        RX packets 4809  bytes 329578 (321.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 4854  bytes 1485513 (1.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:71:d8:f1:e2  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.192.131  netmask 255.255.255.0  broadcast 192.168.192.255
        ether 00:0c:29:8c:51:ba  txqueuelen 1000  (Ethernet)
        RX packets 536379  bytes 581462942 (554.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1359677  bytes 1764989232 (1.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.244.0.0  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 16:0c:14:08:a6:51  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 625548  bytes 102038881 (97.3 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 625548  bytes 102038881 (97.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth350261c9: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        ether f2:95:97:91:06:00  txqueuelen 0  (Ethernet)
        RX packets 2400  bytes 198077 (193.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2424  bytes 741548 (724.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

vethd9ac2bc1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        ether 16:5c:e0:81:25:ba  txqueuelen 0  (Ethernet)
        RX packets 2409  bytes 198827 (194.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2435  bytes 744163 (726.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

node01、node01主机加入节点

kubeadm-init.log日志文件最后一行复制到node01、node02执行

kubeadm join 192.168.192.131:6443 --token abcdef.0123456789abcdef \
    --discovery-token-ca-cert-hash sha256:3236bf910c84de4e1f5ad24b1b627771602d5bad03e7819aad18805c440fd8aa

再到master上查看两个节点还没完全运行

[root@k8s-master01 flannel]# kubectl get node
NAME           STATUS     ROLES    AGE   VERSION
k8s-master01   Ready      master   99m   v1.15.1
k8s-node01     NotReady   <none>   23s   v1.15.1
k8s-node02     NotReady   <none>   20s   v1.15.1

# 等一会node01、node02运行完成后再次看看
[root@k8s-master01 flannel]# kubectl get pod -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
coredns-5c98db65d4-4kj2t               1/1     Running   0          100m
coredns-5c98db65d4-7zsr7               1/1     Running   0          100m
etcd-k8s-master01                      1/1     Running   0          100m
kube-apiserver-k8s-master01            1/1     Running   0          100m
kube-controller-manager-k8s-master01   1/1     Running   0          100m
kube-flannel-ds-amd64-5chsx            1/1     Running   0          109s
kube-flannel-ds-amd64-8bxpj            1/1     Running   0          112s
kube-flannel-ds-amd64-g4gh9            1/1     Running   0          26m
kube-proxy-cznqr                       1/1     Running   0          112s
kube-proxy-mcsdl                       1/1     Running   0          109s
kube-proxy-t7v46                       1/1     Running   0          100m
kube-scheduler-k8s-master01            1/1     Running   0          100m
[root@k8s-master01 flannel]# kubectl get pod -n kube-system -o wide
NAME                                   READY   STATUS    RESTARTS   AGE     IP                NODE           NOMINATED NODE   READINESS GATES
coredns-5c98db65d4-4kj2t               1/1     Running   0          101m    10.244.0.3        k8s-master01   <none>           <none>
coredns-5c98db65d4-7zsr7               1/1     Running   0          101m    10.244.0.2        k8s-master01   <none>           <none>
etcd-k8s-master01                      1/1     Running   0          100m    192.168.192.131   k8s-master01   <none>           <none>
kube-apiserver-k8s-master01            1/1     Running   0          100m    192.168.192.131   k8s-master01   <none>           <none>
kube-controller-manager-k8s-master01   1/1     Running   0          100m    192.168.192.131   k8s-master01   <none>           <none>
kube-flannel-ds-amd64-5chsx            1/1     Running   0          2m20s   192.168.192.129   k8s-node02     <none>           <none>
kube-flannel-ds-amd64-8bxpj            1/1     Running   0          2m23s   192.168.192.130   k8s-node01     <none>           <none>
kube-flannel-ds-amd64-g4gh9            1/1     Running   0          26m     192.168.192.131   k8s-master01   <none>           <none>
kube-proxy-cznqr                       1/1     Running   0          2m23s   192.168.192.130   k8s-node01     <none>           <none>
kube-proxy-mcsdl                       1/1     Running   0          2m20s   192.168.192.129   k8s-node02     <none>           <none>
kube-proxy-t7v46                       1/1     Running   0          101m    192.168.192.131   k8s-master01   <none>           <none>
kube-scheduler-k8s-master01            1/1     Running   0          100m    192.168.192.131   k8s-master01   <none>           <none>

[root@k8s-master01 flannel]# kubectl get node
NAME           STATUS   ROLES    AGE     VERSION
k8s-master01   Ready    master   104m    v1.15.1
k8s-node01     Ready    <none>   5m34s   v1.15.1
k8s-node02     Ready    <none>   5m31s   v1.15.1

到此k8s的1.15.1版本部署成功了，现在还不是高可用的
最后将重要文件保存到/usr/local/下，防止别删除

mv install-k8s/ /usr/local/

harbor主机配置

参考文档：https://github.com/WeiboGe2012/Data/tree/master/Linux/k8s/colony
设置ip地址
vi /etc/sysconfig/network-scripts/ifcfg-ens33
master主机

TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="ea3b61ed-9232-4b69-b6c0-2f863969e750"
DEVICE="ens33"
ONBOOT="yes"
IPADDR=192.168.192.128
NETMASK=255.255.255.0
GATEWAY=192.168.192.1
DNS1=192.168.192.1
DNS2=114.114.114.114

安装docker

卸载旧版本

$ sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

# 设置存储库
# 安装yum-utils包（提供yum-config-manager 实用程序）并设置稳定存储库。
$ sudo yum install -y yum-utils
$ sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

# 安装指定版本的 Docker Engine 和 containerd
$ sudo yum install -y docker-ce-18.09.9 docker-ce-cli-18.09.9 containerd.io

启动docker

$ sudo systemctl start docker

通过运行hello-world 映像验证 Docker Engine 是否已正确安装。

$ sudo docker run hello-world

设置开机定时启动docker

$ sudo systemctl enable docker

关闭docker服务

$ sudo systemctl stop docker

重启docker服务

$ sudo systemctl restart  docker

验证

[root@k8s-master01 ~]# docker version
Client:
 Version:           18.09.9
 API version:       1.39
 Go version:        go1.11.13
 Git commit:        039a7df9ba
 Built:             Wed Sep  4 16:51:21 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.11.13
  Git commit:       039a7df
  Built:            Wed Sep  4 16:22:32 2019
  OS/Arch:          linux/amd64
  Experimental:     false

# 创建/etc/docker目录
mkdir /etc/docker

# 配置daemon
cat > /etc/docker/daemon.json <<EOF
{
	"exec-opts": ["native.cgroupdriver=systemd"],
	"log-driver": "json-file",
	"log-opts": {
		"max-size": "100m"
	},
	"insecure-registries": ["https://hub.atguigu.com"]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# 重启docker服务
systemctl daemon-reload && systemctl restart docker && systemctl enable docker

安装docker-compose

下载

sudo curl -L "https://get.daocloud.io/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

设置权限

sudo chmod +x /usr/local/bin/docker-compose

验证

docker-compose --version

下载harborhttps://github.com/goharbor/harbor/releases

wget https://github.91chi.fun//https://github.com//goharbor/harbor/releases/download/v1.10.11/harbor-offline-installer-v1.10.11.tgz

tar -zxvf harbor-offline-installer-v1.10.11.tgz

mv harbor /usr/local/

cd /usr/local/harbor/

vi harbor.yml

harbor.yml

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: hub.atguigu.com  # 修改点1，通讯地址

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
https:
  # https port for harbor, default is 443
  port: 443
  # The path of cert and key files for nginx
  certificate: /data/cert/server.crt    # 修改并创建/data/cert路径
  private_key: /data/cert/server.key    # 修改并创建/data/cert路径

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345     # harbor登录admin账户密码

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 50
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 100 for postgres.
  max_open_conns: 100

# The default data volume
data_volume: /data      # 数据存储地址，没有需要创建目录

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

# Clair configuration
clair:
  # The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
  updaters_interval: 12

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 1.10.0

# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   clair:
#     host: clair_db_host
#     port: clair_db_port
#     db_name: clair_db_name
#     username: clair_db_username
#     password: clair_db_password
#     ssl_mode: disable
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# Uncomment external_redis if using external Redis server
# external_redis:
#   host: redis
#   port: 6379
#   password:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   chartmuseum_db_index: 3
#   clair_db_index: 4

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
  http_proxy:
  https_proxy:
  # no_proxy endpoints will appended to 127.0.0.1,localhost,.local,.internal,log,db,redis,nginx,core,portal,postgresql,jobservice,registry,registryctl,clair,chartmuseum,notary-server
  no_proxy:
  components:
    - core
    - jobservice
    - clair

根据上面的创建目录

[root@localhost harbor]# mkdir -p /data/cert
[root@localhost harbor]# cd !$
cd /data/cert
[root@localhost cert]#

创建证书

# 生成私钥
[root@localhost cert]# openssl genrsa -des3 -out server.key 2048
Generating RSA private key, 2048 bit long modulus
....................................................................................+++
...........+++
e is 65537 (0x10001)
Enter pass phrase for server.key:					# 输入两次一样的密码
Verifying - Enter pass phrase for server.key:

[root@localhost cert]# openssl req -new -key server.key -out server.csr
Enter pass phrase for server.key:
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN										# 国家
State or Province Name (full name) []:BJ									# 省份
Locality Name (eg, city) [Default City]:BJ									# 城市
Organization Name (eg, company) [Default Company Ltd]:atguigu				# 组织
Organizational Unit Name (eg, section) []:atguigu							# 机构
Common Name (eg, your name or your server's hostname) []:hub.atguigu.com	# 域名
Email Address []:wuyongtao8@126.com											# 管理员邮箱

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:													# 是否改密码，直接回车
An optional company name []:												# 是否改密码，直接回车


# 备份私钥
[root@localhost cert]# cp server.key server.key.org

# 转换成证书，把密码退掉
[root@localhost cert]# openssl rsa -in server.key.org -out server.key
Enter pass phrase for server.key.org:
writing RSA key

# 拿证书请求去签名
[root@localhost cert]# openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt
Signature ok
subject=/C=CN/ST=BJ/L=BJ/O=atguigu/OU=atguigu/CN=hub.atguigu.com/emailAddress=wuyongtao8@126.com
Getting Private key

# 把证书赋予权限
[root@localhost cert]# chmod a+x *

# 运行脚本安装
[root@localhost cert]# cd /usr/local/harbor/
[root@localhost harbor]# ls
common.sh  harbor.v1.10.11.tar.gz  harbor.yml  install.sh  LICENSE  prepare
# 下面这部遇到问题5
[root@localhost harbor]# ./install.sh

到master、node01、node02三台主机，添加harbor主机域名

[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.192.131 k8s-master01
192.168.192.130 k8s-node01
192.168.192.129 k8s-node02
[root@k8s-master01 ~]# echo "192.168.192.128 hub.atguigu.com" >> /etc/hosts
[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.192.131 k8s-master01
192.168.192.130 k8s-node01
192.168.192.129 k8s-node02
192.168.192.128 hub.atguigu.com

harbor主机也添加上域名

[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.192.131 k8s-master01
192.168.192.130 k8s-node01
192.168.192.129 k8s-node02
192.168.192.128 hub.atguigu.com

修改Windows本地hosts文件，添加域名
C:\Windows\System32\drivers\etc\hosts

192.168.192.128 hub.atguigu.com

harbor主机，查看运行的容器

[root@localhost harbor]# docker ps -a
CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS                      PORTS                                         NAMES
cf5d0df2935f        goharbor/nginx-photon:v1.10.11         "nginx -g 'daemon of…"   48 seconds ago      Up 46 seconds (healthy)     0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp   nginx
5f373c689525        goharbor/harbor-jobservice:v1.10.11    "/harbor/harbor_jobs…"   48 seconds ago      Up 46 seconds (healthy)                                                   harbor-jobservice
242b4f35f322        goharbor/harbor-core:v1.10.11          "/harbor/harbor_core"    48 seconds ago      Up 47 seconds (healthy)                                                   harbor-core
6fc46205eccb        goharbor/harbor-registryctl:v1.10.11   "/home/harbor/start.…"   50 seconds ago      Up 48 seconds (healthy)                                                   registryctl
8ca6e340e8b5        goharbor/harbor-db:v1.10.11            "/docker-entrypoint.…"   50 seconds ago      Up 48 seconds (healthy)     5432/tcp                                      harbor-db
bed1ed36df00        goharbor/redis-photon:v1.10.11         "redis-server /etc/r…"   50 seconds ago      Up 48 seconds (healthy)     6379/tcp                                      redis
42f03bcc4fb8        goharbor/harbor-portal:v1.10.11        "nginx -g 'daemon of…"   51 seconds ago      Up 48 seconds (healthy)     8080/tcp                                      harbor-portal
0647d52988cf        goharbor/registry-photon:v1.10.11      "/home/harbor/entryp…"   51 seconds ago      Up 48 seconds (healthy)     5000/tcp                                      registry
229aa32bbc70        goharbor/harbor-log:v1.10.11           "/bin/sh -c /usr/loc…"   51 seconds ago      Up 50 seconds (healthy)     127.0.0.1:1514->10514/tcp                     harbor-log
f349984bf935        91d0ab894aff                           "stf local --public-…"   6 months ago        Exited (1) 6 months ago                                                   stf
3b7be288d1ff        7123ee61b746                           "/sbin/tini -- adb -…"   6 months ago        Exited (143) 6 months ago                                                 adbd
a4bfb45049e4        2a54dcb95502                           "rethinkdb --bind al…"   6 months ago        Exited (0) 6 months ago                                                   rethinkdb

打开浏览器访问：hub.atguigu.com
在这里插入图片描述
输入账户：admin 密码：Harbor12345

验证node01登录harbor

[root@k8s-node01 ~]# docker login https://hub.atguigu.com
Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

[root@k8s-node01 ~]# docker pull wangyanglinux/myapp:v1
v1: Pulling from wangyanglinux/myapp
550fe1bea624: Pull complete 
af3988949040: Pull complete 
d6642feac728: Pull complete 
c20f0a205eaa: Pull complete 
fe78b5db7c4e: Pull complete 
6565e38e67fe: Pull complete 
Digest: sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513
Status: Downloaded newer image for wangyanglinux/myapp:v1

在这里插入图片描述

格式：docker tag SOURCE_IMAGE[:TAG] hub.atguigu.com/library/IMAGE[:TAG]
修改镜像标签，并推送

[root@k8s-node01 ~]# docker tag wangyanglinux/myapp:v1 hub.atguigu.com/library/myapp:v1
[root@k8s-node01 ~]# docker push hub.atguigu.com/library/myapp:v1
The push refers to repository [hub.atguigu.com/library/myapp]
a0d2c4392b06: Pushed 
05a9e65e2d53: Pushed 
68695a6cfd7d: Pushed 
c1dc81a64903: Pushed 
8460a579ab63: Pushed 
d39d92664027: Pushed 
v1: digest: sha256:9eeca44ba2d410e54fccc54cbe9c021802aa8b9836a0bcf3d3229354e4c8870e size: 1569

查看推送成功
在这里插入图片描述
删除镜像

[root@k8s-node01 ~]# docker rmi -f hub.atguigu.com/library/myapp:v1       
Untagged: hub.atguigu.com/library/myapp:v1
Untagged: hub.atguigu.com/library/myapp@sha256:9eeca44ba2d410e54fccc54cbe9c021802aa8b9836a0bcf3d3229354e4c8870e
[root@k8s-node01 ~]# docker rmi -f wangyanglinux/myapp:v1                 
Untagged: wangyanglinux/myapp:v1
Untagged: wangyanglinux/myapp@sha256:9c3dc30b5219788b2b8a4b065f548b922a34479577befb54b03330999d30d513
Deleted: sha256:d4a5e0eaa84f28550cb9dd1bde4bfe63a93e3cf88886aa5dad52c9a75dd0e6a9
Deleted: sha256:bf5594a16c1ff32ffe64a68a92ebade1080641f608d299170a2ae403f08764e7
Deleted: sha256:b74f3c20dd90bf6ead520265073c4946461baaa168176424ea7aea1bc7f08c1f
Deleted: sha256:8943f94f7db615e453fa88694440f76d65927fa18c6bf69f32ebc9419bfcc04a
Deleted: sha256:2020231862738f8ad677bb75020d1dfa34159ad95eef10e790839174bb908908
Deleted: sha256:49757da6049113b08246e77f770f49b1d50bb97c93f19d2eeae62b485b46e489
Deleted: sha256:d39d92664027be502c35cf1bf464c726d15b8ead0e3084be6e252a161730bc82

在这里插入图片描述

docker pull hub.atguigu.com/library/myapp:v1
在master主机上

[root@k8s-master01 ~]# kubectl run nginx-deployment --image=hub.atguigu.com/library/myapp:v1 --prot=80 --replicas=1
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
deployment.apps/nginx-deployment created

[root@k8s-master01 ~]# kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1/1     1            1           3m32s
[root@k8s-master01 ~]# kubectl get rs
NAME                          DESIRED   CURRENT   READY   AGE
nginx-deployment-78b46578cd   1         1         1       4m39s
[root@k8s-master01 ~]# kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-78b46578cd-ttnkc   1/1     Running   0          4m48s

# 可以看到是在node01接点上运行
[root@k8s-master01 ~]# kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
nginx-deployment-78b46578cd-ttnkc   1/1     Running   0          5m19s   10.244.1.2   k8s-node01   <none>           <none>

# 直接访问是通的
[root@k8s-master01 ~]# curl 10.244.1.2
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>
[root@k8s-master01 ~]# curl 10.244.1.2/hostname.html 
nginx-deployment-78b46578cd-ttnkc

我们就到node01查看到hub.atguigu.com/library/myapp 在运行

[root@k8s-node01 ~]# docker ps
CONTAINER ID        IMAGE                           COMMAND                  CREATED             STATUS              PORTS               NAMES
ead8b7b658d0        hub.atguigu.com/library/myapp   "nginx -g 'daemon of…"   6 minutes ago       Up 6 minutes                            k8s_nginx-deployment_nginx-deployment-78b46578cd-ttnkc_default_aba645b2-b0bc-40df-bd3f-74e872c5eb3e_0
2e9604fe6814        k8s.gcr.io/pause:3.1            "/pause"                 6 minutes ago       Up 6 minutes                            k8s_POD_nginx-deployment-78b46578cd-ttnkc_default_aba645b2-b0bc-40df-bd3f-74e872c5eb3e_0
14144597ee6c        4e9f801d2217                    "/opt/bin/flanneld -…"   6 hours ago         Up 6 hours                              k8s_kube-flannel_kube-flannel-ds-amd64-8bxpj_kube-system_cf50e169-2798-496b-ac94-901ae02fc836_3
da5aa7976f63        89a062da739d                    "/usr/local/bin/kube…"   6 hours ago         Up 6 hours                              k8s_kube-proxy_kube-proxy-cznqr_kube-system_4146aaa5-e985-45bb-9f42-871c7671eea2_2
db9c540c9d69        k8s.gcr.io/pause:3.1            "/pause"                 6 hours ago         Up 6 hours                              k8s_POD_kube-proxy-cznqr_kube-system_4146aaa5-e985-45bb-9f42-871c7671eea2_3
b0c135210d4f        k8s.gcr.io/pause:3.1            "/pause"                 6 hours ago         Up 6 hours                              k8s_POD_kube-flannel-ds-amd64-8bxpj_kube-system_cf50e169-2798-496b-ac94-901ae02fc836_2

在这里插入图片描述
删除pod，就自动添加了一个pod

[root@k8s-master01 ~]# kubectl delete pod nginx-deployment-78b46578cd-ttnkc
pod "nginx-deployment-78b46578cd-ttnkc" deleted
[root@k8s-master01 ~]# kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-78b46578cd-vvkd6   1/1     Running   0          29s

进行扩容

[root@k8s-master01 ~]# kubectl scale --replicas=3 deployment/nginx-deployment
deployment.extensions/nginx-deployment scaled
[root@k8s-master01 ~]# kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-78b46578cd-45ndq   1/1     Running   0          11s
nginx-deployment-78b46578cd-r627l   1/1     Running   0          11s
nginx-deployment-78b46578cd-vvkd6   1/1     Running   0          3m17s
[root@k8s-master01 ~]# kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
nginx-deployment-78b46578cd-45ndq   1/1     Running   0          50s     10.244.2.2   k8s-node02   <none>           <none>
nginx-deployment-78b46578cd-r627l   1/1     Running   0          50s     10.244.1.4   k8s-node01   <none>           <none>
nginx-deployment-78b46578cd-vvkd6   1/1     Running   0          3m56s   10.244.1.3   k8s-node01   <none>           <none>

删除一个后再查看依然还是三个

[root@k8s-master01 ~]# kubectl delete pod nginx-deployment-78b46578cd-45ndq
pod "nginx-deployment-78b46578cd-45ndq" deleted
[root@k8s-master01 ~]# kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
nginx-deployment-78b46578cd-4g4cb   1/1     Running   0          34s     10.244.2.3   k8s-node02   <none>           <none>
nginx-deployment-78b46578cd-r627l   1/1     Running   0          2m30s   10.244.1.4   k8s-node01   <none>           <none>
nginx-deployment-78b46578cd-vvkd6   1/1     Running   0          5m36s   10.244.1.3   k8s-node01   <none>           <none>

[root@k8s-master01 ~]# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   23h
[root@k8s-master01 ~]# kubectl get deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           22m
[root@k8s-master01 ~]# kubectl expose deployment nginx-deployment --port=30000 --target-port=80
service/nginx-deployment exposed
[root@k8s-master01 ~]# kubectl get svc
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
kubernetes         ClusterIP   10.96.0.1      <none>        443/TCP     23h
nginx-deployment   ClusterIP   10.97.63.227   <none>        30000/TCP   39s
[root@k8s-master01 ~]# curl 10.97.63.227:30000
Hello MyApp | Version: v1 | <a href="hostname.html">Pod Name</a>

# 轮询的访问
[root@k8s-master01 ~]# curl 10.97.63.227:30000/hostname.html
nginx-deployment-78b46578cd-r627l
[root@k8s-master01 ~]# curl 10.97.63.227:30000/hostname.html
nginx-deployment-78b46578cd-vvkd6
[root@k8s-master01 ~]# curl 10.97.63.227:30000/hostname.html
nginx-deployment-78b46578cd-4g4cb
[root@k8s-master01 ~]# curl 10.97.63.227:30000/hostname.html
nginx-deployment-78b46578cd-r627l
[root@k8s-master01 ~]# ipvsadm -Ln | grep 10.97.63.227
TCP  10.97.63.227:30000 rr
[root@k8s-master01 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.1:443 rr
  -> 192.168.192.131:6443         Masq    1      3          0         
TCP  10.96.0.10:53 rr
  -> 10.244.0.6:53                Masq    1      0          0         
  -> 10.244.0.7:53                Masq    1      0          0         
TCP  10.96.0.10:9153 rr
  -> 10.244.0.6:9153              Masq    1      0          0         
  -> 10.244.0.7:9153              Masq    1      0          0         
TCP  10.97.63.227:30000 rr
  -> 10.244.1.3:80                Masq    1      0          2         
  -> 10.244.1.4:80                Masq    1      0          2         
  -> 10.244.2.3:80                Masq    1      0          2         
UDP  10.96.0.10:53 rr
  -> 10.244.0.6:53                Masq    1      0          0         
  -> 10.244.0.7:53                Masq    1      0          0        
 
# IP和上面的一样 
[root@k8s-master01 ~]# kubectl get pod -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
nginx-deployment-78b46578cd-4g4cb   1/1     Running   0          13m   10.244.2.3   k8s-node02   <none>           <none>
nginx-deployment-78b46578cd-r627l   1/1     Running   0          15m   10.244.1.4   k8s-node01   <none>           <none>
nginx-deployment-78b46578cd-vvkd6   1/1     Running   0          18m   10.244.1.3   k8s-node01   <none>           <none>

外网不能访问问题

[root@k8s-master01 ~]# kubectl get svc
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
kubernetes         ClusterIP   10.96.0.1      <none>        443/TCP     24h
nginx-deployment   ClusterIP   10.97.63.227   <none>        30000/TCP   10m
[root@k8s-master01 ~]# kubectl edit svc nginx-deployment

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-05-26T11:48:32Z"
  labels:
    run: nginx-deployment
  name: nginx-deployment
  namespace: default
  resourceVersion: "72884"
  selfLink: /api/v1/namespaces/default/services/nginx-deployment
  uid: 22af8fb4-9580-4918-9040-ac1de01e39d3
spec:
  clusterIP: 10.97.63.227
  ports:
  - port: 30000
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx-deployment
  sessionAffinity: None
  type: NodePort     # 修改成NodePort
status:
  loadBalancer: {}

[root@k8s-master01 ~]# kubectl get svc
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)           AGE
kubernetes         ClusterIP   10.96.0.1      <none>        443/TCP           24h
nginx-deployment   NodePort    10.97.63.227   <none>        30000:30607/TCP   14m
[root@k8s-master01 ~]# netstat -anpt | grep :30607
tcp6       0      0 :::30607                :::*                    LISTEN      112386/kube-proxy   
[root@k8s-master01 ~]# netstat -anpt | grep :30607
tcp6       0      0 :::30607                :::*                    LISTEN      112386/kube-proxy

外网访问master主机地址加端口
在这里插入图片描述
node01的访问

node02的访问

这样k8s的测试，和仓库的私有连接就完成了

遇到问题1：浏览器192.168.1.1访问不了

解决办法：关闭所有的虚拟机，还原网络配置
在这里插入图片描述

遇到问题2：

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
解决办法：docker少创建了一个目录，环境部署仔细捋一遍

遇到问题3：

[root@k8s-master01 ~]# kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log
Flag --experimental-upload-certs has been deprecated, use --upload-certs instead
[init] Using Kubernetes version: v1.15.1
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.16. Latest validated version: 18.09
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2

解决办法：安装指定版本的docker，参考1：https://blog.csdn.net/mayi_xiaochaun/article/details/123421532
卸载docker，参考2：https://blog.csdn.net/wujian_csdn_csdn/article/details/122421103

遇到问题4：

[root@k8s-master01 ~]# kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log
Flag --experimental-upload-certs has been deprecated, use --upload-certs instead
[init] Using Kubernetes version: v1.15.1
[preflight] Running pre-flight checks
        [WARNING Service-Docker]: docker service is not enabled, please run 'systemctl enable docker.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR NumCPU]: the number of available CPUs 1 is less than the required 2
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

解决办法：虚拟机添加cpu，必须最低保证两个cpu

遇到问题5：

[root@localhost harbor]# ./install.sh 
...
...
...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating harbor-portal ... 
Creating registryctl   ... 
Creating registry      ... error
Creating harbor-portal ... done
Creating registryctl   ... done
ERROR: for registry  Cannot create container for service registry: Conflict. The container name "/registry" is already in use by container "3f42e1c7dd80b96b59848ac10698ef3f5537afeedb718a424dd91d13bb55440b". You have to remove (or renCreating redis         ... done
Creating harbor-db     ... done

ERROR: for registry  Cannot create container for service registry: Conflict. The container name "/registry" is already in use by container "3f42e1c7dd80b96b59848ac10698ef3f5537afeedb718a424dd91d13bb55440b". You have to remove (or rename) that container to be able to reuse that name.
ERROR: Encountered errors while bringing up the project.

解决办法：删除registry容器和镜像

[root@localhost harbor]# docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
goharbor/chartmuseum-photon     v1.10.11            d00df92a5e3e        3 weeks ago         164MB
goharbor/redis-photon           v1.10.11            aa57c8e9fa46        3 weeks ago         151MB
goharbor/clair-adapter-photon   v1.10.11            e87900ea4eb9        3 weeks ago         66.1MB
goharbor/clair-photon           v1.10.11            03cd37f2ca5d        3 weeks ago         178MB
goharbor/notary-server-photon   v1.10.11            801719b38205        3 weeks ago         105MB
goharbor/notary-signer-photon   v1.10.11            005e711802d6        3 weeks ago         102MB
goharbor/harbor-registryctl     v1.10.11            fd34fcc88f68        3 weeks ago         93.4MB
goharbor/registry-photon        v1.10.11            c7076a9bc40b        3 weeks ago         78.6MB
goharbor/nginx-photon           v1.10.11            68e6d0e1c018        3 weeks ago         45MB
goharbor/harbor-log             v1.10.11            06df11c5e8f3        3 weeks ago         108MB
goharbor/harbor-jobservice      v1.10.11            f7d878b39e41        3 weeks ago         84.7MB
goharbor/harbor-core            v1.10.11            69d4874721a3        3 weeks ago         79.6MB
goharbor/harbor-portal          v1.10.11            83b24472c7c8        3 weeks ago         53.1MB
goharbor/harbor-db              v1.10.11            11278dbcadf4        3 weeks ago         188MB
goharbor/prepare                v1.10.11            66d60732b8ff        3 weeks ago         206MB
nginx                           latest              04661cdce581        6 months ago        141MB
rethinkdb                       latest              2a54dcb95502        7 months ago        131MB
192.168.111.129:5000/demo       latest              40fc65df2cf9        14 months ago       660MB
demo                            1.0-SNAPSHOT        40fc65df2cf9        14 months ago       660MB
registry                        latest              678dfa38fcfa        17 months ago       26.2MB
openstf/ambassador              latest              938a816f078a        21 months ago       8.63MB
openstf/stf                     latest              91d0ab894aff        21 months ago       958MB
sorccu/adb                      latest              7123ee61b746        4 years ago         30.5MB
java                            8                   d23bdf5b1b1b        5 years ago         643MB
[root@localhost harbor]# docker rmi -f registry:latest
或者
[root@localhost harbor]# docker rmi 678dfa38fcfa
Error response from daemon: conflict: unable to delete 678dfa38fcfa (must be forced) - image is being used by stopped container 3f42e1c7dd80
[root@localhost harbor]# docker rm 3f42e1c7dd80
3f42e1c7dd80
[root@localhost harbor]# docker rmi 678dfa38fcfa
Error: No such image: 678dfa38fcfa

[root@localhost harbor]# cd /var/lib/docker/image/overlay2/imagedb/content/sha256/
[root@localhost sha256]# ll
总用量 188
-rw-------. 1 root root 4265 5月  26 17:48 005e711802d665903b9216e8c46b46676ad9c2e43ef48911a33f0bf9dbd30a06
-rw-------. 1 root root 4791 5月  26 17:48 03cd37f2ca5d16745270fef771c60cd502ec9559d332d6601f5ab5e9f41e841a
-rw-------. 1 root root 7738 11月 11 2021 04661cdce5812210bac48a8af672915d0719e745414b4c322719ff48c7da5b83
-rw-------. 1 root root 4789 5月  26 17:48 06df11c5e8f3cf1963c236327434cbfe2f5f3d9a9798c487d4e6c8ba1742e5fe
-rw-------. 1 root root 5725 5月  26 17:48 11278dbcadf4651433c7432427bd4877b6840c5d21454c3bf266c12c1d1dd559
-rw-------. 1 root root 5640 3月  18 2021 27fdf005f8f0ddc7837fcb7c1dd61cffdb37f14d964fb4edccfa4e6b60f6e339
-rw-------. 1 root root 3364 11月 11 2021 2a54dcb95502386ca01cdec479bea40b3eacfefe7019e6bb4883eff04d35b883
-rw-------. 1 root root 5771 3月  18 2021 40fc65df2cf909b24416714b4640fa783869481a4eebf37f7da8cbf1f299b2ab
-rw-------. 1 root root 5062 3月  18 2021 4e1d920c1cd61aeaf44f7e17b2dfc3dcb24414770a3955bbed61e53c39b90232
-rw-------. 1 root root 3526 5月  26 17:48 66d60732b8ff10d2f2d237a68044f11fc87497606cf8c5feae2adf6433f3d946
-rw-------. 1 root root 3114 3月  18 2021 678dfa38fcfa349ccbdb1b6d52ac113ace67d5746794b36dfbad9dd96a9d1c43
-rw-------. 1 root root 3501 5月  26 17:48 68e6d0e1c018d66a4cab78d6c2c46c23794e8a06b37e195e8c8bcded2ecc82d2
-rw-------. 1 root root 4058 5月  26 17:48 69d4874721a38f06228c4a2e407e4c82a8847b61b51c2d50a6e3fc645635c233
-rw-------. 1 root root 4284 11月 11 2021 7123ee61b7468b74c8de67a6dfb40c61840f5b910fee4105e38737271741582f
-rw-------. 1 root root 4263 5月  26 17:48 801719b38205038d5a145380a1954481ed4e8a340e914ef7819f0f7bed395df6
-rw-------. 1 root root 4814 5月  26 17:48 83b24472c7c8cc71effc362949d4fa04e3e0a6e4a8f2a248f5706513cbcdfa0a
-rw-------. 1 root root 5316 3月  18 2021 8f36ad1be230ab68cc8337931358734ffedd968a076a9522cd88359fbf4af98d
-rw-------. 1 root root 6501 11月 11 2021 91d0ab894affa1430efb6bd130edb360d9600132bb4187a6cabe403d8ef98bdd
-rw-------. 1 root root 2070 11月 11 2021 938a816f078a0b27753a0f690518754bfbb47e0bb57e61275600757845d7e4b1
-rw-------. 1 root root 3751 5月  26 17:48 aa57c8e9fa464f8124a753facda552af0a79f5aa169901f357a9003c8a65d9c5
-rw-------. 1 root root 5204 3月  18 2021 bb56dc5d8cbd1f41b9e9bc7de1df15354a204cf17527790e13ac7d0147916dd6
-rw-------. 1 root root 4347 5月  26 17:48 c7076a9bc40b43db80afb4699e124c0dd1c777825bd36d504a2233028421a178
-rw-------. 1 root root 4858 3月  18 2021 c70a4b1ffafa08a73e469ed0caa6693111aad55d34df8df7eea6bae0fb542547
-rw-------. 1 root root 4592 5月  26 17:48 d00df92a5e3e330c2231d9f54a0a0c320c420946e04afcfe02ad596d778a8370
-rw-------. 1 root root 4733 3月  18 2021 d23bdf5b1b1b1afce5f1d0fd33e7ed8afbc084b594b9ccf742a5b27080d8a4a8
-rw-------. 1 root root 3715 5月  26 17:48 e87900ea4eb9beafd8264e874f71fcb72ce1df827e999d6bb7b2e47ebb0ca5e4
-rw-------. 1 root root 3677 5月  26 17:48 f7d878b39e411b345d4eb263205cde950ad59ba3102d4ac1079238ef1b3de903
-rw-------. 1 root root 4632 5月  26 17:48 fd34fcc88f68fe44a06c34d3a754f70fd36ee6174e64e65cb1072f0d4a9826d0
[root@localhost sha256]# rm -rf 678dfa38fcfa349ccbdb1b6d52ac113ace67d5746794b36dfbad9dd96a9d1c43
[root@localhost sha256]# docker images
REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
goharbor/chartmuseum-photon     v1.10.11            d00df92a5e3e        3 weeks ago         164MB
goharbor/redis-photon           v1.10.11            aa57c8e9fa46        3 weeks ago         151MB
goharbor/clair-adapter-photon   v1.10.11            e87900ea4eb9        3 weeks ago         66.1MB
goharbor/clair-photon           v1.10.11            03cd37f2ca5d        3 weeks ago         178MB
goharbor/notary-server-photon   v1.10.11            801719b38205        3 weeks ago         105MB
goharbor/notary-signer-photon   v1.10.11            005e711802d6        3 weeks ago         102MB
goharbor/harbor-registryctl     v1.10.11            fd34fcc88f68        3 weeks ago         93.4MB
goharbor/registry-photon        v1.10.11            c7076a9bc40b        3 weeks ago         78.6MB
goharbor/nginx-photon           v1.10.11            68e6d0e1c018        3 weeks ago         45MB
goharbor/harbor-log             v1.10.11            06df11c5e8f3        3 weeks ago         108MB
goharbor/harbor-jobservice      v1.10.11            f7d878b39e41        3 weeks ago         84.7MB
goharbor/harbor-core            v1.10.11            69d4874721a3        3 weeks ago         79.6MB
goharbor/harbor-portal          v1.10.11            83b24472c7c8        3 weeks ago         53.1MB
goharbor/harbor-db              v1.10.11            11278dbcadf4        3 weeks ago         188MB
goharbor/prepare                v1.10.11            66d60732b8ff        3 weeks ago         206MB
nginx                           latest              04661cdce581        6 months ago        141MB
rethinkdb                       latest              2a54dcb95502        7 months ago        131MB
192.168.111.129:5000/demo       latest              40fc65df2cf9        14 months ago       660MB
demo                            1.0-SNAPSHOT        40fc65df2cf9        14 months ago       660MB
openstf/ambassador              latest              938a816f078a        21 months ago       8.63MB
openstf/stf                     latest              91d0ab894aff        21 months ago       958MB
sorccu/adb                      latest              7123ee61b746        4 years ago         30.5MB
java                            8                   d23bdf5b1b1b        5 years ago         643MB