pod容器与镜像管理和k8s私有仓库harbor搭建

前言

一 . pod容器与镜像管理

1.1：pod的容器分类与镜像拉取策略
pod在k8s中是：

1、最小部署单页
2、一组容器的集合
3、一个pod中的容器共享网络命名空间
4、pod是短暂的

pod的容器分类：

1、infrastructure container：基础容器

维护整个pod网络空间：可以在node节点操作查看容器的网络

[root@node01 ~]# cat /opt/k8s/cfg/kubelet

KUBELET_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=192.168.233.132 \
--kubeconfig=/opt/k8s/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/k8s/cfg/bootstrap.kubeconfig \
--config=/opt/k8s/cfg/kubelet.config \
--cert-dir=/opt/k8s/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"	'//是基础容器'

2、initcontainers：初始化容器

先于业务容器开始执行，原先pod中容器是并行开启，现在进行了改进
无论容器写在初始化容器前还是写在初始化容器后，最先执行的都是初始化容器。只有初始化容器执行成功后才可以启动容器。
初始化容器的应用场景一般是多容器，例如：mysql和业务分开两个容器。将业务设为初始化容器，并检查mysql是否启动，若mysql启动，则业务容器启动；否则业务容器等待mysql启动。
3、container：业务容器

业务容器就是我们创建的pod资源内的容器服务，业务容器也叫APP容器，并行启动
镜像拉取策略（image PullPolicy）

1、ifnotpresent：默认值，镜像在宿主机上不存在时会拉取

2、always：每次创建pod都会重新拉取一次镜像

3、never：pod永远不会主动拉取这个镜像

查看镜像拉取策略（master节点查看）：

[root@master ~]# kubectl get pod
NAME                        READY   STATUS    RESTARTS   AGE
nginx-dbddb74b8-5s6h7       1/1     Running   1          10d
nginx-test-d55b94fd-9zmdj   1/1     Running   0          27h
nginx-test-d55b94fd-b8lkl   1/1     Running   0          27h
nginx-test-d55b94fd-w4c5k   1/1     Running   0          27h
[root@master ~]# kubectl edit deploy/nginx

尝试编辑一个pod并指定拉去策略

[root@master ~]# cd test/
[root@master test]# ls
nginx-service-test.yaml  nginx-test02.yaml
nginx-test01.yaml        nginx-test.yaml
[root@master test]# cat > pod1-test.yaml <<EOF
> apiVersion: v1
> kind: Pod
> metadata:
>     name: mypod
> spec:
>     containers:
>       - name: nginx
>         image: nginx:1.14
>         imagePullPolicy: Always
> EOF
[root@master test]# kubectl create -f pod1-test.yaml 	'//如果需要更新容器，需要删除原先的容器：kubectl delete -f pod1-test.yaml，修改yaml文件后使用apply命令重新部署：kubectl apply -f pod1-test.yaml '
pod/mypod created
[root@master test]# kubectl get pod 
NAME                        READY   STATUS    RESTARTS   AGE
mypod                       1/1     Running   0          6m
nginx-dbddb74b8-5s6h7       1/1     Running   1          10d
nginx-test-d55b94fd-9zmdj   1/1     Running   0          27h
nginx-test-d55b94fd-b8lkl   1/1     Running   0          27h
nginx-test-d55b94fd-w4c5k   1/1     Running   0          27h

查看容器详细信息：kubectl describe pod 名称

[root@master test]# kubectl describe pod mypod
Name:               mypod
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               192.168.233.132/192.168.233.132	'//资源被创建在这个ip的node节点上'
Start Time:         Mon, 11 May 2020 19:27:58 +0800
Labels:             <none>
Annotations:        <none>
Status:             Running
IP:                 172.17.26.5	'//可以查看到ip'
...省略信息

可以在相应node节点访问容器

[root@node01 ~]# curl -I 172.17.26.5	'//可以查看到相应的信息'
HTTP/1.1 200 OK
Server: nginx/1.14.2
Date: Mon, 11 May 2020 11:35:54 GMT
Content-Type: text/html
Content-Length: 612
Last-Modified: Tue, 04 Dec 2018 14:44:49 GMT
Connection: keep-alive
ETag: "5c0692e1-264"
Accept-Ranges: bytes

二 . 安装harbor 私有仓库

docker 环境部署

设置阿里云镜像

[root@server 11 ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
设置阿里云镜像


yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
已加载插件：fastestmirror, langpacks
adding repo from: https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
grabbing file https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo to /etc/yum.repos.d/docker-ce.repo
repo saved to /etc/yum.repos.d/docker-ce.repo


[root@server 11 ~]# vim /etc/sysctl.conf
net.ipv4.ip_forward=1
优化
[root@server 11 ~]# sysctl -p
net.ipv4.ip_forward = 1

加速优化 阿里云镜像加速


sudo mkdir -p /etc/docker 
sudo tee /etc/docker/daemon.json <<-'EOF'
 { 
"registry-mirrors": ["https://*******.mirror.aliyuncs.com"]
 } 
EOF
 sudo systemctl daemon-reload 

systemctl restart network

下载安装docker

[root@server 11 ~]# yum install -y docker-ce

sudo systemctl restart docker
----------------------------------------------------------------------------------------

下载软件 docker-compose

[root@docker2 opt]# ls
  docker-compose 

[root@docker2 opt]# chmod +x docker-compose
[root@docker2 opt]# cp -p docker-compose  /usr/local/bin/

---

安装软件 harbor

[root@harbor soft]# tar zxvf harbor-offline-installer-v1.2.2.tgz -C /usr/local

[root@harbor soft]# vim /usr/local/harbor/harbor.cfg

启动docker

[root@harbor harbor]# systemctl start docker


[root@harbor soft]# cd /usr/local/harbor/
[root@harbor harbor]# ls
common                     docker-compose.yml     harbor.v1.2.2.tar.gz  NOTICE
docker-compose.clair.yml   harbor_1_1_0_template  install.sh            prepare
docker-compose.notary.yml  harbor.cfg             LICENSE               upgrade

改为参数需要运行此命令

[root@harbor harbor]# sh install.sh

Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating registry           ... done
Creating harbor-db          ... done
Creating harbor-adminserver ... done
Creating harbor-ui          ... done
Creating harbor-jobservice  ... done
Creating nginx              ... done

✔ ----Harbor has been installed and started successfully.----

Now you should be able to visit the admin portal at http://192.168.100.9.
For more details, please visit https://github.com/vmware/harbor .

网页登录容器

node节点配置连接私有仓库（注意后面的逗号要添加）

两台节点同样操作

[root@node1 ~]# vim /etc/docker/daemon.json

重启docker
[root@node2 ~]# systemctl restart docker

登录仓库

[root@node2 ~]#
[root@node2 ~]# docker login 192.168.100.9
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

查看凭证文件

[root@node2 ~]#
此时会在跟目录生成 docker 仓库凭据文件

[root@node2 ~]# cd .docker
[root@node2 .docker]# ls
config.json
64位解码文件，并不换行输出凭证文件
[root@node2 .docker]# cat config.json | base64 -w 0
ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEwMC45IjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NlNHRnlZbTl5TVRJek5EVT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjEzIChsaW51eCkiCgl9Cn0=[root@node2 .docker]#

[root@node2 .docker]# pwd
/root/.docker

在 master 上创建 安全认证资源，访问仓库时需要认证

[root@master demo]# vim registry-pull-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: registry-pull-secret
data:  .dockerconfigjson: ewoJImF1dGhzIjogewoJCSIxOTIuMTY4LjEwMC45IjogewoJCQkiYXV0aCI6ICJZV1J0YVc0NlNHRnlZbTl5TVRJek5EVT0iCgkJfQoJfSwKCSJIdHRwSGVhZGVycyI6IHsKCQkiVXNlci1BZ2VudCI6ICJEb2NrZXItQ2xpZW50LzE5LjAzLjEzIChsaW51eCkiCgl9Cn0=
type: kubernetes.io/dockerconfigjson

创建secret资源

[root@master demo]# kubectl create -f registry-pull-secret.yaml
secret/registry-pull-secret created

查看secret资源

[root@master demo]# kubectl get secret
NAME                   TYPE                                  DATA   AGE
default-token-rd8b7    kubernetes.io/service-account-token   3      2d7h
registry-pull-secret   kubernetes.io/dockerconfigjson        1      50s
[root@master demo]#

节点镜像的上传

从现网源下载镜像

[root@node1 ~]# docker pull tomcat
标记镜像
[root@node1 ~]# docker tag tomcat 192.168.100.9/sha/tomcat

上传

[root@node1 ~]# docker push 192.168.100.9/sha/tomcat
The push refers to repository [192.168.100.9/sha/tomcat]
b654a29de9ee: Pushed

在另一节点下载镜像

[root@node2 .docker]# docker pull 192.168.100.9/sha/tomcat:latest
latest: Pulling from sha/tomcat
57df1a1f1ad8: Pull complete
71e126169501: Pull complete
1af28a55c3f3: Pull complete
03f1c9932170: Pull complete
881ad7aafb13: Pull complete
9c0ffd4062f3: Pull complete
bd62e479351a: Pull complete
48ee8bc64dbc: Pull complete
07cb85cca4f0: Pull complete
6a78fac8d191: Pull complete
Digest: sha256:99c20ba4ab117d182a0aa2266123b2cfb425777495fd62e2ba37f489c3e2f808
Status: Downloaded newer image for 192.168.100.9/sha/tomcat:latest
192.168.100.9/sha/tomcat:latest
[root@node2 .docker]#

创建yaml 资源文件

Node 节点 192.168.100.6 下载镜像

[root@node2 .docker]# docker pull tomcat:8.0.52
8.0.52: Pulling from library/tomcat
1c7fe136a31e: Pull complete
ece825d3308b: Pull complete
122a54f77455: Pull complete
b0f58081abfa: Pull complete
d87948ea8b09: Pull complete
25934b035c41: Pull complete
c19ad0b452cb: Pull complete
97b2cf7bf1a2: Pull complete
5a118107a2f9: Pull complete
bf2397e2ae9f: Pull complete
4378950c2263: Pull complete
Digest: sha256:32d451f50c0f9e46011091adb3a726e24512002df66aaeecc3c3fd4ba6981bd4
Status: Downloaded newer image for tomcat:8.0.52
docker.io/library/tomcat:8.0.52
[root@node2 .docker]#

创建yaml 资源文件

[root@master demo]#
[root@master demo]# vim tomcat-deployment.yaml

kind: Deployment
metadata:
  name: my-tomcat
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: my-tomcat
    spec:
      containers:
      - name: my-tomcat
        image: docker.io/tomcat:8.0.52
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-tomcat
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    nodePort: 31111
  selector:
    app: my-tomcat

创建资源

[root@master demo]# kubectl create -f tomcat-deployment.yaml
deployment.extensions/my-tomcat created

查看资源

[root@master demo]# kubectl get all
NAME                                  READY   STATUS    RESTARTS   AGE
pod/my-tomcat-57667b9d9-dgkzk         1/1     Running   0          7m38s
pod/my-tomcat-57667b9d9-slhtv         1/1     Running   0          7m38s
pod/nginx-7697996758-gzqms            1/1     Running   1          31h
pod/nginx-7697996758-j6tfj            1/1     Running   1          31h
pod/nginx-7697996758-ldfvx            1/1     Running   1          31h
pod/nginx-deployment-d55b94fd-5zhjt   1/1     Running   1          30h
pod/nginx-deployment-d55b94fd-6f6hm   1/1     Running   1          30h
pod/nginx-deployment-d55b94fd-kr7c6   1/1     Running   1          30h

查看pod 详细信息，创建过程

[root@master demo]# kubectl describe pod/my-tomcat-57667b9d9-dgkzk
Name:               my-tomcat-57667b9d9-dgkzk
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               192.168.100.6/192.168.100.6
Start Time:         Mon, 12 Oct 2020 18:34:26 +0800
Labels:             app=my-tomcat
                    pod-template-hash=57667b9d9
Annotations:        <none>
Status:             Running
IP:                 172.17.71.6
Controlled By:      ReplicaSet/my-tomcat-57667b9d9
Containers:
  my-tomcat:
    Container ID:   docker://2b4d5af458fd70d013317b5615730a526a7397cf5a7cb4a6a276c6239a252087
    Image:          docker.io/tomcat:8.0.52
    Image ID:       docker-pullable://tomcat@sha256:32d451f50c0f9e46011091adb3a726e24512002df66aaeecc3c3fd4ba6981bd4
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 12 Oct 2020 18:34:26 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rd8b7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-rd8b7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rd8b7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                    Message
  ----    ------     ----   ----                    -------
  Normal  Scheduled  8m29s  default-scheduler       Successfully assigned default/my-tomcat-57667b9d9-dgkzk to 192.168.100.6
  Normal  Pulled     8m29s  kubelet, 192.168.100.6  Container image "docker.io/tomcat:8.0.52" already present on machine
  Normal  Created    8m29s  kubelet, 192.168.100.6  Created container
  Normal  Started    8m29s  kubelet, 192.168.100.6  Started container

查看另一个副本创建过程

[root@master demo]# kubectl describe pod/my-tomcat-57667b9d9-slhtv
Name:               my-tomcat-57667b9d9-slhtv
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               192.168.100.5/192.168.100.5
Start Time:         Mon, 12 Oct 2020 18:34:26 +0800
Labels:             app=my-tomcat
                    pod-template-hash=57667b9d9
Annotations:        <none>
Status:             Running
IP:                 172.17.22.5
Controlled By:      ReplicaSet/my-tomcat-57667b9d9
Containers:
  my-tomcat:
    Container ID:   docker://88287ca0724ae3dbc7b9d5b9ad09e9df7e5ccbd25858e9a10bebe64ab4153e08
    Image:          docker.io/tomcat:8.0.52
    Image ID:       docker-pullable://tomcat@sha256:32d451f50c0f9e46011091adb3a726e24512002df66aaeecc3c3fd4ba6981bd4
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Mon, 12 Oct 2020 18:35:08 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rd8b7 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-rd8b7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rd8b7
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                    Message
  ----    ------     ----   ----                    -------
  Normal  Scheduled  8m56s  default-scheduler       Successfully assigned default/my-tomcat-57667b9d9-slhtv to 192.168.100.5
  Normal  Pulling    8m56s  kubelet, 192.168.100.5  pulling image "docker.io/tomcat:8.0.52"
  Normal  Pulled     8m14s  kubelet, 192.168.100.5  Successfully pulled image "docker.io/tomcat:8.0.52"
  Normal  Created    8m14s  kubelet, 192.168.100.5  Created container
  Normal  Started    8m14s  kubelet, 192.168.100.5  Started container

总结：

发现node 节点存在镜像的会直接从本地获取，节省了创建时间。

kubectl edit在线修改资源配置方法

重新创建资源

[root@master demo]#  kubectl create -f tomcat-deployment.yaml
deployment.extensions/my-tomcat created
The Service "my-tomcat" is invalid: spec.ports[0].nodePort: Invalid value: 31111: provided port is already allocated
[root@master demo]# kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
my-tomcat-57667b9d9-467hk         1/1     Running   0          17s
my-tomcat-57667b9d9-6l6cd         1/1     Running   0          17s
nginx-7697996758-gzqms            1/1     Running   1          32h
nginx-7697996758-j6tfj            1/1     Running   1          32h
nginx-7697996758-ldfvx            1/1     Running   1          32h
nginx-deployment-d55b94fd-5zhjt   1/1     Running   1          31h
nginx-deployment-d55b94fd-6f6hm   1/1     Running   1          31h
nginx-deployment-d55b94fd-kr7c6   1/1     Running   1          31h
Edit cancelled, no changes made.
[root@master demo]# kubectl  get node
NAME            STATUS   ROLES    AGE    VERSION
192.168.100.5   Ready    <none>   2d9h   v1.12.3
192.168.100.6   Ready    <none>   2d9h   v1.12.3

在线修改资源配置

[root@master demo]# kubectl edit deployment/my-tomcat  -n default

可以直接进行修改，以 VIM 编辑器的方式进行操作

遇到 Terminating 状态的普通删除不掉，需要强制删除

发现目标 Terminating

[root@master demo]# kubectl get pods
NAME                              READY   STATUS        RESTARTS   AGE
my-tomcat-57667b9d9-pwrhs         0/1     Terminating   0          13m
nginx-7697996758-gzqms            1/1     Running       1          35h
nginx-7697996758-j6tfj            1/1     Running       1          35h
nginx-7697996758-ldfvx            1/1     Running       1          35h
nginx-deployment-d55b94fd-5zhjt   1/1     Running       1          34h
nginx-deployment-d55b94fd-6f6hm   1/1     Running       1          34h
nginx-deployment-d55b94fd-kr7c6   1/1     Running       1          34h

进行强制删除

[root@master demo]#  kubectl delete pod my-tomcat-57667b9d9-pwrhs --force --grace-period=0 -n default
warning: Immediate deletion does not wait for confirmation that the running resource hasbeen terminated. The resource may continue to run on the cluster indefinitely.
Error from server (NotFound): pods "my-tomcat-57667b9d9-pwrhs" not found

已经删除掉了

[root@master demo]# kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
nginx-7697996758-gzqms            1/1     Running   1          35h
nginx-7697996758-j6tfj            1/1     Running   1          35h
nginx-7697996758-ldfvx            1/1     Running   1          35h
nginx-deployment-d55b94fd-5zhjt   1/1     Running   1          35h
nginx-deployment-d55b94fd-6f6hm   1/1     Running   1          35h
nginx-deployment-d55b94fd-kr7c6   1/1     Running   1          35h

修改tomcat yaml文件 从私有仓库下载镜像

编辑yaml 文件

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-tomcat
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: my-tomcat
    spec:
      imagePullSecrets:
      - name: registry-pull-secret
      containers:
      - name: my-tomcat
        image: 192.168.100.9/sha/tomcat
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: my-tomcat
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    nodePort: 31111
  selector:
    app: my-tomcat

开始创建资源

[root@master demo]# kubectl create -f tomcat-deployment.yaml
deployment.extensions/my-tomcat created
service/my-tomcat created
[root@master demo]# kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
my-tomcat-8884884f6-gk7rw         1/1     Running   0          13s
my-tomcat-8884884f6-pt7jt         1/1     Running   0          13s
nginx-7697996758-gzqms            1/1     Running   1          35h
nginx-7697996758-j6tfj            1/1     Running   1          35h
nginx-7697996758-ldfvx            1/1     Running   1          35h
nginx-deployment-d55b94fd-5zhjt   1/1     Running   1          35h
nginx-deployment-d55b94fd-6f6hm   1/1     Running   1          35h
nginx-deployment-d55b94fd-kr7c6   1/1     Running   1          35h

查看service

[root@master demo]# kubectl get svc
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
kubernetes      ClusterIP   10.0.0.1     <none>        443/TCP          2d12h
my-tomcat       NodePort    10.0.0.119   <none>        8080:31111/TCP   2m39s
nginx-service   NodePort    10.0.0.67    <none>        80:38759/TCP     34h

查看后段节点

[root@master demo]# kubectl get ep
NAME            ENDPOINTS                                      AGE
kubernetes      192.168.100.3:6443,192.168.100.8:6443          2d12h
my-tomcat       172.17.22.5:8080,172.17.71.6:8080              2m41s
nginx-service   172.17.22.4:80,172.17.71.3:80,172.17.71.4:80   34h

[root@master demo]#
查看镜像仓库下载情况

私有云 总结：

为了搭建私有云，需要搭配一个habor仓库，将需要的镜像文件上传上去创建资源时，直接由私人仓库下载，从而达到了以后方便使用的目的，安全高效。