k8s分离jdkj环境与java程序研究测试

设计思路：将java程序的jdk环境、配置文件、程序包进行分离，这样可以实现在不影响java程序的情况下进行jdk版本的升级、配置文件的修改等

方案优势：Pod有一个IP，而一个Pod中的所有容器可以共享相同的IP。如果为Pod创建了任何卷，则作为Pod一部分的所有容器都可以安装该卷。因此，容器可以共享存储空间，它们还可以通过本地主机相互通信。

方案测试示例：

创建共享存储卷

#创建共享存储卷pvc，因为k8s集群配置了glusterfs动态存储服务，所以只需创建pvc即可
[root@k8s-master-1 test]# cat centos_pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: centos-pvc
  namespace: test
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: glusterfs

创建基础需求镜像

jdk环境镜像

#上次jdk离线包，并解压重命名为jdk-11，这里选择直接解压是为了规整目录名称。
[root@k8s-master-1 Dockerfile]# ls
dockerfile  jdk  jdk-11.0.15.1_linux-x64_bin.tar.gz

[root@k8s-master-1 Dockerfile]# vim dockerfile
FROM dockerhub.dsj.com:18443/library/centos:centos7.9.2009
ADD jdk /jdk/

#镜像打包
[root@k8s-master-1 Dockerfile]# docker build -t dockerhub.dsj.com:18443/library/centos:jdk_11 .
Sending build context to Docker daemon  451.2MB
Step 1/2 : FROM dockerhub.dsj.com:18443/library/centos:centos7.9.2009
 ---> eeb6ee3f44bd
Step 2/2 : ADD jdk /jdk/
 ---> Using cache
 ---> 0c36cbc3203e
Successfully built 0c36cbc3203e
Successfully tagged dockerhub.dsj.com:18443/library/centos:jdk_11

#镜像上传到镜像仓库
[root@k8s-master-1 Dockerfile]# docker push dockerhub.dsj.com:18443/library/centos:jdk_11
The push refers to repository [dockerhub.dsj.com:18443/library/centos]
dc777d9fd09f: Layer already exists
174f56854903: Layer already exists
jdk_11: digest: sha256:54c62bfedbd20a14953a1c45e1e64540016e1e709b30ebd878a1ea09cfd95515 size: 742

java程序包镜像

[root@k8s-master-1 Dockerfile-1]# ls
dockerfile

#配置java程序模拟镜像的jdk环境变量
[root@k8s-master-1 Dockerfile-1]# cat dockerfile
FROM dockerhub.dsj.com:18443/library/centos:centos7.9.2009
ENV JAVA_HOME /app/jdk/
ENV JRE_HOME /app/jdk
ENV CLASSPATH $JAVA_HOME/lib/
ENV PATH $PATH:$JAVA_HOME/bin

#打包镜像
[root@k8s-master-1 Dockerfile-1]# docker build -t dockerhub.dsj.com:18443/library/centos:java-mode .
Sending build context to Docker daemon  14.85kB
Step 1/5 : FROM dockerhub.dsj.com:18443/library/centos:centos7.9.2009
 ---> eeb6ee3f44bd
Step 2/5 : ENV JAVA_HOME /app/jdk/
 ---> Using cache
 ---> 161f99cd5ce7
Step 3/5 : ENV JRE_HOME /app/jdk
 ---> Using cache
 ---> 0edaa15cf1c1
Step 4/5 : ENV CLASSPATH $JAVA_HOME/lib/
 ---> Using cache
 ---> 0fa619e550ae
Step 5/5 : ENV PATH $PATH:$JAVA_HOME/bin
 ---> Using cache
 ---> 71a6572db1bc
Successfully built 71a6572db1bc
Successfully tagged dockerhub.dsj.com:18443/library/centos:java-mode

#上次镜像
[root@k8s-master-1 Dockerfile-1]# docker push dockerhub.dsj.com:18443/library/centos:java-mode
The push refers to repository [dockerhub.dsj.com:18443/library/centos]
174f56854903: Layer already exists
java-mode: digest: sha256:0364996d9d7f0300fe6cf8d1e3451ba81a7de8d4da01f05579f18fcf94f9fbe4 size: 529

创建测试pod

这里一个pod多容器时可以根据具体需求来选择是否使用initContainers选项，主要区别如下

• init container的镜像被更新时，init container将会重新运行，导致Pod重启。仅更新应用容器的镜像只会使得应用容器被重启。（init container可以是多个）
• 同一级的containers容器下，某一容器镜像更新时只会更新对应的容器。

因此initContainers更适合只在创建时运行一次即可的场景。

创建部署yaml文件

因为多容器需要依靠挂载统一存储卷来共享存储，故这里先将jdk环境包放到/jdk，然后在启动时将jdk复制到共享存储目录，否则在挂载时会将目录内容覆盖

[root@k8s-master-1 test]# cat java_pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: java
  namespace: test
spec:
# initContainers 会比 spec.containers 里定义的容器先启动
#  initContainers:
  containers:
  - image: dockerhub.dsj.com:18443/library/centos:jdk_11
    imagePullPolicy: Always
    name: centos-jdk
    command: ["/bin/sh"]
    args: ["-c", "rm -rf /app/jdk;cp -r -f /jdk/ /app/;tail -f /dev/null"]
    volumeMounts:
    - mountPath: /app
      name: app-volume
  - image: dockerhub.dsj.com:18443/library/centos:java-mode
    imagePullPolicy: Always
    name: centos-java-mode
    command: ["/bin/sh"]
    args: ["-c", "source /etc/profile;java -version > /1.txt;sleep 3000"]
    volumeMounts:
    - mountPath: /app
      name: app-volume
  volumes:
  - name: app-volume
    persistentVolumeClaim:
      claimName: centos-pvc

创建pod

[root@k8s-master-1 test]# kubectl apply -f java_pod.yaml
pod/java11 created
[root@k8s-master-1 test]# kubectl get pod -n test
NAME     READY   STATUS    RESTARTS   AGE
java   2/2     Running   0          7s
#查看pod详情，可以看的俩个容器的详细信息
[root@k8s-master-1 test]# kubectl describe pod/java -n test
Name:         java
Namespace:    test
Priority:     0
Node:         k8s-node-1/10.4.11.32
Start Time:   Thu, 19 May 2022 12:05:55 +0530
Labels:       <none>
Annotations:  cni.projectcalico.org/containerID: ff0d2888248898e95340b30ba49d739d2a6c50ca66261240fc0dc9b8939b4b12
              cni.projectcalico.org/podIP: 10.233.117.69/32
              cni.projectcalico.org/podIPs: 10.233.117.69/32
Status:       Running
IP:           10.233.117.69
IPs:
  IP:  10.233.117.69
Containers:
  centos-jdk:
    Container ID:  docker://2695f146e470bef29754cef51eac04a392ee863825efaf5d2f015a88c0f14580
    Image:         dockerhub.dsj.com:18443/library/centos:jdk_11
    Image ID:      docker-pullable://dockerhub.dsj.com:18443/library/centos@sha256:54c62bfedbd20a14953a1c45e1e64540016e1e709b30ebd878a1ea09cfd95515
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      rm -rf /app/jdk;cp -r -f /jdk/ /app/;tail -f /dev/null
    State:          Running
      Started:      Thu, 19 May 2022 12:05:57 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /app from app-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j8lzh (ro)
  centos-java-mode:
    Container ID:  docker://736dd40537fa5ee0ced0d485939e96759efadaa729c2ccf9007ffb5a2ff237f5
    Image:         dockerhub.dsj.com:18443/library/centos:java-mode
    Image ID:      docker-pullable://dockerhub.dsj.com:18443/library/centos@sha256:0364996d9d7f0300fe6cf8d1e3451ba81a7de8d4da01f05579f18fcf94f9fbe4
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      source /etc/profile;sleep 3000
    State:          Running
      Started:      Thu, 19 May 2022 12:05:58 +0530
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /app from app-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j8lzh (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  app-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  centos-pvc
    ReadOnly:   false
  kube-api-access-j8lzh:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  2m25s  default-scheduler  Successfully assigned test/java to k8s-node-1
  Normal  Pulling    116s   kubelet            Pulling image "dockerhub.dsj.com:18443/library/centos:jdk_11"
  Normal  Pulled     116s   kubelet            Successfully pulled image "dockerhub.dsj.com:18443/library/centos:jdk_11" in 121.550062ms
  Normal  Created    116s   kubelet            Created container centos-jdk
  Normal  Started    116s   kubelet            Started container centos-jdk
  Normal  Pulling    116s   kubelet            Pulling image "dockerhub.dsj.com:18443/library/centos:java-mode"
  Normal  Pulled     116s   kubelet            Successfully pulled image "dockerhub.dsj.com:18443/library/centos:java-mode" in 113.384613ms
  Normal  Created    116s   kubelet            Created container centos-java-mode
  Normal  Started    115s   kubelet            Started container centos-java-mode

java环境验证测试

[root@k8s-master-1 test]# kubectl exec -it pod/java /bin/bash -n test -c centos-java-mode

[root@java /]# cd app/

[root@java app]# ls
jdk

[root@java app]# ls jdk/
README.html  bin  conf  include  jmods  legal  lib  man  release

[root@java11 /]# java -version
java version "11.0.15.1" 2022-04-22 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.15.1+2-LTS-10)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.15.1+2-LTS-10, mixed mode)

容器升级验证测试

jdk18升级镜像制作

#删除jdk目录，上传jdk18离线包并解压为jdk
[root@k8s-master-1 Dockerfile]# ls
dockerfile  jdk  jdk-11.0.15.1_linux-x64_bin.tar.gz  jdk-18_linux-x64_bin.tar.gz
#镜像打包并上传
[root@k8s-master-1 Dockerfile]# docker build -t dockerhub.dsj.com:18443/library/centos:jdk_18 .
Sending build context to Docker daemon  669.7MB
Step 1/2 : FROM dockerhub.dsj.com:18443/library/centos:centos7.9.2009
 ---> eeb6ee3f44bd
Step 2/2 : ADD jdk /jdk/
 ---> 4651c8032169
Successfully built 4651c8032169
Successfully tagged dockerhub.dsj.com:18443/library/centos:jdk_18
[root@k8s-master-1 Dockerfile]# docker push dockerhub.dsj.com:18443/library/centos:jdk_18
The push refers to repository [dockerhub.dsj.com:18443/library/centos]
5e0461302672: Pushed
174f56854903: Layer already exists
jdk_18: digest: sha256:4570740be12cf03fac13990aa8fab53498a5c640b471a79a6546d4655282a14d size: 742

修改容器镜像版本号

[root@k8s-master-1 test]# vim java_pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: java
  namespace: test
spec:
# initContainers 会比 spec.containers 里定义的容器先启动
#  initContainers:
  containers:
  - image: dockerhub.dsj.com:18443/library/centos:jdk_18  #修改容器版本号
    imagePullPolicy: Always
    name: centos-11
    command: ["/bin/sh"]
    args: ["-c", "rm -rf /app/jdk;cp -r -f /jdk/ /app/;tail -f /dev/null"]
    volumeMounts:
    - mountPath: /app
      name: app-volume
  - image: dockerhub.dsj.com:18443/library/centos:java-mode
    imagePullPolicy: Always
    name: centos-java-mode
    command: ["/bin/sh"]
    args: ["-c", "source /etc/profile;sleep 3000"]
    volumeMounts:
    - mountPath: /app
      name: app-volume
  volumes:
  - name: app-volume
    persistentVolumeClaim:
      claimName: centos-pvc

[root@k8s-master-1 test]# kubectl apply -f java11_pod.yaml

结果观察

jdk容器成功从jdk-11升级到jdk-18版本，java程序容器并未发生改变

[root@k8s-master-1 test]# kubectl describe -n test pod/java
Name:         java
Namespace:    test
Priority:     0
Node:         k8s-node-1/10.4.11.32
Start Time:   Thu, 19 May 2022 12:42:13 +0530
Labels:       <none>
Annotations:  cni.projectcalico.org/containerID: 13c8865ae72672d4667e1d00a71e05580927f3e40d65816131a1361315351447
              cni.projectcalico.org/podIP: 10.233.117.74/32
              cni.projectcalico.org/podIPs: 10.233.117.74/32
Status:       Running
IP:           10.233.117.74
IPs:
  IP:  10.233.117.74
Containers:
  centos-jdk:
    Container ID:  docker://3416ea608ab9887d6f3ca98244181d8ac87b561b1cbcca67799151bc71d5d8aa
    Image:         dockerhub.dsj.com:18443/library/centos:jdk_18
    Image ID:      docker-pullable://dockerhub.dsj.com:18443/library/centos@sha256:54c62bfedbd20a14953a1c45e1e64540016e1e709b30ebd878a1ea09cfd95515
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      rm -rf /app/jdk;cp -r -f /jdk/ /app/;tail -f /dev/null
    State:          Running
      Started:      Thu, 19 May 2022 12:55:52 +0530        ######启动时间改变
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Thu, 19 May 2022 12:42:14 +0530
      Finished:     Thu, 19 May 2022 12:55:52 +0530
    Ready:          True
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /app from app-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2mzd (ro)
  centos-java-mode:
    Container ID:  docker://1265f2fd80ef629f23cb9ce80e1fb3d98938b1f61d8918a61d0dfaf352e2f1e8
    Image:         dockerhub.dsj.com:18443/library/centos:java-mode
    Image ID:      docker-pullable://dockerhub.dsj.com:18443/library/centos@sha256:0364996d9d7f0300fe6cf8d1e3451ba81a7de8d4da01f05579f18fcf94f9fbe4
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      source /etc/profile;sleep 3000
    State:          Running
      Started:      Thu, 19 May 2022 12:42:15 +0530          ####启动时间未改变
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /app from app-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-c2mzd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  app-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  centos-pvc
    ReadOnly:   false
  kube-api-access-c2mzd:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age                From               Message
  ----    ------     ----               ----               -------
  Normal  Scheduled  15m                default-scheduler  Successfully assigned test/java to k8s-node-1
  Normal  Pulling    14m                kubelet            Pulling image "dockerhub.dsj.com:18443/library/centos:jdk_11"
  Normal  Pulled     14m                kubelet            Successfully pulled image "dockerhub.dsj.com:18443/library/centos:jdk_11" in 115.933359ms
  Normal  Pulling    14m                kubelet            Pulling image "dockerhub.dsj.com:18443/library/centos:java-mode"
  Normal  Pulled     14m                kubelet            Successfully pulled image "dockerhub.dsj.com:18443/library/centos:java-mode" in 111.263522ms
  Normal  Created    14m                kubelet            Created container centos-java-mode
  Normal  Started    14m                kubelet            Started container centos-java-mode
  Normal  Killing    90s                kubelet            Container centos-11 definition changed, will be restarted	###只杀掉了更新的容器
  Normal  Started    60s (x2 over 14m)  kubelet            Started container centos-jdk
  Normal  Created    60s (x2 over 14m)  kubelet            Created container centos-jdk
  Normal  Pulling    60s                kubelet            Pulling image "dockerhub.dsj.com:18443/library/centos:jdk_18"
  Normal  Pulled     60s                kubelet            Successfully pulled image "dockerhub.dsj.com:18443/library/centos:jdk_18" in 123.467551ms

进入java程序容器，进行jdk版本验证

[root@k8s-master-1 test]# kubectl exec -it pod/java -n test -c centos-java-mode /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

#jdk版本升级到jdk18
[root@java jdk]# java -version
java version "18.0.1.1" 2022-04-22
Java(TM) SE Runtime Environment (build 18.0.1.1+2-6)
Java HotSpot(TM) 64-Bit Server VM (build 18.0.1.1+2-6, mixed mode, sharing)

结论

• 一个pod多个容器可以通过挂载统一存储卷进行存储共享以实现将jdk环境与java程序分离的目的
• 升级jdk版本时可以在不影响java程序的情况下单独进行升级，升级之后java程序包的jdk版本也会随之改变