Kubernetes CKA认证运维工程师笔记-Kubernetes项目部署案例
1. 容器交付流程
本地开发阶段 | 持续集成/持续交付 | 部署应用 | 运维 |
---|---|---|---|
代码编写 | 代码编译 | 持续交付 | 监控告警 |
Dockerfile编写 | 构建镜像 | 规划部署拓扑 | 日志收集 |
编排模板编写 | 推送镜像 | 环境配置 | 故障排查 |
容器化测试 | 部署编排模板 | 扩缩容配置 | 持续优化 |
2. 在K8s平台部署项目流程
制作镜像 | 使用控制器部署镜像 | 对外暴露应用 | 日志、监控 | 日常运维 |
---|---|---|---|---|
dockerfile | Deployment、StatefulSet、DaemonSet | Service、Ingress | Prometheus+Grafana、ELK Stack |
3. 在K8s平台部署Java网站项目
熟悉项目:
- 项目代码构成
- 依赖的服务
- 提供服务的端口
- 配置文件
- 程序在工作中涉及持久化的文件
java:jdk、maven
mysql
Tomcat
src/main/resources/application.yml
pv、pvc
镜像分类:
- 基础镜像,例如:centos、ubuntu
- 环境镜像,例如:jdk、nginx、tomcat
- 项目镜像,例如:dashboard
第一步:制作镜像
yum install java-1.8.0-openjdk maven git -y
git clone https://github.com/lizhenliang/tomcat-java-demo
mvn clean package -Dmaven.test.skip=true # 代码编译构建
unzip target/*.war -d target/ROOT # 解压构建文件
FROM tomcat
LABEL maintainer lizhenliang
COPY target/ROOT /usr/local/tomcat/webapps/ROOT
docker build -t image:tag .
docker push <镜像仓库地址>/<项目名>/image:tag #demo:1.0\demo:1.1\demo:1.2
docker run -d image:tag
第一步:制作镜像
使用镜像仓库(私有仓库、公共仓库):
1、配置可信任(如果仓库是HTTPS访问不用配置)
# vi /etc/docker/daemon.json
{
"insecure-registries": ["10.0.0.65"]
}
2、将镜像仓库认证凭据保存在K8s Secret中
kubectl create secret docker-registry registry-auth --docker-username=admin --docker-password=Harbor12345 --docker-server=10.0.0.65
3、在yaml中使用这个认证凭据
imagePullSecrets:
-name: registry-auth
第二步:使用控制器部署镜像
# 控制器定义
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 1
selector:
matchLabels:
app: web
# 被控制对象
template:
metadata:
labels:
app: web
spec:
containers:
-image: lizhenliang/java-demo
name: java-demo
Pod主要配置启动容器属性:
- 变量
- 资源配额
- 健康检查
- 卷挂载点
使用configmap保存项目配置文件
第三步:对外暴露应用
示例:Nginx作为负载均衡器
upstream java-demo {
server 192.168.31.62:80;
server 192.168.31.63:80;
}
server {
listen 80;
server_name example.ctnrs.com;
location / {
proxy_pass http://java-demo;
proxy_set_header Host $Host;
}
}
应用访问架构图
[root@k8s-master 项目部署]# cd tomcat-java-demo-master/
[root@k8s-master tomcat-java-demo-master]# ls
db LICENSE pom.xml README.md src
[root@k8s-master tomcat-java-demo-master]# cd src/
[root@k8s-master src]# ls
main
[root@k8s-master src]# cd main/
[root@k8s-master main]# ls
java resources
[root@k8s-master main]# ls java/
com
[root@k8s-master main]# ld resources/
resources/: file not recognized: Is a directory
[root@k8s-master main]# ls resources/
application.yml log4j.properties static templates
[root@k8s-master main]# ls db
ls: cannot access db: No such file or directory
[root@k8s-master main]# ls db/
ls: cannot access db/: No such file or directory
[root@k8s-master main]# cd ..
[root@k8s-master src]# ls db/
ls: cannot access db/: No such file or directory
[root@k8s-master src]# cd ..
[root@k8s-master tomcat-java-demo-master]# ls db/
tables_ly_tomcat.sql
[root@k8s-master tomcat-java-demo-master]# cat src/main/resources/application.yml
server:
port: 8080
spring:
datasource:
url: jdbc:mysql://localhost:3306/test?characterEncoding=utf-8
username: root
password: 12345
driver-class-name: com.mysql.jdbc.Driver
freemarker:
allow-request-override: false
cache: true
check-template-location: true
charset: UTF-8
content-type: text/html; charset=utf-8
expose-request-attributes: false
expose-session-attributes: false
expose-spring-macro-helpers: false
suffix: .ftl
template-loader-path:
- classpath:/templates/
[root@k8s-master tomcat-java-demo-master]# yum install java-1.8.0-openjdkit -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.aliyun.com
* extras: mirrors.aliyun.com
* updates: mirrors.aliyun.com
Package 1:java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64 already installatest version
Package maven-3.0.5-17.el7.noarch already installed and latest version
Package git-1.8.3.1-23.el7_8.x86_64 already installed and latest version
Nothing to do
[root@k8s-master tomcat-java-demo-master]# mvn clean package -Dmaven.test.skip=true
...
[INFO] Building war: /root/项目部署/tomcat-java-demo-master/target/ly-simple-tomcat-0.0.1-SNAPSHOT.war
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:57.598s
[INFO] Finished at: Mon Dec 27 10:28:54 CST 2021
[INFO] Final Memory: 26M/140M
[INFO] ------------------------------------------------------------------------
[root@k8s-master tomcat-java-demo-master]# ls
db LICENSE pom.xml README.md src target
[root@k8s-master tomcat-java-demo-master]# ls target/
classes ly-simple-tomcat-0.0.1-SNAPSHOT maven-archiver
generated-sources ly-simple-tomcat-0.0.1-SNAPSHOT.war maven-status
[root@k8s-master tomcat-java-demo-master]# unzip target/ly-simple-tomcat-0.0.1-SNAPSHOT.war -d target/ROOT
Archive: target/ly-simple-tomcat-0.0.1-SNAPSHOT.war
inflating: target/ROOT/META-INF/MANIFEST.MF
...
[root@k8s-master tomcat-java-demo-master]# ls target/
classes maven-archiver
generated-sources maven-status
ly-simple-tomcat-0.0.1-SNAPSHOT ROOT
ly-simple-tomcat-0.0.1-SNAPSHOT.war
[root@k8s-master tomcat-java-demo-master]# ls target/ROOT/
META-INF WEB-INF
[root@k8s-master tomcat-java-demo-master]# docker build -t java-demo .
Sending build context to Docker daemon 65.45MB
Step 1/4 : FROM lizhenliang/tomcat
latest: Pulling from lizhenliang/tomcat
a02a4930cb5d: Pull complete
12ac6c660ca4: Pull complete
675ab56ce773: Pull complete
Digest: sha256:3bd83236a4b6cf4152d0384981ce9ed55063bc5fd5259d206bfdfdb877651db0
Status: Downloaded newer image for lizhenliang/tomcat:latest
---> 143035d83fdc
Step 2/4 : LABEL maintainer www.ctnrs.com
---> Running in cced391ddde7
Removing intermediate container cced391ddde7
---> f645047acc86
Step 3/4 : RUN rm -rf /usr/local/tomcat/webapps/*
---> Running in 2343188f5767
Removing intermediate container 2343188f5767
---> 4ca7cd219431
Step 4/4 : ADD target/ROOT /usr/local/tomcat/webapps/ROOT
---> 844893cae228
Successfully built 844893cae228
Successfully tagged java-demo:latest
[root@k8s-master tomcat-java-demo-master]# docker run -d -p 8888:8080 java-demo
5a787c6a6eb49d303e07dd2769a52287933daf5c6573c8b15fda3769ab7fd09f
[root@k8s-master tomcat-java-demo-master]# vi /usr/lib/systemd/system/docker.service
...
ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock --insecure-registry 10.0.0.65 --insecure-registry harbor.lpk.net
...
[root@k8s-master tomcat-java-demo-master]# systemctl daemon-reload
[root@k8s-master tomcat-java-demo-master]# systemctl restart docker
[root@k8s-master tomcat-java-demo-master]# docker login 10.0.0.65
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@k8s-master tomcat-java-demo-master]# docker push 10.0.0.65/demo/java-demo:v1
The push refers to repository [10.0.0.65/demo/java-demo]
9e7a9c6b2bd4: Pushed
e02e78c5e1a2: Pushed
ceead5ca823f: Pushed
2353c173a26a: Pushed
071d8bd76517: Pushed
v1: digest: sha256:62288c888c740eaeb7786da327d05ea7bda3d14d6669c0e18c822ff69b62f643 size: 1371
[root@k8s-master ~]# vi java-demo.yaml
[root@k8s-master ~]# cat java-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: java-demo
labels:
app: java-demo
spec:
replicas: 3
selector:
matchLabels:
project: www
app: java-demo
template:
metadata:
labels:
project: www
app: java-demo
spec:
containers:
- name: web
image: 10.0.0.65/demo/java-demo:v1
ports:
- containerPort: 8080
[root@k8s-master ~]# kubectl apply -f java-demo.yaml
deployment.apps/java-demo created
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
client1 1/1 Running 9 4d23h
client2 1/1 Running 9 4d23h
configmap-demo-pod 1/1 Running 4 6d9h
java-demo-7d7d8cd565-dsllq 0/1 ImagePullBackOff 0 7h13m
java-demo-7d7d8cd565-h7c84 0/1 ImagePullBackOff 0 7h13m
java-demo-7d7d8cd565-zr465 0/1 ImagePullBackOff 0 7h13m
my-pod2 1/1 Running 15 6d16h
nfs-client-provisioner-58d675cd5-dx7n4 1/1 Running 10 6d11h
pod-taint 1/1 Running 9 12d
secret-demo-pod 1/1 Running 4 6d8h
sh 1/1 Running 6 6d10h
test-76846b5956-gftn9 1/1 Running 2 6d10h
test-76846b5956-r7s9k 1/1 Running 2 6d10h
test-76846b5956-trpbn 1/1 Running 2 6d10h
test2-78c4694588-87b9r 1/1 Running 5 6d11h
web-0 1/1 Running 4 6d10h
web-1 1/1 Running 4 6d10h
web-2 1/1 Running 4 6d10h
web-96d5df5c8-vc9kf 1/1 Running 3 4d23h
[root@k8s-master ~]# kubectl describe pod java-demo-7d7d8cd565-dsllq
Name: java-demo-7d7d8cd565-dsllq
Namespace: default
Priority: 0
Node: k8s-node2/10.0.0.63
Start Time: Mon, 27 Dec 2021 14:19:33 +0800
Labels: app=java-demo
pod-template-hash=7d7d8cd565
project=www
Annotations: cni.projectcalico.org/podIP: 10.244.169.186/32
cni.projectcalico.org/podIPs: 10.244.169.186/32
Status: Pending
IP: 10.244.169.186
IPs:
IP: 10.244.169.186
Controlled By: ReplicaSet/java-demo-7d7d8cd565
Containers:
web:
Container ID:
Image: 10.0.0.65/demo/java-demo:v1
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-8grtj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-8grtj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8grtj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal BackOff 12m (x1850 over 7h12m) kubelet Back-off pulling image "10.0.0.65/demo/java-demo:v1"
Warning Failed 2m54s (x1894 over 7h12m) kubelet Error: ImagePullBackOff
[root@k8s-node1 ~]# docker run -d 10.0.0.65/demo/java-demo:v1
Unable to find image '10.0.0.65/demo/java-demo:v1' locally
docker: Error response from daemon: unauthorized: unauthorized to access repository: demo/java-demo, action: pull: unauthorized to access repository: demo/java-demo, action: pull.
See 'docker run --help'.
[root@k8s-node1 ~]# docker login 10.0.0.65
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@k8s-node1 ~]# docker run -d 10.0.0.65/demo/java-demo:v1
Unable to find image '10.0.0.65/demo/java-demo:v1' locally
v1: Pulling from demo/java-demo
a02a4930cb5d: Already exists
12ac6c660ca4: Already exists
675ab56ce773: Already exists
4c1eea01bd69: Pull complete
26dff194b2a4: Pull complete
Digest: sha256:62288c888c740eaeb7786da327d05ea7bda3d14d6669c0e18c822ff69b62f643
Status: Downloaded newer image for 10.0.0.65/demo/java-demo:v1
9c0b4ea12bb78c0b9984aa4669e9951401b84682514a723d6ce9a3d0d0e754cf
[root@k8s-master ~]# vi java-demo.yaml
[root@k8s-master ~]# cat java-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: java-demo
labels:
app: java-demo
spec:
replicas: 3
selector:
matchLabels:
project: www
app: java-demo
template:
metadata:
labels:
project: www
app: java-demo
spec:
imagePullSecrets:
- name: registry-auth
containers:
- name: web
image: 10.0.0.65/demo/java-demo:v1
ports:
- containerPort: 8080
[root@k8s-master ~]# kubectl delete -f java-demo.yaml
deployment.apps "java-demo" deleted
[root@k8s-master ~]# kubectl apply -f java-demo.yaml
deployment.apps/java-demo created
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
java-demo-bcbfffcb9-f7nfk 0/1 ContainerCreating 0 12s
java-demo-bcbfffcb9-mbxtq 0/1 ContainerCreating 0 12s
java-demo-bcbfffcb9-v4bqj 0/1 ContainerCreating 0 12s
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
java-demo-bcbfffcb9-f7nfk 1/1 Running 0 19s
java-demo-bcbfffcb9-mbxtq 1/1 Running 0 19s
java-demo-bcbfffcb9-v4bqj 0/1 ErrImagePull 0 19s
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
java-demo-bcbfffcb9-f7nfk 1/1 Running 2 19m
java-demo-bcbfffcb9-k5g89 1/1 Running 1 8m45s
java-demo-bcbfffcb9-mbxtq 1/1 Running 2 19m
[root@k8s-master tomcat-java-demo-master]# cat src/main/resources/application.yml
server:
port: 8080
spring:
datasource:
url: jdbc:mysql://localhost:3306/test?characterEncoding=utf-8
username: root
password: 12345
driver-class-name: com.mysql.jdbc.Driver
freemarker:
allow-request-override: false
cache: true
check-template-location: true
charset: UTF-8
content-type: text/html; charset=utf-8
expose-request-attributes: false
expose-session-attributes: false
expose-spring-macro-helpers: false
suffix: .ftl
template-loader-path:
- classpath:/templates/
[root@k8s-master tomcat-java-demo-master]# vi configmap.yaml
[root@k8s-master tomcat-java-demo-master]# cat configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: java-demo-config
data:
application.yml: |
server:
port: 8080
spring:
datasource:
url: jdbc:mysql://localhost:3306/test?characterEncoding=utf-8
username: root
password: 12345
driver-class-name: com.mysql.jdbc.Driver
freemarker:
allow-request-override: false
cache: true
check-template-location: true
charset: UTF-8
content-type: text/html; charset=utf-8
expose-request-attributes: false
expose-session-attributes: false
expose-spring-macro-helpers: false
suffix: .ftl
template-loader-path:
- classpath:/templates/
[root@k8s-master tomcat-java-demo-master]# kubectl apply -f configmap.yaml
configmap/java-demo-config created
[root@java-demo-757f9dcc46-wx9cq tomcat]# cd webapps/ROOT/WEB-INF/classes/
[root@java-demo-757f9dcc46-wx9cq classes]# ls
application.yml
[root@java-demo-757f9dcc46-wx9cq classes]# command terminated with exit code 137
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
java-demo-757f9dcc46-vrdzz 0/1 Running 0 84s
java-demo-757f9dcc46-w9sgc 0/1 Running 0 84s
java-demo-757f9dcc46-wx9cq 0/1 Running 1 84s
[root@k8s-master ~]# vi java-demo.yaml
[root@k8s-master ~]# kubectl delete -f java-demo.yaml
deployment.apps "java-demo" deleted
[root@k8s-master ~]# kubectl apply -f java-demo.yaml
deployment.apps/java-demo created
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
configmap-demo-pod 1/1 Running 5 6d10h
java-demo-6bf4cd9644-4vxmp 0/1 Running 0 5s
java-demo-6bf4cd9644-wqdqf 0/1 Pending 0 4s
java-demo-6bf4cd9644-zdlvw 0/1 ContainerCreating 0 4s
java-demo-757f9dcc46-vrdzz 0/1 Terminating 1 3m21s
java-demo-757f9dcc46-w9sgc 0/1 Terminating 1 3m21s
[root@k8s-master ~]# kubectl get pods
NAME READY STATUS RESTARTS AGE
configmap-demo-pod 1/1 Running 5 6d10h
java-demo-6bf4cd9644-4vxmp 0/1 Running 0 38s
java-demo-6bf4cd9644-wqdqf 0/1 Pending 0 37s
java-demo-6bf4cd9644-zdlvw 0/1 Running 0 37s
java-demo-757f9dcc46-vrdzz 0/1 Terminating 1 3m54s
java-demo-757f9dcc46-w9sgc 0/1 Terminating 1 3m54s
[root@k8s-master ~]# kubectl exec -it java-demo-6bf4cd9644-4vxmp -- bash
[root@java-demo-6bf4cd9644-4vxmp tomcat]# cd webapps/ROOT/WEB-INF/classes/
[root@java-demo-6bf4cd9644-4vxmp classes]# ls
application.yml com log4j.properties static templates
[root@java-demo-6bf4cd9644-4vxmp classes]# exit
exit
[root@k8s-master ~]# vi service.yaml
[root@k8s-master ~]# vi java-demo-service.yaml
[root@k8s-master ~]# kubectl apply -f java-demo-service.yaml
service/java-demo created
[root@k8s-master ~]# kubectl get ep
NAME ENDPOINTS AGE
fuseim.pri-ifs <none> 6d12h
java-demo 10.244.169.150:8080,10.244.169.184:8080,10.244.36.84:8080 42s
kubernetes 10.0.0.61:6443 35d
my-dep <none> 32d
my-service 10.244.36.68:80,10.244.36.71:80,10.244.36.79:80 26d
nginx 10.244.36.68:80,10.244.36.71:80,10.244.36.79:80 6d12h
[root@k8s-master ~]# vi java-demo-ingress.yaml
[root@k8s-master ~]# cat java-demo-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: java-demo
spec:
rules:
- host: example.ctnrs.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: java-demo
port:
number: 80
[root@k8s-master ~]# kubectl apply -f java-demo-ingress.yaml
ingress.networking.k8s.io/java-demo created
[root@k8s-master ~]# kubectl get ingress
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress <none> example.ctnrs.com 80 11d
java-demo <none> example.ctnrs.com 80 11s
[root@k8s-master ~]# kubectl get pods -n ongress
No resources found in ongress namespace.
[root@k8s-master ~]# kubectl get pods -n ingress
No resources found in ingress namespace.
[root@k8s-master ~]# kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
nginx-ingress-controller-7cjvq 1/1 Running 12 11d
nginx-ingress-controller-pk4sq 1/1 Running 11 11d