前提:启动服务
下面内容会涉及自动生成K8S的yaml代码(py撰写) 代码内容下文中不做体现 只提供思路
代码区分前后端服务 且会根据需求生成Ingress deployment svc 等
大佬勿喷 仅作调研实现
# 启动服务
# devops服务器
nohup python3 /opt/scripts/autodeploy_py/WebApi.py &
现状
环境 | jenkins | slave | k8s | 节点名称 |
---|---|---|---|---|
dev | 192.168.1.48 | 192.168.1.26 | 192.168.1.156 | devk8s_context |
uat | 192.168.1.48 | 172.16.1.15(公网IP1) | 172.16.1.16 | uatk8s_context |
fat | 192.168.1.48 | 172.16.2.6(公网IP2) | 172.16.2.17 | fatk8s_context |
lpt | 192.168.1.48 | 172.16.1.32(公网IP3) | 172.16.1.19 | lptk8s_context |
prod | 192.168.1.48 | 10.0.2.42(公网IP4) | 10.0.1.16 | prodk8s_context |
方案一、docker jenkins安装部署 版本[2.222.4]
1、安装jenkins
# 获取镜像
docker pull *****.*****.com/base/jenkins
# 启动容器
docker run -u root -d -p 8080:8080 -p 50000:50000 -v /data/jenkins_data:/var/jenkins_home -v /var/run/docker.sock:/var/run/docker.sock --name jenkins_demo *****.*****.com/base/jenkins
# 服务器请求
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d '{"version":"5c86d41508abddfc8540db99794353579ac0eecb","appname":"***-frontend","environment":"uat","branch":"dev","k8sversion":"1.18"}' | tail -5
2、创建文件夹及job(忽略)
3、 配置job
变量名称 | 参数化 | 默认值 | 可填 |
---|---|---|---|
APPNAME | 文本参数 | 无 | 必填 |
environment | 选项参数 | dev uat fat prd | 必填 |
version | 文本参数 | 无 | 必填 |
k8sversion | 选项参数 | 1.12 1.18 | 必填 |
jenkins job配置如下:
# 生成配置文件
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d {\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\",\"k8sversion\":\"$k8sversion\"}
# 指定对接apiserver
kubectl config set-cluster dev-k8s --server https://192.168.1.156:6443 --certificate-authority=/opt/k8s_config/dev/dev-cluster.ca
# 添加用户 需要指定crt,key文件,上一步有获取
kubectl config set-credentials kubernetes-admin --client-certificate=/opt/k8s_config/dev/dev.crt --client-key=/opt/k8s_config/dev/dev.key
# 指定一个上下文
kubectl config set-context dev --cluster=dev-k8s --namespace=app --user=kubernetes-admin
#激活这个上下文
kubectl config use-context dev
# kubetctl发布
kubectl apply -f $APPNAME-$environment.yaml
方案二、本地化 jenkins安装部署 版本[2.277.4] [√]
1、安装jenkins
# 获取安装yum
sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
# 安装jenkins
yum install jenkins
# 启动jenkins
systemctl start jenkins
# 重启及停止
systemctl restart jenkins
systemctl stop jenkins
2、创建文件夹及job(忽略)
3、 配置job
变量名称 | 参数化 | 默认值 | 可填 |
---|---|---|---|
APPNAME | 文本参数 | 无 | 必填 |
environment | 选项参数 | dev uat fat prd | 必填 |
version | 文本参数 | 无 | 必填 |
k8sversion | 选项参数 | 1.12 1.18 | 必填 |
jenkins job配置如下:此处为shell脚本调用
# 配置文件
APPNAME=`echo $JOB_NAME|awk -F '/' '{print $NF}'`
ENV=`echo $JOB_NAME|awk -F '/' '{print $1}'`
PYTHONAPI=http://devops.******.com/autodeploy/deploy
if [ ` curl -i -X POST $PYTHONAPI -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$ENV\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['code'])" ` -eq "200" ] ;then
echo "生成配置文件"
# 生成配置文件
curl -i -X POST $PYTHONAPI -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$ENV\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['data'])"> $version.yaml
# apply yaml
if [ $ENV == "develop" ]; then
cp $version.yaml $version-fort.yaml
sed -i "s/$APPNAME/$APPNAME-fort/g" $version-fort.yaml
sed -i "s#$ENV/$APPNAME-fort#$ENV/$APPNAME#g" $version-fort.yaml
kubectl apply -f $version-fort.yaml
else
kubectl apply -f $version.yaml
fi
else
echo ` curl -i -X POST $PYTHONAPI -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$ENV\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\}`
exit 1
fi
echo "结束"
可以使用jenkins插件 Configuration Slicing
Tied Label Slicer: 配置节点信息
Execute shell slicer:配置脚本信息
复制job 可以使用Python脚本操作核心代码如下:
# copy_job(“复制项”,“黏贴项”)
jenkinsenkins.Jenkins(Jenkins_server_url, Jenkins_user, Jenkins_passwd).copy_job("fat/single-spa/report-web","fat/single-spa/demo")
目前使用jenkins[本地化部署]
地址:http://192.168.1.251:8080/
用户名:admin
密码:admin
目录结构
目录 | 用途 |
---|---|
/data/yaml | 存放yaml生成文件 |
/opt/k8s_config | 存放各个环境的k8s对接信息证书等 |
/data/yaml/deploy.py | 具体代码文件 |
方案一、K8S多环境调用
安装Kubectl
# 将kubectl和apiserver的版本尽量保持一致 防止出现版本问题 地址如下
https://dl.k8s.io/v1.12.3/kubernetes-client-linux-amd64.tar.gz
如果用户没有证书则执行:
# 生成签发证书的策略ca-config.json
{
"signing": {
"default": {
"expiry": "438000h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "438000h"
}
}
}
}
# 生成ca自签名请求 cat ca-csr.json
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "HangZhou",
"L": "XS",
"O": "k8s",
"OU": "System"
}
],
"ca": {
"expiry": "876000h"
}
}
# 执行下面命令 会生成证书及私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
# 如果命令不存在执行:
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo
# 创建admin用户配置
cat > admin-csr.json <<EOF
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"L": "BeiJing",
"ST": "BeiJing",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://172.16.1.34:6443"
kubectl config set-cluster kubernetes \
--certificate-authority=/opt/kubernetes/ssl/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials cluster-admin \
--client-certificate=./admin.pem \
--client-key=./admin-key.pem \
--embed-certs=true \
--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \
--cluster=kubernetes \
--user=cluster-admin \
--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}
获取master的相关信息
# 到k8s集群master服务器 执行:cat ~/.kube/config
# 将
-cluster
-client-certificate-data
-client-key-data
# 这三个所对应的value进行转化
echo '${替换内容}' | base64 -d
echo '' | base64 -d > dev-cluster.ca
echo '' | base64 -d > dev.crt
echo '' | base64 -d > dev.key
#并记录server的值:https://192.168.1.156:6443
# 将三个保存成文件 分别为 (命名可以随便修改)
-cluster --> dev-cluster.ca
-client-certificate-data --> dev.crt
-client-key-data --> dev.key
DEV环境执行命令:
# 将master的/root/.kube文件复制到client节点
# 指定指定的k8s的链接地址
kubectl config set-cluster dev-k8s --server https://192.168.1.156:6443 --certificate-authority=/opt/k8s_config/dev/dev-cluster.ca
# 添加用户 需要指定crt key文件 上一步有获取
kubectl config set-credentials kubernetes-admin --client-certificate=/opt/k8s_config/dev/dev.crt --client-key=/opt/k8s_config/dev/dev.key
# 指定一个上下文
kubectl config set-context dev --cluster=dev-k8s --namespace=app --user=kubernetes-admin
#激活这个上下文
kubectl config use-context dev
# 已测试可以正常使用 下面命令有输出则正常
kubectl version
问题现状
1、docker容器部署的jenkins 无法调用kubectl 如果可调用多环境的apiserver调用的key ca crt 要做挂载
2、目前dev环境kubectl调用远端apiserver实现 但uat fat prd 的k8s的server均为内网地址 本地无法调用 尝试更改但怕影响环境
3、代码和jenkins及kubectl目前耦合度较强
方案二、jenkins_slave方式
创建从节点
# 1、服务器时间同步
yum -y install ntp ntpdate
ntpdate 0.asia.pool.ntp.org && hwclock --systohc
# 2、创建节点
登录Jenkins服务器-点击系统管理-选择Manage Nodes(节点管理)-点击新建节点-点击ok
# 3、配置节点
名称:dev_master_kubesphere_192.168.1.11
描述:测试
执行器数量:4
远程工作目录:/home
标签:Kubesphere
用法;只允许运行绑定到这台机器的job
启动方式:Launch agents via SSH
主机:192.168.1.11
Credentials: root(后面会说明)
Host Key Verification Strategy:Non verifying verification Stategy
可用性:尽量保持代理在线
#4、服务器创建公私钥
ssh-keygen -t rsa
jenkins添加私钥 id_rsa
添加凭据:SSH Username with private key 私钥复制即可
=======================查看状态节点添加完成=======================
创建Job
General中需要注意一点:
限制项目运行节点
标签表达式
dev_master_kubesphere_192.168.1.11
执行shell(后期优化逻辑)
if [ ` curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\",\"branch\":\"$branch\"\,\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['code'])" ` -eq "200" ] ;then
echo "生成配置文件"
# 生成配置文件
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['data'])"> $APPNAME-$environment-$BUILD_NUMBER.yaml
# apply yaml
kubectl apply -f $APPNAME-$environment-$BUILD_NUMBER.yaml
else
echo ` curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d \{\"version\":\"$version\",\"appname\":\"$APPNAME\",\"environment\":\"$environment\",\"branch\":\"$branch\",\"k8sversion\":\"$k8sversion\"\} | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['msg'])" `
fi
echo "结束"
接口文档
序号 | 环境 | URL |
---|---|---|
1 | dev环境 | htpp://192.168.1.251:8077/deploy |
简要描述:
跟据参数调用数据库生成yaml并保存成文件
请求URL:
htpp://192.168.1.251:8077/deploy
请求方式:
POST
heardes:
Content-Type:application/json
参数:
参数名称 | 必选 | 类型 | 说明 |
---|---|---|---|
appname | 是 | String | pod名称(镜像名称) |
environment | 是 | String | 环境参数 |
version | 是 | String | 版本参数 |
branch | 是 | String | 代码版本 |
k8sversion | 是 | String | k8s版本 |
请求示例(以下实际参数为示例):
linux请求:
# 查看接口返回
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d '{"version":"5c86d41508abddfc8540db99794353579ac0eecb","appname":"***-frontend","environment":"uat","branch":"dev","k8sversion":"1.18"}' | tail -5
# 获取data信息
curl -i -X POST http://192.168.1.251:8077/deploy -H 'Content-type':'application/json' -d '{"version":"5c86d41508abddfc8540db99794353579ac0eecb","appname":"***-frontend","environment":"uat","branch":"dev","k8sversion":"1.18"}' | tail -5|python -c "import sys, json; print(json.load(sys.stdin)['data'])"> ***-frontend.yaml
python请求:
url = "http://127.0.0.1:8077/deploy"
data = {"version" :"7f06e1599e6d78c80302d8032b0d36b8a37f6410", "appname":"******", "environment":"prd","branch":"dev","k8sversion":"1.18"}
data = json.dumps(data)
res = requests.post(url, data=data)
print(res.text)
postman参数:
header: application/json
{
"version" :"7f06e1599e6d78c80302d8032b0d36b8a37f6410", "appname":"******",
"environment":"prd",
"branch":"dev",
"k8sversion":"1.18"
}
返回示例(以下实际参数为示例):
正常
{
"code": "200",
"data": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n annotations:\n deployment.kubernetes.io/revision: '1'\n generation: 3\n labels:\n app: voc-work-weixin\n branch: dev\n version: eced3eadff71665b10231cc7f4010148e5635dfe\n name: voc-work-weixin-eced3eadff71665b10231cc7f4010148e5635dfe\n namespace: app\nspec:\n progressDeadlineSeconds: 600\n replicas: 1\n revisionHistoryLimit: 2\n selector:\n matchLabels:\n app: voc-work-weixin\n branch: dev\n version: eced3eadff71665b10231cc7f4010148e5635dfe\n template:\n metadata:\n annotations:\n prometheus.io/path: api/prometheus\n prometheus.io/port: '8080'\n prometheus.io/scrape: 'true'\n creationTimestamp: null\n labels:\n app: voc-work-weixin\n branch: dev\n version: eced3eadff71665b10231cc7f4010148e5635dfe\n spec:\n containers:\n - env: []\n image: harbor.vocustcloud.com/dev/uat/voc-work-weixin:eced3eadff71665b10231cc7f4010148e5635dfe\n imagePullPolicy: IfNotPresent\n lifecycle:\n postStart:\n exec:\n command:\n - sh\n - -c\n - echo `uname -a | awk '{print $2}'` > /opt/voc-work-weixin/logs/podName\n preStop:\n exec:\n command:\n - /opt/consul/bin/consul\n - leave\n livenessProbe:\n failureThreshold: 3\n httpGet:\n path: /\n port: 8080\n scheme: HTTP\n initialDelaySeconds: 120\n periodSeconds: 60\n successThreshold: 1\n timeoutSeconds: 5\n name: voc-work-weixin\n ports:\n - containerPort: 8080\n name: service-http\n protocol: TCP\n resources:\n limits:\n cpu: 1000m\n memory: 2048Mi\n requests:\n cpu: 50m\n memory: 128Mi\n terminationMessagePath: /dev/termination-log\n terminationMessagePolicy: File\n volumeMounts:\n - mountPath: /opt/ctrip/AppData/\n name: appdata-dir\n - mountPath: /opt/voc-work-weixin/logs\n name: log-dir\n securityContext: {}\n terminationGracePeriodSeconds: 30\n volumes:\n - hostPath:\n path: /opt/ctrip/AppData/\n type: ''\n name: appdata-dir\n - hostPath:\n path: /opt/logs/voc-work-weixin/eced3eadff71665b10231cc7f4010148e5635dfe/\n type: ''\n name: log-dir\n---\napiVersion: extensions/v1beta1\nkind: Ingress\nmetadata:\n annotations:\n kubernetes.io/ingress.class: nginx\n generation: 1\n name: voc-work-weixin-ingress\n namespace: app\nspec:\n rules:\n - host: voc-work-weixin.uat.vocustcloud.com\n http:\n paths:\n - backend:\n serviceName: voc-work-weixin-service\n servicePort: 8080\nstatus:\n loadBalancer: {}\n---\napiVersion: v1\nkind: Service\nmetadata:\n labels:\n app: voc-work-weixin\n service: voc-work-weixin-service\n name: voc-work-weixin-service\n namespace: app\nspec:\n ports:\n - name: service-http\n port: 8080\n protocol: TCP\n targetPort: 8080\n selector:\n app: voc-work-weixin\n sessionAffinity: None\n type: ClusterIP\n---\n",
"msg": "OK"
}
请求参数格式问题
{
"code": "405",
"datas": {请求参数},
"msg": "json decode error 请检查参数格式是否正确"
}
env格式错误导致转dict失败检查env格式
{
"code": "405",
"datas": {},
"msg": "文件写入异常"
}
数据库连接失败
{
"code": "503",
"datas": {},
"msg": "database connection error:"
}
sql执行错误
{
"code": "503",
"datas": {},
"msg": "SQL excute Error:"
}
数据库
-- 地址:192.168.1.157
-- 库:`cloudci_k8s` 表:`ci_deployment`
-- 用户名密码:root/root
-- 建表语句:
CREATE TABLE `ci_deployment` (
`id` int(5) NOT NULL AUTO_INCREMENT,
`appname` varchar(120) NOT NULL COMMENT '服务名称',
`namespace` varchar(20) NOT NULL DEFAULT 'app' COMMENT '命名空间',
`environment` varchar(20) DEFAULT NULL COMMENT '环境分支 区分各个环境',
`branch` varchar(20) NOT NULL COMMENT '代码分支',
`location` int(1) DEFAULT '1' COMMENT '1 后端 0 前端',
`podnum` int(2) NOT NULL DEFAULT '1' COMMENT 'pod数量',
`imageurl` varchar(120) NOT NULL COMMENT '镜像地址',
`env` mediumtext NOT NULL COMMENT 'env变量',
`memory_requests` varchar(5) NOT NULL DEFAULT '512' COMMENT '内存值 单位Mi',
`cpu_requests` varchar(5) NOT NULL DEFAULT '250' COMMENT 'CPU值 单位m',
`memory_limits` varchar(5) NOT NULL DEFAULT '2048' COMMENT '内存值 单位Mi',
`cpu_limits` varchar(5) NOT NULL DEFAULT '1000' COMMENT 'CPU值 单位m',
`flag` int(1) NOT NULL DEFAULT '1' COMMENT '1启动 0未启用',
`deployment` int(1) NOT NULL DEFAULT '1' COMMENT 'deployment 1启动 0未启用',
`ingress` int(1) NOT NULL DEFAULT '1' COMMENT 'ingress 1启动 0未启用',
`service` int(1) NOT NULL DEFAULT '1' COMMENT 'service 1启动 0未启用',
PRIMARY KEY (`id`),
UNIQUE KEY `appname` (`appname`,`environment`,`branch`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
如果你看完了想要代码可以私信