文章目录
java api 操作helm
一、helm架构在云管理平台开发中的不足
helm至今为止,官方仍然没有ga版的api。chart的下载,部署,升级,卸载,全部依赖cli。在多集群环境下cli很难满足平台的业务要求。
通过查看github issue,社区大概有两种解决思路:
- 封装cli成api。这种方式仍然存在每个集群需要通过ssh或者ansible的方式部署helm二进制文件到master节点上,给底层部署工作添加负担
- CRD。将helm的核心能力打包成docker镜像,部署到k8s集群中,以controller的方式提供能力。利用crd的方式完成release的部署,卸载,升级,回滚等业务动作
cli方式最大的问题就在于不符合云原生的思想,而且cli的方式和helm版本锁定,如果要升级helm,需要重新适配解析console内容。 crd的问题在于,官方目前还没有ga。但仍然期待controller的方式
在github搜索helm controller
,发现了两个仓库,一个是rancher提供的controller,一个是灵雀云提供的。经过简单的测试,captain一次性安装并测试成功,并结合内部的讨论,最终决定基于captain进行开发
二、captain介绍
github: https://github.com/alauda/captain
captain是灵雀云开源的helm v3 controller
。其内部依赖helm library。所以核心的逻辑与helm client是一致的。等到后期helm官方正式ga后,可以迁移回官方正式版本
安装captain
kubectl create ns captain-system
kubectl create clusterrolebinding captain --serviceaccount=captain-system:default --clusterrole=cluster-admin
kubectl apply -n captain-system -f https://raw.githubusercontent.com/alauda/captain/master/artifacts/all/deploy.yaml
卸载captain
kubectl delete -n captain-system -f https://raw.githubusercontent.com/alauda/captain/master/artifacts/all/deploy.yaml
kubectl delete ns captain-system
chart repo问题
captain默认自带stable的helm官方仓库,helm官方的仓库地址本身没有问题,但是chart镜像中如果使用了被墙了的docker镜像,无法下载。测试的时候是使用的aliyun提供的仓库地址https://developer.aliyun.com/hub/。这样captain controller才能顺利的将chart镜像下载成功
当测试结束时,我们需要将k8s与内网的chart私库进行打通,需要新建一个ChartRepo的yaml文件
apiVersion: app.alauda.io/v1alpha1
kind: ChartRepo
metadata:
name: bitnami
namespace: captain-system
spec:
url: https://charts.bitnami.com/bitnami
然后使用kubectl create -f fileName
添加到k8s中,需要注意的是,我们使用了harbor做docker镜像和helm镜像的管理,因为docker的问题,我们使用了自签的证书,captain在根据地址同步的时候,会校验证书,这个问题我们也和官方进行了沟通,得到了解决,目前captain已经ga,可以直接使用,不需要担心证书的问题。
这里我添加完成之后查看仓库会发现没有Synced
,过一段时间之后就好啦
命令:
#添加chartRepo
$ kubectl create -f fileName
# 查看已经添加成功了的chartRepo
$ kubectl get ChartRepo -A
三、命令行安装mongodb案例
在一个位置创建一个mongodb.yaml文件
apiVersion: "app.alauda.io/v1alpha1"
kind: "HelmRequest"
metadata:
name: "mongodb-java-1"
spec:
chart: "stable/mongodb"
namespace: "test-hl2"
releaseName: "mongodb-java-1"
version: "7.8.10"
参数解释:
-
apiVersion,kind固定写法
-
namespace:这个是值生成deployment之后放在kubernetes的那个命名空间
-
releaseName:生成deployment的名称
-
version:版本号,这个是指chart version
- chart:举例
stable/mongodb
,前面是chartrepo的name,后面是repo的chart
然后在安装chart
$ kubectl apply -f mongodb.yaml -n test-hl2
查看是否安装成功
root@k8s1:~/huanglei/helm# kubectl get pods -n test-hl2
NAME READY STATUS RESTARTS AGE
mongodb-java-1-7c76c88954-tkv8l 1/1 Running 0 54m
root@k8s1:~/huanglei/helm# kubectl get hr -n test-hl2
NAME CHART VERSION NAMESPACE ALLCLUSTER PHASE AGE
mongodb-java-1 stable/mongodb 7.8.10 test-hl2 Synced 65m
命令
# 这里hr是指HelmRequest
# 查看chart方式是否部署成功
$ kubectl get hr -n test-hl2
# 查看chart方式部署遇到的问题
$ kubectl describe hr mongodb-java-1 -n test-hl2
# 删除chart方式部署
$ kubectl delete hr mongodb-java-1 -n test-hl2
四、java api操作helm
依赖
<dependency>
<groupId>io.kubernetes</groupId>
<artifactId>client-java</artifactId>
<version>12.0.0</version>
<scope>compile</scope>
</dependency>
创建mongo
private static ApiClient getApiClient(){
String master = "https://127.0.0.1:6443";
String oauthToken = "eyJhbGciOiJSUzI1NiIsImtpZCI6InNRRi1VVFpmUE9nQ3VNc25kcVFXV29nVGZWN0hJX1N5WndHX1p4STc2a3cifQ.eyJpc3MiOVhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5jIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdWzhzIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWVlZjQ0MDQtYWY5OS00NDE4LTk2YTctZWFmYzlkNDJhNmYxIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOms4cyJ9.npRtUn7_AM0yPmdHdw6DEwKxNrapN76CKlNOkc8sWJcOuhCKHT-o58oRvg5_kDfCIZyfhV0UjepkLhq1xzP_mHbbUu8_u5SnTbpFhqslOoKywXsI17oDOIQk44nXyRkrGzsM4xNKN9kov4fzSpQqhHNGfXIMA1D0WGD2nZzh2CMklVhVzbWDDnLGgzhzBr9WNDuyBVXlJc40Tz_B0aTurxZ1yZ2P34VSK_vXW8mWWZxfCSRSf6L2vyHfKwhA4ogoqopHANwOpE0O1Fz8q50kclcyxc9a-GD3nPzYISLnGbDAsuKD4qEAi6QhnXVDdUEf9XYvzTvkBeTvL8g4YSGZrQ";
ApiClient apiClient = new ClientBuilder()
//设置 k8s 服务所在 ip地址
.setBasePath(master)
//是否开启 ssl 验证
.setVerifyingSsl(false)
//插入访问 连接用的 Token
.setAuthentication(new AccessTokenAuthentication(oauthToken))
.build();
io.kubernetes.client.openapi.Configuration.setDefaultApiClient(apiClient);
return apiClient;
}
@GetMapping("/create/chart")
public String createChart() throws ApiException, JsonProcessingException {
String name = "mongodb-java-6666";
String namespace = "test-hl2";
String chart = "bitnami/mongodb";
String version = "11.0.4";
JsonObjectBuilder build = new JsonObjectBuilder().set("rootUser", "admin").set("rootPassword", "admin123123");
JsonObjectBuilder type = new JsonObjectBuilder().set("type", "NodePort");
JsonObjectBuilder values = new JsonObjectBuilder().set("service", type).set("auth",build);
JsonObject jsonObjectBuilder = new JsonObjectBuilder()
.set("apiVersion", "app.alauda.io/v1alpha1")
.set("kind", "HelmRequest")
.set("metadata", new JsonObjectBuilder().set("name", name).build())
.set("spec", new JsonObjectBuilder()
.set("chart", chart)
.set("namespace", namespace)
.set("releaseName", name)
.set("values", values)
.set("version", version)
).build();
JsonNode jsonNode = new ObjectMapper().readTree(String.valueOf(jsonObjectBuilder));
String s = new YAMLMapper().writeValueAsString(jsonNode);
System.out.println(s);
ApiClient apiClient = getApiClient();
CustomObjectsApi customObjectsApi = new CustomObjectsApi(apiClient);
Object result = customObjectsApi.createNamespacedCustomObject("app.alauda.io", "v1alpha1", namespace, "helmrequests", jsonObjectBuilder, "true", null, null);
return "ok";
}
注意:
public Object createNamespacedCustomObject(String group, String version, String namespace, String plural, Object body, String pretty, String dryRun, String fieldManager)
上面的几个参数必须唯一:
- group:app.alauda.io
- version:v1alpha1
- plural:helmrequests
测试创建结果
查看自定义设置的密码是否创建成功
删除mongo
@DeleteMapping("/delete/chart")
public String deleteChart() throws ApiException {
String namespace = "test-hl2";
String name = "mongodb-java-2";
CustomObjectsApi customObjectsApi = new CustomObjectsApi(getApiClient());
customObjectsApi.deleteNamespacedCustomObject(
"app.alauda.io",
"v1alpha1",
namespace,
"helmrequests",
name,
0,
null,
null,
null,
new V1DeleteOptions().gracePeriodSeconds(0L).propagationPolicy("Foreground"));
return "ok";
}
测试删除结果
原本这里是有俩个的,现在这里只有一个了表示删除成功
更新helm
@GetMapping("/update/mongodb-sharded")
public String updateChartMongo() throws JsonProcessingException, ApiException {
String name = "mongodb-sharded-java-33";
String namespace = "test-hl2";
String chart = "bitnami/mongodb-sharded";
String version = "4.0.6";
ApiClient apiClient = getApiClient();
CustomObjectsApi customObjectsApi = new CustomObjectsApi(apiClient);
Object object = customObjectsApi.getNamespacedCustomObject("app.alauda.io", "v1alpha1", namespace,"helmrequests", name);
String jsonStr = JSONUtil.toJsonStr(object);
JSONObject jsonObject = JSONUtil.parseObj(jsonStr);
JSONObject metadata = JSONUtil.parseObj(JSONUtil.toJsonStr(jsonObject.get("metadata")));
String resourceVersion = (String)metadata.get("resourceVersion");
JsonObjectBuilder type = new JsonObjectBuilder().set("type", "NodePort").set("nodePort","30010");
JsonObjectBuilder resources = new JsonObjectBuilder()
.set("limits", new JsonObjectBuilder().set("cpu", "500m").set("memory","1Gi"))
.set("requests",new JsonObjectBuilder().set("cpu", "25m").set("memory","512Mi"));
JsonObjectBuilder values = new JsonObjectBuilder()
.set("mongos",new JsonObjectBuilder().set("replicas","3").set("resources",resources))
.set("mongodbRootPassword","root")
.set("service", type)
.set("resources",resources);
JsonObject jsonObjectBuilder = new JsonObjectBuilder()
.set("apiVersion", "app.alauda.io/v1alpha1")
.set("kind", "HelmRequest")
.set("metadata", new JsonObjectBuilder().set("name", name).set("resourceVersion",resourceVersion).build())
.set("spec", new JsonObjectBuilder()
.set("chart", chart)
.set("namespace", namespace)
.set("releaseName", name)
.set("values", values)
.set("version", version)
).build();
try {
customObjectsApi.replaceNamespacedCustomObject("app.alauda.io","v1alpha1",namespace,"helmrequests",name,jsonObjectBuilder,null,null);
} catch (ApiException e) {
e.printStackTrace();
}
return "ok";
}
注意:
这里需要先根据名称和命名空间查出这个helm的resourceVersion他的值为多少,然后根据下图的格式进行赋值,这里通过java代码查询出来的,跟kubernetes看到的不一样,填写根据java代码查询出来的结果
创建CharRepo仓库
@GetMapping("/create/chartRepo")
public String createChartRepo() {
CustomObjectsApi customObjectsApi = new CustomObjectsApi(getApiClient());
JsonObject jsonObjectBuilder = new JsonObjectBuilder()
.set("apiVersion", "app.alauda.io/v1alpha1")
.set("kind", "ChartRepo")
.set("metadata", new JsonObjectBuilder().set("name", "hl").set("namespace","captain-system").build())
.set("spec", new JsonObjectBuilder()
.set("url", "https://charts.bitnami.com/bitnami")
).build();
try {
customObjectsApi.createNamespacedCustomObject("app.alauda.io", "v1alpha1", "captain-system", "chartrepos", jsonObjectBuilder, "true", null, null);
} catch (ApiException e) {
e.printStackTrace();
}
return "ok";
}
结果测试:
这里就是需要等很长时间才会出现Synced的状态
删除ChartRepo仓库
@DeleteMapping("/delete/chartRepo")
public String deleteChartRepo() {
CustomObjectsApi customObjectsApi = new CustomObjectsApi(getApiClient());
try {
customObjectsApi.deleteNamespacedCustomObject(
"app.alauda.io",
"v1alpha1",
"captain-system",
"chartrepos",
"hl",
0,
null,
null,
null,
new V1DeleteOptions().gracePeriodSeconds(0L).propagationPolicy("Foreground"));
} catch (ApiException e) {
e.printStackTrace();
}
return "ok";
}
查看结果:
这里可以看到没有名字为hl的仓库了
注意:
五、java代码操作helm命令
java代码操作helm命令,打包成镜像,这个时候,镜像里面没有helm客户端,需要挂载helm客户端
helm客户端安装
手动下载安装
#从官网下载最新版本的二进制安装包到本地:https://github.com/helm/helm/tags
tar -zxvf helm-2.9.0.tar.gz # 解压压缩包
# 把 helm 指令放到bin目录下
mv helm-2.9.0/helm /usr/local/bin/helm
helm help # 验证
注意:这里一定需要放到
/usr/local/bin/helm
目录下
通过代码操作helm命令
部署镜像
FROM java:8
MAINTAINER huanglei
ADD helm-cmd.jar helm-middleware.jar
CMD java -jar helm-middleware.jar
COPY linux-amd64/helm /usr/local/bin/helm