使用k8s部署StarRocks

---

前言

部署starrocks之前，需要先搭建一个健康的k8s集群，目前搭建的k8s版本为：
Client Version: version.Info{Major:“1”, Minor:“18”, GitVersion:“v1.18.0”, GitCommit:“9e991415386e4cf155a24b1da15becaa390438d8”, GitTreeState:“clean”, BuildDate:“2020-03-25T14:58:59Z”, GoVersion:“go1.13.8”, Compiler:“gc”, Platform:“linux/amd64”}
k8s的容器管理还是基于docker维护的，所以三台k8s集群下都需要安装docker。

部署文件准备

下文中用到的yaml文件从starrocks官网获取：
https://docs.starrocks.io/zh-cn/latest/deployment/sr_operator

一、使用operator部署StarRocks

部署pod过程中，需要拉取starrocks相关镜像，在部署之前我已经将部署所用的fe、be镜像在2个worker节点下的docker中的提前下载好

1.部署starrocks定制资源

定制资源就是Custom Resource Definition。官网starrocks.com_starrocksclusters.yaml文件。主要定义了整个starrocks-cluster集群资源。规划了fe、be、cn 部署策略以及后续怎么样去写自定义资源清单来运维starrcoks集群（这里主要就是指的自己编写starrocks-fe-and-be.yaml）。
部署命令为：

kubectl create -f starrocks.com_starrocksclusters.yaml

2.部署Operator

先来看一下operator.yaml

首先创建了一个starrocks 的命名空间，后续的所有资源全部放到这个命名空间下。
然后创建了用户、角色、权限相关的一些资源限制，即rbac。

在文件最后创建了一个controller，类型是Deployment，用的镜像为 starrocks/operator:1.8.0
然后设置了一些健康检查的策略和资源限制。

部署命令:

kubectl create -f operator.yaml

查看所创建的命名空间：

kubectl get ns

查看详情：

kubectl get deployment -n starrocks

3.部署 Starrocks 集群

对starrocks-fe-and-be.yaml进行修改

apiVersion: starrocks.com/v1alpha1
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
  namespace: starrocks
spec:
  starRocksFeSpec:
    image: starrocks/fe-ubuntu:3.0.5
    replicas: 3
    requests:
      cpu: 2
      memory: 4Gi
    service:
      type: NodePort ##指定为 NodePort
  starRocksBeSpec:
    image: starrocks/be-ubuntu:3.0.5
    replicas: 3
    requests:
      cpu: 2
      memory: 4Gi

可以看到这个集群的名称是StarrocksCluster, 然后fe、be的配置怎么样去编写这些全部是来源于Custom Resource Definition中定义的，必须严格按照那边的文件去配置，否则无法启动pod。
资源名称为：starrockscluster-sample
部署命令：

kubectl create -f starrocks-fe-and-be.yaml

查看pod启动的命令：
kubectl get pod -n starrocks

可以看到这个地方是按照配置文件的顺序进行启动的，即先部署fe pod、后部署be pod

二、配置starrocks FE、 BE参数

1.创建自定义配置

拉取一份fe.conf/be.conf文件，基于该conf文件创建configmap。

kubectl create configmap fe-config --from-file=fe.conf -n starrocks

查看fe-config详情

kubectl describe cm fe-config -n starrocks

Name:         fe-config
Namespace:    starrocks
Labels:       <none>
Annotations:  <none>

Data
====
fe.conf:
----
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

#####################################################################
## The uppercase properties are read and exported by bin/start_fe.sh.
## To see all Frontend configurations,
## see fe/src/com/starrocks/common/Config.java

# the output dir of stderr/stdout/gc
LOG_DIR = ${STARROCKS_HOME}/log

DATE = "$(date +%Y%m%d-%H%M%S)"
JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0  -Xloggc:${LOG_DIR}/fe.gc.log.$DATE"

# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Dlog4j2.formatMsgNoLookups=true -Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"

##
## the lowercase properties are read by main program.
##

# DEBUG, INFO, WARN, ERROR, FATAL
# sys_log_level = INFO   

# store metadata, create it if it is not exist.
# Default value is ${STARROCKS_HOME}/meta
# meta_dir = ${STARROCKS_HOME}/meta

# http_port = 8030         
# rpc_port = 9020      
# query_port = 9030         
# edit_log_port = 9010  
mysql_service_nio_enabled = true

# Enable jaeger tracing by setting jaeger_grpc_endpoint
# jaeger_grpc_endpoint = http://localhost:14250

# Choose one if there are more than one ip except loopback address. 
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
# priority_networks = 10.10.10.0/24;192.168.0.0/16

# Advanced configurations 
# log_roll_size_mb = 1024
# sys_log_dir = ${STARROCKS_HOME}/log
# sys_log_roll_num = 10
# sys_log_verbose_modules = 
# audit_log_dir = ${STARROCKS_HOME}/log
# audit_log_modules = slow_query, query
# audit_log_roll_num = 10
# meta_delay_toleration_second = 10
# qe_max_connection = 1024
# max_conn_per_user = 100
# qe_query_timeout_second = 300
# qe_slow_log_ms = 5000

sys_log_level = INFO
mysql_server_version=5.7.43

max_create_table_timeout_second = 900


BinaryData
====

Events:  <none>

添加了如下2个配置项，后续方便观察是否生效
mysql_server_version=5.7.43
max_create_table_timeout_second = 900

2.FE/BE绑定ConfigMap

修改完毕后执行命令生效

kubectl delete -f starrocks-fe-and-be.yaml 
kubectl apply -f starrocks-fe-and-be.yaml 

查看配置文件是否生效

3. 后续配置修改

后续如果需要更新配置文件，可以利用configMap的热更新功能。
修改fe.conf之后，使用如下命令进行热更新

kubectl create cm fe-config  --from-file=fe.conf -n starrocks --dry-run -o yaml | kubectl replace -f-

–dry-run：试运行的方法
-o yaml :将生成的configMap 转换成yaml结构进行打印
-f-：将|之前结果当作这里的参数输入

执行成功后，使用如下命令进入容器内查看配置文件是否被修改。

 kubectl  exec -it starrockscluster-sample-fe-0 -n starrocks /bin/bash

修改完毕后执行命令生效

kubectl delete -f starrocks-fe-and-be.yaml 
kubectl apply -f starrocks-fe-and-be.yaml