K8s安装doris踩坑记录

最新推荐文章于 2023-11-23 10:02:53 发布

小哲今天也能摸鱼吗

最新推荐文章于 2023-11-23 10:02:53 发布

阅读量1.6k

点赞数

文章标签： kubernetes 容器云原生

本文链接：https://blog.csdn.net/qq_41864321/article/details/134398354

版权

K8s安装Doris

官方文档： https://doris.apache.org/zh-CN/docs/install/k8s-deploy/

整体流程：配置yml文件，k8s执行apply命令

以官方文档为基础，进行修改。官方文档镜像拉取存在问题，直接拉取test会出现拉取异常。

以下是一个fe节点和一个be节点的配置文件参考：

doris_follower.yml:

apiVersion: v1
kind: Service
metadata:
  name: doris-follower-cluster1
  labels:
    app: doris-follower-cluster1
spec:
  ports:
    - port: 8030
      name: http-port
    - port: 9020
      name: rpc-port
    - port: 9030
      name: query-port
    - port: 9010
      name: edit-log-port #This name should be fixed. Doris will get the port information through this name
  clusterIP: None
  selector:
    app: doris-follower-cluster1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: doris-follower-cluster1
  labels:
    app: doris-follower-cluster1
spec:
  selector:
    matchLabels:
      app: doris-follower-cluster1
  serviceName: doris-follower-cluster1
  # 修改创建副本数量为1
  replicas: 1
  template:
    metadata:
      name: doris-follower-cluster1
      labels:
        app: doris-follower-cluster1
    spec:
      ## 给当前pod加入host
      #hostAliases:
      #- ip: "x"
      #  hostnames:
      #  - "apiserver.cluster.local"
      containers:
        - name: doris-follower-cluster1
          #修改镜像 依据https://hub.docker.com/r/apache/doris/tags获取
          image: apache/doris:2.0.0_alpha-fe-x86_64
          imagePullPolicy: IfNotPresent
          env:
            # 增加APP_NAMESPACE与FE_IPADDRESS环境变量 
            - name: APP_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: FE_IPADDRESS
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            #Specify the startup type as k8s to bypass some restrictions of the official image initialization script
            - name: BUILD_TYPE
              value: "k8s"
            #Initialize the fe of three nodes
            - name: FE_INIT_NUMBER
              # fe创建数量改为1
              value: "1"
            #ServiceName of bakend_cn node,（if do not have bakend_cn node,do not configure this environment variable）
            - name: CN_SERVICE
              value: "doris-cn-cluster1"
            #StatefulSetName of bakend_cn node,（if do not have bakend_cn node,do not configure this environment variable)
            - name: CN_STATEFULSET
              value: "doris-cn-cluster1"
            #ServiceName of bakend node,（if do not have bakend node,do not configure this environment variable）
            - name: BE_SERVICE
              value: "doris-be-cluster1"
            #StatefulSetName of bakend node,（if do not have bakend node,do not configure this environment variable)
            - name: BE_STATEFULSET
              value: "doris-be-cluster1"
            #ServiceName of follower node,（if do not have follower node,do not configure this environment variable）
            - name: FE_SERVICE
              value: "doris-follower-cluster1"
            ##StatefulSetName of follower node,（if do not have follower node,do not configure this environment variable)
            - name: FE_STATEFULSET
              value: "doris-follower-cluster1"
          ports:
            - containerPort: 8030
              name: http-port
            - containerPort: 9020
              name: rpc-port
            - containerPort: 9030
              name: query-port
            - containerPort: 9010
              name: edit-log-port
          volumeMounts:
            #Mount the configuration file in the way of configmap
            - name: conf
              mountPath: /opt/apache-doris/fe/conf
              #In order to call the api of k8s
            - name: kube
              mountPath: /root/.kube/config
              readOnly: true
      volumes:
        - name: conf
          configMap:
            name: follower-conf
        - name: kube
          hostPath:
            path: /root/.kube/config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: follower-conf
data:
  fe.conf: |
    # 更改为对应的k8s网络地址
    priority_networks = 172.16.0.0/24
    #It can automatically maintain node information by getting the number of replicas of StatefulSet, similar to alter system add/drop back
    enable_deploy_manager = k8s
    #Automatically adjust the IP of the node according to the domain name (for example, after the pod is restarted, the domain name is still doris-be-cluster1-0-doris-be-cluster1.default.svc.cluster.local, but the IP may change from 172.16.0.9 to 172.16.0.10)
    enable_fqdn_mode = true
    LOG_DIR = ${DORIS_HOME}/log
    sys_log_level = INFO
    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    #Doris needs to generate the log4j configuration file according to the fe.yml configuration information, which is written in the same directory as fe.yml by default, but the config we mount is readonly, so specify this configuration to write the log4j file to another location
    custom_config_dir = /opt/apache-doris/
    #when set to false, the backend will not be dropped and remaining in DECOMMISSION state
    drop_backend_after_decommission = false

k8s对应的网络地址请修改为使用命令获取pod的CIDR参数–cluster-cidr

ps -ef | grep kube-controller-manager

如果您没有直接访问控制平面节点的权限，可以使用 kubectl 命令：

kubectl -n kube-system get pods -l component=kube-controller-manager -o yaml

使用命令启动fe

kubectl create -f doris_follower.yml

可能出现异常和问题处理：

Failed to pull image “apache-doris-fe:test”: rpc error: code = Unknown desc = failed to pull and unpack image
检查镜像是否填写正常
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [StatefulSet] with name: [doris-follower-cluster1] in namespace: [default] failed.
- 首先检查命名空间是否正确，如果设定了命名空间但是sts还是去default获取，检查APP_NAMESPACE是否增加
- 如果已经获取到对应命名空间，但是sts还是报错，请为当前namespace添加role权限
- 如果出现 Caused by: java.net.UnknownHostException: apiserver.cluster.local: Name or service not known 请检查/root/.kube/config 是否写了 server: https://apiserver.cluster.local:6443 地址，并且检查/etc/hosts 对应地址，将它加入hostAliases中
current node is not added to the group. please add it first
检查网络地址priority_networks是否正确

doris_be.yml:

apiVersion: v1
kind: Service
metadata:
  name: doris-be-cluster1
  labels:
    app: doris-be-cluster1
spec:
  ports:
    - port: 9060
      name: be-port
    - port: 8040
      name: webserver-port
    - port: 9050
      name: heartbeat-port #This name should be fixed. Doris will get the port information through this name
    - port: 8060
      name: brpc-port
  clusterIP: None
  selector:
    app: doris-be-cluster1
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: doris-be-cluster1
  labels:
    app: doris-be-cluster1
spec:
  selector:
    matchLabels:
      app: doris-be-cluster1
  serviceName: doris-be-cluster1
  # 修改创建副本数量
  replicas: 1
  template:
    metadata:
      name: doris-be-cluster1
      labels:
        app: doris-be-cluster1
    spec:
      containers:
        - name: doris-be-cluster1
          # 修改镜像地址
          image: apache/doris:2.0.0_alpha-be-x86_64
          imagePullPolicy: IfNotPresent
          env:
            #Specify the startup type as k8s to bypass some restrictions of the official image initialization script
            - name: BUILD_TYPE
              value: "k8s"
              # 写明FE的IP与端口 (FQDN)
            - name: FE_MASTER_IP
              value: "doris-follower-cluster1-0.doris-follower-cluster1.default.svc.cluster.local"
            - name: FE_MASTER_PORT
              value: "9030"
          ports:
            - containerPort: 9060
              name: be-port
            - containerPort: 8040
              name: webserver-port
            - containerPort: 9050
              name: heartbeat-port
            - containerPort: 8060
              name: brpc-port
          volumeMounts:
              #Mount the configuration file in the way of configmap
            - name: conf
              mountPath: /opt/apache-doris/be/conf
              #Ifnot mounted, when enable_profile, error will be reported when querying the data from jdbc catalog
              #Error message: error setting certificate verify locations: CAfile:/etc/pki/tls/certs/ca-bundle.crt CApath: none
            - name: sys
              mountPath: /etc/pki
              readOnly: true
      volumes:
        - name: conf
          configMap:
            name: be-conf
        - name: sys
          hostPath:
            path: /etc/pki
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: be-conf
data:
  be.conf: |
    PPROF_TMPDIR="$DORIS_HOME/log/"
    sys_log_level = INFO

    be_port = 9060
    webserver_port = 8040
    heartbeat_service_port = 9050
    brpc_port = 8060

    # 修改为k8s网络
    priority_networks = 172.16.0.0/24

使用命令启动be

kubectl create -f doris_be.yml

可能出现异常和问题处理：

be找不到fe
检查服务的完全限定域名 (FQDN) 是否配置错误

doris-svc.yaml
对外暴露端口提供访问

apiVersion: v1
kind: Service
metadata:
  name: doris-svc
  namespace: udm
spec:
  type: NodePort
  ports:
    - port: 8030
      nodePort: 8030
      name: "p8030"
    - port: 9030
    
      nodePort: 9030
      name: "p9030"
    - port: 8040
      nodePort: 8040
      name: "p8040"
  selector:
    app: doris-follower-cluster1