K8s集群部署DolphinScheduler
文章目录
一.准备部署文件
1.从Github上下载部署文件
Tags · apache/dolphinscheduler访问这个地址，在该界面找到自己要安装的DolphinScheduler版本。
上传到服务器上后解压：
#此处选择是home目录
cd /home
unzip dolphinscheduler-3.2.2
cd dolphinscheduler-3.2.2/
cd deploy/kubernetes/dolphinscheduler/
就可以看到里面有部署需要的Chart.yaml和values.yaml文件。
开始是不会有
charts和my-values.yaml这几个文件的。
2.确认helm版本
先决条件
- Helm 3.1.0+
- Kubernetes 1.12+
- PV 供应(需要基础设施支持)
helm version
kubectl version
PV（NFS）安装配置，和K8s安装配置，在以下文章中有提及，暂不在此教程做说明：
二.部署前准备
1.准备镜像
helm的安装方式默认的镜像源地址都是docker.io或者ghcr.io`的，都是外网，在部署的时候会出现镜像拉取不到的境况。所以需要提前准备自己的镜像仓库。参考方法如下：阿里云容器镜像服务搭一个自己的镜像仓库_注册阿里云容器镜像服务-CSDN博客
以下是我自己准备的镜像：
2.修改Chart的镜像源地址
原来的Chart.yaml地址默认是https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami，这个地址在执行helm update命令的时候，会因为无法拉取dependencies而报错。所以需要修改为：https://raw.gitmirror.com/bitnami/charts/archive-full-index/bitnami，这个地址是镜像代理地址，可以解决无法在内网拉取github镜像的问题。
修改后的文件如下图：
除了这几个地址，其他的不用做修改。
修改后执行如下命令：
helm repo add bitnami-full-index https://raw.gitmirror.com/bitnami/charts/archive-full-index/bitnami
helm repo add bitnami https://charts.bitnami.com/bitnami
helm dependency update .
在更新完依赖之后，就会在目录中看到charts文件夹。
因为本次安装步骤除了Mysql都需要同步安装，所以先解压minio-11.10.13.tgz，postgresql-10.3.18.tgz，zookeeper-6.5.3.tgz这几个压缩包：
#示例
tar -zxvf zookeeper-6.5.3.tgz
3.准备my-values.yaml配置文件
helm部署流程中，values.yaml配置文件在部署时可以自己调整部署参数， 例如镜像。
下面是我自己的配置文件my-values.yaml，可以做参考，后面会简要解释一下，为什么要这么改。
timezone: "Asia/Shanghai"
initImage:
pullPolicy: "IfNotPresent"
busybox: "registry.cn-hangzhou.aliyuncs.com/docker_image-ljx/busybox:1.30.1"
image:
registry: registry.cn-hangzhou.aliyuncs.com/docker_image-ljx
tag: 3.2.2
pullPolicy: "IfNotPresent"
pullSecret: ""
master: dolphinscheduler-master
worker: dolphinscheduler-worker
api: dolphinscheduler-api
alert: dolphinscheduler-alert-server
tools: dolphinscheduler-tools
datasource:
profile: postgresql
postgresql:
enabled: false
mysql:
enabled: false
registryJdbc:
enabled: false
hikariConfig:
enabled: false
postgresql:
image:
registry: registry.cn-hangzhou.aliyuncs.com
repository: docker_image-ljx/postgresql
tag: 11.11.0-debian-10-r71
pullPolicy: IfNotPresent
enabled: true
postgresqlUsername: "root"
postgresqlPassword: "pacotest"
postgresqlDatabase: "dolphinscheduler"
driverClassName: "org.postgresql.Driver"
params: "characterEncoding=utf8"
persistence:
enabled: false
minio:
image:
registry: registry.cn-hangzhou.aliyuncs.com
repository: docker_image-ljx/minio
tag: 2022.10.29-debian-11-r0
enabled: true
auth:
rootUser: minioadmin
rootPassword: minioadmin
persistence:
enabled: false
defaultBuckets: "dolphinscheduler"
ingress:
tls: false
zookeeper:
image:
registry: registry.cn-hangzhou.aliyuncs.com
repository: docker_image-ljx/zookeeper
tag: 3.6.2-debian-10-r185
enabled: true
service:
port: 2181
fourlwCommandsWhitelist: "srvr,ruok,wchs,cons"
persistence:
enabled: false
registryEtcd:
enabled: false
registryJdbc:
enabled: false
externalRegistry:
registryPluginName: "zookeeper"
registryServers: "127.0.0.1:2181"
master:
enabled: true
podManagementPolicy: "Parallel"
updateStrategy:
type: RollingUpdate
replicas: "1"
resources:
limits:
memory: "2Gi"
cpu: "2"
requests:
memory: "1Gi"
cpu: "500m"
persistentVolumeClaim:
enabled: false
accessModes:
- "ReadWriteOnce"
storageClassName: "-"
storage: "20Gi"
env:
JAVA_OPTS: "-Xms1g -Xmx1g -Xmn512m"
worker:
enabled: true
podManagementPolicy: "Parallel"
updateStrategy:
type: RollingUpdate
replicas: "1"
resources:
limits:
memory: "4Gi"
cpu: "2"
requests:
memory: "1Gi"
cpu: "500m"
persistentVolumeClaim:
enabled: true
dataPersistentVolume:
enabled: true
accessModes:
- "ReadWriteOnce"
storageClassName: "nfs-client"
storage: "3Gi"
logsPersistentVolume:
enabled: true
accessModes:
- "ReadWriteOnce"
storageClassName: "nfs-client"
storage: "3Gi"
alert:
enabled: true
replicas: 1
strategy:
type: "RollingUpdate"
rollingUpdate:
maxSurge: "25%"
maxUnavailable: "25%"
persistentVolumeClaim:
enabled: false
accessModes:
- "ReadWriteOnce"
storageClassName: "-"
storage: "3Gi"
env:
JAVA_OPTS: "-Xms512m -Xmx512m -Xmn256m"
api:
enabled: true
replicas: "1"
-
initImage和image修改# -- Used to detect whether dolphinscheduler dependent services such as database are ready initImage: # -- Image pull policy. Options: Always, Never, IfNotPresent pullPolicy: "IfNotPresent" # -- Specify initImage repository busybox: "busybox:1.30.1" image: # -- Docker image repository for the DolphinScheduler registry: apache # -- Docker image version for the DolphinScheduler tag: latest # -- Image pull policy. Options: Always, Never, IfNotPresent pullPolicy: "IfNotPresent" # -- Specify a imagePullSecrets pullSecret: "" # -- master image master: dolphinscheduler-master # -- worker image worker: dolphinscheduler-worker # -- api-server image api: dolphinscheduler-api # -- alert-server image alert: dolphinscheduler-alert-server # -- tools image tools: dolphinscheduler-tools原配置如上，需要修改
busybox和dolphinscheduler各个组件的镜像地址，源镜像地址是apache的。修改后如下timezone: "Asia/Shanghai" initImage: pullPolicy: "IfNotPresent" busybox: "registry.cn-hangzhou.aliyuncs.com/docker_image-ljx/busybox:1.30.1" image: registry: registry.cn-hangzhou.aliyuncs.com/docker_image-ljx tag: 3.2.2 pullPolicy: "IfNotPresent" pullSecret: "" master: dolphinscheduler-master worker: dolphinscheduler-worker api: dolphinscheduler-api alert: dolphinscheduler-alert-server tools: dolphinscheduler-tools -
修改
postgresql配置，原配置如下datasource: # -- The profile of datasource profile: postgresql postgresql: # -- If not exists external PostgreSQL, by default, the DolphinScheduler will use a internal PostgreSQL enabled: true # -- The username for internal PostgreSQL postgresqlUsername: "root" # -- The password for internal PostgreSQL postgresqlPassword: "root" # -- The database for internal PostgreSQL postgresqlDatabase: "dolphinscheduler" # -- The driverClassName for internal PostgreSQL driverClassName: "org.postgresql.Driver" # -- The params for internal PostgreSQL params: "characterEncoding=utf8" persistence: # -- Set postgresql.persistence.enabled to true to mount a new volume for internal PostgreSQL enabled: false # -- `PersistentVolumeClaim` size size: "20Gi" # -- PostgreSQL data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning storageClass: "-"此处配置了默认的数据库为
profile: postgresql，但是没有postgresql镜像的配置，因为dolphinscheduler依赖于bitnami postgresql的Chart，且已经在上一步 解压了。我们进入解压目录，查看postgresql的values.yaml就可以找到对镜像地址的定义：image: registry: docker.io repository: bitnami/postgresql tag: 11.11.0-debian-10-r71 ## Specify a imagePullPolicy ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent' ## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images ## pullPolicy: IfNotPresent ## Optionally specify an array of imagePullSecrets. ## Secrets must be manually created in the namespace. ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ ## # pullSecrets: # - myRegistryKeySecretName ## Set to true if you would like to see extra information on logs ## It turns BASH and/or NAMI debugging in the image ## debug: false在结合一下，修改之后，
my-values配置如下：postgresql: image: registry: registry.cn-hangzhou.aliyuncs.com repository: docker_image-ljx/postgresql tag: 11.11.0-debian-10-r71 pullPolicy: IfNotPresent enabled: true postgresqlUsername: "root" postgresqlPassword: "pacotest" postgresqlDatabase: "dolphinscheduler" driverClassName: "org.postgresql.Driver" params: "characterEncoding=utf8" persistence: enabled: false -
同理
minio，zookeeper也是参考如上方法修改的。 -
接下来就是
worker,alert,master相关的修改，主要是修改原配置中的，pod数量，运行内存，是否挂载数据等，此处是演示安装流程，资源有限，就没个pod数量为一，内存大小改小：replicas: "1" resources: limits: memory: "2Gi" cpu: "2" requests: memory: "1Gi" cpu: "500m" persistentVolumeClaim: enabled: false accessModes: - "ReadWriteOnce" storageClassName: "-" storage: "20Gi" env: JAVA_OPTS: "-Xms1g -Xmx1g -Xmn512m"
综上配置好，检查好之后，执行如下命令进行部署前最后的检查。
helm template dolphinscheduler -f my-values.yaml ./
输出如上图，该命令会列出安装的所以资源deployment,statefulset,service等。已确认第一句警告不会影响部署。
三.部署并访问测试
1.部署安装
在确定my-vqlues没问题之后，就可以执行如下命令安装dolphinscheduler
kubectl create namespace dolphinscheduler
helm -n dolphinscheduler install dolphinscheduler -f my-values.yaml ./
执行之后，就可以在dolphinscheduler命名空间下查看正在进行的部署流程。
kubectl get all -n dolphinscheduler
全部部署完后如下图：
也可在kubesphere中查看安装流程以及排查问题：
2.新增Nodeport Service访问UI界面
因为我们没有配置Ingress，所以目前无法访问dolphinscheduler的UI界面。
有如下两种部署方法：
-
Yaml部署：apiVersion: v1 kind: Service metadata: name: dolphinscheduler-api-nodeport namespace: dolphinscheduler spec: ports: - name: tcp-12345 protocol: TCP port: 12345 targetPort: 12345 nodePort: 32660 selector: app.kubernetes.io/component: api app.kubernetes.io/instance: dolphinscheduler app.kubernetes.io/name: dolphinscheduler-api type: NodePortkubectl apply -f dolphinscheduler-api-nodeport.yaml就可以通过
<节点IP>:32660/dolphinscheduler/ui访问UI界面了。 -
kubesphere页面部署，步骤如下图：
3.访问UI，简单测试功能
通过<节点IP>:32660/dolphinscheduler/ui访问UI界面。默认的用户是admin，默认的密码是dolphinscheduler123。
登陆后点击监控中心，查看是否有数据：
点击安全，用户管理，测试新增用户。
至此，dolphinscheduler安装部署流程就此结束。
扫一扫
7569
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
