dolphinscheduler资源中心

资源中心

资源中心介绍

资源中心提供文件管理，UDF管理，任务组管理。

文件管理可以访问要执行的hive的sql文件

UDF管理可以放置fllink执行的自定义udf函数jar包，hive自定义的UDF函数jar包

以上的*.sql,*.jar文件可以理解为资源，这些资源需要有个存储的地方，本文以minio存储作为介绍

使用minio作为资源中心的存储

• 编辑 common.properties

vi ${DOPHINSCHEDULER_HOME}/tools/conf/common.properties

• 修改如图中所示

• 修改部分的代码段

# resource storage type: HDFS, S3, OSS, NONE
resource.storage.type=S3
# resource store on HDFS/S3 path, resource file will store to this base path, self configuration, please make sure the directory exists on hdfs and have read write permissions. "/dolphinscheduler" is recommended
resource.storage.upload.base.path=/dolphinscheduler

# The AWS access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.access.key.id=root
# The AWS secret access key. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.secret.access.key=root123456
# The AWS Region to use. if resource.storage.type=S3 or use EMR-Task, This configuration is required
resource.aws.region=cn-north-1
# The name of the bucket. You need to create them by yourself. Otherwise, the system cannot start. All buckets in Amazon S3 share a single namespace; ensure the bucket is given a unique name.
resource.aws.s3.bucket.name=dolphinscheduler
# You need to set this parameter when private cloud s3. If S3 uses public cloud, you only need to set resource.aws.region or set to the endpoint of a public cloud such as S3.cn-north-1.amazonaws.com.cn
resource.aws.s3.endpoint=http://192.168.75.117:9000

注意在修改配置之后，需要在minio创建对应的桶,桶名字dolphinscheduler

• 配置文件生效

  #进入到主目录
  cd ${DOPHINSCHEDULER_HOME}
  #复制配置文件到模块
  cp ./tools/conf/common.properties ./api-server/conf/
  cp ./tools/conf/common.properties ./master-server/conf/
  cp ./tools/conf/common.properties ./worker-server/conf/
  cp ./tools/conf/common.properties ./alert-server/conf/
  # 一键停止集群所有服务
  bash ./bin/stop-all.sh
  #重新安装部署并启动
  bash ./bin/install.sh
  

minio资源预览

dolphinscheduler会在minio中创建文件的结果如下

其中root代表用户所在的租户名

#资源路径，
dolphinscheduler/root/resources/
#udf函数路径
dolphinscheduler/root/udf