datasophon升级海豚调度为3.2.1

已于 2024-08-13 22:27:49 修改
阅读量413 收藏 8
点赞数 6
文章标签： flink
于 2024-08-13 20:34:23 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/weixin_45249411/article/details/141172235
版权

官网下载dolphinscheduler安装包，解压结构如下：

修改安装包名称

mv apache-dolphinscheduler-3.2.1-bin dolphinscheduler-3.2.1

添加mysql驱动

dolphinscheduler3.2.1使用mysql8，安装包中没有mysql8的连接器，每个模块都需要，防止不同场景下报错

dolphinscheduler-3.2.1/api-server/libs
dolphinscheduler-3.2.1/alert-server/libs
dolphinscheduler-3.2.1/master-server/libs
dolphinscheduler-3.2.1/standalone-server/libs
dolphinscheduler-3.2.1/worker-server/libs
dolphinscheduler-3.2.1/tools/libs

添加jmx

将datasophon的jmx文件夹直接放到dolphinscheduler3.2.1

#jmx服务端口配置：

api-server：12356
master-server：12357
worker-server：12358
alert-server：12359

scp -r /opt/datasophon/dolphinscheduler-3.1.8/jmx  /opt/datasophon/dolphinscheduler-3.2.1/

修改api-server的启动脚本

1、添加hdfs配置文件到api-server的conf，用于ds的资源中心上传文件到hdfs使用，否则会报空指针错误

2、在api-server的启动命令中添加api-server的jmx端口号 api-server：12356

3、修改默认启动的jvm值， 否则无法上传大文件（会报内存溢出）
api-server/bin/start.sh

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

BIN_DIR=$(dirname $0)
DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/..; pwd)}

source "$DOLPHINSCHEDULER_HOME/conf/dolphinscheduler_env.sh"

scp ${HADOOP_HOME}/etc/hadoop/core-site.xml ${DOLPHINSCHEDULER_HOME}/conf
scp ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml ${DOLPHINSCHEDULER_HOME}/conf

JVM_ARGS_ENV_FILE=${BIN_DIR}/jvm_args_env.sh
JVM_ARGS="-server"

if [ -f $JVM_ARGS_ENV_FILE ]; then
  while read line
  do
      if [[ "$line" == -* ]]; then
        JVM_ARGS="${JVM_ARGS} $line"
      fi
  done < $JVM_ARGS_ENV_FILE
fi


JAVA_OPTS=${JAVA_OPTS:-"${JVM_ARGS} -javaagent:$BIN_DIR/../../jmx/jmx_prometheus_javaagent-0.16.1.jar=12356:$BIN_DIR/../../jmx/prometheus_config.yml"}

if [[ "$DOCKER" == "true" ]]; then
  JAVA_OPTS="${JAVA_OPTS} -XX:-UseContainerSupport"
fi

echo "JAVA_HOME=${JAVA_HOME}"
echo "JAVA_OPTS=${JAVA_OPTS}"

$JAVA_HOME/bin/java $JAVA_OPTS \
  -cp "$DOLPHINSCHEDULER_HOME/conf":"$DOLPHINSCHEDULER_HOME/libs/*" \
  org.apache.dolphinscheduler.api.ApiApplicationServer

api-server/bin/jvm_args_env.sh,修改内存参数，防止资源中心上传大文件报错 

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

-Xms4g
-Xmx4g
-Xmn2g

-XX:+IgnoreUnrecognizedVMOptions
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-Xloggc:gc.log

-XX:+ExitOnOutOfMemoryError
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=dump.hprof

-Duser.timezone=${SPRING_JACKSON_TIME_ZONE}

修改master-server的启动脚本

修改启动命令，添加jmx与datasophon-api模、模块servive_ddl.json的jmx端口保持一致

master-server: 12357

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

BIN_DIR=$(dirname $0)
DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/..; pwd)}

source "$DOLPHINSCHEDULER_HOME/conf/dolphinscheduler_env.sh"
scp ${HADOOP_HOME}/etc/hadoop/core-site.xml ${DOLPHINSCHEDULER_HOME}/conf
scp ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml ${DOLPHINSCHEDULER_HOME}/conf


JVM_ARGS_ENV_FILE=${BIN_DIR}/jvm_args_env.sh
JVM_ARGS="-server"

if [ -f $JVM_ARGS_ENV_FILE ]; then
  while read line
  do
      if [[ "$line" == -* ]]; then
            JVM_ARGS="${JVM_ARGS} $line"
      fi
  done < $JVM_ARGS_ENV_FILE
fi

JAVA_OPTS=${JAVA_OPTS:-"${JVM_ARGS} -javaagent:$BIN_DIR/../../jmx/jmx_prometheus_javaagent-0.16.1.jar=12357:$BIN_DIR/../../jmx/prometheus_config.yml"}

if [[ "$DOCKER" == "true" ]]; then
  JAVA_OPTS="${JAVA_OPTS} -XX:-UseContainerSupport"
fi

echo "JAVA_HOME=${JAVA_HOME}"
echo "JAVA_OPTS=${JAVA_OPTS}"

$JAVA_HOME/bin/java $JAVA_OPTS \
  -cp "$DOLPHINSCHEDULER_HOME/conf":"$DOLPHINSCHEDULER_HOME/libs/*" \
  org.apache.dolphinscheduler.server.master.MasterServer

修改worker-server的启动脚本

1、同api-server的修改一致，否则任务启动时无法找到hdfs路径文件

2、在worker-server的启动命令中添加api-server的jmx端口号

api-server：12358

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

BIN_DIR=$(dirname $0)
DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/..; pwd)}

source "$DOLPHINSCHEDULER_HOME/conf/dolphinscheduler_env.sh"
scp ${HADOOP_HOME}/etc/hadoop/core-site.xml ${DOLPHINSCHEDULER_HOME}/conf
scp ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml ${DOLPHINSCHEDULER_HOME}/conf

export DOLPHINSCHEDULER_WORK_HOME=${DOLPHINSCHEDULER_HOME}

JVM_ARGS_ENV_FILE=${BIN_DIR}/jvm_args_env.sh
JVM_ARGS="-server"

if [ -f $JVM_ARGS_ENV_FILE ]; then
  while read line
  do
      if [[ "$line" == -* ]]; then
            JVM_ARGS="${JVM_ARGS} $line"
      fi
  done < $JVM_ARGS_ENV_FILE
fi

JAVA_OPTS=${JAVA_OPTS:-"${JVM_ARGS} -javaagent:$BIN_DIR/../../jmx/jmx_prometheus_javaagent-0.16.1.jar=12358:$BIN_DIR/../../jmx/prometheus_config.yml"}

if [[ "$DOCKER" == "true" ]]; then
  JAVA_OPTS="${JAVA_OPTS} -XX:-UseContainerSupport"
fi

echo "JAVA_HOME=${JAVA_HOME}"
echo "JAVA_OPTS=${JAVA_OPTS}"

$JAVA_HOME/bin/java $JAVA_OPTS \
  -cp "$DOLPHINSCHEDULER_HOME/conf":"$DOLPHINSCHEDULER_HOME/libs/*" \
  org.apache.dolphinscheduler.server.worker.WorkerServer
~

修改alert-server的启动脚本

修改启动命令，添加jmx与datasophon-api模、模块servive_ddl.json的jmx端口保持一致

master-server: 12359

#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

BIN_DIR=$(dirname $0)
DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/..; pwd)}
scp ${HADOOP_HOME}/etc/hadoop/core-site.xml ${DOLPHINSCHEDULER_HOME}/conf
scp ${HADOOP_HOME}/etc/hadoop/hdfs-site.xml ${DOLPHINSCHEDULER_HOME}/conf

source "$DOLPHINSCHEDULER_HOME/conf/dolphinscheduler_env.sh"

JVM_ARGS_ENV_FILE=${BIN_DIR}/jvm_args_env.sh
JVM_ARGS="-server"

if [ -f $JVM_ARGS_ENV_FILE ]; then
  while read line
  do
      if [[ "$line" == -* ]]; then
            JVM_ARGS="${JVM_ARGS} $line"
      fi
  done < $JVM_ARGS_ENV_FILE
fi

JAVA_OPTS=${JAVA_OPTS:-"${JVM_ARGS} -javaagent:$BIN_DIR/../../jmx/jmx_prometheus_javaagent-0.16.1.jar=12359:$BIN_DIR/../../jmx/prometheus_config.yml"}

if [[ "$DOCKER" == "true" ]]; then
  JAVA_OPTS="${JAVA_OPTS} -XX:-UseContainerSupport"
fi

echo "JAVA_HOME=${JAVA_HOME}"
echo "JAVA_OPTS=${JAVA_OPTS}"

$JAVA_HOME/bin/java $JAVA_OPTS \
  -cp "$DOLPHINSCHEDULER_HOME/conf":"$DOLPHINSCHEDULER_HOME/libs/*" \
  org.apache.dolphinscheduler.alert.AlertServer

修改全局控制脚本：${DS_HOME}/bin/dolphinscheduler-daemon.sh

在$state == "STOP"条件最下方添加一行 exit 1

(status)
    get_server_running_status
    if [[ $state == "STOP" ]]; then
      #  font color - red
      state="[ \033[1;31m $state \033[0m ]"
      #增加一行，使得DataSophon执行脚本时可以有返回值判断状态
      exit 1
    else
      # font color - green
      state="[ \033[1;32m $state \033[0m ]"
    fi
    echo -e "$command  $state"
    ;;

  (*)
    echo $usage
    exit 1
    ;;

修改UI超时参数

cd dolphinscheduler-3.2.1/
 vim api-server/ui/assets/service.1059ad9a.js
 vim api-server/ui/assets/service.1059ad9a.js.gz

#搜索  baseURL:"/dolphinscheduler",timeout:
#找到该配置项，mt.getApiTimer?mt.getApiTimer:2e4 为更长的超时时间 2e7
#若要上传超大文件，可将值设大 ， 此处我们修改为mt.getApiTimer?mt.getApiTimer:2e7

重新打成二进制安装包

cd /opt/datasophon/DDP/packages
tar czf dolphinscheduler-3.2.1.tar.gz dolphinscheduler-3.2.1

可能会出现的问题：重视！

州周
关注
datasophon1.2.1升级dolphinscheduler3.2.1组件
lisa9048的博客
04-19 469
datasophon1.2.1支持的dolphinscheduler版本是3.1.8，有些海豚的新特性可能用不了，比如数据库支持Starrocks，所以尝试升级了海豚版本，其中踩了不少坑，所以整理下来，供有需要的同学参考。
Hadoop 2.6.0升级至Hadoop 3.2.1及回滚
Deegue
07-20 2880
1、背景 计划升级Hadoop版本从2.6.0-cdh5.16.1升级至开源3.2.1。 Hive版本为2.3.6，与Hadoop3.2.1不兼容，于是测试Hive3.1.3。 2、问题 ① 启动报错 Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V at org.
dolphinscheduler海豚调度升级(1.3.3-＞2.0.5)及问题总结
黑眼圈@~@从不出水文
04-14 7142
文章写于2022-04-14。我们公司用的海豚1.3.3版本，这个版本问题有点多。影响到了生产了，所以准备对海豚进行个升级。升级到海豚最新版本是2.0.5。这个版本还是有点坑，毕竟新增了一些东西。不过相对1.3.3，我更喜欢2.x的海豚。接下来，我将升级过程及升级后出现的BUG记录起来。 官网升级步骤地址：https://dolphinscheduler.apache.org/zh-cn/docs/latest/user_doc/guide/upgrade.html 升级还是比较容易的只需要.....
dolphinscheduler海豚调度升级代码改造-UpgradeDolphinScheduler
04-22 1566
dolphinscheduler数据升级代码改造背景升级代码改造测试准备工作执行upgrade-dolphinscheduler.sh脚本日志报错修改代码再次验证日志不报错查看页面验证代码UpgradeDaoupgrade-dolphinscheduler.sh说明 背景 上周记录了一次数据迁移过程，就是使用海豚调度自带的升级代码，升级完成后，手动批量更新相关表，今天继续迁移的时候，发现存储过程不好批量更新，虽然可以截取更新method参数，但是参数个数名称不好取，只能写死，这样无疑增加了后续工作量。因此还
20230821-记海豚调度2.0最新版本（dolphinscheduler-2.0.9）和3.0最新版本（ dolphinscheduler-3.1.8）的浅尝辄止
08-21 795
看bug修复记录也没啥印象，直接部署下，登录看看页面有没有啥变化吧？（以后有问题了，也可以先到最新版试一下，最新版没问题，说明已修复，再来定位具体代码）修复了这么多bug,看着真头疼，一开始没想着同步这些bug,不过同步也麻烦，二次开发之后，代码已经不一致了。二次开发，后面版本也没关注了，今天主要来瞻仰下最新版本长啥样。最新版本的源码包（3.0是真大！，是不是和指定worker分组比较像，具体用法，有待研究。），3.0最开始发布的时候，也做过简单了解，可参考。看这些bug修复，好像都没遇到过。
clickhouse集群部署
weixin_46676071的博客
08-18 339
【代码】clickhouse集群部署。
dolphinscheduler 3.2.1 集群安装部署笔记
zylohw的博客
03-20 1915
dolphinescheduler 3.2.1 集群部署 +virtualbox 虚拟机
PowerJobv3.2.1 分布式任务调度框架
08-14
PowerJob是全新一代分布式调度与计算框架，支持CRON、API、固定频率、固定延迟等调度策略，提供工作流来编排任务解决依赖关系，使用简单，功能强大，文档齐全，能让您轻松完成作业的调度与繁杂任务的分布式计算。...
ECharts v3.2.1
12-06
ECharts (Enterprise Charts 商业产品图表库) 提供商业产品常用图表，底层基于ZRender（一个全新的轻量级canvas类库），创建了坐标系，图例，提示，工具箱等基础组件，并在此上构建出折线图、柱状图、散点图、K线图...
jQuery 3.2.1
09-10
例如，`$("#id")`用于选取ID为"id"的元素，`$(".class")`则可以选取所有class为"class"的元素。 2. **事件处理**：jQuery简化了事件绑定和解绑，如`$("element").click(function() {...})`用于监听点击事件。此外，...
maven 3.2.1
06-12
Maven 3.2.1 是一个非常重要的版本，它是 Apache Maven 项目的一部分，Maven 是一个基于项目对象模型（Project Object Model，POM）的概念，用于自动化构建、依赖管理和项目信息管理的工具。对于 Java 开发者而言，...
使用Docker安装海豚调度DolphinScheduler
weixin_42454473的博客
06-08 2089
默认已安装Docker和Docker-compose环境本案例安装海豚调度3.1.7版本并配置datax插件。
Docker-compose部署海豚调度
m0_49889089的博客
11-18 1984
Docker-compose部署海豚调度
DolphinScheduler(海豚调度)- docker部署实战
最新发布
core815的专栏
05-15 820
此问题是因为selinux未关闭所致，解决方案是修改/etc/selinux/config文件，将SELINUX设为disabled，重启服务器即可。地址：http://10.86.97.26:12345/dolphinscheduler/ui。版本要求：docker-26.1.2，docker-compose-v2.11.0。默认密码：dolphinscheduler123。默认账号：admin。
海豚调度系列之：认识海豚调度
zhengzaifeidelushang的博客
03-13 2293
简单易用：丰富的使用场景：High Reliability：High Scalability：DolphinScheduler 作为一款开源分布式工作流任务调度系统，可以很好地部署和运行在 Intel 架构服务器及主流虚拟化环境下，并支持主流的Linux操作系统环境Linux 操作系统版本要求 服务器建议配置DolphinScheduler 支持运行在 Intel x86-64 架构的 64 位通用硬件服务器平台。对生产环境的服务器硬件配置有以下建议：生产环境 网络要求DolphinScheduler正常
大数据最新DolphinScheduler海豚调度教程_海豚调度使用说明
2401_84159839的博客
05-04 1979
本文档为dolphinscheduler配置文件说明文档,针对版本为 dolphinscheduler-1.3.x 版本.
使用Docker部署开源分布式任务调度系统DolphinScheduler
Aron的博客
11-09 5660
本篇教程和大家分享一下DolphinScheduler的安装部署及如何实现公网远程访问，结合内网穿透工具实现公网访问DolphinScheduler内网并进行远程办公，帮助开发人员进行远程任务调度及管理，提高工作效率。DolphinScheduler是一款开源的分布式任务调度系统，它可以帮助开发人员更加方便地进行任务调度和管理。DolphinScheduler支持常见的任务类型，包括Shell、Hadoop、Spark、Hive等，同时它也提供了可视化的任务编排和监控，使得任务管理变得更加简单。
DolphinScheduler海豚调度教程_海豚调度使用说明
2401_84160087的博客
04-08 2874
dolphinscheduler-alert 告警模块，提供 AlertServer 服务。dolphinscheduler-api web应用模块，提供 ApiServer 服务。dolphinscheduler-common 通用的常量枚举、工具类、数据结构或者基类dolphinscheduler-dao 提供数据库访问等操作。dolphinscheduler-remote 基于 netty 的客户端、服务端。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值