Zookeeper配置 Client port found: 2181. Client address: localhost. Client SSL: false. Error contacting
8080端口被占用了
lsof -i:8080
kill -9 pid
----------------------------------------------------------------------------------------------------------------
5 配置dolphinscheduler用户sudo免密
#配置sudo免密
sed -i '$adolphinscheduler ALL=(ALL) NOPASSWD: NOPASSWD: ALL' /etc/sudoers
sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
6 配置dolphinscheduler 用户ssh免密
# 添加用户
useradd dolphinscheduler
# 设置密码 12345678
passwd dolphinscheduler
# 生成秘钥
su - dolphinscheduler
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
# 验证免密是否生效
ssh localhost
# 不需要输入密码可以ssh成功，即配置ssh成功
----------------------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------------------
cp mysql-connector-j-8.3.0.jar /home/dolphinscheduler/master-server/libs
cp mysql-connector-j-8.3.0.jar /home/dolphinscheduler/worker-server/libs
cp mysql-connector-j-8.3.0.jar /home/dolphinscheduler/api-server/libs
----------------------------------------------------------------------------------------------------------------
bin/install.sh
----------------------------------------------------------------------------------------------------------------
bin/dolphinscheduler-daemon.sh start master-server
bin/dolphinscheduler-daemon.sh start worker-server
bin/dolphinscheduler-daemon.sh start api-server
----------------------------------------------------------------------------------------------------------------
大坑:
/home/dolphinscheduler/master-server/conf/application.ymal
/home/dolphinscheduler/worker-server/conf/application.ymal
/home/dolphinscheduler/api-server/conf/application.ymal
registry:
type: zookeeper
zookeeper:
retry-policy:
base-sleep-time: 6000ms
block-until-connected: 6000ms
从60ms都改成6000ms
----------------------------------------------------------------------------------------------------------------
Windows version:
在idea工程中双击dolphinscheduler-ui/readme.md, 双击内容中的pnpm run dev
进入网页http://localhost:5173, 网页中图标不停转圈
terminal窗口中提示:vite http proxy error at dolphinscheduler
改正:
dolphinscheduler-ui/.env.development
VITE_APP_DEV_WEB_URL='http://127.0.0.1:5173'
将5173端口改成12345
----------------------------------------------------------------------------------------------------------------
Windows version:
dolphinscheduler-common/common.properties
sudo.enable=false
data.basedir.path=d:/temp/dolphinscheduler
resource.storage.upload.base.path=d:/temp/dolphinscheduler
----------------------------------------------------------------------------------------------------------------
拷贝到 mysql-connector-j-8.3.0.jar /home/dolphinscheduler/master-server/libs
拷贝到 mysql-connector-j-8.3.0.jar /home/dolphinscheduler/worker-server/libs
拷贝到 mysql-connector-j-8.3.0.jar /home/dolphinscheduler/api-server/libs
----------------------------------------------------------------------------------------------------------------
How to run:
Master Server: Dolphinscheduler-master/src/main/java/org.apache.dolphinscheduler.server.master/MasterServer 上点击右键, run ’MasterServer.main()’
Work Server: Dolphinscheduler-worker/src/main/java/org.apache.dolphinscheduler.server.worker/WorkerServer 上点击右键, run ’WorkerServer.main()’
UI Server: 双击Dolphinscheduler-ui/README.md, 双击运行pnpm run dev, 然后可以再网页中访问管理监控界面.
----------------------------------------------------------------------------------------------------------------
run on windows issue:
D:\Work\apache-dolphinscheduler-3.2.0-src\dolphinscheduler-task-plugin\dolphinscheduler-task-api\src\main\java\org\apache\dolphinscheduler\plugin\task\api\shell\ShellInterceptorBuilderFactory.java
private final static String INTERCEPTOR_TYPE = PropertyUtils.getString("shell.interceptor.type", "bash");
D:\Work\apache-dolphinscheduler-3.2.0-src\dolphinscheduler-common\src\main\resources\common.properties
# The interceptor type of Shell task, e.g. bash, sh, cmd
shell.interceptor.type=cmd
3.2.0版才有这个选项, 3.0.6版没有
----------------------------------------------------------------------------------------------------------------
D:\Work\apache-dolphinscheduler-3.2.0-src\dolphinscheduler-task-plugin\dolphinscheduler-task-api\src\main\java\org\apache\dolphinscheduler\plugin\task\api\AbstractCommandExecutor.java
public TaskResponse run(IShellInterceptorBuilder iShellInterceptorBuilder,
process = iShellInterceptor.execute();
----------------------------------------------------------------------------------------------------------------
change application.ymal->datasource
driver-class-name: com.mysql.cj.jdbc.Driver
url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler
username: root
password: root
Cannot load driver class: com.mysql.cj.jdbc.Driver
pom.xml of main project, delete:
<scope>test</scope>
在pom.xml上右键, maven->reload project
----------------------------------------------------------------------------------------------------------------
│ Update available! 8.15.1 → 8.15.4. │
│ Changelog: https://github.com/pnpm/pnpm/releases/tag/v8.15.4 │
│ Run "pnpm add -g pnpm" to update.
----------------------------------------------------------------------------------------------------------------
/task/api/AbstractCommandExecutor.java, 加上/c, 不然CMD窗口不会关闭, 一直执行.
command.add(commandInterpreter());
command.add("/c");
----------------------------------------------------------------------------------------------------------------
introduce jna.jar, idea中打开, 添加下面的依赖后保存, maven自动下载jna.jar
pom.xml
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna</artifactId>
<version>5.10.0</version>
</dependency>
<dependency>
<groupId>net.java.dev.jna</groupId>
<artifactId>jna-platform</artifactId>
<version>5.10.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.33</version>
</dependency>
然后在pom.xml上右键, maven->reload project
-----------------------------------------------------------------
import com.sun.jna.platform.win32.Kernel32;
import com.sun.jna.platform.win32.WinBase.SYSTEM_INFO;
import com.sun.jna.platform.win32.WinNT;
import com.sun.jna.ptr.IntByReference;
import com.sun.jna.Pointer;
搜一下jna的jar包的正确版本. DS3.0.6对应的事5.10.0
dolphinscheduler-task-plugin\dolphinscheduler-task-api\src\main\java\org\apache\dolphinscheduler\plugin\task\api\AbstractCommandExecutor.java
private int getProcessId(Process process) {
int processId = 0;
try {
Field[] fields = process.getClass().getDeclaredFields();
for (Field f : fields)
logger.info("Fields:" + f.getName() + f.toString());
// Field f = process.getClass().getDeclaredField(TaskConstants.PID);
Field f = process.getClass().getDeclaredField("handle");
f.setAccessible(true);
// processId = f.getInt(process);
long handl = f.getLong(process);
// Kernel32 kernel = Kernel32.INSTANCE;
WinNT.HANDLE handle = new WinNT.HANDLE();
handle.setPointer(Pointer.createConstant(handl));
processId = com.sun.jna.platform.win32.Kernel32.INSTANCE.GetProcessId(handle);
// kernel.GetProcessId(handle);
} catch (Exception e) {
logger.error("Get task pid failed", e);
}
return processId;
}
----------------------------------------------------------------------------------------------------------------
遇到过不去的错误时, 挨个把每个工程build一遍.
实在不行重启机器^_^
----------------------------------------------------------------------------------------------------------------
后台代码打包
使用命令行进入项目根目录下，使用命令进行打包（时间会很长）：
第一次用:
mvn clean package -Prelease -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
或
mvn clean install -Prelease -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
很可能编译会出错, 以后用:
mvn package -Prelease -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
或
mvn install -Prelease -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
cd dolphinscheduler-dist
安装包apache-dolphinscheduler-3.0.1-SNAPSHOT-bin.tar.gz，在dist目录下面
期间可能会报错，提示使用命令进行修复：(apache-dolphinscheduler-3.2.0-src目录下运行)
mvn spotless:apply
project dolphinscheduler-ui: Failed to run task: 'pnpm run build:prod' failed.
src/App.tsx - error JSX element implicitly has type 'any' because no JSX.IntrinsicElements' exists
dolphinscheduler-ui/tsconfig.json
"strict": false,
npm install --global yarn
yarn add axios@next
dolphinscheduler-ui/package.json
"build:prod": "vite build",
[vite] Internal server error: unknown: (0 , _genMapping.maybeAddMapping) is not a function
删除
D:\Work\apache-dolphinscheduler-3.0.6-src\dolphinscheduler-ui\node_modules\.pnpm
------------------------------------------------------------------------------------------------------------------------------------------
JAVA_OPTS=${JAVA_OPTS:-"-server -Duser.timezone=${SPRING_JACKSON_TIME_ZONE} -Xms4g -Xmx4g -Xmn2g -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof"}
java $JAVA_OPTS \
-cp ".;d:/dolphinscheduler/apache-dolphinscheduler-3.0.6-bin/master-server/conf;d:/dolphinscheduler/apache-dolphinscheduler-3.0.6-bin/master-server/libs/*" \
org.apache.dolphinscheduler.server.master.MasterServer
------------------------------------------------------------------------------------------------------------------------------------------
安装cygwin, 如果报错:could not mirrors:
cygwin-setup-x86_64.exe setup-x86_64 --site http://mirrors.163.com/cygwin/ --only-site
bat版本, 不需要cygwin64
java -server -Xms4g -Xmx4g -Xmn2g -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof -cp ".;d:/dolphinscheduler/apache-dolphinscheduler-3.0.6-bin/worker-server/conf;d:/dolphinscheduler/apache-dolphinscheduler-3.0.6-bin/worker-server/libs/*" org.apache.dolphinscheduler.server.worker.WorkerServer
最终只有apache-dolphinscheduler-3.0.6 版本制作安装包成功, 用的命令是:mvn clean install -Prelease -Dmaven.test.skip=true -Dcheckstyle.skip=true -Dmaven.javadoc.skip=true
修复完成后继续使用打包命令打包即可。打包完成后，可以在dolphinscheduler-dist下的target文件夹下找到打包好的压缩包，名称大约是 apache-dolphinscheduler-dev-SNAPSHOT-bin.tar.gz。
----------------------------------------------------------------------------------------------------------------
server load balance方式, application.ymal
host-selector: round_robin
----------------------------------------------------------------------------------------------------------------
安装另一台worker, 拷贝worker目录到这台机器就可以, 然后修改conf/application.yaml, conf/common.properties, conf/dolphinscheduler_env.sh. conf/logback-spring.xml中加上<appender-ref ref="STDOUT"/>就可以输出到标准输出窗口
另一台worker机器没反应, server上报错:dolphinscheduler.remote.NettyRemotingClient error io.netty.channel.ConnectTimeoutException: connection timed out
在worker上关闭防火墙, public就可以
----------------------------------------------------------------------------------------------------------------
运行:
c:/cygwin64/Cygwin.bat
cd d:
cd dolphinscheduler/apache-dolphinscheduler-3.0.6/master-server/bin
./start.sh
同样运行workder-server和api-server
----------------------------------------------------------------------------------------------------------------
安装WEB UI
先安装node-v20.11.0-x64.msi
将整个dolphinscheduler-ui目录拷贝到某台机器上
CMD窗口运行:
pnpm install
pnpm run dev
访问http://localhost:3000/
CMD窗口运行:pnpm run dev --host则可以被外部访问
前端初始化,有可能需要运行npm而不是pnpm
dolphinscheduler-dev\dolphinscheduler-ui>npm install
有可能会报错:npm ERR! Cannot read properties of null (reading 'matches'), 则
npm install -g npm (更新最新版本)
npm cache clear --force (强制清除缓存)
npm install (再次安装)
----------------------------------------------------------------------------------------------------------------
gRPC c# prject:
Server端:
launchSettings.json, localhost改为10.106.40.74
"applicationUrl": "https://10.106.40.74:7008;http://10.106.40.74:5031",
关闭防火墙,否则外部可能连接不到grpc server
Client端:
GrpcChannelOptions opt = new GrpcChannelOptions()
{
HttpClient = null,
HttpHandler = new HttpClientHandler
{
ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
}
};
Grpc.Net.Client.GrpcChannel triggerChannel = GrpcChannel.ForAddress("https://10.106.40.74:7008", opt);
----------------------------------------------------------------------------------------------------------------
io.netty.channel.ConnectTimeoutException: connection timed out: /172.18.0.161:5678
NettyClientConfig.java
private int connectTimeoutMillis = 3000;
改为30000
(应该不是这个问题, 是另一台机器访问master, host机上多IP, 把不用的IP禁掉就可以了)
----------------------------------------------------------------------------------------------------------------
另一台机器运行RSReportCreatorGrpcClient 报错
App: C:\dolphinscheduler\ReportServer\RSReportCreatorGrpcClient.exe
Architecture: x64
App host version: 7.0.4
.NET location: Not found
需下载net run time:
https://dotnet.microsoft.com/en-us/download/dotnet/thank-you/runtime-7.0.16-windows-x64-installer?cid=getdotnetcore
----------------------------------------------------------------------------------------------------------------
Idea 代码改动不起作用:
点击导航栏File然后点击invalidate(驳斥)Caches在选择清除缓存并且重启；此方法的截图如下所示：
小心用此方法, 运行这个命令后, web管理页面变空白了.
----------------------------------------------------------------------------------------------------------------
dolphinscheduler-ui, run dev, 报错
ERROR Could not resolve "@dagrejs/graphlib, ERROR: Could not resolve "d3-force"
运行:
pnpm install
----------------------------------------------------------------------------------------------------------------
报错:
-03-12 19:28:22.103 +0800 org.apache.dolphinscheduler.server.worker.rpc.WorkerMessageSender:[73] - [WorkflowInstance-6691][TaskInstance-6805] - Send message error, messageType: TASK_EXECUTE_RESULT, message: TaskExecuteResultCommand(super=BaseCommand(messageSenderAddress=10.75.52.45:1234, messageReceiverAddress=172.17.50.33:5678, messageSendTime=1710242881050), taskInstanceId=6805, processInstanceId=6691, status=7, startTime=Tue Mar 12 19:27:28 SGT 2024, host=10.75.52.45:1234, logPath=D:\dolphinscheduler\apache-dolphinscheduler-3.0.6-bin\worker-server\bin\logs\20240312\12835953908896_10-6691-6805.log, executePath=d:/temp/dolphinscheduler/exec/process/12835948342688/12835953908896_10/6691/6805, endTime=Tue Mar 12 19:28:01 SGT 2024, processId=18412, appIds=, varPool=[])
[INFO] 2024-03-12 19:28:22.106 +0800 org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread:[247] - [WorkflowInstance-6691][TaskInstance-6805] - develop mode is: false
[INFO] 2024-03-12 19:28:22.115 +0800 org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread:[265] - [WorkflowInstance-6691][TaskInstance-6805] - exec local path: d:/temp/dolphinscheduler/exec/process/12835948342688/12835953908896_10/6691/6805 cleared.
[INFO] 2024-03-12 19:32:28.745 +0800 org.apache.dolphinscheduler.server.worker.message.MessageRetryRunner:[126] - [WorkflowInstance-0][TaskInstance-6805] - Begin retry send message to master, message: TaskExecuteRunningCommand(super=BaseCommand(messageSenderAddress=10.75.52.45:1234, messageReceiverAddress=172.17.50.33:5678, messageSendTime=1710242848002), taskInstanceId=6805, processInstanceId=6691, startTime=Tue Mar 12 19:27:28 SGT 2024, host=10.75.52.45:1234, status=1, logPath=D:\dolphinscheduler\apache-dolphinscheduler-3.0.6-bin\worker-server\bin\logs\20240312\12835953908896_10-6691-6805.log, executePath=d:/temp/dolphinscheduler/exec/process/12835948342688/12835953908896_10/6691/6805, processId=0, appIds=null)
[WARN] 2024-03-12 19:32:49.769 +0800 org.apache.dolphinscheduler.remote.NettyRemotingClient:[368] - [WorkflowInstance-0][TaskInstance-6805] - connect to Host{address='172.17.50.33:5678', ip='172.17.50.33', port=5678} error
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: /172.17.50.33:5678
更改source code:
WorkerMessageSender.java
sendMessageWithRetry()
messageReceiverAddress = "10.106.40.74:5678";
更好的办法是:
Control Panel\All Control Panel Items\Network Connections
找到内部地址172.17.50.33的那个网卡, 把IPV4的勾选去掉, 不用这块卡.
----------------------------------------------------------------------------------------------------------------
从程序外部传送参数
1. postman
start-process-instance函数
http://localhost:12345/dolphinscheduler/projects/12835948342688/executors/start-process-instance?failureStrategy=END&processDefinitionCode=12835953908896&processInstancePriority=HIGHEST&projectCode=12835948342688&scheduleTime&warningType=NONE&startParams={"inputparam":"aaaaa"}
startParams填上{"inputparam":"aaaaa"}
2. workflow
c:\dolphinscheduler\ReportServer\RSReportCreatorGrpcClient.exe ${inputparam}
不要创建Custom Parameters. 如果创建了inputparam, 反而不行了. start-process-instance函数中生成的是globalParams,
Custom Parameters如果再创建相同的变量就冲突了.
----------------------------------------------------------------------------------------------------------------
worker.properties，我这里设置为10，意思是一个worker，最多运行10个任务实例；
这样可以控制我的app的数量限制
----------------------------------------------------------------------------------------------------------------
270
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
