dolphinscheduler 3.0.1数据质量

韧小钊

已于 2022-11-14 15:05:26 修改

阅读量4.1k

点赞数 5

分类专栏： # dolphinscheduler 文章标签： hadoop 大数据 spark 质量节点数据质量

于 2022-10-26 18:38:08 首次发布

本文链接：https://blog.csdn.net/qq_36434219/article/details/127520960

版权

dolphinscheduler 专栏收录该内容

62 篇文章 36 订阅

订阅专栏

dolphinscheduler 3.0.1数据质量

🐬环境配置
🐬定义质量节点测试
- 🐠运行失败
🐬源码
其它(spark-submit)20221114补充

🔼上一集：dolphinscheduler 3.0.1 资源中心

*️⃣主目录：dolphinscheduler 3.0.1功能梳理及源码解读

🔽下一集：dolphinscheduler 3.0.1 数据源中心

🐬环境配置

/worker-server/conf/dolphinscheduler_env.sh
- SPARK_HOME2 ：配置spark安装目录
- HADOOP_USER_NAME：增加该变量，填写hadoop集群的部署用户(HADOOP_HOME等不需要配置，目前我是没遇到过)
```
org.apache.hadoop.security.AccessControlException): Permission denied: user=dolphin, access=WRITE
```
/worker-server/conf/common.properties
- data-quality.jar.name=dolphinscheduler-data-quality-3.0.1-SNAPSHOT.jar :保持jar包名称和编译后的名称一致，默认为dolphinscheduler-data-quality-dev-SNAPSHOT.jar
给执行租户赋权，添加hadoop部署用户组，比如租户dolphin,hadoop部署用户组为bigdata
```
sudo usermod -a -G bigdata dolphin
```

🐬定义质量节点测试

直接对着官网操作吧，最开始官网资料还没有3.0，后来就不看了，貌似多走了一些路。

校验公式：[校验方式][操作符][阈值]，如果结果为真，则表明数据不符合期望，执行失败策略
校验方式：
- [Expected-Actual][期望值-实际值] [Actual-Expected][实际值-期望值]
- [Actual/Expected][实际值/期望值]x100%
- [(Expected-Actual)/Expected][(期望值-实际值)/期望值]x100%
操作符：=、>、>=、<、<=、!=
期望值类型：固定值日均值周均值月均值最近7天均值最近30天均值源表总行数目标表总行数
例子：
- 校验方式为：[Expected-Actual][期望值-实际值] [操作符]：> [阈值]：0 期望值类型：固定值=9。
- 假设实际值为10，操作符为 >, 期望值为9，那么结果 10 -9 > 0 为真，那就意味列为空的行数据已经超过阈值，任务被判定为失败

🐠运行失败

按理说应该成功才对，结果不会小于0的
在这里插入图片描述
错误信息：

[INFO] 2022-10-26 08:20:10.909 +0000 [taskAppId=TASK-20221026-7339496693088_2-471-499] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[205] - process has exited, execute path:/tmp/dolphinscheduler3/exec/process/7338799615584/7339496693088_2/471/499, processId:260177 ,exitStatusCode:1 ,processWaitForStatus:true ,processExitValue:1
[INFO] 2022-10-26 08:20:11.582 +0000 [taskAppId=TASK-20221026-7339496693088_2-471-499] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[63] -  -> 22/10/26 16:20:10 INFO Client: Application report for application_1658989604268_0102 (state: FAILED)
	22/10/26 16:20:10 INFO Client: 
		 client token: N/A
		 diagnostics: Application application_1658989604268_0102 failed 1 times (global limit =2; local limit is =1) due to ApplicationMaster for attempt appattempt_1658989604268_0102_000001 timed out. Failing the application.
		 ApplicationMaster host: N/A
		 ApplicationMaster RPC port: -1
		 queue: default
		 start time: 1666771787933
		 final status: FAILED
		 tracking URL: http://bigdata02:8088/cluster/app/application_1658989604268_0102
		 user: hadoop
	22/10/26 16:20:10 ERROR Client: Application diagnostics message: Application application_1658989604268_0102 failed 1 times (global limit =2; local limit is =1) due to ApplicationMaster for attempt appattempt_1658989604268_0102_000001 timed out. Failing the application.
	Exception in thread "main" org.apache.spark.SparkException: Application application_1658989604268_0102 finished with failed status
		at org.apache.spark.deploy.yarn.Client.run(Client.scala:1283)
		at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1677)
		at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
		at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
		at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
		at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
		at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
		at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
		at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
	22/10/26 16:20:10 INFO ShutdownHookManager: Shutdown hook called
	22/10/26 16:20:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-869c7fa1-4136-4458-8626-27961ce772e1
	22/10/26 16:20:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-873afb0a-8352-48c7-981a-7be426e1c145
[INFO] 2022-10-26 08:20:11.582 +0000 [taskAppId=TASK-20221026-7339496693088_2-471-499] TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.dq.DataQualityTask:[57] - FINALIZE_SESSION

在这里插入图片描述
测试用的spark是3.2版本，不知道报错是不是这个原因，暂时没时间研究了，有知道的欢迎指教

🐬源码

在这里插入图片描述

简单过了下，不深究了，后面还会单独研究后台核心代码

其它(spark-submit)20221114补充

spark-submit官网
在这里插入图片描述

组装好的命令没有看到--class配置，因为数据质量jar包中已有main函数，会直接调用该Main-Class: org.apache.dolphinscheduler.data.quality.DataQualityApplication

${SPARK_HOME2}/bin/spark-submit --master yarn --deploy-mode cluster --driver-cores 1 --driver-memory 512M --num-executors 2 --executor-cores 2 --executor-memory 2G --queue default --conf spark.yarn.maxAppAttempts=1 /home/dolphinscheduler/app/dolphinscheduler/worker-server/libs/dolphinscheduler-data-quality-3.0.1-SNAPSHOT.jar

在这里插入图片描述

jar包设置了Main-Class