工作任务

最新推荐文章于 2023-04-27 16:46:36 发布

大壮vip

最新推荐文章于 2023-04-27 16:46:36 发布

阅读量376

点赞数

分类专栏：大数据 linux python ETL 文章标签：开发 python ETL hadoop hive

本文链接：https://blog.csdn.net/qq_33792843/article/details/78832425

版权

大数据同时被 3 个专栏收录

136 篇文章 0 订阅

订阅专栏

python

27 篇文章 1 订阅

订阅专栏

linux

22 篇文章 0 订阅

订阅专栏

追查数据源，找到报表产出数据源。

首先配置好git之后，我们打开页面的url，分配的是先看产检报告。

移除点击此处添加图片说明文字

同时，将数据报表产出的etl部分从git检出，等待查阅代码。

移除点击此处添加图片说明文字

这里的git.exe不用理会，因为我们暂时打算使用命令行，后面可以我们使用客户端吧，看个人喜好，喜欢客户端或者命令行都可以。

再者，我们进入堡垒机，进入hadoop05节点，这个后面补充堡垒机配置信息。

显示权限机器，p，进入hadoop05

移除点击此处添加图片说明文字

切换root，

sudo su -

然后查阅crontab,

crontab -l |grep chanjian

页面的报表涉及到下列脚本信息，这个，后面我们兴许要看

移除点击此处添加图片说明文字

上午，我已经将sh脚本搬迁过来，搬到了/opt/etl/chanjian/

经过了两天的修改，已经将搬迁etl的工作基本完成了，解决了很多坑。比方说数据的丢失，hive没有元数据，

Error: java.io.IOException: java.io.IOException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-41827661-10.8.8.6-1438054542785:blk_1073743587_2763 file=/user/hive/warehouse/antenatal_data.db/mmbang_antenatal_hospital/dt=2015-01-16/000000_0

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)

at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:226)

at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:136)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Caused by: java.io.IOException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-41827661-10.8.8.6-1438054542785:blk_1073743587_2763 file=/user/hive/warehouse/antenatal_data.db/mmbang_antenatal_hospital/dt=2015-01-16/000000_0

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)

at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)

at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:105)

at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)

at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)

at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:224)

... 11 more

Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-41827661-10.8.8.6-1438054542785:blk_1073743587_2763 file=/user/hive/warehouse/antenatal_data.db/mmbang_antenatal_hospital/dt=2015-01-16/000000_0

at org.apache.hadoop.hdfs.DFSInputStream.chooseDataNode(DFSInputStream.java:888)

at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:568)

at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:800)

at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:847)

at java.io.DataInputStream.read(DataInputStream.java:100)

at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)

at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)

at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)

at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:206)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:244)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:47)

at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)