一、问题现象:
hadoop jar /opt/cloudera/parcels/CDH/jars/kudu-client-tools-1.10.0-cdh6.3.2.jar org.apache.kudu.mapreduce.tools.ImportCsv -Dkudu.master.addresses=cdh001:7051 -Dimportcsv.skip.bad.lines=true '-Dimportcsv.separator=\t' 'id,name' impala::kududb.my_first_table hdfs://cluster:8020/user/root/test.csv
WARNING: Use "yarn jar" to launch YARN applications.
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kudu/shaded/org/apache/commons/logging/LogFactory
at org.apache.kudu.mapreduce.KuduTableMapReduceUtil.<clinit>(KuduTableMapReduceUtil.java:68)
at org.apache.kudu.mapreduce.KuduTableMapReduceUtil$AbstractTableOutputFormatConfigurator.configure(KuduTableMapReduceUtil.java:169)
at org.apache.kudu.mapreduce.KuduTableMapReduceUtil$TableOutputFormatConfiguratorWithCommandLineParser.configure(KuduTableMapReduceUtil.java:304)
at org.apache.kudu.mapreduce.tools.ImportCsv.createSubmittableJob(ImportCsv.java:79)
at org.apache.kudu.mapreduce.tools.ImportCsv.run(ImportCsv.java:112)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.kudu.mapreduce.tools.ImportCsv.main(ImportCsv.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.lang.ClassNotFoundException: org.apache.kudu.shaded.org.apache.commons.logging.LogFactory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 14 more
二、问题原因
/opt/cloudera/parcels/CDH/jars/kudu-client-tools-1.10.0-cdh6.3.2.jar 这个库不是一个fatjar,没有包含依赖的库,看了一下错误,缺的是org.apache.kudu.shaded.org.apache.commons.logging.LogFactory包。
三、解决方案
缺依赖的库,那解决方案就是把依赖给补上,有两种方法:
1.重新编译一个包含所有依赖的fatjar包(肥jar包)
2.把依赖的jar包拷贝到HADOOP_CLASSPATH目录下。
这里我们用第二种方法:
查找依赖包
在/opt/cloudera/parcels/CDH/jars/目录下查找一下:
cd /opt/cloudera/parcels/CDH/jars
grep -r "org.apache.kudu.shaded.org.apache.commons.logging.LogFactory"
Binary file kudu-test-utils-1.10.0-cdh6.3.2.jar matches
找到存在org.apache.kudu.shaded.org.apache.commons.logging.LogFactory的jar包。
查看HADOOP_CLASSPATH目录
hadoop classpath
/etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop-yarn/.//*
拷贝依赖包到HADOOP_CLASSPATH
cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/libexec/../../hadoop/lib/
ln -s /opt/cloudera/parcels/CDH/jars/kudu-test-utils-1.10.0-cdh6.3.2.jar ./
四、重新执行,一切正常。
hadoop jar /opt/cloudera/parcels/CDH/jars/kudu-client-tools-1.10.0-cdh6.3.2.jar org.apache.kudu.mapreduce.tools.ImportCsv -Dkudu.master.addresses=cdh001:7051 -Dimportcsv.skip.bad.lines=true '-Dimportcsv.separator=\t' 'id,name' impala::kududb.my_first_table hdfs://cluster:8020/user/root/test.csv
WARNING: Use "yarn jar" to launch YARN applications.
20/05/20 10:40:26 INFO client.RMProxy: Connecting to ResourceManager at cdh001/*.*.*.*:8032
20/05/20 10:40:27 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/root/.staging/job_1589519953669_0071
20/05/20 10:40:28 INFO input.FileInputFormat: Total input files to process : 1
20/05/20 10:40:28 INFO mapreduce.JobSubmitter: number of splits:1
20/05/20 10:40:28 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
20/05/20 10:40:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1589519953669_0071
20/05/20 10:40:28 INFO mapreduce.JobSubmitter: Executing with tokens: [Kind: kudu-authn-data, Service: cdh003:7051, Ident: ]
20/05/20 10:40:29 INFO conf.Configuration: resource-types.xml not found
20/05/20 10:40:29 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
20/05/20 10:40:29 INFO impl.YarnClientImpl: Submitted application application_1589519953669_0071
20/05/20 10:40:29 INFO mapreduce.Job: The url to track the job: http://cdh002:8088/proxy/application_1589519953669_0071/
20/05/20 10:40:29 INFO mapreduce.Job: Running job: job_1589519953669_0071
20/05/20 10:40:38 INFO mapreduce.Job: Job job_1589519953669_0071 running in uber mode : false
20/05/20 10:40:38 INFO mapreduce.Job: map 0% reduce 0%
20/05/20 10:40:45 INFO mapreduce.Job: map 100% reduce 0%
20/05/20 10:40:45 INFO mapreduce.Job: Job job_1589519953669_0071 completed successfully
20/05/20 10:40:45 INFO mapreduce.Job: Counters: 35
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=224860
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=140
HDFS: Number of bytes written=0
HDFS: Number of read operations=2
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
HDFS: Number of bytes read erasure-coded=0
Job Counters
Launched map tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=4675
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4675
Total vcore-milliseconds taken by all map tasks=4675
Total megabyte-milliseconds taken by all map tasks=4787200
Map-Reduce Framework
Map input records=5
Map output records=5
Input split bytes=105
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=114
CPU time spent (ms)=2030
Physical memory (bytes) snapshot=433262592
Virtual memory (bytes) snapshot=2754244608
Total committed heap usage (bytes)=776994816
Peak Map Physical memory (bytes)=433262592
Peak Map Virtual memory (bytes)=2755149824
File Input Format Counters
Bytes Read=35
File Output Format Counters
Bytes Written=0
org.apache.kudu.mapreduce.KuduTableOutputFormat$Counters
ROWS_WITH_ERRORS=0
org.apache.kudu.mapreduce.tools.ImportCsv$Counters
BAD_LINES=0