1.当sqoop导出程序长时间执行未完成,且mr控制台也没有报错
[w4xj@node202 sqoop-1.4.6.bin__hadoop-2.0.4-alpha]$ bin/sqoop export \
> --connect jdbc:mysql://node202:3306/company \
> --username root \
> --password 123456 \
> --table staff \
> --num-mappers 1 \
> --export-dir /user/hive/warehouse/staff_hive \
> --input-fields-terminated-by "\t"
Warning: /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop-1.4.6.bin__hadoop-2.0.4-alpha/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
19/03/31 11:02:55 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
19/03/31 11:02:55 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
19/03/31 11:02:55 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
19/03/31 11:02:55 INFO tool.CodeGenTool: Beginning code generation
19/03/31 11:02:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `staff` AS t LIMIT 1
19/03/31 11:02:56 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `staff` AS t LIMIT 1
19/03/31 11:02:56 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/module/hadoop-2.7.2
注: /tmp/sqoop-w4xj/compile/299e764abb8e42f075ad01d3ab61c92f/staff.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
19/03/31 11:02:57 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-w4xj/compile/299e764abb8e42f075ad01d3ab61c92f/staff.jar
19/03/31 11:02:57 INFO mapreduce.ExportJobBase: Beginning export of staff
19/03/31 11:02:57 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
19/03/31 11:02:58 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.speculative
19/03/31 11:02:58 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
19/03/31 11:02:58 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
19/03/31 11:02:58 INFO client.RMProxy: Connecting to ResourceManager at node203/192.168.26.203:8032
19/03/31 11:03:01 INFO input.FileInputFormat: Total input paths to process : 1
19/03/31 11:03:01 INFO input.FileInputFormat: Total input paths to process : 1
19/03/31 11:03:01 INFO mapreduce.JobSubmitter: number of splits:1
19/03/31 11:03:01 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
19/03/31 11:03:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1553997792849_0005
19/03/31 11:03:01 INFO impl.YarnClientImpl: Submitted application application_1553997792849_0005
19/03/31 11:03:01 INFO mapreduce.Job: The url to track the job: http://node203:8088/proxy/application_1553997792849_0005/
19/03/31 11:03:01 INFO mapreduce.Job: Running job: job_1553997792849_0005
19/03/31 11:03:12 INFO mapreduce.Job: Job job_1553997792849_0005 running in uber mode : false
19/03/31 11:03:12 INFO mapreduce.Job: map 0% reduce 0%
19/03/31 11:03:25 INFO mapreduce.Job: map 100% reduce 0%
19/03/31 11:13:40 INFO mapreduce.Job: Task Id : attempt_1553997792849_0005_m_000000_0, Status : FAILED
AttemptID:attempt_1553997792849_0005_m_000000_0 Timed out after 600 secs
19/03/31 11:13:42 INFO mapreduce.Job: map 0% reduce 0%
19/03/31 11:13:53 INFO mapreduce.Job: map 100% reduce 0%
2.那么我们打开yarn的控制台http://node203:8088/cluster,找到那个正在执行的job,点击ApplicationMaster
3.点击Job_ID,查看当前job的执行情况
4.点击正在运行的map
5.点击任务
6.点击logs
7.点击syslog
8.我们可以看到mysql实际已经报错了,但是sqoop控制台并没有报错,等待最终job失败才返回job失败
9.关于返回错误
[1].返回错误:
19/03/31 11:45:12 INFO mapreduce.Job: Job job_1553997792849_0005 failed with state FAILED due to: Task failed task_1553997792849_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
19/03/31 11:45:12 INFO mapreduce.Job: Counters: 9
Job Counters
Failed map tasks=4
Launched map tasks=4
Other local map tasks=3
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=2512919
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=2512919
Total vcore-milliseconds taken by all map tasks=2512919
Total megabyte-milliseconds taken by all map tasks=2573229056
19/03/31 11:45:12 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
19/03/31 11:45:12 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 2,534.5903 seconds (0 bytes/sec)
19/03/31 11:45:12 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
19/03/31 11:45:12 INFO mapreduce.ExportJobBase: Exported 0 records.
19/03/31 11:45:12 ERROR tool.ExportTool: Error during export: Export job failed!
[2].若不返回错误,程序一直卡在那儿,按Ctrl C 停止后,job还在继续运行,需要在ResourceManager上手动停止job,这里的jobId就是上面的那个Job ID
sh yarn application -kill jobId