Sqoop安装与MapReduce读MySql写HDFS加载不了JDBC驱动问题

                                                                                                                                                         安装Sqoop

1.下载

sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
 sqoop-1.4.6.tar.gz

2.解压

tar -xf sqoop-1.4.6.tar.gz

3.vi /etc/profile

#sqoop
export SQOOP_PATH=/home/softs/sqoop-1.4.6
export PATH=$PATH:$SQOOP_PATH/bin

5.运行

出错：错误：找不到或无法加载主类 org.apache.sqoop.sqoop

解决方法：

解压sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz，进入该文件解压后的文件夹，[root@master sqoop-1.4.6.bin__hadoop-2.0.4-alpha]# cp sqoop-1.4.6.jar /home/softs/hadoop/share/hadoop/yarn/  #复制sqoop-1.4.6.jar到hadoop/share/hadoop/yarn/下

这里我想到了之前我运行MapReduce代码读取Mysql写入到HDFS中出现错误：

[root@master MySqlMapReduceToAvroHDFS]# hadoop jar export-hbase-data-jar-with-dependencies.jar mysql-connector-java-5.1.32.jar MySqlMapReduceToAvroHDFSOutput 
17/09/20 17:27:12 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.9.148.193:8032
17/09/20 17:27:12 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.9.148.193:8032
17/09/20 17:27:12 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/09/20 17:27:20 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1505372208302_0004
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver
at org.apache.hadoop.mapreduce.lib.db.DBInputFormat.setConf(DBInputFormat.java:171)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:689)
at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:330)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:322)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:198)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)

解决方法：

[root@master MySqlMapReduceToAvroHDFS]# cp mysql-connector-java-5.1.32.jar /home/softs/hadoop/share/hadoop/yarn/

复制jdbc的jar包到hadoop里，问题解决

[root@master toTextHDFS]# hadoop jar export-hbase-data-jar-with-dependencies.jar
17/09/21 11:08:54 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.9.148.193:8032
17/09/21 11:08:54 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.9.148.193:8032
17/09/21 11:08:54 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/09/21 11:09:02 INFO mapreduce.JobSubmitter: number of splits:2
17/09/21 11:09:02 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1505372208302_0011
17/09/21 11:09:03 INFO impl.YarnClientImpl: Submitted application application_1505372208302_0011
17/09/21 11:09:03 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1505372208302_0011/
17/09/21 11:09:03 INFO mapreduce.Job: Running job: job_1505372208302_0011
17/09/21 11:09:26 INFO mapreduce.Job: Job job_1505372208302_0011 running in uber mode : false
17/09/21 11:09:26 INFO mapreduce.Job:  map 0% reduce 0%
17/09/21 11:09:36 INFO mapreduce.Job:  map 50% reduce 0%
17/09/21 11:09:39 INFO mapreduce.Job:  map 100% reduce 0%
17/09/21 11:09:47 INFO mapreduce.Job:  map 100% reduce 100%
17/09/21 11:09:47 INFO mapreduce.Job: Job job_1505372208302_0011 completed successfully
17/09/21 11:09:47 INFO mapreduce.Job: Counters: 50
File System Counters
FILE: Number of bytes read=126
FILE: Number of bytes written=412175
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=150
HDFS: Number of bytes written=108
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters 
Killed map tasks=1
Launched map tasks=2
Launched reduce tasks=1
Other local map tasks=2
Total time spent by all maps in occupied slots (ms)=17342
Total time spent by all reduces in occupied slots (ms)=6745
Total time spent by all map tasks (ms)=17342
Total time spent by all reduce tasks (ms)=6745
Total vcore-milliseconds taken by all map tasks=17342
Total vcore-milliseconds taken by all reduce tasks=6745
Total megabyte-milliseconds taken by all map tasks=17758208
Total megabyte-milliseconds taken by all reduce tasks=6906880
Map-Reduce Framework
Map input records=3
Map output records=3
Map output bytes=114
Map output materialized bytes=132
Input split bytes=150
Combine input records=0
Combine output records=0
Reduce input groups=3
Reduce shuffle bytes=132
Reduce input records=3
Reduce output records=3
Spilled Records=6
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=1116
CPU time spent (ms)=4500
Physical memory (bytes) snapshot=801751040
Virtual memory (bytes) snapshot=2680033280
Total committed heap usage (bytes)=477626368
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters 
Bytes Read=0
File Output Format Counters 
Bytes Written=108

6.测试：成功

[root@master MySqlMapReduceToAvroHDFS]# sqoop list-databases --connect jdbc:mysql://10.9.148.193:3306/ --username root --password root123@
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/softs/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/softs/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/09/21 11:05:20 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
17/09/21 11:05:20 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/09/21 11:05:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
information_schema
idb
mysql
performance_schema
sys