Sqoop将mysql数据导入hdfs报错,Exception in thread "main" java.lang.NoSuchFieldError: HADOOP_CLASSPATH怎么解决啊啊啊
是因为sqoop-env.sh文件中环境变量写错:
HADOOP_MAPRED_HOME=Hadoop安装路径/share/等等
正确写法:
HADOOP_MAPRED_HOME=hadoop的安装路径(如/usr/local/hadoop-3.1.3)
完整:
2024-09-26 17:21:38,499 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Enter password:
2024-09-26 17:21:51,538 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2024-09-26 17:21:51,539 INFO tool.CodeGenTool: Beginning code generation
2024-09-26 17:21:51,974 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employees` AS t LIMIT 1
2024-09-26 17:21:52,033 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employees` AS t LIMIT 1
2024-09-26 17:21:52,048 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/hadoop-3.1.4/share/hadoop/mapreduce
注: /tmp/sqoop-root/compile/6baa5b229f0a92634359773b3fbfc4f6/employees.java使用或覆盖了已过时的 API。
注: 有关详细信息, 请使用 -Xlint:deprecation 重新编译。
2024-09-26 17:21:54,297 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/6baa5b229f0a92634359773b3fbfc4f6/employees.jar
2024-09-26 17:21:55,254 INFO tool.ImportTool: Destination directory /myfile/sqoop is not present, hence not deleting.
2024-09-26 17:21:55,254 WARN manager.MySQLManager: It looks like you are importing from mysql.
2024-09-26 17:21:55,254 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
2024-09-26 17:21:55,254 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
2024-09-26 17:21:55,254 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
2024-09-26 17:21:55,274 INFO mapreduce.ImportJobBase: Beginning import of employees
2024-09-26 17:21:55,288 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2024-09-26 17:21:55,320 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2024-09-26 17:21:55,475 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.128.130:8032
2024-09-26 17:21:58,500 INFO db.DBInputFormat: Using read commited transaction isolation
2024-09-26 17:21:58,502 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `employees`
2024-09-26 17:21:58,556 INFO mapreduce.JobSubmitter: number of splits:3
2024-09-26 17:21:58,845 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1727331894654_0008
2024-09-26 17:21:58,941 INFO conf.Configuration: resource-types.xml not found
2024-09-26 17:21:58,941 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2024-09-26 17:21:59,039 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1727331894654_0008
Exception in thread "main" java.lang.NoSuchFieldError: HADOOP_CLASSPATH
at org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:249)
at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:467)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:295)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:242)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:196)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:169)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:266)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)