[hadoop@scnnode01 bin]$ importFdcData.sh
23/09/01 20:01:11 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
23/09/01 20:01:11 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
23/09/01 20:01:11 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
23/09/01 20:01:11 INFO manager.SqlManager: Using default fetchSize of 1000
23/09/01 20:01:11 INFO tool.CodeGenTool: Beginning code generation
23/09/01 20:01:12 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:12 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EQP_SUM_VALUE_TRX_TAS t WHERE 1=0
23/09/01 20:01:12 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:12 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EQP_SUM_VALUE_TRX_TAS t WHERE 1=0
23/09/01 20:01:12 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.10.1
Note: /tmp/sqoop-hadoop/compile/1d4a97f77af9410413a55681f37660f6/EQP_SUM_VALUE_TRX_TAS.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
23/09/01 20:01:13 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/1d4a97f77af9410413a55681f37660f6/EQP_SUM_VALUE_TRX_TAS.jar
23/09/01 20:01:13 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:13 INFO mapreduce.ImportJobBase: Beginning import of EQP_SUM_VALUE_TRX_TAS
23/09/01 20:01:13 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
23/09/01 20:01:13 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:13 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
23/09/01 20:01:13 INFO client.RMProxy: Connecting to ResourceManager at /10.255.148.192:8032
23/09/01 20:01:16 INFO db.DBInputFormat: Using read commited transaction isolation
23/09/01 20:01:16 INFO mapreduce.JobSubmitter: number of splits:1
23/09/01 20:01:16 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1693569574799_0001
23/09/01 20:01:16 INFO conf.Configuration: resource-types.xml not found
23/09/01 20:01:16 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
23/09/01 20:01:16 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
23/09/01 20:01:16 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
23/09/01 20:01:17 INFO impl.YarnClientImpl: Submitted application application_1693569574799_0001
23/09/01 20:01:17 INFO mapreduce.Job: The url to track the job: http://scnnode02:8088/proxy/application_1693569574799_0001/
23/09/01 20:01:17 INFO mapreduce.Job: Running job: job_1693569574799_0001
23/09/01 20:01:23 INFO mapreduce.Job: Job job_1693569574799_0001 running in uber mode : false
23/09/01 20:01:23 INFO mapreduce.Job: map 0% reduce 0%
23/09/01 20:01:29 INFO mapreduce.Job: map 100% reduce 0%
23/09/01 20:01:29 INFO mapreduce.Job: Job job_1693569574799_0001 completed successfully
23/09/01 20:01:29 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=220482
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=10586862
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=4088
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=4088
Total vcore-milliseconds taken by all map tasks=4088
Total megabyte-milliseconds taken by all map tasks=4186112
Map-Reduce Framework
Map input records=163395
Map output records=163395
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=103
CPU time spent (ms)=5170
Physical memory (bytes) snapshot=266260480
Virtual memory (bytes) snapshot=2164719616
Total committed heap usage (bytes)=159383552
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=10586862
23/09/01 20:01:29 INFO mapreduce.ImportJobBase: Transferred 10.0964 MB in 15.474 seconds (668.1351 KB/sec)
23/09/01 20:01:29 INFO mapreduce.ImportJobBase: Retrieved 163395 records.
23/09/01 20:01:29 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table EQP_SUM_VALUE_TRX_TAS
23/09/01 20:01:29 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:29 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EQP_SUM_VALUE_TRX_TAS t WHERE 1=0
23/09/01 20:01:29 WARN hive.TableDefWriter: Column EQP_SUM_RAWID had to be cast to a less precise type in Hive
23/09/01 20:01:29 WARN hive.TableDefWriter: Column SUM_DTTS had to be cast to a less precise type in Hive
23/09/01 20:01:29 INFO hive.HiveImport: Loading uploaded data into Hive
23/09/01 20:01:29 INFO conf.HiveConf: Found configuration file file:/home/hadoop/apache-hive-2.3.6-bin/conf/hive-site.xml
23/09/01 20:01:29 WARN conf.HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist
23/09/01 20:01:32 INFO hive.HiveImport:
23/09/01 20:01:32 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/home/hadoop/apache-hive-2.3.6-bin/lib/hive-common-2.3.6.jar!/hive-log4j2.properties Async: true
23/09/01 20:01:32 INFO hive.HiveImport: Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: mtcluster
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:443)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:142)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:369)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:303)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:159)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3247)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3296)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3264)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:475)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:228)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:583)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:553)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:750)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
23/09/01 20:01:32 INFO hive.HiveImport: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
23/09/01 20:01:32 INFO hive.HiveImport: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
23/09/01 20:01:32 INFO hive.HiveImport: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
23/09/01 20:01:32 INFO hive.HiveImport: at java.lang.reflect.Method.invoke(Method.java:498)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.util.RunJar.run(RunJar.java:244)
23/09/01 20:01:32 INFO hive.HiveImport: at org.apache.hadoop.util.RunJar.main(RunJar.java:158)
23/09/01 20:01:32 INFO hive.HiveImport: Caused by: java.net.UnknownHostException: mtcluster
23/09/01 20:01:32 INFO hive.HiveImport: ... 21 more
23/09/01 20:01:32 ERROR tool.ImportTool: Import failed: java.io.IOException: Hive exited with status 1
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:384)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:337)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:241)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:537)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
23/09/01 20:01:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
23/09/01 20:01:33 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
23/09/01 20:01:33 INFO oracle.OraOopManagerFactory: Data Connector for Oracle and Hadoop is disabled.
23/09/01 20:01:33 INFO manager.SqlManager: Using default fetchSize of 1000
23/09/01 20:01:33 INFO tool.CodeGenTool: Beginning code generation
23/09/01 20:01:34 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:34 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EQP_SUM_TRX_TAS t WHERE 1=0
23/09/01 20:01:34 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/hadoop/hadoop-2.10.1
Note: /tmp/sqoop-hadoop/compile/58b6927777ee20b251fd832f6da140b4/EQP_SUM_TRX_TAS.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
23/09/01 20:01:35 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoop/compile/58b6927777ee20b251fd832f6da140b4/EQP_SUM_TRX_TAS.jar
23/09/01 20:01:35 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:35 WARN manager.OracleManager: The table EQP_SUM_TRX_TAS contains a multi-column primary key. Sqoop will default to the column RAWID only for this job.
23/09/01 20:01:35 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:36 WARN manager.OracleManager: The table EQP_SUM_TRX_TAS contains a multi-column primary key. Sqoop will default to the column RAWID only for this job.
23/09/01 20:01:36 INFO mapreduce.ImportJobBase: Beginning import of EQP_SUM_TRX_TAS
23/09/01 20:01:36 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
23/09/01 20:01:36 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:36 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
23/09/01 20:01:36 INFO client.RMProxy: Connecting to ResourceManager at /10.255.148.192:8032
23/09/01 20:01:38 INFO db.DBInputFormat: Using read commited transaction isolation
23/09/01 20:01:38 INFO mapreduce.JobSubmitter: number of splits:1
23/09/01 20:01:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1693569574799_0002
23/09/01 20:01:38 INFO conf.Configuration: resource-types.xml not found
23/09/01 20:01:38 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
23/09/01 20:01:38 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
23/09/01 20:01:38 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
23/09/01 20:01:38 INFO impl.YarnClientImpl: Submitted application application_1693569574799_0002
23/09/01 20:01:38 INFO mapreduce.Job: The url to track the job: http://scnnode02:8088/proxy/application_1693569574799_0002/
23/09/01 20:01:38 INFO mapreduce.Job: Running job: job_1693569574799_0002
23/09/01 20:01:43 INFO mapreduce.Job: Job job_1693569574799_0002 running in uber mode : false
23/09/01 20:01:43 INFO mapreduce.Job: map 0% reduce 0%
23/09/01 20:01:49 INFO mapreduce.Job: map 100% reduce 0%
23/09/01 20:01:50 INFO mapreduce.Job: Job job_1693569574799_0002 completed successfully
23/09/01 20:01:50 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=220963
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=87
HDFS: Number of bytes written=4251806
HDFS: Number of read operations=4
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Other local map tasks=1
Total time spent by all maps in occupied slots (ms)=3493
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=3493
Total vcore-milliseconds taken by all map tasks=3493
Total megabyte-milliseconds taken by all map tasks=3576832
Map-Reduce Framework
Map input records=14832
Map output records=14832
Input split bytes=87
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=124
CPU time spent (ms)=3880
Physical memory (bytes) snapshot=269402112
Virtual memory (bytes) snapshot=2169745408
Total committed heap usage (bytes)=195035136
File Input Format Counters
Bytes Read=0
File Output Format Counters
Bytes Written=4251806
23/09/01 20:01:50 INFO mapreduce.ImportJobBase: Transferred 4.0548 MB in 13.4237 seconds (309.3158 KB/sec)
23/09/01 20:01:50 INFO mapreduce.ImportJobBase: Retrieved 14832 records.
23/09/01 20:01:50 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners for table EQP_SUM_TRX_TAS
23/09/01 20:01:50 INFO manager.OracleManager: Time zone has been set to GMT
23/09/01 20:01:50 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EQP_SUM_TRX_TAS t WHERE 1=0
23/09/01 20:01:50 WARN hive.TableDefWriter: Column RAWID had to be cast to a less precise type in Hive
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.nameservices</name>
<value>mtcluster</value>
</property>
<property>
<name>dfs.ha.namenodes.mtcluster</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.mtcluster.nn1</name>
<value>scnnode01:8020</value>
</property>
<!-- nn1的HTTP通信地址 -->
<property>
<name>dfs.namenode.http-address.mtcluster.nn1</name>
<value>scnnode01:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.mtcluster.nn2</name>
<value>scnnode02:8020</value>
</property>
<!-- nn2的HTTP通信地址 -->
<property>
<name>dfs.namenode.http-address.mtcluster.nn2</name>
<value>scnnode02:50070</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>指定DataNode存放block副本数</description>
</property>
<property>
<name>dfs.datanode.name.dir.mtcluster</name>
<value>/home/hadoop/hadoop-2.10.1/hadoopdata/namenode</value>
<description>hdfs元数据存储路径</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/hadoop-2.10.1/hadoopdata/datanode</value>
<description>hdfs数据存储路径</description>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://scnnode01:8485;scnnode02:8485/mtcluster</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用隔离机制时需要ssh无秘钥登录-->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<!-- 声明journalnode服务器存储目录-->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hadoop/hadoop-2.10.1/data/jn</value>
</property>
<!-- 访问代理类:client,mycluster,active配置失败自动切换实现方式-->
<property>
<name>dfs.client.failover.proxy.provider.mtcluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
</configuration>
core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mtcluster:8020</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>mtcluster</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-2.10.1/tmp</value>
</property>
<!--配置HDFS网页登录使用的静态用户为hadoop -->
<property>
<name>hadoop.http.staticuser.user</name>
<value>hadoop</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>scnnode01:2181,scnnode02:2181,scdnode01:2181,scdnode02:2181,scdnode03:2181,scdnode04:2181</value>
</property>
<property>
<name>hadoop.http.staticuser.user</name>
<value>hadoop</value>
</property>
</configuration>
hive-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<!-- jdbc 连接的 URL -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://scnnode01:3306/metastore?useSSL=false</value>
</property>
<!-- jdbc 连接的 Driver-->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<!-- jdbc 连接的 username-->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<!-- jdbc 连接的 password -->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hadoop</value>
</property>
<!-- Hive 元数据存储版本的验证 -->
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
<!--元数据存储授权-->
<property>
<name>hive.metastore.event.db.notification.api.auth</name>
<value>false</value>
</property>
<!-- Hive 默认在 HDFS 的工作目录 -->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<!-- 指定 hiveserver2 连接的 host -->
<property>
<name>hive.server2.thrift.bind.host</name>
<value>scnnode01</value>
</property>
<!-- 指定 hiveserver2 连接的端口号 -->
<property>
<name>hive.server2.thrift.port</name>
<value>10000</value>
</property>
</configuration>