hadoop-sqoop安装（五）

node10节点安装sqoop

1. 创建文件夹：mkdir /home/hadoop/sqoop

2. 进入：cd /home/hadoop/sqoop

3. 下载并且上传安装包：http://archive.apache.org/dist/sqoop/1.4.7/

4. 解压：tar -xvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz

5. 配置环境变量：vi /etc/profile

   export SQOOP_HOME=/home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0
   export PATH=$PATH:$SQOOP_HOME/bin
   export CLASSPATH=$CLASSPATH:$SQOOP_HOME/lib
   

6. 刷新环境变量：source /etc/profile

7. 进入目录：cd /home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/conf/

8. 拷贝文件：cp sqoop-env-template.sh sqoop-env.sh

9. 修改配置：vi sqoop-env.sh

   export HADOOP_COMMON_HOME=/home/hadoop/module/hadoop-3.2.2
   export HADOOP_MAPRED_HOME=/home/hadoop/module/hadoop-3.2.2
   export HBASE_HOME=/home/hadoop/hbase/hbase-2.1.0
   export HIVE_HOME=/home/hadoop/module/hive
   export ZOOKEEPER_HOME=/home/hadoop/zookeeper/apache-zookeeper-3.6.3-bin
   export ZOOCFGDIR=/home/hadoop/zookeeper/apache-zookeeper-3.6.3-bin/conf   
   

10. 将mysql的jar包mysql-connector-java-5.1.35.jar上传到sqoop的lib目录下/home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

11. 将java-json.jar上传到sqoop的lib目录下/home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

    下载地址：http://www.java2s.com/Code/Jar/j/Downloadjavajsonjar.htm#google_vignette

12. 将hive的jar包拷贝到sqoop的lib目录下/home/hadoop/sqoop/lib/

    cp /home/hadoop/module/hive/lib/hive-common-3.1.2.jar  /home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/lib/
    

13. 上传commons-lang-2.6.jar包到/home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/lib/，并且移除原来的commons-lang

    ps：下载https://repo.maven.apache.org/maven2/commons-lang/commons-lang/2.6/

14. 将需要的hbase-1.3.5的低版本jar包导入到/home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/lib/下

    [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-W0keBpQV-1672658894405)(C:\Users\86188\AppData\Roaming\Typora\typora-user-images\image-20221213112859484.png)]

15. 赋权：chown hadoop:hadoop /home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

16. 测试连接mysql

    sqoop list-databases --connect jdbc:mysql://192.168.248.12:3306/ --username root --password ffcsict123
    

17. 将mysql数据导入hive和hbase

    //导入hive
    sqoop import   -D mapred.job.queue.name=yarnuser1 --connect jdbc:mysql://192.168.248.12:3306/sqoop_test --username root --password ffcsict123 --table test -m 1  --hive-import  --hive-table 'default.user_infor11'  --create-hive-table
    
     
    //导入hbase
    sqoop import   -D mapred.job.queue.name=yarnuser1 --connect jdbc:mysql://192.168.248.12:3306/sqoop_test --username root --password ffcsict123 --table test -m 1 --hbase-create-table  --hbase-table tbl_users  --column-family detail  --hbase-row-key id  --num-mappers 2
    
    //查看中文转格式
    scan 'tbl_users', {FORMATTER => 'toString'}
    

18. sqoop job作业免密配置

    1） vi /home/hadoop/sqoop/sqoop-1.4.7.bin__hadoop-2.6.0/conf/sqoop-site.xml，放开注释

    <property>
       <name>sqoop.metastore.client.record.password</name>
       <value>true</value>
       <description>If true, allow saved passwords in the metastore.</description>
     </property>
    

    2） 将密码输出到mysqlpwd.pwd文件中

    echo -n "ffcsict123" > mysqlpwd.pwd
    hdfs dfs -mkdir  -p /mysql/pwd
    hdfs dfs -put ./mysqlpwd.pwd /mysql/pwd
    hdfs dfs -chmod 400 /mysql/pwd/mysqlpwd.pwd
    

19. sqoop job作业

    sqoop job -Dmapreduce.job.queuename=yarnuser1  --create sqoon_hive  -- import  --connect jdbc:mysql://192.168.248.12:3306/sqoop_test   --username root   --password-file /mysql/pwd/mysqlpwd.pwd --table test  -m 1  --hive-import   --hive-table 'default.user_infor11'  
    

    #查看job
    sqoop job --list
    
    #删除job
    sqoop job --delete jobname
    
    #查看job的定义
    sqoop job --show jobname
    
    #启动job
    sqoop job --exec jobname