在windows下开发、调试hadoop 2程序

最新推荐文章于 2021-09-26 07:45:46 发布

wackycrazy

最新推荐文章于 2021-09-26 07:45:46 发布

阅读量496

点赞数

一、winutils的windows版本

GitHub上，有牛人提供了winutils的windows的版本，项目地址是：https://github.com/srccodes/hadoop-common-2.2.0-bin

直接下载此项目的zip包，下载后是文件名是hadoop-common-2.2.0-bin-master.zip,解压到一个目录

二、配置环境变量

配置环境变量：
HADOOP_HOME, 值是hadoop-common-2.2.0-bin-master.zip解压后的目录，如D:\Program\hadoop-common-2.2.0-bin-master

三、验证

利用hive JDBC执行show tables

 
        import 
         java.sql.Connection; 
       
        import 
         java.sql.DriverManager; 
       
        import 
         java.sql.ResultSet; 
       
        import 
         java.sql.SQLException; 
       
        import 
         java.sql.Statement; 
       
        public 
         class 
        Demo { 
       
        public 
        final 
        static 
         String hiveJDBC =  
        "jdbc:hive2://172.168.10.12:10000" 
        ; 
       
        public 
        static 
        void 
         main(String[] args)  
        throws 
         Exception { 
       
        Demo demo =  
        new 
        Demo(); 
       
        System.out.println( 
        "测试hive程序" 
        ); 
       
        demo.testHive(); 
       
        } 
       
        public 
        void 
        testHive()  
        throws 
        Exception { 
       
        Class.forName( 
        "org.apache.hive.jdbc.HiveDriver" 
        ); 
       
        //要带上这个用户名密码，就可以执行job任务,否则hive jdbc调用job会产生异常 
       
        Connection conn = DriverManager.getConnection(hiveJDBC, 
        "hive" 
        , 
        "hive" 
        );  
       
        Statement stmt = conn.createStatement(); 
       
        ResultSet rs = stmt.executeQuery( 
        "show tables" 
        ); 
       
        while 
        (rs.next()) { 
       
        for 
        ( 
        int 
         i =  
        1 
        ; i <  
        2 
        ; i++) { 
       
        System.out.print(rs.getString(i) +  
        " " 
        ); 
       
        } 
       
        System.out.println(); 
       
        } 
       
        free(conn, stmt, rs); 
       
        } 
       
        public 
        static 
        synchronized 
        void 
        free(Connection conn, Statement st, 
       
        ResultSet rs) { 
       
        if 
        (rs !=  
        null 
        ) { 
       
        try 
        { 
       
        rs.close(); 
       
        }  
        catch 
        (SQLException e) { 
       
        e.printStackTrace(); 
       
        } 
       
        } 
       
        if 
        (st !=  
        null 
        ) { 
       
        try 
        { 
       
        st.close(); 
       
        }  
        catch 
        (SQLException e) { 
       
        e.printStackTrace(); 
       
        } 
       
        } 
       
        if 
        (conn !=  
        null 
        ) 
       
        try 
        { 
       
        conn.close(); 
       
        }  
        catch 
        (SQLException e)

执行结果是：

测试hive程序
testtable
lzotable

不会再报

 
        [ 
        2014 
        - 
        07 
        - 
        02 
        09 
        : 
        57 
        : 
        05 
        , 
        651 
        ][ERROR][org.apache.hadoop.util.Shell: 
        336 
        ] - Failed to locate the winutils binary in the hadoop binary path 
       
 
        java.io.IOException: Could not locate executable  
        null 
        \bin\winutils.exe in the Hadoop binaries. 
       
 
        at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java: 
        318 
        ) 
       
 
        at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java: 
        333 
        ) 
       
 
        at org.apache.hadoop.util.Shell.<clinit>(Shell.java: 
        326 
        ) 
       
 
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java: 
        924 
        ) 
       
 
        at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java: 
        230 
        ) 
       
 
        at org.apache.hive.jdbc.HiveConnection.isHttpTransportMode(HiveConnection.java: 
        304 
        ) 
       
 
        at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java: 
        181 
        ) 
       
 
        at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java: 
        164 
        ) 
       
 
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java: 
        105 
        ) 
       
 
        at java.sql.DriverManager.getConnection(DriverManager.java: 
        571 
        ) 
       
 
        at java.sql.DriverManager.getConnection(DriverManager.java: 
        215 
        ) 
       
 
        at impala.Demo.testHive(Demo.java: 
        27 
        ) 
       
 
        at impala.Demo.main(Demo.java: 
        19 
        ) 
       

wackycrazy

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
在windows下开发、调试hadoop 2程序

一、winutils的windows版本 GitHub上，有牛人提供了winutils的windows的版本，项目地址是：https://github.com/srccodes/hadoop-common-2.2.0-bin 直接下载此项目的zip包，下载后是文件名是hadoop-common-2.2.0-bin-master.zip,解压到一个目录二、
复制链接

扫一扫