Hadoop学习遇到的问题记录

最新推荐文章于 2024-07-20 17:52:45 发布

zhallalxj

最新推荐文章于 2024-07-20 17:52:45 发布

阅读量607

点赞数 1

分类专栏：大数据 HBase Hadoop 文章标签： hadoop hbase 报错 java.lang.NoSuchMethodError: org.ap

本文链接：https://blog.csdn.net/qq_21940627/article/details/99956347

版权

大数据同时被 3 个专栏收录

1 篇文章 0 订阅

订阅专栏

HBase

1 篇文章 0 订阅

订阅专栏

Hadoop

1 篇文章 0 订阅

订阅专栏

Hadoop学习遇到的问题记录

1. 使用JAVA API操作HBase出现错误

1. 使用JAVA API操作HBase出现错误

集群环境：hadoop2.6.5 ，zookeeper3.4.5，hbase1.3.5，jdk1.8
在本机使用maven+java api操作hbase一直报错：

Exception in thread "main" org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:783)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:911)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:875)
	at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1249)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:35518)
	at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:404)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:211)
	at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:65)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
	at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
	at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
	at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.net.NetUtils.getInputStream(Ljava/net/Socket;)Lorg/apache/hadoop/net/SocketInputWrapper;
	at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:725)
	... 17 more

各种调试，更换集群Hbase版本都不奏效，然后发现报错的这个org.apache.hadoop.net.NetUtils这个类在我POM文件中引入的另一个jar包中也出现过，就想是不是可能Hbase中的和这个jar冲突，然后发现注释了果然就好了

		<!--引起报错的jar-->
		<dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>

下面是修改后的POM文件，和Java代码

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.zh</groupId>
    <artifactId>hadooptest</artifactId>
    <version>1.0</version>

    <dependencies>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-core -->
        <!--引入了这个jar会导致hbase连接出现报错-->
       <!-- <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>1.2.1</version>
        </dependency>-->

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-hdfs -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.6.5</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.6.5</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-auth -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-auth</artifactId>
            <version>2.6.5</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>2.6.5</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.6</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/commons-logging/commons-logging -->
        <dependency>
            <groupId>commons-logging</groupId>
            <artifactId>commons-logging</artifactId>
            <version>1.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.huaban/jieba-analysis -->
        <dependency>
            <groupId>com.huaban</groupId>
            <artifactId>jieba-analysis</artifactId>
            <version>1.0.2</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/com.hankcs/hanlp -->
        <dependency>
            <groupId>com.hankcs</groupId>
            <artifactId>hanlp</artifactId>
            <version>portable-1.7.4</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-client -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-client</artifactId>
            <version>1.3.5</version>
        </dependency>


        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-common -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-common</artifactId>
            <version>1.3.5</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.hbase/hbase-server -->
        <dependency>
            <groupId>org.apache.hbase</groupId>
            <artifactId>hbase-server</artifactId>
            <version>1.3.5</version>
        </dependency>


    </dependencies>

    <build>
        <finalName>hadoopTest</finalName>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <!-- 打jar包插件 -->
                <artifactId>maven-jar-plugin</artifactId>
                <version>2.4</version>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.0</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

package com.zh.hbase;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.client.Admin;
import org.apache.hadoop.hbase.client.ConnectionFactory;

import java.io.IOException;
import java.util.Arrays;
import java.util.List;

/**
 * @author zh
 */
public class CreateTable {


    public static void main(String[] args) throws IOException {

        createTable("java_table", Arrays.asList("data","flag"));



    }


    private static void createTable(String tableName, List<String> cols) throws IOException {
        TableName table = TableName.valueOf(tableName);
        Admin hBaseAdmin = initHbase();
        if(hBaseAdmin.tableExists(TableName.valueOf(tableName))){
            System.out.println("表已存在");
        }else {

            HTableDescriptor hTableDescriptor = new HTableDescriptor(table);

            cols.forEach(s->{
                HColumnDescriptor column = new HColumnDescriptor(s);
                hTableDescriptor.addFamily(column);
            });

            hBaseAdmin.createTable(hTableDescriptor);
            System.out.println("表：".concat(tableName).concat(" 创建成功，列族：").concat(cols.toString()));
        }

    }


    private static Admin initHbase() throws IOException {

        Configuration conf = HBaseConfiguration.create();
        conf.set("hbase.master","master:60000");
        conf.set("hbase.zookeeper.quorum","master,slave1,slave2");
        Admin admin = ConnectionFactory.createConnection(conf).getAdmin();
        return admin;
    }


}

zhallalxj

关注

1
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
Hadoop学习遇到的问题记录

Hadoop学习遇到的问题记录1. 使用JAVA API操作HBase出现错误1. 使用JAVA API操作HBase出现错误集群环境：hadoop2.6.5 ，zookeeper3.4.5，hbase1.3.5，jdk1.8在本机使用maven+java api操作hbase一直报错：Exception in thread "main" org.apache.hadoop.hbase...
复制链接

扫一扫

专栏目录