使用eclipse远程连接hbase

hadoop 专栏收录该内容
4 篇文章 0 订阅
[size=small]基础环境:
CDH 5.4.10
hadoop 2.6.0
hive 1.1.0
hbase 1.0.0
zookeeper 3.4.5
sqoop 1.4.5
jdk 1.7.0_67
os centos6.5

由于我的hive和hbase是一起测试的,因此我就在hive连接的基础上,做hbase的连接测试,上面是hive的连接配置:
# cd /opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/bin/
# ./beeline
Beeline version 1.1.0-cdh5.4.10 by Apache Hive
beeline>

beeline> !connect jdbc:hive2://192.168.1.207:10000
Connecting to jdbc:hive2://192.168.1.207:10000
Enter username for jdbc:hive2://192.168.1.207:10000: root
Enter password for jdbc:hive2://192.168.1.207:10000: **** --root/root直接连上了!
Connected to: Apache Hive (version 1.1.0-cdh5.4.10)
Driver: Hive JDBC (version 1.1.0-cdh5.4.10)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://192.168.1.207:10000>


先来创建几个表,导入点数据,方便后面操作:
# hive
Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/jars/hive-common-1.1.0-cdh5.4.10.jar!/hive-log4j.properties
WARNING: Hive CLI is deprecated and migration to Beeline is recommended.
hive>
hive> create table runningrecord_old(id int,systemno string,longitude string,latitude string,speed string,direction smallint,elevation string,acc string,islocation string,mileage string,oil string,currenttime timestamp,signalname string,currentvalue string) row format delimited fields terminated by ',';
OK
Time taken: 2.607 seconds
hive> load data local inpath '/tmp/rtest.txt' into table runningrecord_old;
Loading data to table default.runningrecord_old
Table default.runningrecord_old stats: [numFiles=1, totalSize=5480004]
OK
Time taken: 1.575 seconds


下一步,我们拷贝jar包到eclipse项目中去:
新建java project:hiveconnect
新建class:hiveconnecttest
新建folder:lib/ 跟src同级

# cd /opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/jars
# sz hive*.jar 放到刚刚那个lib目录下:d:/workspace/hiveconnect/lib/
# sz hadoop*.jar
# ll hive*.jar
hive-accumulo-handler-1.1.0-cdh5.4.10.jar
hive-ant-1.1.0-cdh5.4.10.jar
hive-beeline-1.1.0-cdh5.4.10.jar
hive-cli-1.1.0-cdh5.4.10.jar
hive-common-1.1.0-cdh5.4.10.jar
hive-contrib-1.1.0-cdh5.4.10.jar
hive-exec-1.1.0-cdh5.4.10.jar
hive-hbase-handler-1.1.0-cdh5.4.10.jar
hive-hcatalog-core-1.1.0-cdh5.4.10.jar
hive-hcatalog-pig-adapter-1.1.0-cdh5.4.10.jar
hive-hcatalog-server-extensions-1.1.0-cdh5.4.10.jar
hive-hcatalog-streaming-1.1.0-cdh5.4.10.jar
hive-hwi-1.1.0-cdh5.4.10.jar
hive-jdbc-1.1.0-cdh5.4.10.jar
hive-jdbc-1.1.0-cdh5.4.10-standalone.jar
hive-metastore-1.1.0-cdh5.4.10.jar
hive-serde-1.1.0-cdh5.4.10.jar
hive-service-1.1.0-cdh5.4.10.jar
hive-shims-0.23-1.1.0-cdh5.4.10.jar
hive-shims-1.1.0-cdh5.4.10.jar
hive-shims-common-1.1.0-cdh5.4.10.jar
hive-shims-scheduler-1.1.0-cdh5.4.10.jar
hive-testutils-1.1.0-cdh5.4.10.jar
hive-webhcat-1.1.0-cdh5.4.10.jar
hive-webhcat-java-client-1.1.0-cdh5.4.10.jar
hadoop-annotations-2.6.0-cdh5.4.10.jar
hadoop-ant-2.6.0-cdh5.4.10.jar
hadoop-ant-2.6.0-mr1-cdh5.4.10.jar
hadoop-archives-2.6.0-cdh5.4.10.jar
hadoop-auth-2.6.0-cdh5.4.10.jar
hadoop-aws-2.6.0-cdh5.4.10.jar
hadoop-azure-2.6.0-cdh5.4.10.jar
hadoop-capacity-scheduler-2.6.0-mr1-cdh5.4.10.jar
hadoop-common-2.6.0-cdh5.4.10.jar
hadoop-common-2.6.0-cdh5.4.10-tests.jar
hadoop-core-2.6.0-mr1-cdh5.4.10.jar
hadoop-datajoin-2.6.0-cdh5.4.10.jar
hadoop-distcp-2.6.0-cdh5.4.10.jar
hadoop-examples-2.6.0-mr1-cdh5.4.10.jar
hadoop-examples.jar
hadoop-extras-2.6.0-cdh5.4.10.jar
hadoop-fairscheduler-2.6.0-mr1-cdh5.4.10.jar
hadoop-gridmix-2.6.0-cdh5.4.10.jar
hadoop-gridmix-2.6.0-mr1-cdh5.4.10.jar
hadoop-hdfs-2.6.0-cdh5.4.10.jar
hadoop-hdfs-2.6.0-cdh5.4.10-tests.jar
hadoop-hdfs-nfs-2.6.0-cdh5.4.10.jar
hadoop-kms-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-app-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-common-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-core-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-hs-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.4.10-tests.jar
hadoop-mapreduce-client-nativetask-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-client-shuffle-2.6.0-cdh5.4.10.jar
hadoop-mapreduce-examples-2.6.0-cdh5.4.10.jar
hadoop-nfs-2.6.0-cdh5.4.10.jar
hadoop-rumen-2.6.0-cdh5.4.10.jar
hadoop-sls-2.6.0-cdh5.4.10.jar
hadoop-streaming-2.6.0-cdh5.4.10.jar
hadoop-streaming-2.6.0-mr1-cdh5.4.10.jar
hadoop-test-2.6.0-mr1-cdh5.4.10.jar
hadoop-tools-2.6.0-mr1-cdh5.4.10.jar
hadoop-yarn-api-2.6.0-cdh5.4.10.jar
hadoop-yarn-applications-distributedshell-2.6.0-cdh5.4.10.jar
hadoop-yarn-applications-unmanaged-am-launcher-2.6.0-cdh5.4.10.jar
hadoop-yarn-client-2.6.0-cdh5.4.10.jar
hadoop-yarn-common-2.6.0-cdh5.4.10.jar
hadoop-yarn-registry-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-applicationhistoryservice-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-common-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-nodemanager-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-resourcemanager-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-tests-2.6.0-cdh5.4.10.jar
hadoop-yarn-server-web-proxy-2.6.0-cdh5.4.10.jar

添加到build path

测试ok,测试内容下方:
连接:org.apache.hive.jdbc.HiveConnection@43a25848
是否有数据:true
Running:show tables 'tinatest'
执行“show tables”运行结果:
tinatest
Running:describe tinatest
执行“describe table”运行结果:
key int
value string
Running:load data local inpath '/tmp/test2.txt' into table tinatest
Running:select * from tinatest
执行“select * query”运行结果:
1 a
2 b
3 tina
====================================================================================================================

在hive连接的基础上,我们来创建一个hbase的连接实例:
新建class:hbaseconnecttest

sz hbase*.jar
hbase-annotations-1.0.0-cdh5.4.10.jar
hbase-annotations-1.0.0-cdh5.4.10-tests.jar
hbase-checkstyle-1.0.0-cdh5.4.10.jar
hbase-client-1.0.0-cdh5.4.10.jar
hbase-client-1.0.0-cdh5.4.10-tests.jar
hbase-common-1.0.0-cdh5.4.10.jar
hbase-common-1.0.0-cdh5.4.10-tests.jar
hbase-examples-1.0.0-cdh5.4.10.jar
hbase-hadoop2-compat-1.0.0-cdh5.4.10.jar
hbase-hadoop2-compat-1.0.0-cdh5.4.10-tests.jar
hbase-hadoop-compat-1.0.0-cdh5.4.10.jar
hbase-hadoop-compat-1.0.0-cdh5.4.10-tests.jar
hbase-indexer-cli-1.5-cdh5.4.10.jar
hbase-indexer-common-1.5-cdh5.4.10.jar
hbase-indexer-demo-1.5-cdh5.4.10.jar
hbase-indexer-engine-1.5-cdh5.4.10.jar
hbase-indexer-model-1.5-cdh5.4.10.jar
hbase-indexer-morphlines-1.5-cdh5.4.10.jar
hbase-indexer-mr-1.5-cdh5.4.10.jar
hbase-indexer-mr-1.5-cdh5.4.10-job.jar
hbase-indexer-server-1.5-cdh5.4.10.jar
hbase-it-1.0.0-cdh5.4.10.jar
hbase-it-1.0.0-cdh5.4.10-tests.jar
hbase-prefix-tree-1.0.0-cdh5.4.10.jar
hbase-protocol-1.0.0-cdh5.4.10.jar
hbase-rest-1.0.0-cdh5.4.10.jar
hbase-sep-api-1.5-cdh5.4.10.jar
hbase-sep-impl-1.5-hbase1.0-cdh5.4.10.jar
hbase-sep-impl-common-1.5-cdh5.4.10.jar
hbase-sep-tools-1.5-cdh5.4.10.jar
hbase-server-1.0.0-cdh5.4.10.jar
hbase-server-1.0.0-cdh5.4.10-tests.jar
hbase-shell-1.0.0-cdh5.4.10.jar
hbase-testing-util-1.0.0-cdh5.4.10.jar
hbase-thrift-1.0.0-cdh5.4.10.jar

第一次执行报错:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/zookeeper/KeeperException
at hiveconnect.hbaseconnecttest.main(hbaseconnecttest.java:17)

# sz zookeeper-3.4.5-cdh5.4.10.jar

第二次执行报错:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/Level
at org.apache.hadoop.mapred.JobConf.<clinit>(JobConf.java:348)


# sz *log4j*.jar
apache-log4j-extras-1.1.jar
apache-log4j-extras-1.2.17.jar
flume-ng-log4jappender-1.5.0-cdh5.4.10.jar
flume-ng-log4jappender-1.5.0-cdh5.4.10-jar-with-dependencies.jar
log4j-1.2.16.jar
log4j-1.2.17.jar
slf4j-log4j12-1.7.5.jar


# hbase shell
16/06/12 10:56:09 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.0.0-cdh5.4.10, rUnknown, Tue Apr 12 11:10:23 PDT 2016

hbase(main):001:0>

第三次执行报错:
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:218)

# ll *trace*.jar
-rw-r--r-- 1 root root 117409 Apr 13 02:42 accumulo-trace-1.6.0.jar
-rw-r--r-- 1 root root 30457 Apr 13 02:40 htrace-core-2.00.jar
-rw-r--r-- 1 root root 31212 Apr 13 02:40 htrace-core-3.0.4.jar
-rw-r--r-- 1 root root 1475955 Apr 13 02:39 htrace-core-3.1.0-incubating.jar
# sz *trace*.jar


报错:
16/06/12 13:35:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/12 13:35:24 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

新建目录:conf
/opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/etc/hbase/conf.dist/hbase-site.xml
# cd /opt/cloudera/parcels/CDH-5.4.10-1.cdh5.4.10.p0.16/etc/hbase/conf.dist/

# cd /etc/hbase/conf.cloudera.hbase/
# ll
total 28
-rw-r--r-- 1 root root 21 Jun 8 10:07 __cloudera_generation__
-rw-r--r-- 1 root root 3538 Jun 8 10:07 core-site.xml
-rw-r--r-- 1 root root 360 Jun 8 10:07 hbase-env.sh
-rw-r--r-- 1 root root 1984 Jun 8 10:07 hbase-site.xml
-rw-r--r-- 1 root root 1610 Jun 8 10:07 hdfs-site.xml
-rw-r--r-- 1 root root 0 Jun 8 10:07 jaas.conf
-rw-r--r-- 1 root root 312 Jun 8 10:07 log4j.properties
-rw-r--r-- 1 root root 315 Jun 8 10:07 ssl-client.xml
# sz hbase-site.xml
*B00000000000000
#

选中项目名称hbaseconnect,右键选择Propertie->Java Build Path->Libraries->Add Class Folder(选中conf,将conf目录添加进这个项目)。

报错:主机不识别
16/06/12 14:11:25 WARN zookeeper.RecoverableZooKeeper: Unable to create ZooKeeper Connection
java.net.UnknownHostException: master01

编辑文件C:\WINDOWS\system32\drivers\etc\hosts
192.168.1.207 master01


还是缺少包,干脆把jars下面的全拷贝过来算了!
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/cli/ParseException
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:636)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:608)

sz protobuf*.jar slf4j*.jar commons*.jar


还是存在报错:
16/06/12 15:05:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/06/12 15:05:10 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.

Hadoop2.x下 Windows下eclipse开发hadoop程序会报错,原因是因为hadoop2.x没有发布winutils.exe造成的,

提供了winutils的windows的版本,项目地址是:https://codeload.github.com/amihalik/hadoop-common-2.6.0-bin/zip/master
将包解压到D:\hadoop-common-2.6.0-bin-master

设置环境变量,并重启电脑
HADOOP_HOME=D:\hadoop-common-2.6.0-bin-master

不再报错!


测试的java代码:
package hiveconnect;

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HColumnDescriptor;
import org.apache.hadoop.hbase.HTableDescriptor;
import org.apache.hadoop.hbase.MasterNotRunningException;
import org.apache.hadoop.hbase.TableName;
import org.apache.hadoop.hbase.ZooKeeperConnectionException;
import org.apache.hadoop.hbase.client.HBaseAdmin;
import org.apache.hadoop.hbase.client.HTable;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.util.Bytes;

public class hbaseconnect {

private static Configuration conf = null;
static {
conf = HBaseConfiguration.create();
conf.set("hbase.zookeeper.quorum", "master01");// 使用eclipse时必须添加这个,否则无法定位master需要配置hosts
conf.set("hbase.zookeeper.property.clientPort", "2181");
}

public static void main(String[] args) throws IOException {
String[] cols = new String[1];
String[] colsValue = new String[1];
cols[0] = "title";
colsValue[0] = "AboutYunArticle";

// 创建表
createTable();
// 添加值
addData("www.aboutyun.com", "blog01", cols, colsValue);
}

private static void createTable() throws MasterNotRunningException,
ZooKeeperConnectionException, IOException {

HBaseAdmin admin = new HBaseAdmin(conf);// 新建一个数据库管理员//新api
if (admin.tableExists(TableName.valueOf("LogTable"))) {
System.out.println("table is not exist!");
System.exit(0);
} else {

HTableDescriptor desc = new HTableDescriptor(
TableName.valueOf("blog01"));
desc.addFamily(new HColumnDescriptor("article"));
admin.createTable(desc);
admin.close();
System.out.println("create table Success!");
}
}
private static void addData(String rowKey, String tableName,String[] column1, String[] value1) throws IOException {
Put put = new Put(Bytes.toBytes(rowKey));// 设置rowkey
HTable table = new HTable(conf, Bytes.toBytes(tableName));// HTabel负责跟记录相关的操作如增删改查等//
// 获取表
HColumnDescriptor[] columnFamilies = table.getTableDescriptor() // 获取所有的列族
.getColumnFamilies();

for (int i = 0; i < columnFamilies.length; i++) {
String familyName = columnFamilies[i].getNameAsString(); // 获取列族名
if (familyName.equals("article")) { // article列族put数据
for (int j = 0; j < column1.length; j++) {
put.add(Bytes.toBytes(familyName),
Bytes.toBytes(column1[j]), Bytes.toBytes(value1[j]));
}
}
}
table.put(put);
System.out.println("add data Success!");
}

}


测试结果:
hbase(main):003:0> scan 'blog'
ROW COLUMN+CELL
www.aboutyun.com column=article:title, timestamp=1465721042702, value=AboutYunArticle
1 row(s) in 0.1010 seconds


将项目打包导出:
选中项目名称hiveconnect--export--runnable jar file--(hiveconnecttest-hiveconnect)--C:\Users\Administrator\Desktop\hiveconnect.jar--


将jar包导入:
新建一个项目:testhive--build path--configure build path--add external jars--选中C:\Users\Administrator\Desktop\hiveconnect.jar
新建类hivetest:

package testhive;
import hiveconnect.hiveconnecttest;

public class hivetest {

public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
hiveconnect.hiveconnecttest hiveconnecttest = new hiveconnecttest();
hiveconnecttest.main(null);
}

}
就可以直接执行了!


学习阶段,不对的地方欢迎指正。
QQ:906179271
tina

[/size]
  • 1
    点赞
  • 0
    评论
  • 0
    收藏
  • 一键三连
    一键三连
  • 扫一扫,分享海报

©️2021 CSDN 皮肤主题: 大白 设计师:CSDN官方博客 返回首页
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、C币套餐、付费专栏及课程。

余额充值