昨天开始才知道IDEA Intellij中可以在打包的jar包中设置断点进行debug,于是决定把之前遗留下的连接远程hbase的问题重新捡起解决。
现在的环境:client端:是windows10的系统,开发工具为IDEA Intellij
远程的服务器端:是非本地机的centos系统。使用的hadoop为2.7.2版本,hbase为1.2.1版本。且hadoop和hbase都配置了kerberos认证。
前期准备:在client端的windows系统中,设置hosts文件,路径为:C:\Windows\System32\drivers\etc\hosts
需要添加服务器的主机名,因为在hbase-site.xml配置文件中最好是都用主机名,而非ip。示例如下:
# localhost name resolution is handled within DNS itself.
# 127.0.0.1 localhost
# ::1 localhost
10.2.41.207 nativemaster
10.2.41.195 mj2
10.2.41.236 mj1
其中的10.2.41.207,10.2.41.195,10.2.41.236都是我添加的。
在服务器端的hosts文件中同样要加上client端的主机名(其实发现不加也可以),如下:
10.2.46.96 DESKTOP-GT6JK53
最后只要在windows中可以ping通服务器端,在服务器端同样可以ping通windows端即可。
用java开发hbase需要的jar包如下:
绝大多数的包我是从hbase的lib库或者hadoop的lib库中复制过来的,主要是考虑到版本兼容问题,因为之前在网上下了其他版本的包,运行时总是出错。
接下来就是我的工程的目录图:
注意,此处的hbase.keytab是从服务器端拷贝来的,后面会上帖我服务器端的hbase-site.xml配置内容
krb5.conf是从kdc服务器端拷贝来的,里面主要是说明了kdc的主机名,告诉程序要去哪里验证身份。
具体的krb5.conf的内容如下:
[logging] default = FILE:D:\IDEA\hbaseTest\logs\krb5libs.log kdc = FILE:D:\IDEA\hbaseTest\logs\krb5kdc.log admin_server = FILE:D:\IDEA\hbaseTest\logs\kadmind.log [libdefaults] default_realm = BDSM.CMCC dns_lookup_realm = false dns_lookup_kdc = false ticket_lifetime = 24h renew_lifetime = 7d forwardable = true [realms] YOUR.DOMAIN_NAME = { kdc = mj1 admin_server = mj1 } [domain_realm] .example.com = YOUR.DOMAIN_NAME example.com =
YOUR.DOMAIN_NAME
接下来就是服务器端的hbase-site.xml的配置文件内容:接下来是client端的hbase-site.xml的配置内容:<configuration> <property> <name>hbase.rootdir</name> <value>hdfs://mj1:9000/hbase</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <property> <name>hbase.tmp.dir</name> <value>/opt/hbase-1.2.1/tmp</value> </property> <property> <name>hbase.zookeeper.quorum</name> <value>mj1</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> </property> <property> <name>hbase.zookeeper.property.dataDir</name> <!--<value>/opt/hbase-1.2.1/zookeeper</value>--> <value>/opt/zookeeper-3.4.6</value> </property> <property> <name>hbase.security.authorization</name> <value>true</value> </property> <property> <name>hbase.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.regionserver.kerberos.principal</name> <value>hbase/mj1@YOUR.DOMAIN_NAME</value> </property> <property> <name>hbase.regionserver.keytab.file</name> <value>/opt/hbase-1.2.1/conf/hbase.keytab</value> </property> <property> <name>hbase.master.kerberos.principal</name> <value>hbase/mj1@YOUR.DOMAIN_NAME</value> </property> <property> <name>hbase.master.keytab.file</name> <value>/opt/hbase-1.2.1/conf/hbase.keytab</value> </property> </configuration>
接下来就是java程序:<configuration> <property> <name>hbase.zookeeper.quorum</name> <value>mj1</value> </property> <property> <name>hbase.zookeeper.property.clientPort</name> <value>2181</value> </property> <property> <name>hbase.security.authorization</name> <value>true</value> </property> <property> <name>hbase.security.authentication</name> <value>kerberos</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hbase.master.maxclockskew</name> <value>200000</value> <description>Time difference of regionserver from master</description> </property> <property> <name>hbase.regionserver.kerberos.principal</name> <value>hbase/mj1@YOUR.DOMAIN_NAME</value> </property> <property> <name>hbase.master.kerberos.principal</name> <value>hbase/mj1@YOUR.DOMAIN_NAME</value> </property> </configuration>
public static void main(String[] args){ System.setProperty("java.security.krb5.conf","D:\\IDEA\\hbaseTest\\src\\krb5.conf"); Configuration conf = HBaseConfiguration.create(); // conf.set("hbase.zookeeper.quorum",DEFAULT_AUTHORIZE_ZOOKEEPER_QUORUM); // conf.set("hbase.zookeeper.property.clientPort",DEFAULT_AUTHORIZE_ZOOKEEPER_PORT); conf.addResource("D:\\IDEA\\hbaseTest\\src\\hbase-site.xml"); UserGroupInformation.setConfiguration(conf); try { UserGroupInformation.loginUserFromKeytab("hbase/mj1@YOUR.DOMAIN_NAME","D:\\IDEA\\hbaseTest\\src\\hbase.keytab"); HTable table = new HTable(conf,"emp_client"); Put put = new Put(Bytes.toBytes("row1")); put.add(Bytes.toBytes("personal data"),Bytes.toBytes("col1"),Bytes.toBytes("intellij-1")); Get get = new Get(Bytes.toBytes("myrow-1")); Result t = table.get(get); for(KeyValue kv : t.list()){ System.out.println(Bytes.toString(kv.getValue())); } // table.put(put); table.close(); } catch (IOException e) { e.printStackTrace(); } }
结果如下:需要注意的是,因为服务器的hbase配置了kerberos,所以必须要保证windows和服务器的系统时间是一致的(包括时区),否则,kerberos认证是通不过的。D:\Java\jdk1.7.0_67\bin\java -Didea.launcher.port=7533 "-Didea.launcher.bin.path=D:\Program Files (x86)\IntelliJ IDEA Community Edition 2016.1.1\bin" -Dfile.encoding=UTF-8 -classpath "D:\Java\jdk1.7.0_67\jre\lib\charsets.jar;D:\Java\jdk1.7.0_67\jre\lib\deploy.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\access-bridge-64.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\dnsns.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\jaccess.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\localedata.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\sunec.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\sunjce_provider.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\sunmscapi.jar;D:\Java\jdk1.7.0_67\jre\lib\ext\zipfs.jar;D:\Java\jdk1.7.0_67\jre\lib\javaws.jar;D:\Java\jdk1.7.0_67\jre\lib\jce.jar;D:\Java\jdk1.7.0_67\jre\lib\jfr.jar;D:\Java\jdk1.7.0_67\jre\lib\jfxrt.jar;D:\Java\jdk1.7.0_67\jre\lib\jsse.jar;D:\Java\jdk1.7.0_67\jre\lib\management-agent.jar;D:\Java\jdk1.7.0_67\jre\lib\plugin.jar;D:\Java\jdk1.7.0_67\jre\lib\resources.jar;D:\Java\jdk1.7.0_67\jre\lib\rt.jar;D:\IDEA\hbaseTest\out\production\hbaseTest;D:\IDEA\hbaseTest\lib\unnamed.jar;D:\IDEA\hbaseTest\lib\guava-12.0.1.jar;D:\IDEA\hbaseTest\lib\log4j-1.2.16.jar;D:\IDEA\hbaseTest\lib\commons-io-2.4.jar;D:\IDEA\hbaseTest\lib\slf4j-api-1.7.7.jar;D:\IDEA\hbaseTest\lib\zookeeper-3.4.6.jar;D:\IDEA\hbaseTest\lib\commons-lang-2.6.jar;D:\IDEA\hbaseTest\lib\hadoop-nfs-2.7.2.jar;D:\IDEA\hbaseTest\lib\commons-codec-1.4.jar;D:\IDEA\hbaseTest\lib\hadoop-auth-2.7.2.jar;D:\IDEA\hbaseTest\lib\hadoop-hdfs-2.7.2.jar;D:\IDEA\hbaseTest\lib\jackson-xc-1.9.13.jar;D:\IDEA\hbaseTest\lib\hbase-client-1.2.1.jar;D:\IDEA\hbaseTest\lib\hbase-common-1.2.1.jar;D:\IDEA\hbaseTest\lib\metrics-core-2.2.0.jar;D:\IDEA\hbaseTest\lib\commons-lang3-3.3.2.jar;D:\IDEA\hbaseTest\lib\hadoop-common-2.7.2.jar;D:\IDEA\hbaseTest\lib\protobuf-java-2.5.0.jar;D:\IDEA\hbaseTest\lib\hbase-protocol-1.2.1.jar;D:\IDEA\hbaseTest\lib\jackson-jaxrs-1.9.13.jar;D:\IDEA\hbaseTest\lib\mysql-connector-java.jar;D:\IDEA\hbaseTest\lib\commons-logging-1.1.1.jar;D:\IDEA\hbaseTest\lib\netty-all-4.0.23.Final.jar;D:\IDEA\hbaseTest\lib\commons-beanutils-1.7.0.jar;D:\IDEA\hbaseTest\lib\jackson-core-asl-1.9.13.jar;D:\IDEA\hbaseTest\lib\commons-collections-3.2.2.jar;D:\IDEA\hbaseTest\lib\commons-configuration-1.6.jar;D:\IDEA\hbaseTest\lib\jackson-mapper-asl-1.9.13.jar;D:\IDEA\hbaseTest\lib\htrace-core-3.1.0-incubating.jar;D:\Program Files (x86)\IntelliJ IDEA Community Edition 2016.1.1\lib\idea_rt.jar" com.intellij.rt.execution.application.AppMain connection.ConnectionTest log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.Shell). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. client1 client2
最后继续说一句:加油吧,少年!