hbase之Reverse DNS问题

最新推荐文章于 2023-04-01 17:02:26 发布

weixin_34240657

最新推荐文章于 2023-04-01 17:02:26 发布

阅读量212

点赞数

文章标签：运维大数据操作系统

原文链接：http://blog.51cto.com/yaoyinjie/650902

版权

今天是周五，明天周末，晚上正好有时间倒腾下hbase, 在笔记本上安装了虚拟机，并安装了 ubuntu-11.04-server-i386.iso 系统，下载了jdk-7-linux-i586.tar.gz， hadoop-0.20.203.0rc1.tar.gz， hbase-0.90.4.tar.gz，接着安装jdk, 配置环境变量, 准备就绪后开始启动hadoop, 正常，接着启动hbase, 发现报错，查看日志，大意是hbase下的hadoop版本与运行中的hadoop版本不同

OK，来检查下, hbase_home/lib下的hadoop版本是 hadoop-core-0.20-append-r1056497.jar, hadoop-0.20.203 下的为hadoop-core-0.20.203.0.jar. 将hbase_home/lib下的hadoop版本替换为hadoop-core-0.20.203.0.jar，重新启动hbase, 发现还是有错误，大意是某些类找不到，这个好解决，在hbase启动脚本中把hadoop下的jar包设置到classpath路径，重启后错误解决，可以建表和查询了。

因为ubuntu-11.04-server-i386 没有界面，我的eclipse安装在本地windows系统中，需要通过虚拟机连接hbase

查看下虚拟机IP地址:

yinjie@ubuntu:~/soft/hadoop-0.20.203.0/bin$ ifconfig
eth0      Link encap:Ethernet HWaddr 00:0c:29:bd:62:7b
          inet addr:192.168.203.129 Bcast:192.168.203.255 Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:febd:627b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:300269 errors:0 dropped:0 overruns:0 frame:0
          TX packets:78941 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:417200572 (417.2 MB) TX bytes:7264996 (7.2 MB)
          Interrupt:19 Base address:0x2000

ip是 192.168.203.129
创建一张表试试:

Configuration HBASE_CONFIG = new Configuration();
HBASE_CONFIG.set("hbase.zookeeper.quorum", "192.168.203.129");
HBASE_CONFIG.set("hbase.zookeeper.property.clientPort", "2181");
cfg = new HBaseConfiguration(HBASE_CONFIG);
/**
* 创建一张表
*/
public static void creatTable(String tablename) throws Exception {
HBaseAdmin admin = new HBaseAdmin(cfg);
if (admin.tableExists(tablename)) {
System.out.println("table Exists!!!");
} else {
HTableDescriptor tableDesc = new HTableDescriptor(tablename);
HColumnDescriptor hColumnDescriptor=null;
hColumnDescriptor = new HColumnDescriptor("f1");
hColumnDescriptor.setMaxVersions(1);
tableDesc.addFamily(hColumnDescriptor);
hColumnDescriptor = new HColumnDescriptor("f2");
hColumnDescriptor.setMaxVersions(2);
tableDesc.addFamily(hColumnDescriptor);
hColumnDescriptor = new HColumnDescriptor("f3");
hColumnDescriptor.setMaxVersions(3);
tableDesc.addFamily(hColumnDescriptor);
admin.createTable(tableDesc);
System.out.println("create table ok .");
}
}
public static void main(String[] args) {
try {
String tablename = "t1";
Main.creatTable(tablename);
} catch (Exception e) {
e.printStackTrace();
}

执行后发现总是异常
查看zookeeper连接是否正常: 192.168.203.129:2181
zkCli -server 192.168.203.129:2181

[zk: 192.168.203.129:2181(CONNECTED) 5] ls /
[hbase, zookeeper]
[zk: 192.168.203.129:2181(CONNECTED) 6] ls /hbase
[root-region-server, rs, table, unassigned, master, shutdown]
[zk: 192.168.203.129:2181(CONNECTED) 7]
[zk: 192.168.203.129:2181(CONNECTED) 7]
[zk: 192.168.203.129:2181(CONNECTED) 7] get /hbase/master
localhost:60000
cZxid = 0x15
ctime = Sat Aug 27 19:43:27 CST 2011
mZxid = 0x15
mtime = Sat Aug 27 19:43:27 CST 2011
pZxid = 0x15
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x1320b0d2e2f0000
dataLength = 12
numChildren = 0

问题很明显，通过zookeeper拿到的hmaster地址是 localhost:60000, 这样当然就连接不上hmaster了
接下去看看 hbase中的默认配置文件hbase-default.xml
挖几个配置项出来，在hbase-site.xml中重新设置:

<property>
<name>hbase.master.dns.interface</name>
<value>eth0</value>
<description>The name of the Network Interface from which a master
should report its IP address.
</description>
</property>
<property>
<name>hbase.master.dns.nameserver</name>
<value>192.168.203.129</value>
<description>The host name or IP address of the name server (DNS)
which a master should use to determine the host name used
for communication and display purposes.
</description>
</property>
<property>
<name>hbase.regionserver.dns.interface</name>
<value>eth0</value>
<description>The name of the Network Interface from which a region server
should report its IP address.
</description>
</property>
<property>
<name>hbase.regionserver.dns.nameserver</name>
<value>192.168.203.129</value>
<description>The host name or IP address of the name server (DNS)
which a region server should use to determine the host name used by the
master for communication and display purposes.
</description>
</property>

上面明确指定了使用192.168.203.129地址. 以为这样就没问题了，但事实上zookeeper里面注册的hmaster 地址仍旧是localhost，为什么会返回localhost而不是192.168.203.129, 真的有点沉不住气了，那就直接翻源码吧。
翻源码过程就不说了，截几个重要的函数来看下:
在HMaster.java中有一个函数:

/*
* @return This masters' address.
* @throws UnknownHostException
*/
private static String getMyAddress(final Configuration c)
throws UnknownHostException {
// Find out our address up in DNS.
String s = Strings.domainNamePointerToHostName(DNS.getDefaultHost(c.get(
"hbase.master.dns.interface", "default"), c.get(
"hbase.master.dns.nameserver", "default")));
s += ":" + c.get(HConstants.MASTER_PORT,
Integer.toString(HConstants.DEFAULT_MASTER_PORT));
return s;
}

来看看DNS.getDefaultHost方法

public static String getDefaultHost(String strInterface, String nameserver)
throws UnknownHostException {
if (strInterface.equals("default"))
return InetAddress.getLocalHost().getCanonicalHostName();
if (nameserver != null && nameserver.equals("default"))
return getDefaultHost(strInterface);
String[] hosts = getHosts(strInterface, nameserver);
return hosts[0];
}

public static String[] getHosts(String strInterface, String nameserver)
throws UnknownHostException {
String[] ips = getIPs(strInterface);
Vector<String> hosts = new Vector<String>();
for (int ctr = 0; ctr < ips.length; ctr++)
try {
hosts.add(reverseDns(InetAddress.getByName(ips[ctr]),
nameserver));
} catch (Exception e) {
}
if (hosts.size() == 0) // 为空，返回localhost
return new String[] { InetAddress.getLocalHost().getCanonicalHostName() };
else
return hosts.toArray(new String[] {});
}

步步跟踪，最后来到:

public static String reverseDns(InetAddress hostIp, String ns)
throws NamingException {
//
// Builds the reverse IP lookup form
// This is formed by reversing the IP numbers and appending in-addr.arpa
//
String[] parts = hostIp.getHostAddress().split("\\.");
String reverseIP = parts[3] + "." + parts[2] + "." + parts[1] + "."
+ parts[0] + ".in-addr.arpa";
DirContext ictx = new InitialDirContext();
Attributes attribute =
ictx.getAttributes("dns://" // Use "dns:///" if the default
+ ((ns == null) ? "" : ns) +
// nameserver is to be used
"/" + reverseIP, new String[] { "PTR" });
ictx.close();
return attribute.get("PTR").get().toString();
}

从这个函数可以看出，hbase会根据ip来得到域名，如果中间有异常则在 getHosts函数中返回localhost既然是ip->域名反向查找过程有问题，那就来看看DNS服务吧

yinjie@ubuntu:~/soft/hadoop-0.20.203.0/bin$ nslookup
> 192.168.203.129
Server: 192.168.203.2
Address: 192.168.203.2#53
** server can't find 129.203.168.192.in-addr.arpa.: NXDOMAIN

问题很明显，是192.168.203.129这个IP没有被DNS反向解析成域名，也就是说192.168.203.129在dns里找不到对应的域名

真是一路坎坷啊，那就来安装一个DNS

yinjie@ubuntu:~/soft/hbase-0.90.4/conf$ sudo apt-get install bind9

安装完毕后来查看下/etc/bind下有哪些文件

yinjie@ubuntu:~/soft/hbase-0.90.4/conf$ cd /etc/bind
yinjie@ubuntu:/etc/bind$ ls
bind.keys db.0 db.127 db.255 db.empty db.local db.root named.conf named.conf.default-zones named.conf.local named.conf.options rndc.key zones.rfc1918
yinjie@ubuntu:/etc/bind$

接着配置 192.168.203.129 IP地址与域名的对应关系(这里把域名设置成yinjie)：
照样画葫芦,创建db.129 文件：
vi db.129 输入:

;
; BIND reverse data file
;
$TTL 604800
@ IN SOA yinjie. root.localhost. (
1 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS yinjie.
129.203.168 IN PTR yinjie.

创建db.yinjie文件
vi db.yinjie 输入:

;
; BIND data file
;
$TTL 604800
@ IN SOA yinjie. root.localhost. (
2 ; Serial
604800 ; Refresh
86400 ; Retry
2419200 ; Expire
604800 ) ; Negative Cache TTL
;
@ IN NS yinjie.
@ IN A 192.168.203.129
@ IN AAAA ::1

然后编辑: named.conf.default-zones 添加:

zone "yinjie" {
type master;
file "/etc/bind/db.yinjie";
};
zone "192.in-addr.arpa" {
type master;
file "/etc/bind/db.129";
};

这样 192.168.203.129就与域名yinjie关联起来了，通过yinjie可以解析出地址 192.168.203.129, 通过地址192.168.203.129可以解析出域名yinjie

重启一下DNS服务，使配置生效:

yinjie@ubuntu:/etc/bind$ sudo /etc/init.d/bind9 restart

好了，来验证下：

yinjie@ubuntu:~/soft/hadoop-0.20.203.0/bin$ nslookup
> 192.168.203.129
Server: 192.168.203.129
Address: 192.168.203.129#53
129.203.168.192.in-addr.arpa name = yinjie.
>
>
> yinjie
Server: 192.168.203.129
Address: 192.168.203.129#53
Name: yinjie
Address: 192.168.203.129
>
>

从上面可以看出，输入192.168.203.129返回的域名确实是yinjie, 而通过域名yinjie可以得到IP地址192.168.203.129好了，DNS已经解决，成功已经接近，不过最后还要做些配置上的改动
修改hadoop配置core-site.xml

<property>
<name>fs.default.name</name>
<value>hdfs://yinjie:9090</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>

修改hadoop配置mapred-site.xml

<property>
<name>mapred.job.tracker</name>
<value>yinjie:9999</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>

修改hbase配置 hbase-site.xml

<property>
<name>hbase.master.dns.interface</name>
<value>eth0</value>
<description>The name of the Network Interface from which a master
should report its IP address.
</description>
</property>
<property>
<name>hbase.master.dns.nameserver</name>
<value>192.168.203.129</value>
<description>The host name or IP address of the name server (DNS)
which a master should use to determine the host name used
for communication and display purposes.
</description>
</property>
<property>
<name>hbase.regionserver.dns.interface</name>
<value>eth0</value>
<description>The name of the Network Interface from which a region server
should report its IP address.
</description>
</property>
<property>
<name>hbase.regionserver.dns.nameserver</name>
<value>192.168.203.129</value>
<description>The host name or IP address of the name server (DNS)
which a region server should use to determine the host name used by the
master for communication and display purposes.
</description>
</property>

最后修改/etc/hosts文件
增加:
192.168.203.129 yinjie

OK了，启动hadoop, hbase，然后在windows下运行创建表程序, 部分日志打印如下：
11/08/27 19:45:18 INFO zookeeper.ZooKeeper: Client environment:user.name=Administrator
11/08/27 19:45:18 INFO zookeeper.ZooKeeper: Client environment:user.home=C:\Documents and Settings\Administrator
11/08/27 19:45:18 INFO zookeeper.ZooKeeper: Client environment:user.dir=D:\Development\Source\workspace\hbase-test
11/08/27 19:45:18 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=yinjie:2181 sessionTimeout=180000 watcher=hconnection
11/08/27 19:45:19 INFO zookeeper.ClientCnxn: Opening socket connection to server yinjie/192.168.203.129:2181
11/08/27 19:45:19 INFO zookeeper.ClientCnxn: Socket connection established to yinjie/192.168.203.129:2181, initiating session
11/08/27 19:45:19 INFO zookeeper.ClientCnxn: Session establishment complete on server yinjie/192.168.203.129:2181, sessionid = 0x1320b0d2e2f0005, negotiated timeout = 180000
create table ok .

成功！！！
HADOOP-5339.doc 附件是问题解决过程中找到的一些资料，是关于
reverse DNS doesnt resolve local loop address DNS解析问题

转载于:https://blog.51cto.com/yaoyinjie/650902

weixin_34240657

关注

0
点赞
踩
0

收藏

觉得还不错? 一键收藏
0
评论
hbase之Reverse DNS问题

今天是周五，明天周末，晚上正好有时间倒腾下hbase, 在笔记本上安装了虚拟机，并安装了ubuntu-11.04-server-i386.iso 系统，下载了jdk-7-linux-i586.tar.gz， hadoop-0.20.203.0rc1.tar.gz， hbase-0.90.4.tar.gz，接着安装jdk, 配置环境变量, 准备就绪后开始启...
复制链接

扫一扫