系统基础设置
系统的请关闭ipv6
linux检测ipv6 关闭方式
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
如果是0 的话就是关闭是1就是开启
防火墙关闭
Ubuntu sudo ufw disable
Linux sudo service iptable
修改/etc/hosts 文件
你要加入的主机的ip地址名称加入其中
并在其他主机做同样操作
修改主机名称
Vi /etc/hostsname
安装ssh 并设定无验证连接
Ubuntu apt-get install ssh
Linux yum install openssh*
生成秘钥
Ssh-keygen –t rsa –P “”
在你所在权限下会出现
复制 id_rsa.pub 到其他的datanode 上
Scp authorized_keys Hadoop@datanode1:~/.ssh
安装Java
用alien 转rpm 到deb
设置/etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_25
export CLASSPATH=.:$JAVA_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin
Hadoop 应用于64位安装
安装 svn
下载 Hadoop 源代
Yum install autofaunto make libtool cmake
Yum install ncurses-devel
Yum install opensll-ssl
Yum intall gcc*
安装maven
安装 ant
安装 protobuf
mvn install -DskipTests
mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
hbase 选择,选择 hbase-x.x.x-hadoop2
完全进程启动
hbase regionserver 安装系统时间要同步
ntpdate -u time.nist.gov
并使用 date 查看所有节点的时间是否正常
配置好安装好系统,看到所有的进程都已经起来了,那么请查看hbase下的logs 是否会出现什么问题,
例如:
2014-11-28 09:51:41,249 WARN [regionserver60020-SendThread(hd3.hthorizon.com:2181)] zookeeper.ClientCnxn: Session 0x149f41a1cfb0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
这是说明 zookeeper 没有起来,需要去zookeeper 的bin 下查看zookeeper.out, zookeeper 出了什么问题
eclipse 连接HBASE
配置好hbase 之后
从windows 系统上连接服务器上的hadoop,我只是使用了网上的代码,应用全部hbase 目录下lib 中所有的jar 包,然后开始测试,会出现以下问题
系统不显示log 进程所以需要配置log4j.properties,如果日志不出现查看日志文件把所有的log4j.appender.myLogFile.threshold=info
log4j.appender.myConsole.threshold=info
windows 下系统连接,调试hdaoop 系统那 windows系统需要下载安装
winutils.exe 并在系统环境下配置HADOOP_HOME,winutils.exe 在此路径的bin 文件下,如果没有 bin 文件夹,要建立一个.
如果发现和hadoop 系统连接上了,但是操作hbase 的数据库命令运转超时,那么他们会报各种错误比如连接超时等,估计是找不到host文件的主机名,请完全复制到windows 上, Hadoop 服务器上的主机配置在windows 下设定好,hosts 文件在 C:\WINDOWS\system32\drivers\etc
错误:
找不到ssh 服务
sed: -e expression #1, char 6: unknown option to `s'
-c: Unknown cipher type 'cd'
master: namenode running as process 4191. Stop it first.
guard.: ssh: Could not resolve hostname guard.: Name or service not known
now.: ssh: Could not resolve hostname now.: Name or service not known
noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Name or service not known
try: ssh: Could not resolve hostname try: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Name or service not known
The authenticity of host 'java (127.0.53.53)' can't be established.
ECDSA key fingerprint is bf:cb:c6:3a:cc:92:df:85:0f:d3:0f:f0:2d:f5:b4:9d.
Are you sure you want to continue connecting (yes/no)? <libfile>',: ssh: Could not resolve hostname <libfile>',: Name or service not known
it: ssh: Could not resolve hostname it: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
loaded: ssh: Could not resolve hostname loaded: Name or service not known
highly: ssh: Could not resolve hostname highly: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
It's: ssh: Could not resolve hostname It's: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
recommended: ssh: Could not resolve hostname recommended: Name or service not known
'-z: ssh: Could not resolve hostname '-z: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
which: ssh: Could not resolve hostname which: Name or service not known
Client: ssh: Could not resolve hostname Client: Name or service not known
or: ssh: Could not resolve hostname or: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
will: ssh: Could not resolve hostname will: Name or service not known
that: ssh: Could not resolve hostname that: Name or service not known
to: ssh: connect to host to port 22: Connection refused
with: ssh: Could not resolve hostname with: Name or service not known
'execstack: ssh: Could not resolve hostname 'execstack: Name or service not known
You: ssh: Could not resolve hostname You: Name or service not known
disabled: ssh: Could not resolve hostname disabled: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
you: ssh: Could not resolve hostname you: Name or service not known
might: ssh: Could not resolve hostname might: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
The: ssh: Could not resolve hostname The: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
warning:: ssh: Could not resolve hostname warning:: Name or service not known
guard: ssh: Could not resolve hostname guard: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
link: ssh: Could not resolve hostname link: Name or service not known
解决方法:
在hadoop-evn.xml加入
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true - Djava.library.path=$HADOOP_PREFIX/lib"
测试命令
shirdrn@master:~/cloud/hadoop2/hadoop-2.0.4-alpha$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.4-alpha.jar randomwriter out
hdfs-size 参数表
参数 取值范例 详细说明
dfs.default.chunk.view.size 32768 namenode的http访问页面中针对每个文件的内容显示大小,通常无需设置。
dfs.datanode.du.reserved 1073741824 每块磁盘所保留的空间大小,需要设置一些,主要是给非hdfs文件使用,默认是不保留,0字节
dfs.name.dir /opt/data1/hdfs/name,
/opt/data2/hdfs/name,
/nfs/data/hdfs/name NN所使用的元数据保存,一般建议在nfs上保留一份,作为1.0的HA方案使用,也可以在一台服务器的多块硬盘上使用
dfs.web.ugi nobody,nobody NN,JT等所使用的web tracker页面服务器所使用的用户和组
dfs.permissions true | false dfs权限是否打开,我一般设置false,通过开发工具培训别人界面操作避免误操作,设置为true有时候会遇到数据因为权限访问不了。
dfs.permissions.supergroup Supergroup 设置hdfs超级权限的组,默认是supergroup,启动hadoop所使用的用户通常是superuser。
dfs.data.dir
/opt/data1/hdfs/data,
/opt/data2/hdfs/data,
/opt/data3/hdfs/data, 真正的datanode数据保存路径,可以写多块硬盘,逗号分隔
dfs.datanode.data.dir.perm 755 datanode所使用的本地文件夹的路径权限,默认755
dfs.replication 3 hdfs数据块的复制份数,默认3,理论上份数越多跑数速度越快,但是需要的存储空间也更多。有钱人可以调5或者6
dfs.replication.max 512 有时dn临时故障恢复后会导致数据超过默认备份数。复制份数的最多数,通常没什么用,不用写配置文件里。
dfs.replication.min 1 最小份数,作用同上。
dfs.block.size 134217728 每个文件块的大小,我们用128M,默认是64M。这个计算需要128*1024^2,我碰上过有人直接写128000000,十分浪漫。
dfs.df.interval 60000 磁盘用量统计自动刷新时间,单位是毫秒。
dfs.client.block.write.retries 3 数据块写入的最多重试次数,在此次数之前不会捕获失败。
dfs.heartbeat.interval 3 DN的心跳检测时间间隔。秒
dfs.namenode.handler.count 10 NN启动后展开的线程数。
dfs.balance.bandwidthPerSec 1048576 做balance时所使用的每秒最大带宽,使用字节作为单位,而不是bit
dfs.hosts /opt/hadoop/conf/hosts.allow 一个主机名列表文件,这里的主机是允许连接NN的,必须写绝对路径,文件内容为空则认为全都可以。
dfs.hosts.exclude /opt/hadoop/conf/hosts.deny 基本原理同上,只不过这里放的是禁止访问NN的主机名称列表。这在从集群中摘除DN会比较有用。
dfs.max.objects 0 dfs最大并发对象数,HDFS中的文件,目录块都会被认为是一个对象。0表示不限制
dfs.replication.interval 3 NN计算复制块的内部间隔时间,通常不需写入配置文件。默认就好
dfs.support.append true | false 新的hadoop支持了文件的APPEND操作,这个就是控制是否允许文件APPEND的,但是默认是false,理由是追加还有bug。
dfs.datanode.failed.volumes.tolerated 0 能够导致DN挂掉的坏硬盘最大数,默认0就是只要有1个硬盘坏了,DN就会shutdown。
dfs.secondary.http.address 0.0.0.0:50090 SNN的tracker页面监听地址和端口
dfs.datanode.address 0.0.0.0:50010 DN的服务监听端口,端口为0的话会随机监听端口,通过心跳通知NN
dfs.datanode.http.address 0.0.0.0:50075 DN的tracker页面监听地址和端口
dfs.datanode.ipc.address 0.0.0.0:50020 DN的IPC监听端口,写0的话监听在随机端口通过心跳传输给NN
dfs.datanode.handler.count 3 DN启动的服务线程数
dfs.http.address 0.0.0.0:50070 NN的tracker页面监听地址和端口
dfs.https.enable true | false NN的tracker是否监听在HTTPS协议,默认false
dfs.datanode.https.address 0.0.0.0:5047 DN的HTTPS的tracker页面监听地址和端口
dfs.https.address 0.0.0.0:50470 NN的HTTPS的tracker页面监听地址和端口
参考文章
http://www.cnblogs.com/huligong1234/p/4136331.html
Habse 相关网站是:
http://hbase.apache.org/book.html#basic.prerequisites
系统的请关闭ipv6
linux检测ipv6 关闭方式
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
如果是0 的话就是关闭是1就是开启
防火墙关闭
Ubuntu sudo ufw disable
Linux sudo service iptable
修改/etc/hosts 文件
你要加入的主机的ip地址名称加入其中
并在其他主机做同样操作
修改主机名称
Vi /etc/hostsname
安装ssh 并设定无验证连接
Ubuntu apt-get install ssh
Linux yum install openssh*
生成秘钥
Ssh-keygen –t rsa –P “”
在你所在权限下会出现
复制 id_rsa.pub 到其他的datanode 上
Scp authorized_keys Hadoop@datanode1:~/.ssh
安装Java
用alien 转rpm 到deb
设置/etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_25
export CLASSPATH=.:$JAVA_HOME/lib:$HADOOP_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin
Hadoop 应用于64位安装
安装 svn
下载 Hadoop 源代
Yum install autofaunto make libtool cmake
Yum install ncurses-devel
Yum install opensll-ssl
Yum intall gcc*
安装maven
安装 ant
安装 protobuf
mvn install -DskipTests
mvn package -Pdist,native -DskipTests -Dtar -Dmaven.javadoc.skip=true
hbase 选择,选择 hbase-x.x.x-hadoop2
完全进程启动
hbase regionserver 安装系统时间要同步
ntpdate -u time.nist.gov
并使用 date 查看所有节点的时间是否正常
配置好安装好系统,看到所有的进程都已经起来了,那么请查看hbase下的logs 是否会出现什么问题,
例如:
2014-11-28 09:51:41,249 WARN [regionserver60020-SendThread(hd3.hthorizon.com:2181)] zookeeper.ClientCnxn: Session 0x149f41a1cfb0000 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:716)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
这是说明 zookeeper 没有起来,需要去zookeeper 的bin 下查看zookeeper.out, zookeeper 出了什么问题
eclipse 连接HBASE
配置好hbase 之后
从windows 系统上连接服务器上的hadoop,我只是使用了网上的代码,应用全部hbase 目录下lib 中所有的jar 包,然后开始测试,会出现以下问题
系统不显示log 进程所以需要配置log4j.properties,如果日志不出现查看日志文件把所有的log4j.appender.myLogFile.threshold=info
log4j.appender.myConsole.threshold=info
windows 下系统连接,调试hdaoop 系统那 windows系统需要下载安装
winutils.exe 并在系统环境下配置HADOOP_HOME,winutils.exe 在此路径的bin 文件下,如果没有 bin 文件夹,要建立一个.
如果发现和hadoop 系统连接上了,但是操作hbase 的数据库命令运转超时,那么他们会报各种错误比如连接超时等,估计是找不到host文件的主机名,请完全复制到windows 上, Hadoop 服务器上的主机配置在windows 下设定好,hosts 文件在 C:\WINDOWS\system32\drivers\etc
错误:
找不到ssh 服务
sed: -e expression #1, char 6: unknown option to `s'
-c: Unknown cipher type 'cd'
master: namenode running as process 4191. Stop it first.
guard.: ssh: Could not resolve hostname guard.: Name or service not known
now.: ssh: Could not resolve hostname now.: Name or service not known
noexecstack'.: ssh: Could not resolve hostname noexecstack'.: Name or service not known
try: ssh: Could not resolve hostname try: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
HotSpot(TM): ssh: Could not resolve hostname HotSpot(TM): Name or service not known
The authenticity of host 'java (127.0.53.53)' can't be established.
ECDSA key fingerprint is bf:cb:c6:3a:cc:92:df:85:0f:d3:0f:f0:2d:f5:b4:9d.
Are you sure you want to continue connecting (yes/no)? <libfile>',: ssh: Could not resolve hostname <libfile>',: Name or service not known
it: ssh: Could not resolve hostname it: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
loaded: ssh: Could not resolve hostname loaded: Name or service not known
highly: ssh: Could not resolve hostname highly: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
It's: ssh: Could not resolve hostname It's: Name or service not known
have: ssh: Could not resolve hostname have: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
recommended: ssh: Could not resolve hostname recommended: Name or service not known
'-z: ssh: Could not resolve hostname '-z: Name or service not known
fix: ssh: Could not resolve hostname fix: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
which: ssh: Could not resolve hostname which: Name or service not known
Client: ssh: Could not resolve hostname Client: Name or service not known
or: ssh: Could not resolve hostname or: Name or service not known
the: ssh: Could not resolve hostname the: Name or service not known
will: ssh: Could not resolve hostname will: Name or service not known
that: ssh: Could not resolve hostname that: Name or service not known
to: ssh: connect to host to port 22: Connection refused
with: ssh: Could not resolve hostname with: Name or service not known
'execstack: ssh: Could not resolve hostname 'execstack: Name or service not known
You: ssh: Could not resolve hostname You: Name or service not known
disabled: ssh: Could not resolve hostname disabled: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
you: ssh: Could not resolve hostname you: Name or service not known
might: ssh: Could not resolve hostname might: Name or service not known
with: ssh: Could not resolve hostname with: Name or service not known
The: ssh: Could not resolve hostname The: Name or service not known
VM: ssh: Could not resolve hostname VM: Name or service not known
stack: ssh: Could not resolve hostname stack: Name or service not known
warning:: ssh: Could not resolve hostname warning:: Name or service not known
guard: ssh: Could not resolve hostname guard: Name or service not known
library: ssh: Could not resolve hostname library: Name or service not known
link: ssh: Could not resolve hostname link: Name or service not known
解决方法:
在hadoop-evn.xml加入
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true - Djava.library.path=$HADOOP_PREFIX/lib"
测试命令
shirdrn@master:~/cloud/hadoop2/hadoop-2.0.4-alpha$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.0.4-alpha.jar randomwriter out
hdfs-size 参数表
参数 取值范例 详细说明
dfs.default.chunk.view.size 32768 namenode的http访问页面中针对每个文件的内容显示大小,通常无需设置。
dfs.datanode.du.reserved 1073741824 每块磁盘所保留的空间大小,需要设置一些,主要是给非hdfs文件使用,默认是不保留,0字节
dfs.name.dir /opt/data1/hdfs/name,
/opt/data2/hdfs/name,
/nfs/data/hdfs/name NN所使用的元数据保存,一般建议在nfs上保留一份,作为1.0的HA方案使用,也可以在一台服务器的多块硬盘上使用
dfs.web.ugi nobody,nobody NN,JT等所使用的web tracker页面服务器所使用的用户和组
dfs.permissions true | false dfs权限是否打开,我一般设置false,通过开发工具培训别人界面操作避免误操作,设置为true有时候会遇到数据因为权限访问不了。
dfs.permissions.supergroup Supergroup 设置hdfs超级权限的组,默认是supergroup,启动hadoop所使用的用户通常是superuser。
dfs.data.dir
/opt/data1/hdfs/data,
/opt/data2/hdfs/data,
/opt/data3/hdfs/data, 真正的datanode数据保存路径,可以写多块硬盘,逗号分隔
dfs.datanode.data.dir.perm 755 datanode所使用的本地文件夹的路径权限,默认755
dfs.replication 3 hdfs数据块的复制份数,默认3,理论上份数越多跑数速度越快,但是需要的存储空间也更多。有钱人可以调5或者6
dfs.replication.max 512 有时dn临时故障恢复后会导致数据超过默认备份数。复制份数的最多数,通常没什么用,不用写配置文件里。
dfs.replication.min 1 最小份数,作用同上。
dfs.block.size 134217728 每个文件块的大小,我们用128M,默认是64M。这个计算需要128*1024^2,我碰上过有人直接写128000000,十分浪漫。
dfs.df.interval 60000 磁盘用量统计自动刷新时间,单位是毫秒。
dfs.client.block.write.retries 3 数据块写入的最多重试次数,在此次数之前不会捕获失败。
dfs.heartbeat.interval 3 DN的心跳检测时间间隔。秒
dfs.namenode.handler.count 10 NN启动后展开的线程数。
dfs.balance.bandwidthPerSec 1048576 做balance时所使用的每秒最大带宽,使用字节作为单位,而不是bit
dfs.hosts /opt/hadoop/conf/hosts.allow 一个主机名列表文件,这里的主机是允许连接NN的,必须写绝对路径,文件内容为空则认为全都可以。
dfs.hosts.exclude /opt/hadoop/conf/hosts.deny 基本原理同上,只不过这里放的是禁止访问NN的主机名称列表。这在从集群中摘除DN会比较有用。
dfs.max.objects 0 dfs最大并发对象数,HDFS中的文件,目录块都会被认为是一个对象。0表示不限制
dfs.replication.interval 3 NN计算复制块的内部间隔时间,通常不需写入配置文件。默认就好
dfs.support.append true | false 新的hadoop支持了文件的APPEND操作,这个就是控制是否允许文件APPEND的,但是默认是false,理由是追加还有bug。
dfs.datanode.failed.volumes.tolerated 0 能够导致DN挂掉的坏硬盘最大数,默认0就是只要有1个硬盘坏了,DN就会shutdown。
dfs.secondary.http.address 0.0.0.0:50090 SNN的tracker页面监听地址和端口
dfs.datanode.address 0.0.0.0:50010 DN的服务监听端口,端口为0的话会随机监听端口,通过心跳通知NN
dfs.datanode.http.address 0.0.0.0:50075 DN的tracker页面监听地址和端口
dfs.datanode.ipc.address 0.0.0.0:50020 DN的IPC监听端口,写0的话监听在随机端口通过心跳传输给NN
dfs.datanode.handler.count 3 DN启动的服务线程数
dfs.http.address 0.0.0.0:50070 NN的tracker页面监听地址和端口
dfs.https.enable true | false NN的tracker是否监听在HTTPS协议,默认false
dfs.datanode.https.address 0.0.0.0:5047 DN的HTTPS的tracker页面监听地址和端口
dfs.https.address 0.0.0.0:50470 NN的HTTPS的tracker页面监听地址和端口
参考文章
http://www.cnblogs.com/huligong1234/p/4136331.html
Habse 相关网站是:
http://hbase.apache.org/book.html#basic.prerequisites