HDFS学习总结

（1）安装配置过程
a.构建配置文件
conf/hadoop-env.sh
conf/hadoop-site.xml
fs.default.name
hadoop.tmp.dir
conf/slaves,conf/master
b.将安装包分发至各个机器上
chown 改成自己的
c.格式化HDFS
hadoop namenode format
Trouble Shooting:
(1)IOException:mybe permission denied
(2)wipe the directiory before reformat
d.启动HDFS
bin/start-dfs.sh
Troubleshooting:
(1) connect to host XXX port22:no route to host: unable ssh passwordless
(2) No such file or directory : not distribute the installation
(3) XXX.pid :No such file or directory: specified in hadoop-env.sh for the server PID file was not writable
e.确保HDFS正在运行
(1)NameNode
${JAVA-HOME}/bin/jps
[should be] XXX NameNode
(2)DataNode
Bin/slaves.sh jps | grep Datanode | sort
[should be] +{XXX:XX DataNode}
TroubleShooting:
(1) Invalid directory in XXX: can not create directory : Permission problem
(2) Retrying to connect server: XXX, Already tried 0 time(s) : network connecting problem
(3) whole part
bin/hadoop dfsadmin –report

（2）性能因素
a.文件描述符。即同时打开文件的数量
per-user: /etc/security/limits.conf
*hard file 64000
per-system: /etc/sysctl/conf
fs.file-max = 640000 [fresh by command sysctl -p]
b.Datanode的服务线程数
<property>
<name>dfs.datanode.handler.count</name>
<value>30</value>
</property>
c.NameNode的服务线程数
<property>
<name>dfs.namenode.handler.count</name>
<value>512</value>
</property>
d.Disk保留空间
mapred.local.dir.minspacestart
mapred.local.dir.minspacekill
dfs.datanode.du.reserved
e.Disk I/O
Secondary NameNode:
Fs.checkpoint.edit.dir
Fs.checkpoint.dir
NameNode:
Dfs.name.edit.dir
Dfs.name.dir:
DataNode:
Dfs.data.dir:
Dfs.replication
Dfs.block.size
f.NetWork
dfs.datanode.dns.interface
dfs.datanode.dns.nameserver
[data intensive => compass]
（3）DataNode的增添和移除
增添：hadoop-daemon.sh start datanode
移除：
A. create a file, contain one hostname ot IP address per line
B. hadoop-site.xml
<property>
<namenode>dfs.host.exclude</namenode>
<value>%file position%</value>
</property>
C.hadoop dfsadmin -refreshNodes