Ubuntu环境Hadoop安装总结

1 ssh连接失败

1.0 连接测试

ssh localhost

1.2 问题

ssh: connect to host localhost port 22: Connection refused

1.3 原因

未安装ssh-server.
测试:

# 命令
ps -e | grep ssh

# 结果
1153 ?        00:00:07 ssh-agent

1.4 解决

安装ssh-server

sudo apt-get install openssh-server

1.5 测试效果

# 命令
ps -e | grep ssh

# 结果
  764 ?        00:00:00 sshd
 1153 ?        00:00:07 ssh-agent

1.6 连接

# 命令
ssh localhost

# 结果
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

2 启动hdfs失败

2.1 启动

./sbin/start-dfs.sh

2.2 报错

Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [xhwl]
2019-01-01 12:21:04,362 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2.3 解决

在hadoop-3.0.2/etc/hadoop/log4j.properties文件中添加

log4j.logger.org.apache.hadoop.util.NativeCodeLoader=ERROR

3 hdfs50070端口无法访问

3.1 原因

• Hadoop3.x版本hdfs端口为9870;
• Hadoop2.x版本hdfs端口为50070;

3.2 解决

对应Hadoop版本,启动后访问对应接口;

4 用户组及用户

4.1 创建

• 创建用户组

sudo addgroup group_name

• 创建用户

sudo adduser -ingroup group_name user_name

• 为user_name添加权限

sudo gedit /etc/sudoers
# 添加
hduser ALL=(ALL) ALL

• 重启

sudo reboot

• 切换用户

sudo su user_name

4.2 用户组文件配置

• core-site.xml
  主机文件配置,使用localhost,端口号大于9000,9000只能内网访问.

<property>
		<name>fs.defaultFS</name>
		<value>hdfs://localhost:9820</value>
    <description>hdfs host:IP</description>
</property>

• hdfs-site.xml
  用户文件配置,使用用户名username，才能新建文件。

<property>
		<name>dfs.namenode.secondary.http-address</name>
		<value>xdq:9001</value>
</property>

---

[参考文献]
[1]https://www.linuxidc.com/Linux/2015-01/112045.htm
[2]https://blog.csdn.net/qq_36219266/article/details/81232315
[3]https://blog.csdn.net/ajing454266432/article/details/79893457
[4]https://blog.csdn.net/yjc_1111/article/details/53817750

---