CentOS7 + Hadoop2.7.3开发环境搭建
前言
本文主要参考http://blog.csdn.net/fxdaniel/article/details/48315805这篇博客。作者在CentOS7 下搭建hadoop2.7.3,亲测可用。后面将继续更新hadoop+eclipse的开发环境。
0.新建hadoop用户,并添加sudo权限
#useradd -m hadoop -s /bin/bash
#passwd hadoop
vi /etc/sudoers
添加 hadoop ALL=(ALL) ALL
1.关闭防火墙
sudo systemctl status firewalld
sudo systemctl disable firewalld
sudo systemctl stop firewalld
2.关闭SELinux
sestatus
sudo setenforce 0
sudo vi /etc/sysconfig/selinux
修改:SELINUX=disabled
3.安装openssh-server
CentOS7默认安装了Openssh服务,当然有些老版本没有的也需要手动安装:
sudo yum install openssh-server
4.安装java
sudo yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
采用yum安装模式,java的环境变量已经自动添加到/usr/bin/下,但是不知道hadoop里能不能识别,所以保险起见还是手动添加一下java环境变量吧。采用yum安装的话,java的安装目录在/usr/lib/jvm/下
vi ~/.bash_profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
5.配置ssh免密码登陆
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub>> authorized_keys
6.解压、修改hadoop配置文件
从http://hadoop.apache.org/releases.html下载hadoop二进制包。作者不小心放在了/usr/下,建议还是放在/home/haddop/下比较好。创建dfs、tmp文件夹,不知道hadoop会不会自动创建,反正作者是手动创建的。注意用户权限关系
tar -zxvf hadoop2.7.3 -C /usr/
mv /usr/hadoop2.7.3/ /usr/hadoop
mkdir /usr/hadoop/tmp
mkdir /usr/hadoop/dfs
mkdir /usr/hadoop/dfs/data
mkdir /usr/hadoop/dfs/name
mkdir /usr/hadoop/dfs/namesecondary
chown -R hadoop:hadoop /usr/hadoop/
修改core-site.xml
vi /usr/hadoop/hadoop/etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.108.201:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/hadoop/tmp</value>
<description>Abase for othertemporary directories.
</description>
</property>
</configuration>
修改hdfs-site.xml,由于是伪分布,这里需要改一下副本数
vi /usr/hadoop/hadoop/etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.namenode.checkpoint.dir</name> <value>file:/usr/hadoop/dfs/namesecondary</value>
</property>
</configuration>
修改mapred-site.xml
vi /usr/hadoop/hadoop/etc/hadoop/mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.staging-dir</name>
<value>/usr/hadoop/staging</value>
</property>
</configuration>
修改yarn-site.xml
vi /usr/hadoop/hadoop/etc/hadoop/yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>
</configuration>
修改/etc/hosts
楼主在运行wordcount的时候出现了错误,后来通过修改host解决,可惜的是当时没有截图
7.添加hadoop环境变量
vi ~/.bash_profile
source /home/.bash_profile
修改hadoop-env.sh
vi /usr/hadoop/hadoop/etc/hadoophadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
8.运行hadoop
hadoop namenode -format
./usr/hadoop/hadoop/sbin/start-all.sh
jps
9.运行wordcount
创建2文件,并分别输入hello hadoop、hello hadoop
vi /usr/hadoop/file1.txt
vi /usr/hadoop/file2.txt
在Hdfs根目录下新建input目录将file1.txt和file2.txt两个文件上传到HDFS中的input目录
hdfs dfs -mkdir /input
hdfs dfs -put file1.txt /input
hdfs dfs -put file2.txt /input
执行hadoop自带的WordCount程序,统计单词数
cd /usr/hadoop/hadoop/share/hadoop/mapreduce
hadoop jar hadoop-mapreduce-examples-2.7.1.jar wordcount /input /output
等个几分钟
hdfs dfs -ls /
hdfs dfs -ls /output
hdfs dfs -cat /output/part-r-00000
10.安装过程中出现的错误
Unable to load native-hadoop library for your platform:
参考这篇博客:
http://blog.csdn.net/jack85986370/article/details/51902871