环境Hadoop-2.6.0-cdh5.7.0 centos 6.5 jdk1.7
我们将会根据官方文档来进行hadoop的安装文档地址
首先
1.配置ssh
ssh-keygen -t rsa
cp .ssh/id_rsa.pub ~/.ssh/authorized_keys
看到隐藏文件
ls -la
cd .ssh/
2.配置jdk
vi ~/.bash_profile
export JAVA_HOME=/usr/java/jdk1.7.0_80
export PATH=$ JAVA_HOME/bin:$ PATH
使brash.profile生效
source ~/.brash.profile
#Assuming your installation directory is /usr/local/hadoop
export HADOOP_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
export PATH=$HADOOP_HOME/bin: $HADOOP_HOME/sbin: $ PATH
3.解压hadoop到 app目录下
(tar -zxvf -C)
tar -zxvf /home/hadoop/software/hadoop-2.6.0-cdh5.7.0.tar.gz -C /home/hadoop/app/
1)edit the file etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.7.0_80
2)Configuration(配置)
Use the following:
etc/hadoop/core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:8020</value>
#如果想使用之际hostname 比如hadoop000 需要在 etc/hosts中 映射hosename ip地址
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/app/tmp</value>
</property>
</configuration>
etc/hadoop/hdfs-site.xml: 修改副本系数为一
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
在slaves文件中配置上你的datanode 只有一个所以
4.Execution运行
1)第一次运行时需要格式化文件系统Format the filesystem:
$ bin/hdfs namenode -format
2)启动 Start NameNode daemon and DataNode daemon:
$ sbin/start-dfs.sh(在sbin目录下执行 ./start-dfs.sh )
cd sbin
./start-dfs.sh
[root@hadoop000 sbin]# ./start-dfs.sh
19/01/10 11:07:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop000]
hadoop000: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-root-namenode-hadoop000.out
hadoop000: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-root-datanode-hadoop000.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-root-secondarynamenode-hadoop000.out
19/01/10 11:07:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
运行jps(所有java进程pid的命令) 查看进程、
出现则运行成功:
7820 NameNode
7914 DataNode
8613 Jps
3)停止hdfs:
$ sbin/stop-dfs.sh
YARN 单节点启动
yarn相当于一个操作系统级别的调度框架,它使得spark 内存计算框架 strom流计算框架 。。能够在hdoop2.X 同一个集群上共享同一份hdfs资源
配置:
vi etc/hadoop/mapred-site.xml:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
vi etc/hadoop/yarn-site.xml:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
启动yarn:
$ sbin/start-yarn.sh
cd /sbin
./start-yarn.sh
停止:
$ sbin/stop-yarn.sh
执行完成后我们可以在浏览器中输入http://localhost:50070/查看
active 活跃的