Hadoop单击版测试,搭建集群节点

一.Hadoop技术原理：

Hdfs主要模块：NameNode、DataNode
Yarn主要模块：ResourceManager、NodeManager

二.hadoop单机版测试

1.安装hadoop,创建hadoop用户

tar zxf jdk-8u181-linux-x64.tar.gz
tar zxf hadoop-3.0.3.tar.gz 
ln -s jdk1.8.0_181/ java					##制作软链接,便于启动
ln -s hadoop-3.0.3 hadoop
[hadoop@server1 ~]$ ls
hadoop-3.0.3  hadoop-3.0.3.tar.gz  jdk1.8.0_181  jdk-8u181-linux-x64.tar.gz

2.配置环境变量

  [hadoop@server1 hadoop]$ pwd
/home/hadoop/hadoop/etc/hadoop
[hadoop@server1 hadoop]$ vim hadoop-env.sh 
 54 export JAVA_HOME=/home/hadoop/java

[hadoop@server1 ~]$ vim .bash_profile 
PATH=$PATH:$HOME/.local/bin:$HOME/bin:$HOME/java/bin
[hadoop@server1 ~]$ source .bash_profile 
[hadoop@server1 ~]$ jps    ##配置成功可以调用

3.测试

[hadoop@server1 hadoop]$ pwd
/home/hadoop/hadoop
[hadoop@server1 hadoop]$ mkdir input
[hadoop@server1 hadoop]$ cp etc/hadoop/*.xml input/
[hadoop@server1 hadoop]$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.3.jar grep input output 'dfs[a-z.]+'
[hadoop@server1 hadoop]$ ls input/
capacity-scheduler.xml  hadoop-policy.xml  httpfs-site.xml  kms-site.xml     yarn-site.xml
core-site.xml           hdfs-site.xml      kms-acls.xml     mapred-site.xml
[hadoop@server1 hadoop]$ cd output/
[hadoop@server1 output]$ ls
part-r-00000  _SUCCESS

二.伪分布式

1.编辑文件

hadoop@server1 hadoop]$ pwd
/home/hadoop/hadoop/etc/hadoop
[hadoop@server1 hadoop]$ vim core-site.xml
    <configuration>
        <property>