hadoop安装
官网:http://hadoop.apache.org/
下载并安装到opt目录下(3.x要求jdk1.8版本)
tar -zxvf hadoop-3.2.0.tar.gz
常用命令
jps
格式化
bin/hdfs namenode -format
启动namenode
sbin/hadoop-daemon.sh start namenode
启动datanode
sbin/hadoop-daemon.sh start datanode
启动nodemanager
sbin/yarn-daemon.sh start nodemanager
启动resourcemanager
sbin/yarn-daemon.sh start resourcemanager
启动historyserver
sbin/mr-jobhistory-daemon.sh start historyserver
dfs命令
bin/hdfs dfs 后面加linux命令
删除output
bin/hdfs dfs -rm -r /user/output
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /user/input /user/output
本地模式
- grep测试
进入hadoop目录
cd hadoop
创建input文件夹
mkdir input
将etc里的所有xml文件复制到input文件夹里(此处etc是hadoop目录下的etc)
cp etc/hadoop/*.xml input/
执行以下命令(测试 获取符合’dfs[a-z.]+'该正则表达式的个数(可在output文件夹里一个文件看到))
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar grep input/ output ‘dfs[a-z.]+’
注释:使用该命令时output文件夹不能有,他会自行创建,否则会报错
- WordCount测试
进入hadoop目录
cd hadoop
创建wcinput文件夹
mkdir wcinput
cd wc.input
创建wc.input文件并编辑
touch wc.input
vim wc.input
添加以下内容
hadoop yarn
hadoop mapreduce
hzdy
hzdy
测试
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount wcinput wcoutput
查看
cat wcoutput/part-r-00000
伪分布式HDFS
配置etc/hadoop/core-site.xml
vim etc/hadoop/core-site.xml
如下
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop100:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/hadoop-3.2.0/data/tmp</value>
</property>
配置etc/hadoop/hdfs-site.xml
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
如下
配置etc/hadoop/hadoop-env.sh(不配置的集群不能识别)
export JAVA_HOME=/opt/jdk1.8.0_201
启动集群
(a)格式化NameNode(第一次启动时格式化,以后就不要总格式化了)
bin/hdfs namenode -format
(b)启动NameNode
sbin/hadoop-daemon.sh start namenode
(c)启动DataNode
sbin/hadoop-daemon.sh start datanode
查看集群
jps
访问localhost:9870(2.x版本端口号为50070,3.x版本端口号为9870)
点击Utilities下的Browse the file system
里面相当于一个linux文件目录
创建目录
bin/hdfs dfs -mkdir -p /user/input
浏览器可以看到新建的文件目录
上传文件(前面参数是上传文件路径 后面参数是目的路径)
bin/hdfs dfs -put wcinput/wc.input /user/input
浏览器可以看到上传的文件
值得注意的是hdfs的根目录是hdfs://localhost:9000/user/root,创建的文件都在这下面
这时候可以测试一下:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /user/input /user/output
YARN启动mapreduce
编辑 yarn-env配置java_home
vim etc/hadoop/yarn-env.sh
配置etc/hadoop/yarn-site.xml
vim etc/hadoop/yarn-site.xml
配置etc/hadoop/mapred-env.sh
vim etc/hadoop/mapred-env.sh
配置etc/hadoop/mapred-site.xml(使其在yarn上运行,默认是local)
vim etc/hadoop/mapred-site.xml
内容如下(HADOOP_MAPRED_HOME的值可以通过hadoop classpath获得再贴上去)
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0/etc/hadoop:/opt/hadoop-3.2.0/share/hadoop/common/lib/*:/opt/hadoop-3.2.0/share/hadoop/common/*:/opt/hadoop-3.2.0/share/hadoop/hdfs:/opt/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.0/share/hadoop/hdfs/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/*:/opt/hadoop-3.2.0/share/hadoop/yarn:/opt/hadoop-3.2.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.0/share/hadoop/yarn/*</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0/etc/hadoop:/opt/hadoop-3.2.0/share/hadoop/common/lib/*:/opt/hadoop-3.2.0/share/hadoop/common/*:/opt/hadoop-3.2.0/share/hadoop/hdfs:/opt/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.0/share/hadoop/hdfs/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/*:/opt/hadoop-3.2.0/share/hadoop/yarn:/opt/hadoop-3.2.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.0/share/hadoop/yarn/*</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/opt/hadoop-3.2.0/etc/hadoop:/opt/hadoop-3.2.0/share/hadoop/common/lib/*:/opt/hadoop-3.2.0/share/hadoop/common/*:/opt/hadoop-3.2.0/share/hadoop/hdfs:/opt/hadoop-3.2.0/share/hadoop/hdfs/lib/*:/opt/hadoop-3.2.0/share/hadoop/hdfs/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.2.0/share/hadoop/mapreduce/*:/opt/hadoop-3.2.0/share/hadoop/yarn:/opt/hadoop-3.2.0/share/hadoop/yarn/lib/*:/opt/hadoop-3.2.0/share/hadoop/yarn/*</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop100:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop100:19888</value>
</property>
</configuration>
启动集群
启动之前先确认nameNode和dataNode已经运行
开启resourcemanager
sbin/yarn-daemon.sh start resourcemanager
开启nodemanager
sbin/yarn-daemon.sh start nodemanager
开启历史服务器historyserver
sbin/mr-jobhistory-daemon.sh start historyserver
浏览器打开http://localhost:8088查看程序运行进程
删除之前的output
bin/hdfs dfs -rm -r /user/output
测试:
hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar wordcount /user/input /user/output
hadoop(完全分布式配置)
进入hadoop主目录下的etc/hadoop/修改配置文件
配置core-site.xml地址为主ip
配置hadoop-env.sh的java路径
配置hdfs-site.xml辅助名称节点主机(辅助节点是另外一台主机)
配置yarn-env.sh的java路径
配置yarn-site.xml的resourceManager地址
地址为除主机和辅助节点外的另外一台
配置mapred-env.sh的java路径
配置mapred-site.xml
分发所有hadoop配置文件
xsync /opt/hadoop-3.2.0/
分发完成后其他主机不需要修改内容了
启动集群
清空三台主机的logs
rm -rf data/logs/
格式化namenode(只格式化master)
bin/hdfs namenode -format
开启namenode(只开启master)
sbin/hadoop-daemon.sh start namenode
开启datanode(三台都开)
sbin/hadoop-daemon.sh start datanode