Hadoop分布式集群

Hadoop分布式集群

单机安装部署

购买云主机

主机	IP地址	配置
hadoop1	192.168.1.50	最低配置2核2G

hadoop安装

拷贝云盘 public/hadoop/hadoop-2.7.7.tar.gz 到 hadoop1 上

[root@hadoop1 ~]# yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel
[root@hadoop1 ~]# tar -zxf hadoop-2.7.7.tar.gz
[root@hadoop1 ~]# mv hadoop-2.7.7 /usr/local/hadoop
[root@hadoop1 ~]# chown -R 0.0 /usr/local/hadoop

配置JAVA运行环境

[root@hadoop1 ~]# vim /etc/hosts
192.168.1.50	hadoop1
[root@hadoop1 ~]# rpm -qa | grep java
[root@hadoop1 ~]# rpm -ql java-1.8.0-openjdk
[root@hadoop1 ~]# vim /usr/local/hadoop/etc/hadoop/hadoop-env.sh
25:  export JAVA_HOME="java-1.8.0-openjdk安装路径"
33:  export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop"
[root@hadoop1 ~]# /usr/local/hadoop/bin/hadoop version

热点词汇分析

[root@hadoop1 ~]# cd /usr/local/hadoop
[root@hadoop1 hadoop]# mkdir input
[root@hadoop1 hadoop]# cp *.txt input/
[root@hadoop1 hadoop]# ./bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.7.jar wordcount ./input ./output
[root@hadoop1 hadoop]# cat ./output/*

集群安装部署

购买云主机

主机	IP地址	配置
hadoop1	192.168.1.50	最低配置2核2G
node-0001	192.168.1.51	最低配置2核2G
node-0002	192.168.1.52	最低配置2核2G
node-0003	192.168.1.53	最低配置2核2G

HDFS架构图