搭建采用windows108G内存 centos6.6 jdk1.8 4台虚拟机
1 上传JDK至虚拟机解压文件
tar -zxvf jdk-8u141-linux-x64.tar.gz
2 配置环境变量增加JAVA环境变量
vi /etc/profile
export JAVA_HOME=/app/jdk1.8
export PATH=.:$PATH:$JAVA_HOME/bin
3 配置完成后:wq退出输入在etc目录下输入此命令让配置文件生效
[root@centos01 etc]# source profile
4 输入java -version命令查看版本信息
[root@centos01 etc]# java -version
java version "1.8.0_141"
Java(TM) SE Runtime Environment (build 1.8.0_141-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.141-b15, mixed mode)
jdk配置就完成了
5 配置主机名与域名映射(后面使用主机名完成访问)修改主机名为centos01
[root@centos01 etc]# vi sysconfig/network
增加如下配置
NETWORKING=yes
HOSTNAME=centos01
6 修改hosts文件配置域名映射
[root@centos01 etc]# vi hosts
增加如下配置
192.168.131.3 centos01 linux机器Ip地址映射
192.168.131.2 windows windows ip地址映射
检查是否配置成功
[root@centos01 etc]# ping centos01
PING centos01 (192.168.131.3) 56(84) bytes of data.
64 bytes from centos01 (192.168.131.3): icmp_seq=1 ttl=64 time=0.028 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=2 ttl=64 time=0.040 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=3 ttl=64 time=0.040 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=4 ttl=64 time=0.039 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=5 ttl=64 time=0.039 ms
64 bytes from centos01 (192.168.131.3): icmp_seq=6 ttl=64 time=0.042 ms
ping windows
PING windows (192.168.131.2) 56(84) bytes of data.
64 bytes from windows (192.168.131.2): icmp_seq=1 ttl=128 time=16.0 ms
64 bytes from windows (192.168.131.2): icmp_seq=2 ttl=128 time=0.284 ms
64 bytes from windows (192.168.131.2): icmp_seq=3 ttl=128 time=0.330 ms
64 bytes from windows (192.168.131.2): icmp_seq=4 ttl=128 time=0.267 ms
^C
域名映射就配置完成了。
7 克隆出3台虚拟机(以centos01为例)
7.1修改Ip地址【centos01:192.168.131.3】【centos02:192.168.131.4】【centos03:192.168.131.5】【centos03:192.168.131.6】 centos02到04 分别需要修改网卡名称为ifcfg-eth1
[root@centos01 etc]# cat sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
TYPE="Ethernet"
ONBOOT="yes"
NM_CONTROLLED="yes"
BOOTPROTO="static"
IPADDR=192.168.131.3
NETMASK=255.255.255.0
GETAWAY=192.168.131.2
DNS1=8.8.8.8
[root@centos01 etc]#
7.2 修改主机名 vi sysconfig/network centos01 centos02 centos03 centos04
[root@centos01 etc]# cat sysconfig/network
NETWORKING=yes
HOSTNAME=centos01
GETAWAY=192.168.131.2
7.3 配置域名映射
[root@centos01 etc]# cat hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.131.3 centos01
192.168.131.4 centos02
192.168.131.5 centos03
192.168.131.6 centos04
192.168.131.2 windows
7.4 安装jdk(已完成)克隆的母机已安装
7.5 关闭防火墙
[root@centos01 /]# service iptables status
iptables: Firewall is not running.
7.6 ssh免密登陆
本地免密码登录
ssh-keygen
ssh-copy-id centos02
7. 7 时间同步(date -s '2019-03-11 20:23:10')
8 配置本地yum源(https://www.cnblogs.com/gaoxu007/p/7173112.html)
mkdir /mnt/cdrom 创建文件
# mount -o loop /dev/cdrom /mnt/cdrom/
# cd /etc/yum.repos.d/
# rm -rf * //删除下面的所有文件或者命名为其他名字
# touch yum.repo 创建一个文件
# vi yum.repo 编辑文件 加入以下内容
[rhel]
name =Red Hat enterprise Linux 6
baseurl=file:///mnt/cdrom
enable=1
gpgkey=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-CentOS-6 此文件在/mnt/cdrom/目录下
修改 /etc/fstab 这个文件 再最后面加入
/dev/cdrom/ /mnt/cdrom iso9660 ro 0 0
[root@centos01 /]# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Sun Mar 10 05:58:00 2019
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_centos01-lv_root / ext4 defaults 1 1
UUID=b29da312-2468-4116-b915-a57868afaf49 /boot ext4 defaults 1 2
/dev/mapper/vg_centos01-lv_swap swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/cdrom/ /mnt/cdrom iso9660 ro 0 0
9 安装 yum install openssh-clients
用scp命令拷贝文件至其他机器
10 上传hadoop2.8.3的压缩包 并解压
11 修改配置文件hdfs-site.xml文件
配置nameNode 在哪台环境上运行
配置nameNode的数据存储目录
配置dataNode的数据目录
配置secondarynamenode在哪台机器上运行
<configuration>
<property>
<name>dfs.nameode.rpc-address</name>
<value>centos01:9000</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/hdpdata/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/hdpdata/data</value>
</property>
</configuration>
12 修改vi hadoop-env.sh 修改本地jdk配置 export JAVA_HOME=/app/jdk1.8
13 修改etc/profile文件加上hadoop的环境变量
done
unset i
unset -f pathmunge
export JAVA_HOME=/app/jdk1.8
export HADOOP_HOME=/hadoop/hadoop-2.8.3
export PATH=.:$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin
14 通过scp命令拷贝文件至centos02 centos03 centos04 机器
scp profile centos04:/etc/
15 格式化存储目录 hadoop namenode -format
19/03/12 23:39:31 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
19/03/12 23:39:31 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
19/03/12 23:39:31 INFO util.GSet: Computing capacity for map NameNodeRetryCache
19/03/12 23:39:31 INFO util.GSet: VM type = 64-bit
19/03/12 23:39:31 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
19/03/12 23:39:31 INFO util.GSet: capacity = 2^15 = 32768 entries
19/03/12 23:39:31 INFO namenode.FSImage: Allocated new BlockPoolId: BP-393041277-192.168.131.3-1552405171820
19/03/12 23:39:31 INFO common.Storage: Storage directory /hadoop/hdpdata/name has been successfully formatted.
19/03/12 23:39:31 INFO namenode.FSImageFormatProtobuf: Saving image file /hadoop/hdpdata/name/current/fsimage.ckpt_0000000000000000000 using no compression
19/03/12 23:39:32 INFO namenode.FSImageFormatProtobuf: Image file /hadoop/hdpdata/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds.
19/03/12 23:39:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
19/03/12 23:39:32 INFO util.ExitUtil: Exiting with status 0
19/03/12 23:39:32 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at centos01/192.168.131.3
16 启动hdfs的namenode 和datanode
hadoop-daemon.sh start namenode
-bash: hadoop-daemon.sh: command not found
错误原因未配置hadoop/sbin的环境目录
17 配置sbin的环境变量 vi etc/profile 文件 Path后增加:$HADOOP_HOME/sbin
18 启动namenode 报错 没有权限
.defaultFS is file:///
2019-03-12 23:52:36,318 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-03-12 23:52:36,615 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:644)
at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddressCheckLogical(DFSUtilClient.java:673)
at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:635)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:532)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:663)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:683)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:906)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:885)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1626)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1694)
2019-03-12 23:52:36,622 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2019-03-12 23:52:36,629 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at centos01/192.168.131.3
************************************************************/
18 未配置core-site.xml文件
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://centos01:9000</value>
</property>
</configuration>
19 启动namenode和3台datanode机器
hadoop-daemon.sh start datanode
hadoop-daemon.sh start namenode
20 修改本机hosts文件 增加ip映射
C:\Windows\System32\drivers\etc
192.168.131.3 centos01
192.168.131.4 centos02
192.168.134.5 centos03
192.168.131.6 centos04
21 浏览器访问http://centos01:50070/dfshealth.html#tab-datanode
22 环境搭建完毕 在浏览器中可以看到节点信息