这里写自定义目录标题
1 安装部署Hadoop
关闭防火墙
systemctl stop firewalld.service
systemctl disable firewalld.service
配置主机名
vi /etc/hosts
ip bigdata2
免密码登陆(不对称:公钥、私钥-----> 字符串)
ssh-keygen -t rsa
ssh-copy-id -i .ssh/id_rsa.pub root@bigdata2
安装jdk
mkdir tools
mkdir training
tar -zxvf jdk-8u181-linux-x64.tar.gz -C /root/training/
设置环境变量(zsh终端)
vi ~/.zshrc
export JAVA_HOME=/root/training/jdk1.8.0_241
export PATH=$JAVA_HOME/bin:$PATH
source ~/.zshrc
hadoop部署
tar -zxvf hadoop-3.1.2.tar.gz -C ~/training/
设置环境变量
vim ~/.zshrc
export HADOOP_HOME=/root/training/hadoop-3.1.2
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HDFS_DATANODE_USER=root
export HDFS_DATANODE_SECURE_USER=root
export HDFS_NAMENODE_USER=root
export HDFS_SECONDARYNAMENODE_USER=root
export YARN_RESOURCEMANAGER_USER=root
export YARN_NODEMANAGER_USER=root
source ~/.zshrc
配置hadoop
HDFS:NameNode、SecondaryNameNode、DataNode
Yarn:ResourceManager、NodeManager
hdfs-site.xml
<!--配置数据块的冗余度-->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!--禁用了HDFS的权限-->
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
core-site.xml
<!--NameNode的地址-->
<property>
<name>fs.defaultFS</name>
<value>hdfs://bigdata111:9000</value>
</property>
<!--HDFS对应的操作系统目录-->
<property>
<name>hadoop.tmp.dir</name>
<value>/root/training/hadoop-3.1.2/tmp</value>
</property>
mapred-site.xml
<!--MapReduce运行的框架-->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
yarn-site.xml
<!--配置Yarn的主节点-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bigdata111</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
HDFS格式化
hdfs namenode -format
启动hadoop,由于hadoop与spark的start-all.sh与stop-all.sh相同,为了避免冲突,将hadoop中start-all.sh改为start-hadoop-all.sh,stop-all.sh改为stop-hadoop-all.sh
start-hadoop-all.sh = start-dfs.sh + start-yarn.sh
hadoop3中:
HDFS端口:9870
Yarn:8088
hadoop2中
HDFS端口:50070
Yarn:8088
2 安装部署spark
spark的安装教程
启动spark
start_all
关闭spark
stop_all
spark网页端访问
http://ip:8080/