参照Hadoop包里面的share的doc的离线文档
说明:这里搭建HA时一并把分布式zookeeper分布式spark一并搭建了,
以三台主机为主 master slave1 slave2
-
解压 重命名 环境变量
1. tar -zxvf 解压包名(jdk的包,hadoop包,zookeeper包,spark包) 2. mv 文件名 重命名 3. vi /etc/profile(环境变量对所有用户有效)vi ~/.bash_profile(只对当前用户有效) export JAVA_HOME=/usr/local/src/jdk export HADOOP_HOME=/usr/local/src/hadoop export ZOOKEEPER_HOME=/usr/local/src/zookeeper export SPARK_HOME=/usr/local/src/spark export PATH=$JAVA_HOME/bin:$PATH export PATH=$HADOOP_HOME/bin:$PATH export PATH=$HADOOP_HOME/sbin:$PATH export PATH=$ZOOKEEPER_HOME/bin:$PATH export PATH=$SPARK_HOME/bin:$PATH 4. 刷新环境变量 source ~/.bash_profile
-
hadoop-env.sh
1. 进入到hadoop/etc/hadoop 2. vi hadoop-env.sh 修 export JAVA_HOME=/usr/local/src/jdk
-
core.site.xml
vi core.site.xml <property> <name>fs.defaultFS</name> <value>hdfs://mycluster</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>master:2181,slave1:2181,slave2:2181</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/src/hadoop/tmp</value> </property>
-
hdfs-site.xml
<property> <name>dfs.nameservices</name> <value>mycluster</value> </property> <property> <name>dfs.ha.namenodes.mycluster</name> <value>nn1,nn2</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn1</name> <value>master:8020</value> </property> <property> <name>dfs.namenode.rpc-address.mycluster.nn2</name> <value>slave1:8020</value> </property> <property> <name>dfs.namenode.http-address.mycluster.nn1</name> <value>master:50070</value> </property> <property> <name>dfs.namenode.http-address.mycluster.nn2</name> <value>slave1:50070</value> </property> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://master:8485;slave1:8485;slave2:8485/mycluster</value> </property> <property> <name>dfs.client.failover.proxy.provider.mycluster</name> <value>org.apache.hadoop.hdfs.server.namenode.ha .ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.fencing.methods</name> <value>shell(/bin/true)</value> </property> <property> <name>dfs.journalnode.edits.dir</name> <value>/usr/local/src/hadoop/journaldata</value> </property> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property>
-
mapred-site.xml
cp mapred-site.xml.template mapred-site.xml <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property>
-
yarn-site.xml
<property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>cluster1</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>master</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>slave1</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>master:2181,slave1:2181,slave2:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair .FairScheduler</value> </property> <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property> <property> <name>yarn.nodemanager.vmem-check-enabled</name> <value>false</value> </property>
-
修改slaves
vi slaves master slave1 slave2
-
配置zookeeper
1. cd zookeeper/conf 2. cp zoo_sample.cfg zoo.cfg 3. vi zoo.cfg 修 dataDir=/usr/local/src/zookeeper/data 添加 server.1=master:2888:3888 server.2=slave1:2888:3888 server.3=slave2:2888:3888 4. 配置zookeeper的myid(执行完十步然后执行,三台机上面都要执行) vi zookeeper/data echo 1 > zookeeper/data/myid(slave1上数字改为2) 5. 启动和查看zookeeper zkServer.sh start/zkServer.sh stop zkServer.sh status
-
高可用搭建spark
1. cd spark/conf 2. cp spark-env.sh.template spark-env.sh 3. vi spark-env.sh 添加 export JAVA_HOME=/usr/local/src/jdk export SPARK_HOME=/usr/local/src/spark export HADOOP_CONF_DIR=/usr/local/src/hadoop/etc/hadoop 4. cp slaves.template slaves 5. vi slaves master slave1 slave2
-
分发配置文件
1. scp -r /usr/local/src/hadoop spark zookeeper jdk slava1:/usr/local/src 2. scp ~/.bash_profile slave1:~
-
格式化并启动
1. 启动所有的journalnode(三台机器都要) hadoop-daemon.sh start journalnode 2. 格式化master上的namenode并启动 hdfs namenode -format hadoop-daemon.sh start namenode 3. slave1复制master的namenode并启动 hdfs namenode -bootstrapStandby hadoop-daemon.sh start namenode 4. 在master上的namenode上格式化ZKFC hdfs zkfc -formatZK 5. 启动hadoop集群 cd hadoop/sbin start-all.sh/stop-all.sh 6. 启动JobHistoryServer节点 cd hadoop/sbin mr-jobhistory-daemon.sh start historyserver
-
spark运行jar包
hdsf下运行spark代码jar包: spark-submit --class 类名 --master yarn --deploy-mode client -- executor-memory 1G -- num-executors 3 hotels.jar hdfs://master:8020 第二种方式进行cluster: spark-submit --class HotelTask1 --master yarn --deploy-mode cluster -- executor-memory 1G --num-executors 3 hotels.jar hdfs://master:8020
需要全套搭建最简化的视频与全套安装包的请添加群924338546