一 : 修改机器名
1. 修改3台机器的机器名,注意名字不要带下划线
修改机器名命令:
hostnamectl set-hostname xxxx
然后退出shell重新登陆
修改3台机器的hosts文件
vim /etc/hosts
添加以下内容
192.107.53.157 hadoop-master
192.107.53.158 hadoop-slave1
192.107.53.159 hadoop-slave2
二:主从节点免密码登陆
1. 免密钥登陆本机
1. 关闭防火墙
查看防火墙状态
service iptables status
关闭防火墙
service iptables stop
chkconfig iptables off
2. 免密码登录本机
1)生产秘钥
ssh-keygen -t rsa
2)将公钥追加到”authorized_keys”文件
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
3)赋予权限
chmod 600 .ssh/authorized_keys
4)验证本机能无密码访问
ssh hadoop-master
依次配置hadoop-slave1, hadoop-slave2 免密钥登陆本机
2. hadoop-master本机无密码登录hadoop-slave1、hadoop-slave2,以hadoop-master无密码登录hadoop-slave1为例进行讲解:(主节点免密码登陆从节点)
1)登录hadoop-slave1 ,复制hadoop-master服务器的公钥”id_rsa.pub”到hadoop-slave1服务器的”root”目录下。
scp root@192.107.53.157:/root/.ssh/id_rsa.pub /root/ 或者
scp root@hadoop-master:/root/.ssh/id_rsa.pub /root/
2)将hadoop-master的公钥(id_rsa.pub)追加到hadoop-slave1的authorized_keys中
cat id_rsa.pub >> .ssh/authorized_keys
rm -rf id_rsa.pub
3)在 hadoop-master上面测试
ssh hadoop-slave1
登录hadoop-slave2 ,执行上面同样的操作
3. 配置hadoop-slave1,hadoop-slave2本机无密码登录hadoop-master(从节点免密码登陆主节点)
1)登录hadoop-master,复制hadoop-slave1服务器的公钥”id_rsa.pub”到hadoop-master服务器的”/root/”目录下。
scp root@hadoop-slave1:/root/.ssh/id_rsa.pub /root/
2)将hadoop-slave1的公钥(id_rsa.pub)追加到hadoop-master的authorized_keys中。
cat id_rsa.pub >> .ssh/authorized_keys
rm -rf id_rsa.pub //删除id_rsa.pub
3)在 hadoop-slave1上面测试
ssh hadoop-master
重复上述步骤,使得hadoop-slave2本机也可以无密码登陆主节点
至此,主从节点的免密码登陆完成
三:hadoop安装
1. hadoop-master的安装和配置
1) 安装jdk
#下载
jdk-8u171-linux-x64.tar.gz
#解压
tar -xzvf
jdk-8u171-linux-x64.tar.gz -C /usr/local
#重命名
mv jdk-8u171-linux-x64 java
2) 安装hadoop
#下载
hadoop-3.1.0.tar.gz
#解压
tar -xzvf hadoop-3.1.0.tar.gz -C /usr/local
#重命名
mv hadoop-3.1.0.tar.gz hadoop
3) 配置环境变量
vim /etc/profile
JAVA_HOME="/usr/local/java"
export PATH="$JAVA_HOME/bin:$PATH"
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile
4) hadoop相关配置
cd /usr/local/hadoop/etc/hadoop
a) 配置core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>file:/usr/local/hadoop/tmp</value>
<description>Abase for other temporary directories.</description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.107.53.157:9000</value>
</property>
</configuration>
b) 配置hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/local/hadoop/hdfs/name</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/hadoop/hdfs/data</value>
</property>
<property>
<name>dfs.namenode.rpc-address</name>
<value>192.107.53.157:9000</value>
</property>
</configuration>
c) 配置 mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>http://192.107.53.157:9001</value>
</property>
<property>
<name>mapreduce.application.classpath</name>
<value>
/usr/local/hadoop/etc/hadoop,
/usr/local/hadoop/share/hadoop/common/*,
/usr/local/hadoop/share/hadoop/common/lib/*,
/usr/local/hadoop/share/hadoop/hdfs/*,
/usr/local/hadoop/share/hadoop/hdfs/lib/*,
/usr/local/hadoop/share/hadoop/mapreduce/*,
/usr/local/hadoop/share/hadoop/mapreduce/lib/*,
/usr/local/hadoop/share/hadoop/yarn/*,
/usr/local/hadoop/share/hadoop/yarn/lib/*
</value>
</property>
</configuration>
d) 配置yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-master</value>
</property>
</configuration>
e) 配置workers文件
hadoop-slave1
hadoop-slave2
f) 配置hadoop-env.sh
export JAVA_HOME=/usr/local/java
2. hadoop-slave1的安装和配置(其他从节点操作一样)
1)复制hadoop和java到hadoop-slave1节点
scp -r /usr/local/hadoop hadoop-slave1:/usr/local/
scp -r /usr/local/java hadoop-slave1:/usr/local/
2) 登录hadoop-slave1服务器,删除workers内容
rm -rf /usr/local/hadoop/etc/hadoop/workers
3) 配置环境变量
vim /etc/profile
JAVA_HOME="/usr/local/java"
export PATH="$JAVA_HOME/bin:$PATH"
export HADOOP_HOME=/usr/local/hadoop
export PATH=$PATH:$HADOOP_HOME/bin
source /etc/profile
四:启动hadoop集群
修改sh的用户,不然启动会报错
vim start-dfs.sh 以及 vim stop-dfs.sh 分别添加下面4行
HDFS_DATANODE_USER=root
HADOOP_SECURE_DN_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root
vim start-yarn.sh 以及 vim stop-yarn.sh 分别添加下面4行
YARN_RESOURCEMANAGER_USER=root
YARN_NODEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
1) 首次启动需要format namenode
hdfs namenode -format
2) 启动hadoop:
sbin/start-all.sh
3) 使用jps命令查看运行情况
#master 执行 jps查看运行情况
25928 SecondaryNameNode
25742 NameNode
26387 Jps
26078 ResourceManager
#slave 执行 jps查看运行情况
24002 NodeManager
23899 DataNode
24179 Jps
4) 跑计算圆周率的程序,说明hadoop可以正常运行
hadoop jar /usr/local/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.0.jar pi 5 10
5) 登陆浏览器 http://192.107.53.157:8088/cluster/apps
五:问题处理
1. Hadoop集群配置之后浏览器无法访问问题
https://blog.csdn.net/csdn_chuxuezhe/article/details/73322068
修改主机名:
vi /etc/sysconfig/network
在下边修改:
NETWORKING=yes
HOSTNAME=hadoop-master
同时,修改hosts vi /etc/hosts
192.107.53.157 hadoop-master
192.107.53.158 hadoop-slave1
192.107.53.159 hadoop-slave2
重启!!!!!
参考文献
http://www.ityouknow.com/hadoop/2017/07/24/hadoop-cluster-setup.html
六: spark 集群安装
1. 以hadoop-master节点为例
1. 安装scala
2. 安装spark
3. 配置环境变量
#scala
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin
#spark
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
4. spark配置
cp spark-env.sh.template
spark-env.sh
vim spark-env.sh
export JAVA_HOME=/usr/local/java
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_HOME=/usr/local/spark
export SCALA_HOME=/usr/local/scala
export SPARK_MASTER_IP=hadoop-master
export SPARK_EXECUTOR_MEMORY=1G
export SPARK_WORKER_CORES=2
export SPARK_WORKER_INSTANCES=1
JAVA_HOME:Java安装目录
SCALA_HOME:Scala安装目录
HADOOP_HOME:hadoop安装目录
HADOOP_CONF_DIR:hadoop集群的配置文件的目录
SPARK_MASTER_IP:spark集群的Master节点的ip地址
SPARK_WORKER_MEMORY:每个worker节点能够最大分配给exectors的内存大小
SPARK_WORKER_CORES:每个worker节点所占有的CPU核数目
SPARK_WORKER_INSTANCES:每台机器上开启的worker节点的数目
5. 编辑slaves
cp slaves.template slaves
vi slaves加入Worker节点如下配置
hadoop-slave1
hadoop-slave2
2. hadoop-slave1,hadoop-slave2两个节点将scala,spark包复制过去即可
1.登陆hadoop-svale1
scp -r root@192.107.53.157:/usr/local/scala /usr/local
scp -r root@192.107.53.157:/usr/local/spark /usr/local
2. 配置环境变量
#scala
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin
#spark
export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin
登陆hadoop-svale2执行一样的操作
3. 启动spark集群
1 cd /usr/local/spark/sbin
./start-all.sh
4. 运行样例
下面链接各种提交模式都有,可参考
http://zhenggm.iteye.com/blog/2358324
./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client examples/jars/spark-examples_2.11-2.1.0.jar
七 问题解决
问题1:Container xxx is running beyond physical memory limits
-
日志:
问题分析:Container [pid=134663,containerID=container_1430287094897_0049_02_067966] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 1.5 GB of 10 GB virtual memory used. Killing container. Dump of the process-tree for
从日志可以看出,container使用内存超过虚拟内存的限制,导致如上问题。默认2.1;
NodeManager端设置,类似系统层面的overcommit问题,需要调节yarn.nodemanager.vmem-pmem-ratio相关参数,在yarn-site.xml修改:
<property>
<name>yarn.nodemanager.vmem-pmem-ratio</name>
<value>10</value>
</property>
–或者yarn.nodemanager.vmem-check-enabled,false掉
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
问题2:datanode进程挂掉
重启进程
sbin/hadoop-daemon.sh start datanode
问题3:配置spark-history并启动进程
只有日志文件往往是不够的,有时候我们要查看历史记录,这就需要在driver节点启动History Server
在$SPARK_CONF_DIR下面的spark-defaults.conf文件中添加EventLog和History Server的配置
# EventLog
spark.eventLog.enabled true
spark.eventLog.dir file:///opt/spark/current/spark-events
# History Server
spark.history.provider org.apache.spark.deploy.history.FsHistoryProvider
spark.history.fs.logDirectory file:/opt/spark/current/spark-events
这里注意要创建/opt/spark/current/spark-events路径,application的执行历史才会保存到该路径。
执行启动命令
./sbin/start-history-server.sh
可参考文献
http://www.leonlu.cc/profession/14-spark-log-and-history/
问题4 日志配置
https://blog.csdn.net/stark_summer/article/details/46929481
spark的日志一方面打印到控制台,一方面写入到/home/hadoop/spark.log中了,这是日志的继承特性,后面再来改进,目前把log4j.rootCategory=INFO, console,FILE改为log4j.rootCategory=INFO, FILE即可
cd /usr/local/spark/conf
vim log4j.properties
log4j.rootCategory=INFO, console,FILE
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Settings to quiet third party logs that are too verbose
log4j.logger.org.eclipse.jetty=WARN
log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender
log4j.appender.FILE.Threshold=DEBUG
log4j.appender.FILE.file=/usr/local/spark/logs/spark.log
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
log4j.appender.FILE.layout.ConversionPattern=[%-5p] [%d{yyyy-MM-dd HH:mm:ss}] [%C{1}:%M:%L] %m%n
# spark
log4j.logger.org.apache.spark=INFO
问题5 spark连接ssl的kafka
spark集群的3台机器分别都要生成
将这4个文件放在一个目录下,直接运行2_ServerGenKey.sh脚本即可
八: 配置定时执行脚本
vim /etc/crontab
*/1 * * * * root /bin/sh /usr/local/jars/run.sh
若不想打印运行日志,可这样写 */1 * * * * root /bin/sh /usr/local/jars/run.sh /dev/null 2>&1
查看日志路径 vim /var/log/cron