大数据一件装机脚本

一、大数据装机脚本

该装机脚本包括hadoop、hbase、hive、spark、sqoop、zeppelin、zookeeper等软件的安装；

• 网盘分享：https://pan.baidu.com/s/1ESVIcQISH5HwOBd4wWypVQ
• 提取码：n08k

二、文件结构

• 先在本机创建一个soft目录
• 再soft目录下创建以下文件夹
• installs 中放所有软件安装包
• jar 中为必要的 jar 包
• text 中为脚本和帮助文档
• 其他均为各软件的配置文件信息
  

三、各配置文件修改

1、hadoop配置文件

1）core-site.xml

• IP为虚拟机的IP地址，注意修改

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://IP:9000</value>
  </property>
  
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/hadoop/tmp</value>
  </property>
  
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>*</value>
  </property>
  
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>
  
  <property>
    <name>hadoop.native.lib</name>
    <value>false</value>
    <description>Should native hadoop libraries, if present, be used.
    </description>
  </property>
</configuration>

2）hadoop-env.sh

export JAVA_HOME=/opt/jdk
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"
export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_IDENT_STRING=$USER

3）hdfs-site.xml

• 注意hostname为主机名，注意修改

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>hostname:50090</value>
  </property>
</configuration>

4）mapred-site.xml

• IP为虚拟机的IP地址，注意修改

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>IP:10020</value>
  </property>
  
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>IP:19888</value>
  </property>
</configuration>

5）yarn-site.xml

• 注意hostname为主机名，注意修改

<?xml version="1.0"?>
<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hostname</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <!-- 日志聚集功能使用 -->
  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

  <!-- 日志保留时间设置7天 -->
  <property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
  </property>
</configuration>

2、hbase

1）hbase-env.sh

export JAVA_HOME=/opt/jdk
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
export HBASE_MANAGES_ZK=true

2）hbase-site.xml

• hostname为主机名，注意修改

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>hbase.rootdir</name>
    <value>hdfs://hostname:9000/hbase</value>
</property>
<property>
    <name>hbase.cluster.distributed</name>
    <value>false</value>
</property>
<property>
   <name>hbase.regionserver.wal.codec</name>
   <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
</property>

<property>
  <name>phoenix.schema.isNamespaceMappingEnabled</name>
  <value>true</value>
</property>

<property>
  <name>phoenix.schema.mapSystemTablesToNamespace</name>
  <value>true</value>
</property>

<property>
  <name>phoenix.functions.allowUserDefinedFunctions</name>
  <value>true</value>
  <description>enable UDF functions</description>
</property>
</configuration>

3、hive

1）hive-env.sh

HADOOP_HOME=/opt/hadoop
export HIVE_CONF_DIR=/opt/hive/conf
export HIVE_AUX_JARS_PATH=/opt/hive/lib

2）hive-site.xml

<configuration>
<!-- 在configuration中加入配置 -->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/opt/hive/warehouse</value>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<!-- 如果是远程mysql数据库的话需要在这里写入远程的IP或hosts -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>ok</value>
</property>
</configuration>

4、spark

1）spark-env.sh

#!/usr/bin/env bash
export JAVA_HOME=/opt/jdk
export SCALA_HOME=/opt/scala
export SPARK_HOME=/opt/spark
export SPARK_MASTER_IP=hadoop
export SPARK_EXECUTOR_MEMORY=1G

5、sqoop

1）sqoop-env.sh

export HADOOP_COMMON_HOME=/opt/hadoop
export HADOOP_MAPRED_HOME=/opt/hadoop
export HIVE_HOME=/opt/hive
export ZOOKEEPER_HOME=/opt/zookeeper
export ZOOCFGDIR=/opt/zookeeper
export HBASE_HOME=/opt/hbase

2）configure-sqoop

• 网盘中已给出，无需修改，内容太多不做展示

6、zeppelin

1）zeppelin-env.sh

export JAVA_HOME=/opt/jdk
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop

2）zeppelin-site.xml

• 网盘中已给出，无需修改，内容太多不做展示

7、zookeeper

1）zoo.cfg

• hostname为主机名，注意修改

# The number of milliseconds of each tick
tickTime=2000
#最大访问数:不限制
maxClientCnxns=0
#初始化最小进程数:50
initLimit=50
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
#数据目录
dataDir=/opt/zookeeper/zkdata
#日志目录
dataLogDir=/opt/zookeeper/zkdatalogs
clientPort=2181
server.1=hostname:2888:3888

四、脚本内容

创建脚本时需传入一个参数为主机名

#!/bin/bash
#禁用防火墙,脚本中禁用执行结束后需重启虚拟机
systemctl disable firewalld
#修改主机名
hostnamectl set-hostname $1
#解压所有软件安装压缩包，并重命名
cd /opt/soft/installs
tar -zxvf jdk-8u221-linux-x64.tar.gz -C /opt
tar -zxvf hadoop-2.6.0-cdh5.14.2.tar.gz -C /opt
tar -zxvf zookeeper-3.4.6.tar.gz -C /opt
tar -zxvf hbase-1.2.0-cdh5.14.2.tar.gz -C /opt
tar -zxvf hive-1.1.0-cdh5.14.2.tar.gz -C /opt
tar -zxvf zeppelin-0.8.1-bin-all.tgz -C /opt
tar -zxvf apache-phoenix-4.14.0-cdh5.14.2-bin.tar.gz -C /opt
tar -zxvf sqoop-1.4.6-cdh5.14.2.tar.gz -C /opt
tar -zxvf scala-2.11.12.tgz -C /opt
tar -zxvf spark-2.4.5-bin-hadoop2.6.tgz -C /opt
cd /opt
mv jdk1.8.0_221 jdk
mv sqoop-1.4.6-cdh5.14.2 sqoop
mv zeppelin-0.8.1-bin-all zeppelin
mv zookeeper-3.4.6 zookeeper
mv hive-1.1.0-cdh5.14.2 hive
mv hbase-1.2.0-cdh5.14.2 hbase
mv hadoop-2.6.0-cdh5.14.2 hadoop
mv apache-phoenix-4.14.0-cdh5.14.2-bin phoenix
#scala安装完成
mv scala-2.11.12 scala
mv spark-2.4.5-bin-hadoop2.6 spark

#解压查看历史文件压缩包
cd /opt/soft/installs
tar -xvf hadoop-native-64-2.6.0.tar -C /opt/hadoop/lib/native
tar -xvf hadoop-native-64-2.6.0.tar -C /opt/hadoop/lib

#安装Tomcat8
yum install -y unzip
unzip apache-tomcat-8.5.45.zip
mv apache-tomcat-8.5.45 /opt/tomcat
chmod +x /opt/tomcat/bin/*

#安装mysql
yum install -y perl
yum install -y net-tools
yum install -y autoconf
mar=`rpm -qa | grep mariadb`
rpm -e $mar --nodeps
rpm -ivh MySQL-client-5.6.46-1.el7.x86_64.rpm
rpm -ivh MySQL-server-5.6.46-1.el7.x86_64.rpm
m='[client]\n
default-character-set=utf8\n
skip-grant-tables\n
character_set_server=utf8\n
collation_server=utf8_general_ci'
echo -e $m >> /usr/my.cnf

#配置hadoop
cat /opt/soft/hadoop/core-site.xml > /opt/hadoop/etc/hadoop/core-site.xml
cat /opt/soft/hadoop/hdfs-site.xml > /opt/hadoop/etc/hadoop/hdfs-site.xml
cat /opt/soft/hadoop/hadoop-env.sh > /opt/hadoop/etc/hadoop/hadoop-env.sh
mv /opt/hadoop/etc/hadoop/mapred-site.xml.template /opt/hadoop/etc/hadoop/mapred-site.xml
cat /opt/soft/hadoop/mapred-site.xml > /opt/hadoop/etc/hadoop/mapred-site.xml
cat /opt/soft/hadoop/yarn-site.xml > /opt/hadoop/etc/hadoop/yarn-site.xml

#秘钥
#执行脚本时，手动回车生成秘钥
ssh-keygen
cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys
#需要手动输入密码
ssh-copy-id -i ~/.ssh/id_rsa.pub -p22 root@$1

#格式化hadoop
cd /opt/hadoop
rm -rf tem
hadoop namenode -format

#zookeeper配置
mv /opt/zookeeper/conf/zoo_sample.cfg /opt/zookeeper/conf/zoo.cfg
cat /opt/soft/zookeeper/zoo.cfg > /opt/zookeeper/conf/zoo.cfg
mkdir /opt/zookeeper/zkdata
mkdir /opt/zookeeper/zkdatalogs
cd /opt/zookeeper/zkdata
touch myid
echo 1 > myid

#hbase配置
cat /opt/soft/hbase/hbase-env.sh > /opt/hbase/conf/hbase-env.sh
cat /opt/soft/hbase/hbase-site.xml > /opt/hbase/conf/hbase-site.xml

#hive配置
cd /opt/hive
mkdir warehouse
cd /opt/hive/conf
mv /opt/hive/conf/hive-env.sh.template /opt/hive/conf/hive-env.sh
cat /opt/soft/hive/hive-env.sh > /opt/hive/conf/hive-env.sh
touch hive-site.xml
cat /opt/soft/hive/hive-site.xml > /opt/hive/conf/hive-site.xml
#上传mysql驱动包到hive/lib目录下
\cp /opt/soft/tar/mysql-connector-java-5.1.38.jar /opt/hive/lib/

#zeppelin配置
mv /opt/zeppelin/conf/zeppelin-site.xml.template /opt/zeppelin/conf/zeppelin-site.xml
mv /opt/zeppelin/conf/zeppelin-env.sh.template /opt/zeppelin/conf/zeppelin-env.sh
cat /opt/soft/zeppelin/zeppelin-site.xml > /opt/zeppelin/conf/zeppelin-site.xml
cat /opt/soft/zeppelin/zeppelin-env.sh > /opt/zeppelin/conf/zeppelin-env.sh
#配置hive解释器
\cp /opt/hive/conf/hive-site.xml /opt/zeppelin/conf/
\cp /opt/hadoop/share/hadoop/common/hadoop-common-2.6.0-cdh5.14.2.jar /opt/zeppelin/interpreter/jdbc
\cp /opt/hive/lib/hive-jdbc-1.1.0-cdh5.14.2-standalone.jar /opt/zeppelin/interpreter/jdbc

#phoenix配置
\cp /opt/phoenix/phoenix-4.14.0-cdh5.14.2-server.jar /opt/hbase/lib
cat /opt/soft/hbase/hbase-site.xml > /opt/hbase/conf/hbase-site.xml
\cp /opt/hbase/conf/hbase-site.xml > /opt/phoenix/bin

#sqoop配置
mv /opt/sqoop/conf/sqoop-env-template.sh /opt/sqoop/conf/sqoop-env.sh
cat /opt/soft/sqoop/sqoop-env.sh > /opt/sqoop/conf/sqoop-env.sh
\cp /opt/soft/jar/*.jar /opt/sqoop/lib/
\cp /opt/hive/lib/*.jar /opt/sqoop/lib/
#隐藏警告信息
cat /opt/soft/sqoop/configure-sqoop > /opt/sqoop/bin/configure-sqoop

#spark配置
mv /opt/spark/conf/spark-env.sh.template /opt/spark/conf/spark-env.sh
mv /opt/spark/conf/slaves.template /opt/spark/conf/slaves
cat /opt/soft/spark/spark-env.sh > /opt/spark/conf/spark-env.sh

#配置环境变量
pro='export JAVA_HOME=/opt/jdk\n
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar\n
export HADOOP_HOME=/opt/hadoop\n
export HADOOP_MAPRED_HOME=$HADOOP_HOME\n
export HADOOP_COMMON_HOME=$HADOOP_HOME\n
export HADOOP_HDFS_HOME=$HADOOP_HOME\n
export YARN_HOME=$HADOOP_HOME\n
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native\n
export HADOOP_INSTALL=$HADOOP_HOME\n
export HBASE_HOME=/opt/hbase\n
export HIVE_HOME=/opt/hive\n
export ZK_HOME=/opt/zookeeper\n
export SQOOP_HOME=/opt/sqoop\n
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin\n
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HBASE_HOME/bin:$HIVE_HOME/bin:$PATH\n
export PATH=$PATH:$ZK_HOME/bin\n
export PATH=$SQOOP_HOME/bin:$PATH\n
export SCALA_HOME=/opt/scala\n
export PATH=$PATH:$SCALA_HOME/bin\n
export SPARK_HOME=/opt/spark\n
export PATH=$PATH:$SPARK_HOME/bin'
echo -e $pro >> /etc/profile

#最后删除安装包
cd /opt
rm -rf soft

五、执行脚本

1、将下载的装机脚本文件做以上修改，并上传到虚拟机/opt目录下
2、创建脚本后给脚本赋执行权限
3、执行脚本时传入一个参数为主机名
4、执行过程中生成秘钥时需要手动回车生成秘钥
5、执行结束后source /etc/profile，或者执行脚本时source脚本