shell脚本安装jdk、mysql、hadoop、zookeeper、hbase、hive、spark...大一统

已于 2024-02-10 20:52:11 修改
阅读量6.3k 收藏 368
点赞数 44
文章标签： shell脚本
于 2020-10-18 11:06:07 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/and52696686/article/details/109141043
版权

shell脚本安装jdk、mysql、hadoop、zookeeper、hbase、hive、spark...大一统

一、准备工作

笔者安装上述软件的版本分别为：

  • jdk: jdk-8u221
  • mysql : mysql-5.6.46
  • hadoop: hadoop-2.6.0-cdh5.14.2
  • zookeeper: zookeeper-3.4.6
  • hbase: hbase-1.2.0-cdh5.14.2
  • hive: hive-1.1.0-cdh5.14.2
  • spark: spark-2.2.0-bin-hadoop2.7

链接：安装包及脚本
提取码：25dh

下载好安装包之后需要在虚拟机中创建3个文件夹

mkdir /opt/install
mkdir /opt/software
mkdir /opt/scripts
  • install是安装文件的根目录
  • 将安装包及mysql连接hive相关jar包都放进software文件夹
  • 将安装脚本及配置文件都放入 scripts 文件夹

二、安装脚本及相关配置文件

说明： 笔者使用的虚拟机主机名为： hadoop-001
虚拟机IP： 192.168.206.140

主机名 和 IP 每个人不一样，脚本及配置文件中这两个设置项必须要修改，其他的配置项可因需修改

注：如果使用window编辑的脚本传入虚拟机，会提示报错信息
-bash: ./abc.sh: /bin/bash^M: bad interpreter: No such file or directory

原因是windows下的文件是dos格式，即每一行结尾以\r\n来标识，而linux下的文件是unix格式，行尾则以\n来标识
解决方案：

  1. vim ，编辑文件，执行“: set ff=unix”，将文件设置为unix格式，然后执行“:wq”，保存退出
  2. 或者使用Notepad++ 编辑，如下图：
    在这里插入图片描述

2.1 安装 jdk

  • ①将安装包 jdk-8u221-linux-x64.tar.gz 放入虚拟机 /opt/software/ 目录下

  • ②将 jdk_180_221.sh 脚本放至虚拟机 /opt/scipts/ 目录下

  • ③ 给脚本赋权

chmod u +x /opt/scripts/jdk_180_221.sh
  • ④ 执行脚本
/opt/scripts/jdk_180_221.sh

安装详细步骤参考：Linux安装jdk

脚本内容如下：

jdk_180_221.sh 

#!/bin/bash
#jdk
tar -zxvf /opt/software/jdk-8u221-linux-x64.tar.gz -C /opt/install/
#配置环境变量
echo "export JAVA_HOME=/opt/install/jdk1.8.0_221" >> /etc/profile
echo 'export CLASSPATH=.:$JAVA_HOME/rt.jar:$JAVA_HOME/tools.jar:$JAVA_HOME/dt.jar' >> /etc/profile
echo 'export JRE_HOME=$JAVA_HOME/jre' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin' >> /etc/profile
source /etc/profile
java -version

2.2 安装 mysql

  • ①将安装包 MySQL-client-5.6.46-1.el7.x86_64.rpm、MySQL-server-5.6.46-1.el7.x86_64.rpm 放入虚拟机 /opt/software/ 目录下

  • ②将 mysql_5_6_46.sh 脚本放至虚拟机 /opt/scipts/ 目录下

  • ③ 给脚本赋权

chmod u +x /opt/scripts/mysql_5_6_46.sh
  • ④ 执行脚本
/opt/scripts/mysql_5_6_46.sh

安装详细步骤参考：Linux安装mysql–超简单的安装教程

脚本内容如下：

mysql_5_6_46.sh

#!/bin/bash
#mysql
#找到服务端冲突目录并卸载
a=`rpm -qa | grep -i Mariadb`	
rpm -e ${a} --nodeps
#下载依赖包
yum install -y net-tools
yum install -y perl
yum install -y autoconf
#解压安装
rpm -ivh /opt/software/MySQL-client-5.6.46-1.el7.x86_64.rpm
rpm -ivh /opt/software/MySQL-server-5.6.46-1.el7.x86_64.rpm
#配置文件修改
sed -i '3a [client]' /usr/my.cnf
sed -i '4a default-character-set = utf8' /usr/my.cnf
sed -i '6a skip-grant-tables' /usr/my.cnf
sed -i '7a character_set_server = utf8' /usr/my.cnf
sed -i '8a collation_server = utf8_general_ci' /usr/my.cnf
sed -i '9a lower_case_table_names' /usr/my.cnf
#重启数据库
service mysql restart
#更新root用户的密码
mysql -e "update mysql.user set password=password('ok')"
#设置临时密码不过期
mysql -e "update mysql.user set password_expired='N'"
#将第七行 skip-grant-tables跳过权限验证 注释掉
sed -i  '7s/skip-grant-tables/#skip-grant-tables/g' /usr/my.cnf
service mysql restart
#再一次设置root用户密码
mysql -uroot -pok -e "set password=password('ok')"
#创建新用户bigdata，密码为ok
mysql -uroot -pok -e "create user 'bigdata'@'%' IDENTIFIED BY 'ok'"
#给用户bigdata、root赋权
mysql -uroot -pok -e "grant all on *.* to 'bigdata'@'%'"
mysql -uroot -pok -e "grant all on *.* to 'root'@'%'"
#刷新
mysql -uroot -pok -e "flush privileges"
mysql --version

2.3 安装 hadoop

  • ①将安装包 hadoop-2.6.0-cdh5.14.2.tar.gz 放入虚拟机 /opt/software/ 目录下

  • ②将 hadoop_2_6_0.sh 脚本及配置文件 core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml 放至虚拟机 /opt/scipts/ 目录下

  • ③ 给脚本赋权

chmod u +x /opt/scripts/hadoop_2_6_0.sh
  • ④ 执行脚本
/opt/scripts/hadoop_2_6_0.sh

注： 这里说明一下执行 hadoop脚本 安装过程中需要输入一次回车，这是因为设置免密登录的时候默认要输入一次回车键决定的，以及第一次格式化hdfs的时候需要连续输入几次 yes 确认

安装详细步骤参考：搭建hadoop单机环境（伪分布式）

脚本内容如下：

hadoop_2_6_0.sh

#!/bin/bash
#hadoop
#免登录配置
#删除原先密钥文件
rm -rf /root/.ssh/id_rsa
#生成私钥
ssh-keygen -t rsa -P ""
#复制私钥拷贝到公钥
cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
#给公钥授权，可读可写
chmod 600 /root/.ssh/authorized_keys
#解压安装包
tar -zxvf /opt/software/hadoop-2.6.0-cdh5.14.2.tar.gz -C /opt/install/
#改名
mv /opt/install/hadoop-2.6.0-cdh5.14.2 /opt/install/hadoop-260
#添加环境变量
echo "export HADOOP_HOME=/opt/install/hadoop-260" >> /etc/profile
echo 'export HADOOP_MAPRED_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_COMMON_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_HDFS_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export YARN_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' >> /etc/profile
echo 'export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin' >> /etc/profile
source /etc/profile
#改变hadoop的Java路径
sed -i  '24,26s/\${JAVA_HOME}/\/opt\/install\/jdk1.8.0_221/g' /opt/install/hadoop-260/etc/hadoop/hadoop-env.sh
#注1：上述语句表示匹配24~26行之间的数据
#替换格式为：sed -i '24,26s/str1/str2/g' /path    ，将str1替换成str2
#注2：$、/是特殊字符需要加斜杠转义
#修改配置文件
cat $PWD/core-site.xml > /opt/install/hadoop-260/etc/hadoop/core-site.xml
cat $PWD/hdfs-site.xml > /opt/install/hadoop-260/etc/hadoop/hdfs-site.xml
cat $PWD/mapred-site.xml > /opt/install/hadoop-260/etc/hadoop/mapred-site.xml
cat $PWD/yarn-site.xml > /opt/install/hadoop-260/etc/hadoop/yarn-site.xml
tar -xvf /opt/software/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/lib/native
tar -xvf /opt/software/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/lib
#查看版本信息
echo 'hadoop 版本信息:' 
hadoop version
#格式化hdfs，首先删除用于存放临时文件的目录tmp，安全
rm -rf /opt/install/hadoop-260/tmp/
#格式化
hadoop namenode -format
#启动hdfs和yarn服务
start-all.sh
echo '*******查看jps状态*******'
jps

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<!-- 设置默认节点ip地址接口 -->
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.206.140:9000</value>
</property>

<!-- 设置用于存放hadoop临时文件的目录 -->
<property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/install/hadoop-260/tmp</value>
</property>

<!-- 设置允许其他主机root用户可以访问 -->
<property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>*</value>
</property>

<!-- 设置其他组下的root用户可以访问 -->
<property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
</property>

</configuration>
hdfs-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>

<!-- 设置免权限验证 -->
<property>
        <name>dfs.permissions.enabled</name>
        <value>false</value>
</property>

</configuration>
mapred-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>

<!-- 设置mapreduce的工作模式：yarn -->
<property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>

<!-- 设置mapreduce的工作地址 -->
<property>
        <name>mapreduce.jobhistory.address</name>
        <value>hadoop-001:10020</value>
</property>

<!-- 设置web页面历史服务端口的配置 -->
<property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>hadoop-001:19888</value>
</property>

</configuration>
yarn-site.xml 
<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->
<configuration>

<!-- Site specific YARN configuration properties -->


<!-- reducer获取数据方式 -->
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
    <name>yarn.resourcemanager.hostname</name>
    <value>hadoop-001</value>
</property>

</configuration>

2.4 安装 zookeeper

  • ①将安装包 zookeeper-3.4.6.tar.gz 放入虚拟机 /opt/software/ 目录下

  • ②将 zookeeper_3_4_6.sh 脚本及配置文件 zoo.cfg 放至虚拟机 /opt/scipts/ 目录下

  • ③ 给脚本赋权

chmod u +x /opt/scripts/zookeeper_3_4_6.sh
  • ④ 执行脚本
/opt/scripts/zookeeper_3_4_6.sh

安装详细步骤参考：Zookeeper环境搭建

脚本内容如下：
zookeeper_3_4_6.sh 

#!/bin/bash
#zookeeper
tar -zxvf /opt/software/zookeeper-3.4.6.tar.gz -C /opt/install/
echo 'export ZK_HOME=/opt/install/zookeeper-3.4.6' >> /etc/profile
echo 'export PATH=$PATH:$ZK_HOME/bin' >> /etc/profile
cat $PWD/zoo.cfg > /opt/install/zookeeper-3.4.6/conf/zoo.cfg
mkdir /opt/install/zookeeper-3.4.6/zkData
mkdir /opt/install/zookeeper-3.4.6/zkLog
touch /opt/install/zookeeper-3.4.6/zkData/myid
echo '1' > /opt/install/zookeeper-3.4.6/zkData/myid
source /etc/profile
zkServer.sh start
zkServer.sh status
zoo.cfg 
# The number of milliseconds of each tick
tickTime=2000
#最大访问数:不限制
maxClientCnxns=0
# The number of ticks that the initial
# synchronization phase can take
#初始化最小进程数:50
initLimit=50
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
#数据目录
dataDir=/opt/install/zookeeper-3.4.6/zkData
#日志目录
dataLogDir=/opt/install/zookeeper-3.4.6/zkLog
# the port at which the clients will connect
clientPort=2181
#配置三台以上的奇数台可用机器主机名或者ip,注如果不配集群不需要添加以下内容
server.1=hadoop-001:2888:3888

2.5 安装 hbase

  • ①将安装包 hbase-1.2.0-cdh5.14.2.tar.gz 放入虚拟机 /opt/software/ 目录下

  • ②将 hbase_1_2_0.sh 脚本及配置文件 hbase-site.xml 放至虚拟机 /opt/scipts/ 目录下

  • ③ 给脚本赋权

chmod u +x /opt/scripts/hbase_1_2_0.sh
  • ④ 执行脚本
/opt/scripts/hbase_1_2_0.sh

安装详细步骤参考：Hbase环境搭建

脚本内容如下：

hbase_1_2_0.sh 

#!/bin/bash
#hbase
tar -zxvf /opt/software/hbase-1.2.0-cdh5.14.2.tar.gz -C /opt/install/
mv /opt/install/hbase-1.2.0-cdh5.14.2 /opt/install/hbase-120
echo 'export HBASE_HOME=/opt/install/hbase-120' >> /etc/profile
echo 'export PATH=$PATH:$HBASE_HOME/bin' >> /etc/profile
source /etc/profile
cat $PWD/hbase-site.xml > /opt/install/hbase-120/conf/hbase-site.xml
echo 'export JAVA_HOME=/opt/install/jdk1.8.0_221' >> /opt/install/hbase-120/conf/hbase-env.sh
echo 'export HADOOP_HOME=/opt/install/hadoop-260' >> /opt/install/hbase-120/conf/hbase-env.sh
echo 'export HBASE_HOME=/opt/install/hbase-120' >> /opt/install/hbase-120/conf/hbase-env.sh
#使用外部zookeeper
echo 'export HBASE_MANAGES_ZK=false' >> /opt/install/hbase-120/conf/hbase-env.sh
source /etc/profile
#启动habse
start-hbase.sh
jps | grep -i hmaster
jps | grep -i hre
hbase-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
-->
<configuration>

<!-- hbase文件存储目录 -->
<property>
         <name>hbase.rootdir</name>
         <value>hdfs://192.168.206.140:9000/hbase</value>
</property>

<!-- 配置hbase为分布式值改为true -->
<property>
         <name>hbase.cluster.distributed</name>
         <value>true</value>
</property>

<property>
         <name>hbase.zookeeper.property.dataDir</name>
         <value>/opt/install/hbase-120/data</value>
</property>


</configuration>

2.6 安装hive

  • ①将安装包 hive-1.1.0-cdh5.14.2.tar.gz 放入虚拟机 /opt/software/ 目录下

  • ②将 hive_1_1_0.sh、hive-env.sh 脚本及配置文件 hive-site.xml 放至虚拟机 /opt/scipts/ 目录下

  • ③ 给脚本赋权

chmod u +x /opt/scripts/hive_1_1_0.sh
  • ④ 执行脚本
/opt/scripts/hive_1_1_0.sh

安装详细步骤参考：Hive环境搭建

脚本内容如下：

hive_1_1_0.sh

#!/bin/bash
#hive
tar -zxvf /opt/software/hive-1.1.0-cdh5.14.2.tar.gz -C /opt/install/
mv /opt/install/hive-1.1.0-cdh5.14.2 /opt/install/hive-110
echo 'export HIVE_PATH=/opt/install/hive-110' >> /etc/profile
echo 'export PATH=$PATH:$HIVE_PATH/bin' >> /etc/profile
source /etc/profile
cat $PWD/hive-site.xml > /opt/install/hive-110/conf/hive-site.xml
cat $PWD/hive-env.sh > /opt/install/hive-110/conf/hive-env.sh
mv /opt/install/hive-110/conf/hive-log4j.properties.template /opt/install/hive-110/conf/hive-log4j.properties
sed -i  '20,22s/\${java.io.tmpdir}\/\${user.name}/\/opt\/install\/hive-110\/logs/g' /opt/install/hive-110/conf/hive-log4j.properties
mkdir /opt/install/hive-110/logs
source /etc/profile
echo 'hive版本信息:'
hive --version
#移动jar包
cp /opt/software/mysql-connector-java-5.1.48-bin.jar /opt/install/hive-110/lib/
chmod  777 /opt/install/hive-110
#新建hdfs存储目录
hdfs dfs -mkdir -p /hive/warehouse
#赋权，然后在可以在hive中创建表
hdfs dfs -chmod 777 /hive
hdfs dfs -chmod 777 /hive/warehouse
#初始化数据库
schematool -initSchema -dbType mysql
#后台启动hive原服务
nohup hive --service metastore &
#测试，观察表是否能创建成功
hive -e "create database if not exists test"

hive-env.sh 
export HADOOP_HOME=/opt/install/hadoop-260
export HIVE_HOME=/opt/install/hive-110
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HIVE_AUX_JARS_PATH=/opt/install/hive-110/lib
export JAVA_HOME=/opt/install/jdk1.8.0_221
export HIVE_CONF_DIR=/opt/install/hive-110/conf
hive-site.xml 
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
        <property>
                <name>hive.metastore.warehouse.dir</name>
                <value>hdfs://hadoop-001:9000/hive/warehouse</value>
               <description>管理表存储的位置,可以是linux中的目录,也可以是相对于fs.default.name有关的目录</description>
        </property>
        <property>
                <name>hive.metastore.local</name>
                <value>true</value>
        </property>
        <!-- 指定hive元数据存储的MySQL地址 -->
        <property>
                <name>javax.jdo.option.ConnectionURL</name>
                <value>jdbc:mysql://127.0.0.1:3306/hive?createDatabaseIfNotExist=true</value>
        </property>
        <!-- 元数据存储数据库的驱动 -->
        <property>
                <name>javax.jdo.option.ConnectionDriverName</name>
                <value>com.mysql.jdbc.Driver</value>
        </property>
        <!-- 元数据存储数据库的用户名 -->
        <property>
                <name>javax.jdo.option.ConnectionUserName</name>
                <value>root</value>
        </property>
        <!-- 元数据存储数据库的密码,(注:这里是mysql自己root用户的密码) -->
        <property>
                <name>javax.jdo.option.ConnectionPassword</name>
                <value>ok</value>
        </property>
		<!-- 远程连接元数据库 -->
		<property>
				<name>hive.metastore.uris</name>
				<value>thrift://192.168.206.140:9083</value>
		</property>

</configuration>

2.7 安装 spark

  • ①将安装包 spark-2.2.0-bin-hadoop2.7.tgz 放入虚拟机 /opt/software/ 目录下

  • ②将 spark_2_2_0.sh 脚本及配置文件 spark-env.sh 放至虚拟机 /opt/scipts/ 目录下

  • ③ 给脚本赋权

chmod u +x /opt/scripts/spark_2_2_0.sh
  • ④ 执行脚本
/opt/scripts/spark_2_2_0.sh

脚本内容如下：

spark_2_2_0.sh 

#!/bin/bash
#spark
tar -zxvf /opt/software/spark-2.2.0-bin-hadoop2.7.tgz -C /opt/install/
mv /opt/install/spark-2.2.0-bin-hadoop2.7/ /opt/install/spark-220
echo "export SPARK_HOME=/opt/install/spark-220" >> /etc/profile
echo 'export PATH=$PATH:$SPARK_HOME/bin' >> /etc/profile
source /etc/profile
mv /opt/install/spark-220/conf/slaves.template /opt/install/spark-220/conf/slaves
cat $PWD/spark-env.sh > /opt/install/spark-220/conf/spark-env.sh
#集成spark on hive
cp /opt/software/mysql-connector-java-5.1.48-bin.jar /opt/install/spark-220/jars/
cp /opt/install/hive-110/conf/hive-site.xml /opt/install/spark-220/conf/
cp /opt/install/hadoop-260/etc/hadoop/core-site.xml /opt/install/spark-220/conf/
cp /opt/install/hadoop-260/etc/hadoop/hdfs-site.xml /opt/install/spark-220/conf/
/opt/install/spark-220/sbin/start-all.sh
spark-sql -e "show databases"
jps
spark-env.sh 
#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# This file is sourced when running various Spark programs.
# Copy it as spark-env.sh and edit that to configure Spark for your site.

# Options read when launching programs locally with
# ./bin/run-example or ./bin/spark-submit
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program

# Options read by executors and drivers running inside the cluster
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
# - MESOS_NATIVE_JAVA_LIBRARY, to point to your libmesos.so if you use Mesos

# Options read in YARN client mode
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_EXECUTOR_CORES, Number of cores for the executors (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Executor (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Driver (e.g. 1000M, 2G) (Default: 1G)

# Options for the daemons used in the standalone deploy mode
# - SPARK_MASTER_HOST, to bind the master to a different IP address or hostname
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master
# - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y")
# - SPARK_WORKER_CORES, to set the number of cores to use on this machine
# - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g)
# - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker
# - SPARK_WORKER_DIR, to set the working directory of worker processes
# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")
# - SPARK_DAEMON_MEMORY, to allocate to the master, worker and history server themselves (default: 1g).
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
# - SPARK_SHUFFLE_OPTS, to set config properties only for the external shuffle service (e.g. "-Dx=y")
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers

# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR      Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR       Where log files are stored.  (Default: ${SPARK_HOME}/logs)
# - SPARK_PID_DIR       Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING  A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS      The scheduling priority for daemons. (Default: 0)
# - SPARK_NO_DAEMONIZE  Run the proposed command in the foreground. It will not output a PID file.

export HADOOP_HOME=/opt/install/hadoop-260
export HADOOP_CONF_DIR=/opt/install/hadoop-260/etc/hadoop/
export JAVA_HOME=/opt/install/jdk1.8.0_221
#指定master的主机
export SPARK_MASTER_HOST=hadoop-001
#指定master的端口
export SPARK_MASTER_PORT=7077

三、大一统安装jdk、mysql、hadoop、zookeeper、hbase、hive、spark…

  • ①将下载链接中 software 整体打包放入 linux: /opt 文件夹中
    在这里插入图片描述

  • ②将下载链接中 scripts 整体打包放入 linux: /opt 文件夹中
    在这里插入图片描述

  • ③ 给脚本赋权

chmod u +x /opt/scripts/hadoop-all.sh
  • ④ 执行脚本
/opt/scripts/hadoop-all.sh

注： 在上文已备注过， 在单独安装 hadoop时，执行hadoop脚本 过程中需要输入一次回车以及第一次格式化hdfs的时候需要连续输入几次 yes 确认，这里安装统一脚本也需要进行同样的操作

脚本内容如下：
hadoop-all.sh 

#!/bin/bash
#jdk mysql hadoop zookeeper hive hbase spark

#jdk
tar -zxvf /opt/software/jdk-8u221-linux-x64.tar.gz -C /opt/install/
echo "export JAVA_HOME=/opt/install/jdk1.8.0_221" >> /etc/profile
echo 'export CLASSPATH=.:$JAVA_HOME/rt.jar:$JAVA_HOME/tools.jar:$JAVA_HOME/dt.jar' >> /etc/profile
echo 'export JRE_HOME=$JAVA_HOME/jre' >> /etc/profile
echo 'export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin' >> /etc/profile
source /etc/profile
java -version

#mysql
#找到服务端冲突目录并卸载
a=`rpm -qa | grep -i Mariadb`	
rpm -e ${a} --nodeps
#下载依赖包
yum install -y net-tools
yum install -y perl
yum install -y autoconf
#解压安装
rpm -ivh /opt/software/MySQL-client-5.6.46-1.el7.x86_64.rpm
rpm -ivh /opt/software/MySQL-server-5.6.46-1.el7.x86_64.rpm
#配置文件修改
sed -i '3a [client]' /usr/my.cnf
sed -i '4a default-character-set = utf8' /usr/my.cnf
sed -i '6a skip-grant-tables' /usr/my.cnf
sed -i '7a character_set_server = utf8' /usr/my.cnf
sed -i '8a collation_server = utf8_general_ci' /usr/my.cnf
sed -i '9a lower_case_table_names' /usr/my.cnf
#重启数据库
service mysql restart
mysql -e "update mysql.user set password=password('ok')"
mysql -e "update mysql.user set password_expired='N'"
sed -i  '7s/skip-grant-tables/#skip-grant-tables/g' /usr/my.cnf
service mysql restart
mysql -uroot -pok -e "set password=password('ok')"
mysql -uroot -pok -e "create user 'bigdata'@'%' IDENTIFIED BY 'ok'"
mysql -uroot -pok -e "grant all on *.* to 'bigdata'@'%'"
mysql -uroot -pok -e "grant all on *.* to 'root'@'%'"
mysql -uroot -pok -e "flush privileges"
mysql --version

#hadoop
##免登录配置
#删除原先密钥文件
rm -rf /root/.ssh/id_rsa
#生成私钥
ssh-keygen -t rsa -P ""
#复制私钥拷贝到公钥
cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
#给公钥授权，可读可写
chmod 600 /root/.ssh/authorized_keys
#解压安装包
tar -zxvf /opt/software/hadoop-2.6.0-cdh5.14.2.tar.gz -C /opt/install/
#改名
mv /opt/install/hadoop-2.6.0-cdh5.14.2 /opt/install/hadoop-260
#添加环境变量
echo "export HADOOP_HOME=/opt/install/hadoop-260" >> /etc/profile
echo 'export HADOOP_MAPRED_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_COMMON_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_HDFS_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export YARN_HOME=$HADOOP_HOME' >> /etc/profile
echo 'export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native' >> /etc/profile
echo 'export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"' >> /etc/profile
echo 'export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin' >> /etc/profile
source /etc/profile
#改变hadoop的Java路径
sed -i  '24,26s/\${JAVA_HOME}/\/opt\/install\/jdk1.8.0_221/g' /opt/install/hadoop-260/etc/hadoop/hadoop-env.sh
#注1： 24,26s/str1/str2/g 表示匹配24~26行之间的数据，将str1替换成str2
#注2： \/斜杠表示转义反斜杠，反斜杠/是匹配文件路径用的
#修改配置文件
cat $PWD/core-site.xml > /opt/install/hadoop-260/etc/hadoop/core-site.xml
cat $PWD/hdfs-site.xml > /opt/install/hadoop-260/etc/hadoop/hdfs-site.xml
cat $PWD/mapred-site.xml > /opt/install/hadoop-260/etc/hadoop/mapred-site.xml
cat $PWD/yarn-site.xml > /opt/install/hadoop-260/etc/hadoop/yarn-site.xml
tar -xvf /opt/software/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/lib/native
tar -xvf /opt/software/hadoop-native-64-2.6.0.tar -C $HADOOP_HOME/lib
#查看版本信息
echo 'hadoop 版本信息:' 
hadoop version
#格式化hdfs，首先删除用于存放临时文件的目录tmp，安全
rm -rf /opt/install/hadoop-260/tmp/
#格式化
hadoop namenode -format
#启动hdfs和yarn服务
start-all.sh
echo '*******查看jps状态*******'
jps

#zookeeper
tar -zxvf /opt/software/zookeeper-3.4.6.tar.gz -C /opt/install/
echo 'export ZK_HOME=/opt/install/zookeeper-3.4.6' >> /etc/profile
echo 'export PATH=$PATH:$ZK_HOME/bin' >> /etc/profile
cat $PWD/zoo.cfg > /opt/install/zookeeper-3.4.6/conf/zoo.cfg
mkdir /opt/install/zookeeper-3.4.6/zkData
mkdir /opt/install/zookeeper-3.4.6/zkLog
touch /opt/install/zookeeper-3.4.6/zkData/myid
echo '1' > /opt/install/zookeeper-3.4.6/zkData/myid
source /etc/profile
zkServer.sh start
zkServer.sh status

#hbase
tar -zxvf /opt/software/hbase-1.2.0-cdh5.14.2.tar.gz -C /opt/install/
mv /opt/install/hbase-1.2.0-cdh5.14.2 /opt/install/hbase-120
echo 'export HBASE_HOME=/opt/install/hbase-120' >> /etc/profile
echo 'export PATH=$PATH:$HBASE_HOME/bin' >> /etc/profile
source /etc/profile
cat $PWD/hbase-site.xml > /opt/install/hbase-120/conf/hbase-site.xml
echo 'export JAVA_HOME=/opt/install/jdk1.8.0_221' >> /opt/install/hbase-120/conf/hbase-env.sh
echo 'export HADOOP_HOME=/opt/install/hadoop-260' >> /opt/install/hbase-120/conf/hbase-env.sh
echo 'export HBASE_HOME=/opt/install/hbase-120' >> /opt/install/hbase-120/conf/hbase-env.sh
#使用外部zookeeper
echo 'export HBASE_MANAGES_ZK=false' >> /opt/install/hbase-120/conf/hbase-env.sh
source /etc/profile
#启动habse
start-hbase.sh
jps | grep -i hmaster
jps | grep -i hre

#hive
tar -zxvf /opt/software/hive-1.1.0-cdh5.14.2.tar.gz -C /opt/install/
mv /opt/install/hive-1.1.0-cdh5.14.2 /opt/install/hive-110
echo 'export HIVE_PATH=/opt/install/hive-110' >> /etc/profile
echo 'export PATH=$PATH:$HIVE_PATH/bin' >> /etc/profile
source /etc/profile
cat $PWD/hive-site.xml > /opt/install/hive-110/conf/hive-site.xml
cat $PWD/hive-env.sh > /opt/install/hive-110/conf/hive-env.sh
mv /opt/install/hive-110/conf/hive-log4j.properties.template /opt/install/hive-110/conf/hive-log4j.properties
sed -i  '20,22s/\${java.io.tmpdir}\/\${user.name}/\/opt\/install\/hive-110\/logs/g' /opt/install/hive-110/conf/hive-log4j.properties
mkdir /opt/install/hive-110/logs
source /etc/profile
echo 'hive版本信息:'
hive --version
#移动jar包
cp /opt/software/mysql-connector-java-5.1.48-bin.jar /opt/install/hive-110/lib/
chmod  777 /opt/install/hive-110
#新建hdfs存储目录
hdfs dfs -mkdir -p /hive/warehouse
#赋权，然后在可以在hive中创建表
hdfs dfs -chmod 777 /hive
hdfs dfs -chmod 777 /hive/warehouse
#初始化数据库
schematool -initSchema -dbType mysql
#后台启动hive原服务
nohup hive --service metastore &
#测试，观察表是否能创建成功
hive -e "create database if not exists test"

#spark
tar -zxvf /opt/software/spark-2.2.0-bin-hadoop2.7.tgz -C /opt/install/
mv /opt/install/spark-2.2.0-bin-hadoop2.7/ /opt/install/spark-220
echo "export SPARK_HOME=/opt/install/spark-220" >> /etc/profile
echo 'export PATH=$PATH:$SPARK_HOME/bin' >> /etc/profile
source /etc/profile
mv /opt/install/spark-220/conf/slaves.template /opt/install/spark-220/conf/slaves
cat $PWD/spark-env.sh > /opt/install/spark-220/conf/spark-env.sh
#集成spark on hive
cp /opt/software/mysql-connector-java-5.1.48-bin.jar /opt/install/spark-220/jars/
cp /opt/install/hive-110/conf/hive-site.xml /opt/install/spark-220/conf/
cp /opt/install/hadoop-260/etc/hadoop/core-site.xml /opt/install/spark-220/conf/
cp /opt/install/hadoop-260/etc/hadoop/hdfs-site.xml /opt/install/spark-220/conf/
/opt/install/spark-220/sbin/start-all.sh
spark-sql -e "show databases"
jps
我玩的很开心
关注
  • 44
    点赞
  • 踩
  • 368
    收藏
    觉得还不错? 一键收藏
  • 10
    评论
Linux自动安装jdk（shell脚本）
07-24
安装jdk1.8，直接使用这个脚本，需要vim 安装jdk.sh，然后按Esc进入一般模式，使用 :set ff=unix 然后按Enter，然后按Esc，输入:wq 然后使用./安装jdk.sh,然后使用source /etc/profile
shell脚本安装jdk
maxchenBug的博客
02-19 541
shell脚本安装jdk #!/bin/bash BASE_SERVER=192.168.1.11 yum install -y wget wget $BASE_SERVER/soft-all/jdk-7u45-linux-x64.tar.gz tar -zxvf jdk-7u45-linux-x64.tar.gz -C /usr/local ...
Shell脚本之——自动安装JDK
雷神乐乐的博客
02-10 2195
Shell脚本之——自动安装JDK
CentOS7JDK的shell安装脚本
weixin_55076626的博客
03-01 1885
#安装JDK setupJDK(){ if [ ! -d '/opt/soft' ];then mkdir -p /opt/soft fi if [ ! -d '/opt/soft/jdk180' ];then mkdir -p /opt/soft/jdk180 tar-zxfjdk-8u111-linux-x64.tar.gz-C/opt/s...
Shell脚本实现在Linux系统中自动安装JDK
09-15
主要介绍了Shell脚本实现在Linux系统中自动安装JDK,本文直接给出实现代码,需要的朋友可以参考下
Shell脚本 一键安装JDK
futeng
05-09 386
背景 最近面对十几台“嗷嗷待配”JDK环境的机器，实在不想一个个配。生命如此美好，怎能浪费在无聊的重复劳动上，遂编写了个一键安装JDK的Shell脚本。脚本很简单，而且对于简单的环境配置也很实用。完整的代码贴在下面，希望也能帮助大家减轻工作量。 单步安装，移步这里 需求 能自动检测可能默认安装的openJDK，并能删除之。 能自动配置JDK在 /etc/profile的环境...
使用shell安装jdk
zxczb的博客
05-12 516
注意事项 编辑时需替换为 unix(LF) 换行格式, 否则linux执行时会报 $\r 错误( \ 在linux显示可能为加了横杠的W) 编码格式根据linux系统适配就好,一般utf-8即可 仅需配置 jdk_install_path 确认 ls -l|awk '{print$9} 获取的是文件名 remove_openjdk可以注释不执行(openjdk删不删无所谓,配置好环境变量或用户环境变量就好) 该shell和jdk安装包放同一个目录,chmod +x 脚本,然后执行即可 source /et
shell自动安装JDK
weixin_72036027的博客
03-27 292
shell自动安装JDK
Linux 脚本安装jdk
u013071014的博客
01-11 619
创建文件jdk.sh #!/bin/bash jdkTargz="/opt/jdk-8u144-linux-x64.tar.gz" # 检查原先是否已配置java环境变量 checkExist(){ jdk1=$(grep -n "export JAVA_HOME=.*" /etc/profile | cut -f1 -d':') if [ -n "$jdk1" ];then echo "JAVA_HOME已配置，删除内容" sed -i "${jdk1}d"
ubuntu 自动安装配置jdk的shell脚本
01-30
ubuntu deepin系统下自动安装配置jdk的shell脚本，省去繁琐重复配置
使用shell脚本在Linux虚拟机上安装jdk环境
最新发布
zzy66666c的博客
03-10 535
下载jdk-8u171-linux-x64.tar.gz。
CentOS 6.5 用shell脚本安装jdk
M_SIGNALs的博客
10-15 2033
使用shell 脚本自动化安装 jdk #! /bin/bash # shell script tp install jdk(version:jdk1.8.0_91) # version 1.0 # created by signalfeng@gmail.com # Sat Oct 15 11:46:08 CST 2016# Step1
编写shell脚本安装jdk
jingzhou111的博客
02-28 156
【代码】编写shell脚本安装jdk。
Shell脚本一键安装JDK
生活不只是眼前的不安还有诗和远方
06-21 595
Shell脚本一键安装JDK
shell脚本一键安装JDK1.7
三劫散仙
09-03 307
搞java的朋友们，一般都会避免不了在linux安装jdk，手动安装几台即可，如果机器数，那么我们就可以利用shell来自动化安装，下面是散仙写的一个比较简单的shell安装脚本，安装需要的条件： 1，下载好的jdk安装包 2，注意脚本里面的解压出来的文件名，有必要可以修改一下 3，默认的jdk安装路径，散仙这里是演示方便，就放在/root/java里面了，一般会放在/usr/local...
Shell脚本的安装、基本命令操作
DeathAlthas的博客
09-27 2366
SSH Secure Shell Client是一款不错的远程linux的工具。本文介绍它的用法和基本命令。 具体安装步骤 1、解压之后，双击exe文件，如下图所示， 点击“Next” 2、同意安装协议，点击“Yes”按钮 3、选择安装路径，然后点击下一步 4、填写目录名称，然后点击“Next” 5、选择是否创建桌面图标、文档及其他，然后点击“Next” 6、最后点“Finish”即可 使用方法 打开界面 ![在这里插入图片描述](https://img-blog.csd
shell脚本入门和nginx安装
thillsoft的博客
06-28 615
Nginx是一款轻量级的Web服务器/反向代理服务器及电子邮件（IMAP/POP3）代理服务器，在BSD-like 协议下发行。其特点是占有内存少，并发能力强, 高达 50,000 个并发连接数的响应 ，事实上nginx的并发能力在同类型的网页服务器中表现较好。Nginx是一款自由的、开源的、高性能的HTTP服务器和反向代理服务器。
shell 脚本自动安装jdk-6u34-linux-x64-rpm.bin 跳过回车自动安装
weixin_34306593的博客
02-04 119
上代码：#!/bin/bash #Engineserverinstall #Writebyguojr #SetEngineserverdir jdk=jdk-6u34-linux-x64-rpm.bin chmodo+x$jdk&&echo-e"\n"|./$jdk; ( cat<<EOF #set...
评论 10
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值