Shell脚本一次性安装JDK&MySql&Hadoop&Hive&Zeepelin&Zookeeper&Hbase&Sqoop

最新推荐文章于 2024-02-03 16:16:46 发布
最新推荐文章于 2024-02-03 16:16:46 发布
阅读量2.8k 收藏 15
点赞数 9
分类专栏： Linux 文章标签： 数据库 shell
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.csdn.net/wsjslient/article/details/110525380
版权

目录

前言

        在使用计算机的时候，我们的程序越来越多，服务也越来越多，所以我们在拿到一台新机器的时候总是习惯性的去安装那些我们常用的程序&服务；使用虚拟机时也是一样的，所以为了方便以后的操作，我们可以尝试写shell脚本来一次性完成我们需要的所有安装&配置操作。

  •         MySql服务用wget获取MySql安装包安装的，这一部分时间较长，所以我单独列出来安装MySql的脚本，其他的服务通过本地安装包解压之后配置安装的所以统一放在一起（如果还是想一次性执行脚本安装，不想分两次执行脚本安装，可以考虑在大的脚本里的最前面先执行MySQL的脚本，这样就不用执行两次脚本了）；
  •         由于hive初始化时后台操作进入MySql并创建了数据库，所以我推荐先安装MySql之后再运行配置剩余全部的shell脚本；
  •         如果在MySql服务未安装时先执行整体配置的脚本则会报hive初始化失败的报错；
  •         如果安装一半提示Hive初始化失败，可能原因是Hive初始化需要后台进入MySQL中创建的数据库已经存在了，所以在安装前务必确认hive配置文档中的数据库名在MySQL中不存在才可进行安装！
       代码提取： 百度云链接
       提  取  码：gg4x

一、安装前的准备

  • 一台网络通畅的虚拟机
  • 所有服务的安装包(上传至虚拟机)

        这里列出本文安装的所有服务的安装包信息

  • jdk-8u111-linux-x64.tar.gz
  • mysql-community-release-el7-5.noarch.rpm
  • hadoop-2.6.0-cdh5.14.2.tar.gz
  • mysql-connector-java-5.1.32.jar
  • hadoop-native-64-2.6.0.tar
  • hive-1.1.0-cdh5.14.2.tar.gz
  • hbase-1.2.0-cdh5.14.2.tar.gz
  • zeppelin-0.8.2-bin-all.tgz
  • zookeeper-3.4.5-cdh5.14.2.tar.gz
  • sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
  • scala-2.12.10.tgz

二、shell安装MySQL

        这一部分在安装前为检测是否有上一次安装过的MySQL服务的痕迹，后面将补上这一部分内容（已经加入自动删除旧版本的校验并增加修改root用户密码的功能）；

  • 请在/root目录下运行安装mysql的脚本
  • UNINSTALL MARIADB =>卸载mariadb
  • UNINSTALL PAST MYSQL =>卸载旧版本MySQL
  • yum update =>更新yum
  • SEARCH WGET AND ISTALL WGET =>安装wget
  • WGET MYSQL PACKAGE =>wget获取MySQL安装包
  • RMP INSTALL MySQL =>安装MySQL
  • INSTALL MYSQL-SERVER I=>安装MySQL服务
  • MODIFY PASSWORD =>修改root用户密码
  • CHINESE GARBLED PROBLEM =>修改MySQL字符集，解决中文乱码问题
#!/bin/bash
#------------------UNINSTALL MARIADB------------------------
echo "First,we need to unistall mariadb"
var=`rpm -qa | grep mariadb`
if [[ $var =~ ^mariadb ]]
then
        echo -e Find "\e[1;31m $var \e[0m"
        `rpm -e --nodeps $var`
        rst=`rpm -qa | grep mariadb`
        if [[ $rst =~ "" ]]
        then
                echo "Mariadb uninstall successfully"
        fi
else
        echo "Not find mariadb,just direct install mysql"
fi

#-------------------------UNINSTALL PAST MYSQL---------------------------
echo
echo "Second,we need to check that you pc has past mysql and uninstall"
echo "Start to uninstall mysql server"
VAR=`rpm -qa|grep -i mysql`
VAR=($VAR)
for server in ${VAR[@]}
do
    echo -e uninstall: "\e[1;31m $VAR \e[0m"
    `rpm -e --nodeps $server`
done
VAR=`rpm -qa|grep -i mysql`
if [ ${#VAR} -eq 0 ]
then
    echo "All mysql server were uninstall!"
else
    echo "Find remains mysql server,please reuninstall remain mysql server"
    exit 1
fi
echo
echo "Next,we need to remove remain mysql properties and denpencies"
VAR=`rm -rf /var/lib/mysql`
echo "Mysql propertied and denpencies were all droped"
echo
echo -e Last,we had to remove: "\e[1;31m my.cnf \e[0m"
VAR=`rm -rf /etc/my.cnf`
echo -e File: "\e[1;31m $VAR \e[0m"  has removed!
echo "Your pc with all about mysql has remove, we start to install mysql"

#-------------------------YUM UPDATE---------------------------------------
echo
echo "start to update yum ..."
`yum -y update>/root/test.log 2>&1`
VAR=`awk -F '!'  'END{print $1}' /root/test.log`
if [[ $VAR =~ ^Complete ]]
then
    echo "update yum success!"
elif [[ $VAR =~ ^No ]]
then
    echo "yum has been the newest!"
else
    echo "update yum failed,please reinstall..."
        exit 1
fi

#------------------SEARCH WGET AND INSTALL WGET--------------------
echo
VAR=`yum search wget|grep wget.|awk -F' ' '{print $1}'`
VAR=${VAR##*=}
echo -e yum search wget find: "\e[1;31m $VAR \e[0m"
echo -e start to install: "\e[1;31m $VAR \e[0m"
VAR="yum -y install $VAR>/root/test.log 2>&1"
`eval $VAR`
#验证wget是否安装成功
VAR=`awk -F '!'  'END{print $1}' /root/test.log`
VAR2=`cat /root/test.log |grep wget.|awk -F' ' '{print $0}'`
VAR2=`echo $var|grep "already installed"`
if [[ $VAR =~ ^Complete ]]
then
    echo "install wget success"
elif [[ $VAR2 != NULL ]]
then
    echo "wget already install"
else
    echo "install wget failed,please reinstall..."
        exit 1
fi

#------------------WGET MYSQL PACKAGE-----------------------
echo
echo "Checking past version of mysql package"
VAR=`ls|grep mysql.*`
VAR=($VAR)
for package in ${VAR[@]}
do
    `rm -rf $package`
done
echo "Old version of package, they all been removed"
echo
echo "Start to download mysql package"
VAR=http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
`eval "wget $VAR>/root/test.log 2>&1"`
#验证wget获取MySQL package结果
ls>test.log
VAR=`cat test.log`
VAR=($VAR)
COUNT=0
for var in ${VAR[@]}
do
    if [[ $var =~ ^mysql ]]
    then
        echo "Download package $var success!"
        COUNT=0
        break
    else
        ((COUNT++))
    fi
done

if [ $COUNT -ne 0 ]
then
    echo "Download package failed .... please redownload package!"
    exit 1
fi
echo

#----------------RMP INSTALL-----------------------
echo "Start to install mysql"
VAR=`ls ./|grep mysql*`
OPER=`eval "rpm -ivh $VAR>/root/test.log 2>&1"`
#验证MySQL安装是否成功
VAR=`awk -F' ' 'END{print $1}' /root/test.log`
if [[ $VAR =~ ^mysql ]]
then
    echo "Install mysql successfully"
else
    echo "Install mysql failed,please reinstall..."
    echo "Install log in /root/test.log,please check it and find reason"
        exit 1
fi
echo

#---------------MYSQL-SERVER INSTALL-----------------
echo "start to install mysql server..."
VAR="yum -y install mysql-server>/root/test.log 2>&1"
`eval $VAR`
#验证MySQL服务是否安装成功
VAR=`rpm -qa|grep -i mysql`
VAR=($VAR)
if [[ ${#VAR[*]} -ge 4 ]]
then
    echo "mysql server install success"
else
    echo "mysql server install fail"
    echo "Install log in /root/test.log,please check it and find reason"
        exit 1
fi

#------------------MODIFY PASSWORD------------------------------
echo
echo "Start to modify password"
#通过配置文件的方法跳过MySQL的登录检查，修改后要删掉次配置
`sed -i '/sql_mode=/i\\skip-grant-tables' /etc/my.cnf`
`systemctl restart mysql`
#modify password
`mysql -uroot -e "use mysql;update user set password=password('root') where user='root';flush privileges;grant all on *.* to root@'%' identified by 'root';"`
echo "User of 'root' password has changed into 'root'"
`sed -i '/skip-grant-tables/d' /etc/my.cnf`
echo "Modify password successfully"

#------------------CHINESE GARBLED PROBLEM------------------------
echo
echo "Start to modify mysql-my.cnf"
`echo "#mysql-setup" > /etc/my.cnf`
sen="[client]
default-character-set=utf8
[mysqld]
default-storage-engine=INNODB
character-set-server=utf8
collation-server=utf8_general_ci
[mysql]
default-character-set=utf8
"
lines=`echo $sen`
for line in $lines
do
    VAR=`eval "sed -i '/#mysql-setup/i\$line' /etc/my.cnf"`
done
#验证修改是否完成
VAR=`awk -F' ' 'END{print $1}' /etc/my.cnf`
if [[ $VAR =~ ^#mysql ]]
then
        echo "modify my.cnf success"
else
        echo "modify my.cnf failed,please check shell..."
        exit 1
fi
echo
echo "Restart mysql"
`systemctl restart mysql`
echo "Install mysql finished!"

三、shell安装&配置

        先将整体的安装包解压到/opt/download文件夹下(若无此文件夹，需要手动创建),图中我解压到hadoop文件夹内;接下来就可以启用shell脚本了；

在这里插入图片描述

1.删除重复文件&配置

  • 删除服务所在的目录文件
            本文默认所有服务安装在/opt/software目录下，所以只要递归删除目录software即可；

  • 剔除配置的环境变量
            修改环境变量在/etc/profile下，因为本文插入的环境变量都在done 这一行和unset i这一行之间，所以将这两行中间的内容删除即可；

#!/bin/bash
#------------------REMOVE ALL RELATIVE RESOURCES---------------------
echo "##### start to remove relative resources,included env-vars and software-installation ..."
#获取配置环境变量的开始行和结束行
BEGIN=`cat -n /etc/profile|grep done|awk '{print $1}'`
END=`cat -n /etc/profile|grep "unset i"|awk '{print $1}'`
RST=$(((--END)-(++BEGIN)))
if [ $RST -gt 0 ]
then
	RST="'"$BEGIN,$END" d'"
	RST="sed -i $RST /etc/profile"
	RST=`eval $RST`
	echo "... /etc/profile cleared"
else
	echo "... /etc/profile empty"
fi
#删除服务所在目录下的所有文件
if [ -e /opt/software ]
then
	rm -rf /opt/software
	echo "... /etc/profile delete"
else
	echo "... /etc/profile not existed"
fi
echo

2.解压安装包&重命名&授权

  • 验证安装包是否缺失
            这里防止安装一半时缺失安装包的情况，先验证安装包是否完整；

  • 安装目录是否存在
            统一解压到/opt/software目录下，若没有此目录则新建此目录；

  • 解压&重命名&授权
            存在.tar和.gz的文件，所以解压方式也不同；解压完之后根据安装包的前半部分和重命名集合里的是否一致，一致则替换名称，最后将权限修改；

#------------------NAME LIST FOR INSTALLATION------------------------
echo "----------------------------------------------"
echo
echo "#### start to unpack all resources ..."
INSTALL_LIST="jdk-8u111-linux-x64.tar.gz scala-2.12.10.tgz hadoop-2.6.0-cdh5.14.2.tar.gz hive-1.1.0-cdh5.14.2.tar.gz zeppelin-0.8.2-bin-all.tgz zookeeper-3.4.5-cdh5.14.2.tar.gz hbase-1.2.0-cdh5.14.2.tar.gz sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz"

#---------------------SOURCE PACKAGE VAILIDATION---------------------
echo "start to check whether all the resource needed exists?"
LIST=`ls /opt/download/hadoop`


for item in $INSTALL_LIST
do
	if [[ $LIST =~ $item ]]
	then
		continue
	else
		echo "ERROR and will exit for the lack of $item"
		exit 1
	fi
done
echo "unpack all finish."
echo "start to installing..."
echo "first,we'll unpack all the items ..."

#---------------------ENSURE THE UNPACK DESTINATION DIR-----------------------
echo -n "start to check destination directory ..."
DEST_DIR="/opt/software/"

if [ ! -e $DEST_DIR ]
then
	RST=`mkdir $DEST_DIR`
	echo "not exist and has been created"
else
	echo "ok"
fi

cd /opt/software
echo "                                  ---------------"
echo "current dir has been changed into |/opt/software|"
echo "                                  ---------------"

#--------------------UNPACK AND RENAME AND CHOWN---------------------------------------
RENAME_LIST="java180 scala21210 hadoop260 hive110 zeppelin082 zookeeper345 hbase120 sqoop146"
RENAME_LIST=($RENAME_LIST)
IX=0

for item in $INSTALL_LIST
do
	if [[ $item =~ tar|gz$ ]]
	then
		echo "start to unpack $item ..."
		OPT=${item##*.tar}
		OPT="-"$([ $OPT ] && echo "z" || echo "" )"xf"
		RST=`eval "tar $OPT /opt/download/hadoop/$item"`
		RST=`eval "mv ${item%%-*}* ${RENAME_LIST[$IX]}"`
		echo "unpack $item and rename ${RENAME_LIST[$IX]} ok!"
		`chown -R root:root ${RENAME_LIST[$IX]}`
	fi
	((IX++))
done
echo

3.配置环境变量

  • 目录名转大写并判断配置path时是否包含sbin
  • 插入时以unset i 这一行为基准，一直往这一行上方插入即可
  • CLASSPATH和LOGDIR单独插入
#---------------------ENV_VAR : ALL_XXX_HOME--------------------------------
echo "----------------------------------------------"
echo
echo "#### start to config environment variables ..."
RENAME_ITEMS="java180_bin scala21210_bin hadoop260_bin_sbin hive110_bin zeppelin082_bin zookeeper345_sbin hbase120_bin sqoop146_bin"
ROAD="export PATH="
for item in $RENAME_ITEMS
do
	HOME=${item%%[0-9]*}
	#拼出export对应的字符串
	HOME=${HOME^^}"_HOME"
	#名称转大写
	T="${item//_/ }"
	FIRST=1
	for var in $T
	do
		if [ $FIRST -eq 1 ]
		then
			FIRST=0
		else
			#插入path前拼出字符串
			ROAD+="$"$HOME"/"$var":"
		fi
	done
	echo " .... $HOME finished"
	HOME="export $HOME=$DEST_DIR${item%%_*}"
	#往unset i 上方插入
	RST=`eval "sed -i '/unset i/i\$HOME' /etc/profile"`
done
ROAD+="$""PATH"
RST=`eval "sed -i '/unset i/i\$ROAD' /etc/profile"`
echo " .... PATH finished"
#--------------------------CLASS_PATH---------------------
ROAD="export CLASSPATH=.:$""JAVA_HOME/lib/tools.jar:$""JAVA_HOME/lib/dt.jar"
RST=`eval "sed -i '/unset i/i\$ROAD' /etc/profile"`
echo " .... JAVA_CLASSPATH finished"
#---------------------ZOOKEEPER LOGS---------------------
ROAD="export LOGDIR=$""SQOOP_HOME/mylog/"
RST=`eval "sed -i '/unset i/i\$ROAD' /etc/profile"`
echo " .... ZOOLEEPER LOGDIR finished"

#---------------------ACTIVATE ENV_VAR-------------------
#激活环境变量
source /etc/profile
echo "environment variables have been written into /etc/profile and activated"
echo
#获取虚拟机ip地址
HOST_IP=`ip addr|awk -v FS='/' '/noprefixroute ens33/{print $1}'|awk -F' ' '{print $2}'`
echo
echo "----------------------------------------------"
echo "$HOSTNAME IP is $HOST_IP"
echo "----------------------------------------------"

4.Hadoop配置

  • hadoop-native-64-2.6.0.tar解压至hadoop根目录的lib目录下的native文件夹
  • 修改hadoop-env.sh；
  • core-site.xml hdfs-site.xml mapred-site.xml.template yarn-site.xml在同一目录下，使用循环修改；
  • 格式化Namenode并验证
#---------------------hadoop-----------------------------
echo
echo "Starting to modify hadoop..."
cd /opt/download/hadoop
VAR=`ls -F | grep "^hadoop-native"`
TMP=`eval "tar -xvf $VAR -C ${HADOOP_HOME}/lib/native"` 
DEST_DIR=$HADOOP_HOME"/etc/hadoop/"
REP="export JAVA_HOME=${JAVA_HOME}"
`sed -i "/^export JAVA_HOME=/c$REP" ${HADOOP_HOME}/etc/hadoop/hadoop-env.sh`
CONFS="core-site.xml hdfs-site.xml mapred-site.xml.template yarn-site.xml"

readonly CONFI1="
<property>
<name>hadoop.tmp.dir</name>
<value>$HADOOP_HOME</value>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://${HOST_IP}:9000</value>
</property>
"

RST=`mkdir $HADOOP_HOME"/tmp"`
RST=`mkdir $HADOOP_HOME"/tmp/name" $HADOOP_HOME"/tmp/data"`

readonly CONFI2="
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>$HADOOP_HOME/tmp/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>$HADOOP_HOME/tmp/data</value>
</property>
"

readonly CONFI3="
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
"

readonly CINFI4="
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
"

IX=1
for file in $CONFS
do
	if [[ $file =~ .template ]]
	then
		CMD="mv $DEST_DIR$file $DEST_DIR${file%.*}"
		RST=`eval $CMD`
		file=${file%.*}
	fi
	CMD="sed -i '/<\/configuration>/i\REPL\' $DEST_DIR$file"
	V="CONFI"$IX
	LINES=`eval echo "$"$V`
	echo -n " ---- config $file ..."
	for line in $LINES
	do
		RST=${CMD/REPL/$line}
		RST=`eval $RST`	
	done
	echo " ok"
	((IX++)) 
done

#hadoop namenode format
echo -n " .... start to format namenode ..."
RST=`eval "cd $HADOOP_HOME/tmp/name"`
hdfs namenode -format>/root/nnfmt.log 2>&1
RST=`cat /root/nnfmt.log|grep util.ExitUtil`
RST=${RST##* }
echo $([ $RST -eq 0 ] && echo "ok" || "error")
if [ $RST -ne 0 ]
then
	exit 1
fi
echo
echo "Hadoop modify finish."
echo

5.Hive配置

  • 创建目录warehouse
  • 创建并修改hive-site.xml
  • 拷贝mysql的jar包
  • 初始化hive
#-------------------------hive-----------------------
echo "----------------------------------------------"
echo
echo "Let's modify hive..."
#-------------create warehouse----------------------
cd $HIVE_HOME
#验证路径是否正确
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ hive110$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
#创建目录warehouse
echo "first,we need to create directory:warehouse..."
mkdir warehouse/
echo "Directory warehouse create finish!"
echo

#-------------hive-site.xml-------------------------
echo "second,we'll modify hive-site.xml...."
cd ./conf
#验证路径是否正确
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ conf$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
TEXT="<?xml#version=\"1.0\"#encoding=\"UTF-8\"#standalone=\"no\"?>
<?xml-stylesheet#type=\"text/xsl\"#href=\"configuration.xsl\"?>
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>$HIVE_HOME/warehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://${HOST_IP}:3306/hive110?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>root</value>
</property>
</configuration>
"
#hive-site.xml不存在，创建并插入；考虑到sed无法插入空文件，先echo一行进入再sed插入，最后将那一行删除
touch hive-site.xml
`echo "#setup">./hive-site.xml`
lines=`echo $TEXT`
for line in $lines
do
    VAR=`eval "sed -i '/#setup/i\${line//#/ }' hive-site.xml"`
done
VAR=`awk -F'#' 'END{print $2}' hive-site.xml`
if [[ $VAR =~ ^setup ]]
then
    echo "modify hive-site.xml success"
else
    echo "modify hive-site.xml failed,please check shell..."
    exit 1
fi
sed -i '$d' hive-site.xml
echo
#---------------------------copy mysql-connection---------------
echo "Third,we should copy drive to hive..."
cd ../lib
#验证路径是否正确
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ lib$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
#拷贝jar包
cp /opt/download/hadoop/mysql*.jar ./
echo "copy  drive ok!"
echo

#-----------------------Initialize the environment---------------------------
echo "last,we must initialize the environment...."
cd ../bin
#验证路径是否正确
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ bin$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
echo "start to initialize..."
#初始化hive
VAR="./schematool -dbType mysql -initSchema>/root/test.log 2>&1"
`eval $VAR`
#验证初始化结果
VAR=`awk -F' ' 'END{print $2}' /root/test.log`
if [[ $VAR =~ ^completed ]]
then
    echo "initialize successfully"
else
    echo "initalize failed,please reinitalize..."
    exit 1
fi
echo
echo "hive Configurate finish!"
echo

6.Zeppelin配置

  • 配置zeppelin-site.xml
  • 配置zeppelin-env.xml
  • 拷贝hive-site.xml&jar包
#---------------------zeppelin---------------------------
#--------------------authorize---------------------
echo "----------------------------------------------"
echo
echo "Next,we start to deploy zeppelin..."
#授权和上面重复，但不影响程序，可以删掉这一部分
echo "First,we need to authorize directory with its files...."
cd $ZEPPELIN_HOME
cd ../
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ software$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
VAR=`ls|grep zeppe*`
`chown -R root:root $VAR`
echo "authorized success!"
echo
#-----------modify zeppelin-site.xml---------
echo "Second,we need to modify zeppelin-site.xml..."
cd $ZEPPELIN_HOME
cd ./conf
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ conf$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
VAR=`ls|grep zeppelin-site*`
echo "We need to copy $VAR one more to modify"
cp $VAR ${VAR%.*}
echo "Copy success,next to modify zeppelin-site.xml..."
VAR="sed -i '24s/127.0.0.1/$HOST_IP/' zeppelin-site.xml"
`eval $VAR`
VAR="sed -i '30s/8080/8000/' zeppelin-site.xml"
`eval $VAR`
VAR=`awk 'NR==30{print $0}' zeppelin-site.xml`
VAR=`echo $VAR | tr -cd "[0-9]"`
if [[ $VAR =~ 8000 ]]
then
    echo "modify zeppelin-site.xml success!"
else
    echo  "modify zeppelin-site.xml failed,please check out..."
    exit 1
fi
echo

#-----------modify zeppelin-env.xml---------
echo "Then,let's modify zeppelin-env.sh..."
VAR=`ls|grep zeppelin-env.sh*`
echo "We need to copy $VAR one more to modify"
cp $VAR ${VAR%.*}
echo "Copy success,next to modify zeppelin-env.sh..."
#TEMP=`grep -n "# export JAVA_HOME=" zeppelin-env.sh`
#TEMP=`echo $a | tr -cd "[0-9]"`
#VAR="sed -i '${TEMP}s/#/ /' zeppelin-env.sh"
#`eval $VAR`
#VAR="sed -i '${TEMP}s/$""/${JAVA_HOME//\//\\/}' zeppelin-env.sh"
#`eval $VAR`
REP="export JAVA_HOME=${JAVA_HOME}"
`sed -i "/^# export JAVA_HOME=/c$REP" zeppelin-env.sh`
#TEMP=`grep -n "# export HADOOP_CONF_DIR.*" zeppelin-env.sh`
#TEMP=`echo $a | tr -cd "[0-9]"`
#VAR="sed -i '${TEMP}s/#/ /' zeppelin-env.sh"
#`eval $VAR`
#VAR="sed -i '${TEMP}s/"="/'='${HADOOP_HOME//\//\\/}' zeppelin-env.sh"
#`eval $VAR`
REP="export HADOOP_CONF_DIR=${HADOOP_HOME}"
`sed -i "/^# export HADOOP_CONF_DIR/c$REP" zeppelin-env.sh`
echo "modify zeppelin-env.sh success!"
echo
#--------------cp jar-----------------------
echo "Next,we need to copy .jar and .xml file..."
#拷贝xml和jar包
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ conf$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
`cp ${HIVE_HOME}/conf/hive-site.xml ./`
echo "copy hive-site.xml to current path success!"
`cp ${HADOOP_HOME}/share/hadoop/common/hadoop-common-2.6.0-cdh5.14.2.jar ./`
echo "echo hadoop.jar to current path success!"
`cp ${HIVE_HOME}/lib/hive-jdbc-1.1.0-cdh5.14.2-standalone.jar ./`
echo "copy hive.jar to current path success!"
echo
#-----------------start to switch on service-------------
echo "Finally,we'll start to switch on service..."
cd ../bin
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ bin$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
#启动服务
./zeppelin-daemon.sh start
echo
echo "Zeppelin modify finish."
echo

7.Zookeeper配置

  • 创建directory mydata&myid
  • 修改 zoo.cfg文件
#---------------------zookeeper--------------------------
echo "----------------------------------------------"
echo
echo "After,we start to modify zookeeper..."
echo "First,we need to create directory mydata and myid..."
cd $ZOOKEEPER_HOME
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
echo "currentpath: $VAR"
`mkdir mydata myid`
`echo "1">./myid`
echo "create directory finish!"
echo

echo "Second,we'll modify zoo.cfg..."
cd ./conf
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ conf$ ]]
then
    echo -n ""
else
    echo "path error,please check curren path!"
    exit 1
fi
echo "currentpath: $VAR"
VAR=`ls|grep zoo.sample*`
echo "We need to copy $VAR one more to modify"
cp $VAR zoo.cfg
VAR="zoo.cfg"
echo "Copy success,next to modify $VAR..."
REP="dataDir=${ZOOKEEPER_HOME}/mydata"
`sed -i "/^dataDir=/c$REP" $VAR`
TEMP=`grep -n "dataDir=" zoo.cfg`
TEMP=`echo $a | tr -cd "[0-9]"`
SEN="sed -i 'N;${TEMP}aserver.1=master:2888:3888' $VAR"
`eval $SEN`
echo "Modify zoo.cfg success"
echo
echo "Zookeeper modify finish!"
echo

8.Hbase配置

  • 修改hbase.env.sh
  • 修改hbase-site.xml
#---------------------hbase------------------------------
echo "----------------------------------------------"
echo
echo "Afterwards,we start to deploy hbase"
echo "First,we'll modify hbase.env.sh..."
cd ${HBASE_HOME}/conf
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ conf$ ]]
then
    echo -n ""
else
    echo "path error,please check current path!"
    exit 1
fi
echo "currentpath: $VAR"
REP="export JAVA_HOME=${JAVA_HOME}"
`sed -i "/^# export JAVA_HOME=/c$REP" hbase-env.sh`
REP="export HBASE_MANAGES_ZK=false"
`sed -i "/^# export HBASE_MANAGES_ZK/c$REP" hbase-env.sh`
echo "hbase-env.sh has been modified."
echo
echo "Second,we should modify hbase-site.xml..."
TEXT="<property>
<name>hbase.rootdir</name>
<value>hdfs://${HOST_IP}:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.property.clientport</name>
<value>2181</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>${ZOOKEEPER_HOME}/mydata</value>
</property>
"
lines=`echo $TEXT`
for line in $lines
do
    VAR=`eval "sed -i '/<\/configuration>/i\$line' hbase-site.xml"`
done
echo "hbase-site.xml has been modified."
echo
echo "Hbase  deploy finish!"

9.Sqoop配置

  • 拷贝jar包
  • *配置sqoop-env.sh
#----------------------sqoop-------------------------
echo
echo "----------------------------------------"
echo "Let's start to deploy sqoop..."
echo "First,we need to copy more .jar to path of lib..."
cd ${SQOOP_HOME}/lib
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ lib$ ]]
then
    echo -n ""
else
    echo "path error,please check current path!"
    exit 1
fi
echo "currentpath: $VAR"
`cp /opt/download/hadoop/mysql-connector-java-5.1.32.jar ./`
echo "copy mysql-connector-java-5.1.32.jar to current path success!"
`cp ${HADOOP_HOME}/share/hadoop/common/hadoop-common-2.6.0-cdh5.14.2.jar ./`
echo "copy hadoop-common-2.6.0-cdh5.14.2.jar to current path success!"
`cp ${HADOOP_HOME}/share/hadoop/hdfs/hadoop-hdfs-2.6.0-cdh5.14.2.jar ./`
echo "copy hadoop-hdfs-2.6.0-cdh5.14.2.jar to current path success!"
`cp ${HADOOP_HOME}/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.6.0-cdh5.14.2.jar ./`
echo "copy hadoop-mapreduce-client-core-2.6.0-cdh5.14.2.jar to current path success!"
echo
echo "Second ,we start to modify sqoop.env.sh..."
cd ../conf
`eval "pwd>/root/test.log"`
VAR=`cat /root/test.log`
if [[ $VAR =~ conf$ ]]
then
    echo -n ""
else
    echo "path error,please check current path!"
    exit 1
fi
echo "currentpath: $VAR"
VAR=`ls|grep sqoop-env-template.sh`
echo "We need to copy $VAR one more to modify"
`cp $VAR sqoop-env.sh`
echo "Copy success,next to modify sqoop-env.sh..."
REP="export HADOOP_COMMON_HOME=$HADOOP_HOME"
`sed -i "/^#export HADOOP_COMMON_HOME=/c$REP" sqoop-env.sh`
REP="export HADOOP_MAPRED_HOME=${HADOOP_HOME}/share/common/mapreuce"
`sed -i "/^#export HADOOP_MAPRED_HOME=/c$REP" sqoop-env.sh`
REP="export HBASE_HOME=$HBASE_HOME"
`sed -i "/^#export HBASE_HOME=/c$REP" sqoop-env.sh`
REP="export HIVE_HOME=$HIVE_HOME"
`sed -i "/^#export HIVE_HOME=/c$REP" sqoop-env.sh`
REP="export ZOOCFGDIR=$ZOOKEEPER_HOME"
`sed -i "/^#export ZOOCFGDIR=/c$REP" sqoop-env.sh`
echo "sqoop-env.sh has been modified."
echo
echo "sqoop deploy finish!"
echo
echo "----------------------------------------------"

PS：如果有写错或者写的不好的地方，欢迎各位大佬在评论区留下宝贵的意见或者建议，敬上！如果这篇博客对您有帮助，希望您可以顺手帮我点个赞！不胜感谢！


原创作者：wsjslient

作者主页：https://blog.csdn.net/wsjslient

NightFall丶
关注
  • 9
    点赞
  • 踩
  • 15
    收藏
    觉得还不错? 一键收藏
  • 16
    评论
专栏目录
shell中安装Mysql
wd_87的博客
07-26 1674
因为版本原因，安装airflow必须在shell中安装mysql
Hadoop概述&集群搭建.md
04-08
- **安装和配置JDK**：Java是Hadoop的核心开发语言，确保正确安装并配置环境变量。 - **安装Hadoop**：下载Hadoop安装包，并解压缩到指定目录。 - **创建工作目录**：在用户的家目录下创建名为Bigdata的目录。 - **...
shell 安装MYSQL
zj0910
08-08 2006
安装脚本  #!/bin/bash    MYSQL_HOME=/usr/local/mysql  # 将你的mysql安装程序解压后的文件名设置为MYSQL_DIR，将安装程序(mysql-xxx.tar.gz)放到当前目录  MYSQL_DIR=mysql-standard-5.0.20-linux-i686-glibc23  MYSQL_DIR_OLD=$PWD/${MYSQL_
shell 安装mysql数据库_shell安装mysql数据库
weixin_35019134的博客
02-11 262
此文为mysql 服务器安装脚本。#!/bin/bash#ver 0.2#by zm 2011/04/23#安装 mysql_server;#注意！！#参数解释## CFLAGS="-O3" CXX=gcc CXXFLAGS="-O3 -felide-constructors \# -fno-exceptions -fno-rtti -fomit-frame-pointer -ffixed-ebp...
使用Shell脚本安装MySQL
微笑的段嘉许的博客
10-20 1952
随着Linux系统在企业中的应用越来越多，服务器的自动化管理也变得越来越重要。在Linux服务器的自动化维护工作中，除了计划任务的设置以外，Shell脚本的应用也是非常重要的一部分。本文将主要介绍Shell脚本的基础，变量使用等知识。
shell脚本安装mysql(关联项目)
阳宗德的博客
01-07 869
shell脚本安装mysql安装环境脚本能力脚本内容安装和重装命令生成项目启动配置参考链接 安装包下载路径：https://dev.mysql.com/downloads/mysql/ 安装环境 CentOS-7 / 银河麒麟 mysql-5.7.26-linux-glibc2.12-x86_64.tar.gz 脚本 脚本能力 重复安装 卸载 停止 启动 查看状态 脚本内容 安装和重装 #!/bin/sh #路径 projecthome=`pwd` setup=$projecthome/set
hadoop大数据生态圈工具配置与应用.pdf
12-09
Hadoop生态系统的组件众多，包括Zookeeper、Hive、Flume和Sqoop等，它们协同工作，提供了大数据处理的完整解决方案。 **第二章 Hadoop** 在安装Hadoop时，首先需要在CentOS7上进行基础配置，包括创建必要的文件夹...
10.Hadoop入门进阶课程_第10周_HBase介绍、安装与应用案例.pdf
04-03
- **生态系统集成**：HBase与其他Hadoop生态系统组件紧密集成，如Pig和Hive为HBase提供了高级的数据处理语言支持，而Sqoop则简化了传统关系型数据库与HBase之间的数据迁移过程。 #### 三、HBase的访问接口 HBase...
sqoop安装手册
04-11
在安装 Sqoop 之前，确保服务器上已经安装了 Hadoop、Hive 和 JDK 环境，因为这些是 Sqoop 运行所必需的。如果缺少任何组件，你需要先安装它们。此外，检查服务器的防火墙状态，如果防火墙开启，你需要关闭它，以...
部分大数据生态组 件安装 包
01-27
sqoop-1.4.7.bin__hadoop-2.6.0.tar zookeeper-3.4.14.tar scala-2.11.12 jdk-8u141-linux-x64.tar kafka_2.11-2.4.0 apache-hive-2.3.3-bin.tar hadoop-2.7.7.tar spark-2.4.4-bin-hadoop2.7 hbase-1.3.1-bin.tar ...
shell脚本安装mysql-5.6
04-01
该资源为mysql的安装shell脚本，请自行下载mysql-xxx-xxx.tar.gz安装包，并与脚本一同放置在/opt/db下，修改脚本权限，chmod 777 startupmysql.sh，执行脚本即可完成mysql的安装。如果需要修改安装路径，请适当 修改脚本的第5，7，8，9行
mysql shell安装包以及制作的脚本代码
01-27
shell 脚本一键安装mysql数据库（已打包成图形化安装），内涵.x类型可执行文件。可在linux桌面双击运行配置安装mysql 博客地址 ： http://blog.csdn.net/u013255206/article/details/79176070 本资源是txt，内有下载链接
Linux自动安装Hive（单机，含安装MySQL）（Shell脚本）
12-04
要自己先安装jdk和Hadoop，也可以去这里https://download.csdn.net/download/weixin_44033210/12657718，https://download.csdn.net/download/weixin_44033210/12657738，这里有自动安装jdk和Hadoop的脚本
linux使用shell一键安装mysql
henry
03-16 1232
#!/bin/shgroup=mysqluser=mysql#create group if not existsegrep "^$group" /etc/group &gt;&amp; /dev/nullif [ $? -ne 0 ]then  groupadd $groupfi#create user if not existsegrep "^$user" /etc/passwd &gt;&a...
shell一键安装mysql5.5.35
weixin_36521716的博客
07-13 210
#!/bin/bashVERSION=5.5.35program_path=$HOME/work/odp/programs/mysql-${VERSION}tmp_dir=$HOME/work/odp/tmpmkdir -p ${tmp_dir}cd ${tmp_dir}#wget http://downloads.mysql.com/archives/get/file/mysql-5.5.35....
Hadoop 与 Hadoop-native 所有版本下载
m0_67401761的博客
04-03 2583
Hadoop所有版本：https://archive.apache.org/dist/hadoop/core/ Hadoop Native包所有版本： http://dl.bintray.com/sequenceiq/sequenceiq-bin/ Apache 的 国内镜像地址： 地址1：http://mirror.bit.edu.cn/apache/ 地址2：https://mirrors.tuna.tsinghua.edu.cn/apache ...
shell脚本：一键安装mysql数据库
weixin_45783664的博客
10-11 1973
【代码】shell脚本：一键安装mysql数据库。
Hadoop2.6.0本地库（Native Library）
liweihope的博客
04-30 1230
最近尝试搭建伪分布式hadoop环境，遇到了一些有关hadoop本地库的问题，在此做一个整理，近针对hadoop2.6.0,ubuntu系统。 遇到问题：WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applic...
太强了！写了个自动化安装 MySQL 数据库拿来就用的 Shell 脚本！
最新发布
JackTian
02-03 1328
高效、快速安装 MySQL 数据库
Hadoop集群安装配置与使用教程：HBase, Hive, Sqoop
"这篇文档主要介绍了如何在Hadoop、HBase、Hive和Sqoop的集群环境中进行安装配置以及使用的基本步骤。" 在大数据处理领域，Hadoop、HBase、Hive和Sqoop是四个非常重要的组件。以下是这些组件的简要介绍以及集群环境...
评论 16
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值