linux安装大数据相关软件

最新推荐文章于 2024-06-24 20:12:07 发布

Toner_唐纳

最新推荐文章于 2024-06-24 20:12:07 发布

阅读量286

点赞数

分类专栏： linux 大数据

本文链接：https://blog.csdn.net/qq_37012496/article/details/104051884

版权

这篇博客详细介绍了在Linux环境下安装大数据相关软件的步骤，包括安装ipconfig、wget、Scala、Zookeeper集群，解决telnet连接问题，安装Hadoop并进行格式化和启动，配置HBase以及安装Spark的过程。内容涵盖了防火墙设置、MySQL安装和集群的分布式部署。

摘要由CSDN通过智能技术生成

1.安装ipconfig

yum install net-tools.x86_64

2.安装wget

yum install wget

3.安装Scala

mkdir /usr/local/scala
mv scala-2.13.0.tgz /usr/local/scala
tar -zxvf scala-2.13.0.tgz
vim /etc/profile
#增加配置
export SCALA_HOME=/usr/local/scala/scala-2.13.0
export SCALA_PATH=${SCALA_HOME}/bin

export JAVA_HOME=/usr/local/java/jdk1.8
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin

export PATH=$PATH:${JAVA_PATH}:${SCALA_PATH}

. /etc/profile
scala -version

4.搭建zookeeper集群

mkdir /usr/local/zookeeper
mv zookeeper-3.4.6.tar.gz /usr/local/zookeeper
cd /usr/local/zookeeper
tar -zxvf zookeeper-3.4.6.tar.gz
cd zookeeper-3.4.6/conf/
cp zoo_sample.cfg zoo.cfg
vi zoo.cfg
###########配置文件内容############
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
####此处需要修改
dataDir=/data/zookeeper
# the port at which the clients will connect
clientPort=2181
####此处需要新增，
server.1=bigdata1:2888:3888
server.2=bigdata2:2888:3888
server.3=bigdata3:2888:3888
############配置文件结束#############
mkdir -p /data/zookeeper
cd /data/zookeeper
echo 1 > myid

vi /etc/profile
######追加内容#######
export ZOOKEEPER_HOME=/usr/local/zookeeper/zookeeper-3.4.6
export ZOOKEEPER_PATH=${ZOOKEEPER_HOME}/bin
######追加内容#######
. /etc/profile
#启动zookeeper
zkServer.sh start

修改主机名

vi /etc/hosts
#########文件内容########
192.168.162.11 bigdata1
192.168.162.12 bigdata2
192.168.162.13 bigdata3

安装telnet

yum install telnet

问题：

telnet虚拟机No route to host的解决方法
知道虚拟机的ip地址，而且可以ping的通
但是telnet连不通

解决方案

把虚拟机中的防火墙清一下

sudo iptables -F

hadoop安装

1.将hadoop拷贝至各服务器

for i in {
   1..3}; do 
scp  ~/Downloads/hadoop-2.7.7.tar.gz bigdata${i}:/usr/local/hadoop/hadoop-2.7.7.tar.gz; 
done

2.解压安装包

for i in {
   1..3}; do 
ssh bigdata${i} "tar -xzvf /usr/local/hadoop/hadoop-2.7.7.tar.gz" 
done

3.配置环境变量

# vim ~/.bashrc  

export HADOOP_HOME=/usr/local/hadoop/hadoop-2.7.7
export HADOOP_INSTALL=$HADOOP_HOME 
export HADOOP_MAPRED_HOME=$HADOOP_HOME 
export HADOOP_HDFS_HOME=$HADOOP_HOME 
export HADOOP_COMMON_HOME=$HADOOP_HOME 
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop  
export YARN_HOME=$HADOOP_HOME 
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop  

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin 


for i in {
   1..3}; do 
scp  ~/.bashrc bigdata${i}:/usr/local/hadoop 
ssh bigdata${i}  "cat /usr/local/hadoop/.bashrc >> /etc/profile & source /etc/profile & rm -rf /usr/local/hadoop/.bashrc"  
done

配置Hadoop

# vim $HADOOP_CONF_DIR/core-site.xml  

<configuration>
  <!-- 指定hdfs的nameservices名称为mycluster，与hdfs-site.xml的HA配置相同 -->
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://mycluster</value>
  </property>
	
  <!-- 指定缓存文件存储的路径 -->
  <property>
    <name>hadoop.tmp.dir</name>
    <value>/home/hadoop/app/cdh/hadoop-2.6.0-cdh5.7.1/data/tmp</value>
  </property>
 
  <!-- 配置hdfs文件被永久删除前保留的时间（单位：分钟），默认值为0表明垃圾回收站功能关闭 -->
  <property>
    <name>fs.trash.interval</name>
    <value>1440</value>
  </property>
  
  <!-- 指定zookeeper地址，配置HA时需要 -->
  <property>
    <name>ha.zookeeper.quorum</name>
    <value>hadoop-slave1:2181,hadoop-slave2:2181,hadoop-slave3:2181</value>
  </property>
</configuration>
#ds.defaultFS指定了hdfs的默认路径；hadoop.tmp.dir指定了hadoop文件存放路径的根目录。

5.配置HDFS

# vim $HADOOP_CONF_DIR/hdfs-site.xml

<configuration>
  <!-- 指定hdfs元数据存储的路径 -->
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/home/hadoop/app/cdh/hadoop-2.6.0-cdh5.7.1/data/namenode</value