一文学会搭建大数据开发环境之离线分析 Hadoop/Hive/Zookeeper/Hbase

2019独角兽企业重金招聘Python工程师标准>>>

hadoop 单机

我们基于docker centos7 构建单机hadoop 环境

docker run -it centos bash


#安装必备软件
yum install -y java-1.8.0-openjdk java-1.8.0-openjdk-devel wget  openssh-server openssh-clients which



# 设置并启动ssh
ssh-keygen -q -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key -N ''
ssh-keygen -q -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key -N ''
ssh-keygen -t dsa -f /etc/ssh/ssh_host_ed25519_key  -N ''
/usr/sbin/sshd 

# 设置免密码登陆
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys


#安装hadoop
wget http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz

tar zxvf /root/hadoop-2.7.7.tar.gz


#配置环境变量
vi /etc/profile
 
export PATH=/usr/local/hadoop-2.7.7/bin:$PATH
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64
source /etc/profile

vi etc/hadoop/hadoop-env.sh
export  JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64


#启动HDFS 
# vi etc/hadoop/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
</configuration>


# vi etc/hadoop/hdfs-site.xml
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>


hdfs namenode -format

/root/hadoop-2.7.7/sbin/start-dfs.sh


# 启动yarn


# cp etc/hadoop/mapred-site.xml.template etc/hadoop/mapred-site.xml
# vi etc/hadoop/mapred-site.xml
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

# vi etc/hadoop/yarn-site.xml
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

/root/hadoop-2.7.7/sbin/start-yarn.sh

#全部停止

/root/hadoop-2.7.7/sbin/stop-all.sh

hadoop对应端口

50070 namenode web管理页面 9000 hdfs 端口 8088 ResourceManger 端口 19888 Job History Server

这样我们单机的hadoop环境就搭建完成了

Hive

我们继续搭建Hive的环境

wget http://mirror.bit.edu.cn/apache/hive/hive-2.3.4/apache-hive-2.3.4-bin.tar.gz

tar -zxvf apache-hive-2.3.4-bin.tar.gz 

# vi /etc/profile

export HIVE_HOME=/root/apache-hive-2.3.4-bin
export PATH=$PATH:$HIVE_HOME/bin

source /etc/profile



# copy mysql.jar 到 lib 

wget http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.47/mysql-connector-java-5.1.47.jar


# 修改配置

cp hive-env.sh.template  hive-env.sh
vi  hive-env.sh


cp hive-default.xml.template hive-site.xml

vi hive-site.xml


  <!-- 插入一下代码 放在最下面 -->
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>xx</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name> 
        <value>xx </value>
    </property>
   <property>
        <name>javax.jdo.option.ConnectionURL</name>mysql
        <value>jdbc:mysql://xx:3306/hive</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name> 
        <value>com.mysql.jdbc.Driver</value>
    </property>
        <!-- 到此结束代码 -->
        

# init db schema
schematool -dbType mysql -initSchema


#启动
nohup hive --service hiveserver2 > hiveserver2.log 2>&1 &

Hive的环境搭建完毕 client 连接的端口为10000

zookeeper

#下载

http://archive.apache.org/dist/zookeeper/zookeeper-3.4.9.tar.gz 

tar -zxf zookeeper-3.4.9.tar.gz

cd ./zookeeper-3.4.9/conf


cp zoo_sample.cfg   zoo.cfg

#启动
 ./bin/zkServer.sh start

也可以直接使用zookeeper 镜像启动

docker run --name zk --restart always -d -p 2181:2181 zookeeper

到此zookeeper搭建完成 zookeeper的对应端口为2181

Hbase

wget https://mirrors.cnnic.cn/apache/hbase/2.1.4/hbase-2.1.4-bin.tar.gz 

tar -zxvf  hbase-2.1.4-bin.tar.gz 

# 修改环境变量
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64
export HBASE_HOME=/root/hbase-2.1.4
export PATH=$HBASE_HOME/bin:$PATH

# 修改配置
vi conf/hbase-env.sh



export HBASE_CLASSPATH=/root/hbase-2.1.4/lib

#这里使用公用的zookeeper 不用内置的
export HBASE_MANAGES_ZK=false

vi conf/hbase-site.xml


 <property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>localhost</value>
</property>

echo  "localhost" > regionservers 



 

#启动
./bin/start-hbase.sh 

到此hbase搭建完成

habase的主要端口号为

16010 web管理页面 16000 master ipc端口 16020 region server ipc 端口

转载于:https://my.oschina.net/jayqqaa12/blog/3043996