版权声明:本文为作者原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文作者首次发表地址:https://www.cnblogs.com/gjycn/p/14438354.html
软件依赖版本列表:
- oracle jdk8u281
- zookeeper3.5.9
- hadoop3.2.1
- hbase2.3.4
- phoenix-hbase-2.3-5.1.0
环境准备
-
安装jdk
#解压 tar -zxvf jdk-8u281-linux-x64.tar.gz #复制 sudo cp jdk1.8.0_281 /usr/java -r #配置环境变量 #jdk8 export JAVA_HOME=/usr/java/jdk1.8.0_281 export PATH=${PATH}:${JAVA_HOME}/bin export CLASSPATH=${CLASSPATH}:.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
-
安装ssh\pdsh
#安装 sudo apt install ssh sudo apt install pdsh #配置pdsh #方法一 cd /etc/pdsh sudo gedit rcmd_default #写入 ssh #或 #方法二 mkdir -p /etc/pdsh echo "ssh" > /etc/pdsh/rcmd_default
-
ssh免密
#Now check that you can ssh to the localhost without a passphrase: ssh localhost #If you cannot ssh to localhost without a passphrase, execute the following commands: ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys chmod 0600 ~/.ssh/authorized_keys
-
修改
/etc/hosts
#添加本机外网IP 127.0.0.1 localhost 192.168.200.8 master
-
构建python软连接
sudo ln -s /usr/bin/python3 /usr/bin/python
上传软件
-
上传软件包至~/下载/
-
解压软件
tar -zxvf apache-zookeeper-3.5.9-bin.tar.gz tar -zxvf hadoop-3.2.1.tar.gz tar -zxvf hbase-2.3.4-bin.tar.gz tar -zxvf phoenix-hbase-2.3-5.1.0-bin.tar.gz
-
复制软件至/opt/
sudo cp apache-zookeeper-3.5.9-bin /opt/zookeeper-3.5.9 -r sudo cp hadoop-3.2.1 /opt/ -r sudo cp hbase-2.3.4 /opt/ -r sudo cp phoenix-hbase-2.3-5.1.0-bin /opt/phoenix-hbase-2.3-5.1.0-bin -r
-
授权
sudo chmod -R 755 /opt/zookeeper-3.5.9 sudo chown -R june /opt/zookeeper-3.5.9 sudo chmod -R 755 /opt/hadoop-3.2.1 sudo chown -R june /opt/hadoop-3.2.1 sudo chmod -R 755 /opt/hbase-2.3.4 sudo chown -R june /opt/hbase-2.3.4 sudo chmod -R 755 /opt/phoenix-hbase-2.3-5.1.0-bin sudo chown -R june /opt/phoenix-hbase-2.3-5.1.0-bin
-
配置环境变量
#在/etc/profile追加 #Zookeeper export ZOOKEEPER_HOME=/opt/zookeeper-3.5.9 export PATH=${PATH}:${ZOOKEEPER_HOME}/bin export CLASSPATH=${CLASSPATH}:${ZOOKEEPER_HOME}/lib #Hadoop export HADOOP_HOME=/opt/hadoop-3.2.1 export PATH=${PATH}:${HADOOP_HOME}/bin:${HADOOP_HOME}/sbin #HBase export HBASE_HOME=/opt/hbase-2.3.4 export PATH=${PATH}:${HBASE_HOME}/bin #Phoenix export PHOENIX_HOME=/opt/phoenix-hbase-2.3-5.1.0-bin export PATH=${PATH}:${PHOENIX_HOME}/bin #生效 source /etc/profile
软件安装
Zookeeper 3.5.9
-
在
/opt/zookeeper-3.5.9/conf/
修改配置文件zoo.cfg
cp zoo_sample.cfg zoo.cfg
-
修改配置
#zk1 dataDir=/home/june/zookeeper-3.5.9/zkData/zk1 clientPort=2181 autopurge.snapRetainCount=3 autopurge.purgeInterval=1 server.1=master:2888:3888
-
构建文件夹,写入id
mkdir -p /home/june/zookeeper-3.5.9/zkData/zk1 echo 1 > /home/june/zookeeper-3.5.9/zkData/zk1/myid
-
启动脚本
#!/bin/bash zkServer.sh start
Hadoop 3.2.1
-
在
/opt/hadoop
下修改etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_281
-
在
/opt/hadoop
下修改etc/hadoop/mapred-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_281
-
在
/opt/hadoop
下修改workers
master
-
在
/opt/hadoop
下修改etc/hadoop/core-site.xml
<configuration> <property> <name>fs.default.name</name> <value>hdfs://master:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/home/june/hadoop-3.2.1/data</value> </property> <property> <name>fs.trash.interval</name> <value>10080</value> </property> </configuration>
-
在
/opt/hadoop
下修改etc/hadoop/hdfs-site.xml
<configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>master:9868</value> </property> <!-- 指定namenode的访问地址和端口 --> <property> <name>dfs.namenode.http-address</name> <value>master:9870</value> </property> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
-
在
/opt/hadoop
下修改etc/hadoop/mapred-site.xml
<configuration> <!-- 指定分布式计算使用的框架是yarn --> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <!-- 开启MapReduce小任务模式 --> <property> <name>mapreduce.job.ubertask.enable</name> <value>true</value> </property> <!-- 设置历史任务的主机和端口 --> <property> <name>mapreduce.jobhistory.address</name> <value>master:10020</value> </property> <!-- 设置网页访问历史任务的主机和端口 --> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>master:19888</value> </property> <property> <name>mapreduce.application.classpath</name> <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value> </property> </configuration>
-
在
/opt/hadoop
下修改etc/hadoop/yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <!-- 配置yarn主节点的位置 --> <property> <name>yarn.resourcemanager.hostname</name> <value>master</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <!-- 开启日志聚合功能 --> <property> <name>yarn.log-aggregation-enable</name> <value>true</value> </property> <!-- 设置聚合日志在hdfs上的保存时间 --> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> <property> <name>yarn.nodemanager.env-whitelist</name> <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value> </property> </configuration>
-
格式化hdfs
hdfs namenode -format
-
启动脚本
#!/bin/bash start-dfs.sh start-yarn.sh
HBase 2.3.4
-
在
/opt/hbase/conf
下修改hbase-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_281 export HBASE_MANAGES_ZK=false
-
在
/opt/hbase/conf
下修改hbase-site.xml
<configuration> <!-- HBase数据在HDFS中的存放的路径 --> <property> <name>hbase.rootdir</name> <value>hdfs://master:9000/hbase</value> </property> <!-- Hbase的运行模式。false是单机模式,true是分布式模式。若为false,Hbase和Zookeeper会运行在同一个JVM里面 --> <property> <name>hbase.cluster.distributed</name> <value>true</value> </property> <!-- ZooKeeper的地址 --> <property> <name>hbase.zookeeper.quorum</name> <value>master</value> </property> <!-- ZooKeeper快照的存储位置 --> <property> <name>hbase.zookeeper.property.dataDir</name> <value>/home/june/zookeeper-3.5.9/zkData/zk1</value> </property> <!-- phoenix配置 --> <!-- 支持HBase命名空间映射 --> <property> <name>phoenix.schema.isNamespaceMappingEnabled</name> <value>true</value> </property> <!-- 支持索引预写日志编码 --> <property> <name>hbase.regionserver.wal.codec</name> <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value> </property> </configuration>
-
在
/opt/hbase/conf
下修改regionservers
master
-
启动脚本
#!/bin/bash start-hbase.sh
phoenix-hbase-2.3-5.1.0
-
复制jar包
sudo cp /opt/phoenix-hbase-2.3-5.1.0-bin/phoenix-server-hbase-2.3-5.1.0.jar /opt/hbase-2.3.4/lib sudo cp /opt/phoenix-hbase-2.3-5.1.0-bin/phoenix-pherf-5.1.0.jar /opt/hbase-2.3.4/lib
-
修改配置文件,在
/opt/hbase/conf
下修改hbase-site.xml
(在hbase里配置过的跳过此步)<!-- phoenix配置 --> <!-- 支持HBase命名空间映射 --> <property> <name>phoenix.schema.isNamespaceMappingEnabled</name> <value>true</value> </property> <!-- 支持索引预写日志编码 --> <property> <name>hbase.regionserver.wal.codec</name> <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value> </property>
-
复制配置文件
sudo cp /opt/hbase-2.3.4/conf/hbase-site.xml /opt/phoenix-hbase-2.3-5.1.0-bin/bin
-
启动
sqlline.py master:2181 # 输入!table查看Phoenix中的表 !table
测试脚本
-
启动
#!/bin/bash zkServer.sh start start-dfs.sh start-hbase.sh
-
关闭
#!/bin/bash stop-hbase.sh stop-dfs.sh zkServer.sh stop
-
测试