zookeeper+hbase+hive配置

zookeeper配置

配置zoo.cfg

• cd /opt/zookeeper/conf
• vi zoo.cfg (文件名不对则需要修改文件名)

# The number of milliseconds of each tick
tickTime=2000
maxClientCnxns=0    //添加
# The number of ticks that the initial
# synchronization phase can take
initLimit=50    //修改
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/opt/hadoop/zookeeper    //修改存储目录
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
#http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=hadoop101:2888:3888
server.2=hadoop102:2888:3888
server.3=hadoop103:2888:3888    //添加信息，集群内有多少虚拟机就添加多少条，并修改对应的主机名

• 配置完成后将配置文件传输到集群内其他的虚拟机对应目录下，覆盖其原有zoo.cfg文件

scp zoo.cfg root@192.168.44.102:/opt/zookeeper/conf
scp zoo.cfg root@192.168.44.103:/opt/zookeeper/conf

在3个虚拟机上创建zookeeper文件夹

mkdir /opt/hadoop/zookeeper
vi myid  //myid内填充内容,为上边对应的server号,hadoop101里面就填充 1

启动zookeeper(在每台机器上)

cd /opt/zookeeper/bin
./zkServer.sh start

• jps检查zookeeper是否启动成功,查看是否有QuorumPeerMain进程

hbase配置

配置hbase-site.xml

• cd /opt/hbase/conf
• vi hbase-site.xml

<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://192.168.109.101:9000/hbase</value>  //hbase要部署在的namenodeIP
<description>The directory shared by region servers.</description>
</property>
<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
</property>
<property>
  <name>hbase.master.port</name>
  <value>60000</value>
</property>
<property>
  <name>hbase.zookeeper.quorum</name>
  <value>hadoop101,hadoop102,hadoop103</value>    //主机列表
</property>
<property>
  <name>hbase.regionserver.handler.count</name>
  <value>300</value>
</property>
<property>
  <name>hbase.hstore.blockingStoreFiles</name>
  <value>70</value>
</property>
<property>
  <name>zookeeper.session.timeout</name>
  <value>60000</value>
</property>
<property>
<name>hbase.regionserver.restart.on.zk.expire</name>
<value>true</value>
<description>
Zookeeper session expired will force regionserver exit.
Enable this will make the regionserver restart.
</description>
</property>
<property>
  <name>hbase.replication</name>
  <value>false</value>
</property>
<property>
  <name>hfile.block.cache.size</name>
  <value>0.4</value>
</property>
<property>
  <name>hbase.regionserver.global.memstore.upperLimit</name>
  <value>0.35</value>
</property>
<property>
  <name>hbase.hregion.memstore.block.multiplier</name>
  <value>8</value>
</property>
<property>
  <name>hbase.server.thread.wakefrequency</name>
  <value>100</value>
</property>
<property>
  <name>hbase.master.distributed.log.splitting</name>
  <value>false</value>
</property>
<property>
  <name>hbase.regionserver.hlog.splitlog.writer.threads</name>
  <value>3</value>
</property>
<property>
  <name>hbase.hstore.blockingStoreFiles</name>
  <value>20</value>
</property>
<property>
<name>hbase.hregion.memstore.flush.size</name>
<value>134217728</value>
</property>
<property>
  <name>hbase.hregion.memstore.mslab.enabled</name>
  <value>true</value>
</property>
</configuration>

配置hbase.env.sh

• vi hbase.env.sh

export JAVA_HOME=/opt/jdk8    //java路径
export HBASE_CLASSPATH=/opt/hadoop/etc/hadoop    //hadoop配置文件
export HBASE_HEAPSIZE=4000
export HBASE_OFFHEAPSIZE=1G
export HBASE_OPTS="-Xmx4g -Xms4g -Xmn128m -XX:+UseParNewGC  -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -verbose:gc  -XX:+PrintGCDetails -XX:+PrintGCTimeStamps  -Xloggc:$HBASE_HOME/logs/gc-$(hostname)-hbase.log"    //不能超过虚拟机最大内存
export HBASE_MANAGES_ZK=false

配置log4j.properties

• 修改以下代码,更改日志提醒

hbase.root.logger=WARN,console
log4j.logger.org.apache.hadoop.hbase=WARN

配置regionservers

• 添加集群所有主机名

复制所有文件到集群内其他虚拟机内覆盖其文件

scp hbbase-site.xml root@192.168.44.102:/opt/hbase/conf
scp hbbse-env.sh root@192.168.44.102:/opt/hbase/conf
scp log4j.properties root@192.168.44.101:/opt/hbase/conf
scp regionservers root@192.168.44.101:/opt/hbase/conf

启动hbase

cd /opt/hbase/bin
./start-hbase.sh

校验配置是否正确

//jps查看每个节点是否有hbase进程
//输入命令再次校验
hbase shell
hbase>list

配置hive

前提:liunx内有数据库

配置hive.env.sh(若名字不对应则mv改名)

cd /opt/hive/conf
vi hive.env.sh

HADOOP_HOME=/opt/hadoop    //hadoop路径
export HIVE_CONF_DIR=/opt/hive/conf    //hive的conf路径
export HIVE_AUX_JARS_PATH=/opt/hive/lib    //hive的jar包路径
export JAVA_HOME=/opt/jdk8    //jdk安装路径

配置hive-site.xml(若不存在直接新建)

• vi hive-site.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/opt/hive/warehouse</value>
</property>
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<!-- 如果是远程mysql数据库的话需要在这里写入远程的IP或hosts -->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://192.168.44.101:3306/hive_matadata?createDatabaseIfNotExist=true</value>    //数据库所在主机的IP
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>ok</value>
</property>
</configuration>

将java连接数据库的jar包移入hive的lib目录下

• mv /software/mysql-connector-java-5.1.0-bin.jar /opt/hive/lib/

按照hive-site.xml配置的warehouse目录创建warehouse文件夹

• hadoop fs -mkdir -p /opt/hive/warehouse

给warehouse目录和hive目录赋权

• hadoop fs -chmod 777 /opt/hive/warehouse
• hadoop fs -chmod -R 777 /opt/hive/

初始化hive

schematool -dbType mysql -initSchema

启动hive

• hive