Hadoop+hbase+zookeeper+spark+sqoop

单机环境安装Hadoop+hbase+zookeeper+spark+sqoop

软件：

maven bin  version:3.3.9 

hadoop bin version:2.7.1

hbase bin version:1.1.2

ojdbc6

scala scala-2.11.7.tgz

Spark spark-1.5.2-bin-hadoop2.6.tgz

zookeeper 3.4.7

jdk version:8u65-linux-x64

1.linux修改:

创建免密：

root 和spark 通用

ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa

cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys

ssh localhost 

(测试免密是否可行)

修改hosts

cp /etc/hosts /etc/hosts.bak

vim /etc/hosts

Like This

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

127.0.0.1   spark

10.9.0.91   spark

当前用户为Root：

mkdir -p /usr/local/software

关闭防火墙：

service iptables stop

chkconfig iptables off

传输数据文件

解压缩到当前文件夹并mv 到/usr/local/bigdata

创建用户

groupadd -g 500 spark

useradd -u 500 -g spark -d /home/spark spark -s /bin/bash

更新系统：

yum update -y 

yum install -y zlib* python-devel rsync tigervnc tigervnc-server fontforge

安装VNC软件：

"yum install tigervnc tigervnc-server -y

yum install fontforge -y " 更新系统时已经安装

yum groupinstall Desktop -y

编辑VNC:

vim /etc/sysconfig/vncservers

去掉注释：

VNCSERVERS="2:root 3:spark"

VNCSERVERARGS[2]="-geometry 1024x768 -nolisten tcp -localhost"

VNCSERVERARGS[3]="-geometry 1024x768 -nolisten tcp"

设置密码：

切换到当前用户使用：vncserver 命令设置密码。

例如：oracle用户：

su - oracle

vncserver

重启VNCServer

service vncserver stop

service vncserver start

如果出现could not open default font 'fixed'错误

使用yum install libXfont xorg-x11-xinit xorg-x11-fonts-Type1 安装字符补丁

客户端连接时使用：192.168.1.123:3 此处三代表某个用户 与上边配置数字相同。

编辑.bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

        . ~/.bashrc

fi

# User specific environment and startup programs

PATH=$PATH:$HOME/bin

export HADOOP_HOME=/usr/local/bigdata/hadoop

export HBASE_HOME=/usr/local/bigdata/hbase

export JAVA_HOME=/usr/local/bigdata/jdk

export MAVEN_HOME=/usr/local/bigdata/maven

export SCALA_HOME=/usr/local/bigdata/scala

export SPARK_HOME=/usr/local/bigdata/spark

export ZOOKEEPER_HOME=/usr/local/bigdata/zookeeper

export ORACLE_HOME=/usr/local/bigdata/oracleapp

export HADOOP_PREFIX=/usr/local/bigdata/hadoop

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

export PATH=$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN/bin:$HOME/bin:$HADOOP/bin:$HADOOP/sbin:$ORACLE_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$HBASE_HOME/bin:$PATH:$SQOOP_HOME/bin:$HIVE_HOME/bin:$ZOOKEEPER_HOME/bin

export SPARK_EXAMPLES_JAR=/usr/local/bigdata/spark/lib/spark-examples-1.5.2-hadoop2.2.0.jar

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/jre/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar:/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/lib:$SCALA_HOME/lib:$MAVEN_HOME/lib:/usr/local/lib:/usr/lib:$ORACLE_HOME/lib:$ORACLE_HOME:$SPARK_HOME/lib:$HADOOP_HOME/lib:$HADOOP_HOME/lib/native:$HBASE_HOME/lib:$ZOOKEEPER_HOME/lib

       

2.配置hadoop

1.修改hadoop-env.sh

vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh

1). export JAVA_HOME=/usr/local/bigdata/jdk 绝对路径

2). export HADOOP_COMMON_HOME=/usr/local/bigdata/hadoop 目前没有发现这个参数

3). export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native" 添加

2.修改/etc/environment

vim /etc/environment

/usr/local/bigdata/hadoop/bin:/usr/local/bigdata/hadoop/sbin:

or 

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bigdata/hadoop/bin:/usr/local/bigdata/hadoop/sbin"

启动：

source /etc/environment

3.测试：

mkdir -p $HADOOP_HOME/input

cp $HADOOP_HOME/etc/hadoop/*.xml $HADOOP_HOME/input

4.切换至$HADOOP_HOME执行

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'

ll output

可以看到当前文件

-rw-r--r--. 1 root root 11 Dec 14 18:01 part-r-00000

-rw-r--r--. 1 root root  0 Dec 14 18:01 _SUCCESS

5.修改etc/hadoop/core-site.xml文件 

含义：接收Client连接的RPC端口，用于获取文件系统metadata信息。

vim $HADOOP_HOME/etc/hadoop/core-site.xml

输入：

<configuration>

  <property>

        <name>fs.defaultFS</name>

        <value>hdfs://localhost:9000</value>

    </property>

</configuration>

6.修改etc/hadoop/hdfs-site.xml

含义：备份只有一份

vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml

输入：

<configuration>

    <property>

        <name>dfs.replication</name>

        <value>1</value>

    </property>

<property>

        <name>dfs.namenode.name.dir</name>

        <value>file:/home/spark/dfs/name</value>

    </property>

    <property>

        <name>dfs.datanode.data.dir</name>

         <value>file:/home/spark/dfs/data</value>

    </property>

</configuration>

7.格式化namenode

$HADOOP_HOME/bin/hdfs namenode -format

启动dfs

$HADOOP_HOME/sbin/start-dfs.sh

8.验证

浏览器内输入：http://localhost:50070/ 可显示页面

9.配置yarn源

编辑etc/hadoop/mapred-site.xml

cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml

vim $HADOOP_HOME/etc/hadoop/mapred-site.xml

输入：

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>

</configuration>

10.修改yarn-site.xml

vim $HADOOP_HOME/etc/hadoop/yarn-site.xml

输入：

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

11 启动yarn

$HADOOP_HOME/sbin/start-yarn.sh

浏览器输入：http://10.9.0.91:8088/cluster 查看状态

三、安装外部Zookeeper

1.复制zookeeper的zoo.cfg文件

cp $ZOOKEEPER_HOME/conf/zoo_sample.cfg $ZOOKEEPER_HOME/conf/zoo.cfg

vim $ZOOKEEPER_HOME/conf/zoo.cfg

将dataDir修改为其他文件夹避免机器重启后被删除

原：dataDir=/tmp/zookeeper

替换

新：dataDir=/usr/local/data/zookeeper

mkdir -p /usr/local/data/zookeeper

启动zookeeper

$ZOOKEEPER_HOME/bin/zkServer.sh start

验证：

$ZOOKEEPER_HOME/bin/zkServer.sh status

ZooKeeper JMX enabled by default

Using config: /usr/local/bigdata/zookeeper/bin/../conf/zoo.cfg

Starting zookeeper ... already running as process 24011.

四.安装Hbase

1.修改配置文件

vim $HBASE_HOME/conf/hbase-env.sh

添加jdk绝对路径

export JAVA_HOME=/usr/local/bigdata/jdk

告诉hbase使用外部Zookeeper

export HBASE_MANAGES_ZK=false

2.修改hbase-site.xml文件

vim $HBASE_HOME/conf/hbase-site.xml

<configuration>

     <property>

         <name>hbase.master</name>

         <value>60000</value>

     </property> 

    <property>

         <name>hbase.rootdir</name>

         <value>file:///usr/local/data/hbase</value>

     </property> 

    <property>

        <name>hbase.cluster.distributed</name> <!--Standalone-->

        <value>false</value>

    </property>

    <property>

        <name>hbase.master.info.port</name><!--HBase Master web-->

        <value>60010</value>

    </property>

<property>

<name>hbase.regionserver.port</name><!--HBase RegionServer-->

<value>60020</value>

</property>

    <property>

        <name>hbase.regionserver.info.port</name><!--HBase RegionServer web-->

        <value>60030</value>

    </property>

<property>

        <name>hbase.regionserver.info.port</name>

        <value>60030</value>

    </property>

<property>

        <name>hbase.zookeeper.leaderport</name>

        <value>3888</value>

    </property>

<!--

<property>

<name>hbase.zookeeper.property.clientPort</name>

<value>2181</value>

</property>

-->

</configuration>

安装配置SPARK：

注意使用Hadoop时一定要使用Spark-with-hadoop版本 不能使用without-hadoop版本否则报错

cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh

cp $SPARK_HOME/conf/log4j.properties.template $SPARK_HOME/conf/log4j.properties

vim $SPARK_HOME/conf/spark-env.sh

export SPARK_LOCAL_IP=10.9.0.91

export SPARK_MASTER_PORT=7077

export SPARK_MASTER_OPTS="-Dx=y"

export SPARK_WORKER_CORES=4

export SPARK_WORKER_MEMORY=30g

export SPARK_WORKER_PORT=7077

export SPARK_API_ID=MobileGameAnalyze

export SPARK_WORKER_DIR=/usr/local/data/spark/data

export SPARK_WORKER_OPTS="-Dx=y"

export SPARK_DAEMON_MEMORY=5g

export SPARK_HISTORY_OPTS="-Dx=y"

export SPARK_SHUFFLE_OPTS="-Dx=y"

export SPARK_DAEMON_JAVA_OPTS="-Dx=y"

export JAVA_HOME=${JAVA_HOME}

export SCALA_HOME=${SCALA_HOME}

export HADOOP_HOME=${HADOOP_HOME}

vim $SPARK_HOME/conf/log4j.properties

log4j.rootCategory=INFO, console

替换成

log4j.rootCategory=WARN, console

问题1：

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/usr/local/bigdata/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

hadoop的slf4j和hbase lib下的slf4j冲突

修改方案

mv $HBASE_HOME/lib/slf4j-log4j12-1.7.5.jar $HBASE_HOME/lib/slf4j-log4j12-1.7.5.jar.bak

重启hbase后不再报错