单机环境安装Hadoop+hbase+zookeeper+spark+sqoop
软件:
maven
bin version:3.3.9
hadoop
bin version:2.7.1
hbase
bin version:1.1.2
ojdbc6
scala
scala-2.11.7.tgz
Spark
spark-1.5.2-bin-hadoop2.6.tgz
zookeeper 3.4.7
jdk
version:8u65-linux-x64
1.linux修改:
创建免密:
root 和spark 通用
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cp ~/.ssh/id_dsa.pub ~/.ssh/authorized_keys
ssh localhost
(测试免密是否可行)
修改hosts
cp /etc/hosts /etc/hosts.bak
vim /etc/hosts
Like This
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
127.0.0.1 spark
10.9.0.91 spark
当前用户为Root:
mkdir -p /usr/local/software
关闭防火墙:
service iptables stop
chkconfig iptables off
传输数据文件
解压缩到当前文件夹并mv 到/usr/local/bigdata
创建用户
groupadd -g 500 spark
useradd -u 500 -g spark -d /home/spark spark -s /bin/bash
更新系统:
yum update -y
yum install -y zlib* python-devel rsync tigervnc tigervnc-server fontforge
安装VNC软件:
"yum install tigervnc tigervnc-server -y
yum install fontforge -y " 更新系统时已经安装
yum groupinstall Desktop -y
编辑VNC:
vim /etc/sysconfig/vncservers
去掉注释:
VNCSERVERS="2:root 3:spark"
VNCSERVERARGS[2]="-geometry 1024x768 -nolisten tcp -localhost"
VNCSERVERARGS[3]="-geometry 1024x768 -nolisten tcp"
设置密码:
切换到当前用户使用:vncserver 命令设置密码。
例如:oracle用户:
su - oracle
vncserver
重启VNCServer
service vncserver stop
service vncserver start
如果出现could not open default font 'fixed'错误
使用yum install libXfont xorg-x11-xinit xorg-x11-fonts-Type1 安装字符补丁
客户端连接时使用:192.168.1.123:3 此处三代表某个用户 与上边配置数字相同。
编辑.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export HADOOP_HOME=/usr/local/bigdata/hadoop
export HBASE_HOME=/usr/local/bigdata/hbase
export JAVA_HOME=/usr/local/bigdata/jdk
export MAVEN_HOME=/usr/local/bigdata/maven
export SCALA_HOME=/usr/local/bigdata/scala
export SPARK_HOME=/usr/local/bigdata/spark
export ZOOKEEPER_HOME=/usr/local/bigdata/zookeeper
export ORACLE_HOME=/usr/local/bigdata/oracleapp
export HADOOP_PREFIX=/usr/local/bigdata/hadoop
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export PATH=$JAVA_HOME/bin:$SCALA_HOME/bin:$MAVEN/bin:$HOME/bin:$HADOOP/bin:$HADOOP/sbin:$ORACLE_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$HBASE_HOME/bin:$PATH:$SQOOP_HOME/bin:$HIVE_HOME/bin:$ZOOKEEPER_HOME/bin
export SPARK_EXAMPLES_JAR=/usr/local/bigdata/spark/lib/spark-examples-1.5.2-hadoop2.2.0.jar
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/jre/lib/dt.jar:$JAVA_HOME/jre/lib/tools.jar:/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$JAVA_HOME/lib:$SCALA_HOME/lib:$MAVEN_HOME/lib:/usr/local/lib:/usr/lib:$ORACLE_HOME/lib:$ORACLE_HOME:$SPARK_HOME/lib:$HADOOP_HOME/lib:$HADOOP_HOME/lib/native:$HBASE_HOME/lib:$ZOOKEEPER_HOME/lib
2.配置hadoop
1.修改hadoop-env.sh
vim $HADOOP_HOME/etc/hadoop/hadoop-env.sh
1). export JAVA_HOME=/usr/local/bigdata/jdk 绝对路径
2). export HADOOP_COMMON_HOME=/usr/local/bigdata/hadoop 目前没有发现这个参数
3).
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib:$HADOOP_HOME/lib/native" 添加
2.修改/etc/environment
vim /etc/environment
/usr/local/bigdata/hadoop/bin:/usr/local/bigdata/hadoop/sbin:
or
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bigdata/hadoop/bin:/usr/local/bigdata/hadoop/sbin"
启动:
source /etc/environment
3.测试:
mkdir -p $HADOOP_HOME/input
cp $HADOOP_HOME/etc/hadoop/*.xml $HADOOP_HOME/input
4.切换至$HADOOP_HOME执行
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar grep input output 'dfs[a-z.]+'
ll output
可以看到当前文件
-rw-r--r--. 1 root root 11 Dec 14 18:01 part-r-00000
-rw-r--r--. 1 root root 0 Dec 14 18:01 _SUCCESS
5.修改etc/hadoop/core-site.xml文件
含义:接收Client连接的RPC端口,用于获取文件系统metadata信息。
vim $HADOOP_HOME/etc/hadoop/core-site.xml
输入:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
6.修改etc/hadoop/hdfs-site.xml
含义:备份只有一份
vim $HADOOP_HOME/etc/hadoop/hdfs-site.xml
输入:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/spark/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/spark/dfs/data</value>
</property>
</configuration>
7.格式化namenode
$HADOOP_HOME/bin/hdfs namenode -format
启动dfs
$HADOOP_HOME/sbin/start-dfs.sh
8.验证
浏览器内输入:http://localhost:50070/ 可显示页面
9.配置yarn源
编辑etc/hadoop/mapred-site.xml
cp $HADOOP_HOME/etc/hadoop/mapred-site.xml.template $HADOOP_HOME/etc/hadoop/mapred-site.xml
vim $HADOOP_HOME/etc/hadoop/mapred-site.xml
输入:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
10.修改yarn-site.xml
vim $HADOOP_HOME/etc/hadoop/yarn-site.xml
输入:
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
11 启动yarn
$HADOOP_HOME/sbin/start-yarn.sh
浏览器输入:http://10.9.0.91:8088/cluster 查看状态
三、安装外部Zookeeper
1.复制zookeeper的zoo.cfg文件
cp $ZOOKEEPER_HOME/conf/zoo_sample.cfg $ZOOKEEPER_HOME/conf/zoo.cfg
vim $ZOOKEEPER_HOME/conf/zoo.cfg
将dataDir修改为其他文件夹避免机器重启后被删除
原:dataDir=/tmp/zookeeper
替换
新:dataDir=/usr/local/data/zookeeper
mkdir -p /usr/local/data/zookeeper
启动zookeeper
$ZOOKEEPER_HOME/bin/zkServer.sh start
验证:
$ZOOKEEPER_HOME/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/bigdata/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... already running as process 24011.
四.安装Hbase
1.修改配置文件
vim $HBASE_HOME/conf/hbase-env.sh
添加jdk绝对路径
export JAVA_HOME=/usr/local/bigdata/jdk
告诉hbase使用外部Zookeeper
export HBASE_MANAGES_ZK=false
2.修改hbase-site.xml文件
vim $HBASE_HOME/conf/hbase-site.xml
<configuration>
<property>
<name>hbase.master</name>
<value>60000</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>file:///usr/local/data/hbase</value>
</property>
<property>
<name>hbase.cluster.distributed</name> <!--Standalone-->
<value>false</value>
</property>
<property>
<name>hbase.master.info.port</name><!--HBase Master web-->
<value>60010</value>
</property>
<property>
<name>hbase.regionserver.port</name><!--HBase RegionServer-->
<value>60020</value>
</property>
<property>
<name>hbase.regionserver.info.port</name><!--HBase RegionServer web-->
<value>60030</value>
</property>
<property>
<name>hbase.regionserver.info.port</name>
<value>60030</value>
</property>
<property>
<name>hbase.zookeeper.leaderport</name>
<value>3888</value>
</property>
<!--
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
-->
</configuration>
安装配置SPARK:
注意使用Hadoop时一定要使用Spark-with-hadoop版本 不能使用without-hadoop版本否则报错
cp $SPARK_HOME/conf/spark-env.sh.template $SPARK_HOME/conf/spark-env.sh
cp $SPARK_HOME/conf/log4j.properties.template $SPARK_HOME/conf/log4j.properties
vim $SPARK_HOME/conf/spark-env.sh
export SPARK_LOCAL_IP=10.9.0.91
export SPARK_MASTER_PORT=7077
export SPARK_MASTER_OPTS="-Dx=y"
export SPARK_WORKER_CORES=4
export SPARK_WORKER_MEMORY=30g
export SPARK_WORKER_PORT=7077
export SPARK_API_ID=MobileGameAnalyze
export SPARK_WORKER_DIR=/usr/local/data/spark/data
export SPARK_WORKER_OPTS="-Dx=y"
export SPARK_DAEMON_MEMORY=5g
export SPARK_HISTORY_OPTS="-Dx=y"
export SPARK_SHUFFLE_OPTS="-Dx=y"
export SPARK_DAEMON_JAVA_OPTS="-Dx=y"
export JAVA_HOME=${JAVA_HOME}
export SCALA_HOME=${SCALA_HOME}
export HADOOP_HOME=${HADOOP_HOME}
vim $SPARK_HOME/conf/log4j.properties
log4j.rootCategory=INFO, console
替换成
log4j.rootCategory=WARN, console
问题1:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/bigdata/hbase/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/bigdata/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
hadoop的slf4j和hbase lib下的slf4j冲突
修改方案
mv $HBASE_HOME/lib/slf4j-log4j12-1.7.5.jar $HBASE_HOME/lib/slf4j-log4j12-1.7.5.jar.bak
重启hbase后不再报错