hadoop spark hbase 单机安装

关闭防火墙

如是在虚拟机或是docker里安装,请一定先关闭防火墙,否则外部系统无法访问.

systemctl status firewalld.service #检查防火墙状态
systemctl stop firewalld.service #关闭防火墙
systemctl disable firewalld.service #禁止开机启动防火墙

vim /etc/hosts

如果没有足够的权限，可以切换用户为root。
三台机器的内容统一增加以下host配置：
可以通过hostname来修改服务器名称为master、slave1、slave2 ,此步也是为将来集群服务的

192.168.71.242 master

时间同步

yum install -y ntp #安装ntp服务
ntpdate cn.pool.ntp.org #同步网络时间

需要开放的外网端口

50070，8088，60010 ，7077

解压安装包

tar -zxvf /usr/jxx/scala-2.12.4.tgz -C /usr/local/

tar -zxvf /usr/jxx/spark-2.2.0-bin-hadoop2.7 -C /usr/local/

tar -zxvf /usr/jxx/hbase-1.3.1-bin.tar.gz -C /usr/local/

tar -zxvf /usr/jxx/hadoop-2.8.2.tar.gz -C /usr/local/

创建文件目录

为了便于管理，给Master的hdfs的NameNode、DataNode及临时文件，在用户目录下创建目录

mkdir -p /data/hdfs/name
mkdir -p /data/hdfs/data
mkdir -p /data/hdfs/tmp

如果是集群那将这些目录通过scp命令拷贝到Slave1和Slave2的相同目录下

设置环境变量

vim /etc/profile 添加

export JAVA_HOME=/usr/local/jdk1.8.0 #jdk如果已有就不用添加
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export SCALA_HOME=/usr/local/scala-2.12.4
export PATH=$PATH:$SCALA_HOME/bin
export HADOOP_HOME=/usr/local/hadoop-2.8.2
export PATH=$PATH:$HADOOP_HOME/bin
export HBASE_HOME=/usr/local/hbase-1.3.1
export PATH=$PATH:$HBASE_HOME/bin
export SPARK_HOME=/usr/local/spark-2.2.0-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin

然后执行

source /etc/profile

修改配置

vim /usr/local/hadoop-2.8.2/etc/hadoop/hadoop-env.sh
修改

export JAVA_HOME=/usr/local/java/jdk1.8.0

vim /usr/local/hadoop-2.8.2/etc/hadoop/core-site.xml

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>file:/data/hdfs/tmp</value>
  <description>A base for other temporary directories.</description>
</property>
<property>
  <name>io.file.buffer.size</name>
  <value>131072</value>
</property>
<property>
  <name>fs.default.name</name>
  <value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
</configuration>

vim /usr/local/hadoop-2.8.2/etc/hadoop/hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
  <value>2</value>
</property>
<property>
  <name>dfs.namenode.name.dir</name>
  <value>file:/data/hdfs/name</value>
  <final>true</final>
</property>
<property>
  <name>dfs.datanode.data.dir</name>
  <value>file:/data/hdfs/data</value>
  <final>true</final>
</property>
<property>
  <name>dfs.namenode.secondary.http-address</name>
  <value>master:9001</value>
</property>
<property>
  <name>dfs.webhdfs.enabled</name>
  <value>true</value>
</property>
<property>
  <name>dfs.permissions</name>
  <value>false</value>
</property>
</configuration>

vim /usr/local/hadoop-2.8.2/etc/hadoop/yarn-site.xml

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.resourcemanager.address</name>
  <value>master:18040</value>
</property>
<property>
  <name>yarn.resourcemanager.scheduler.address</name>
  <value>master:18030</value>
</property>
<property>
  <name>yarn.resourcemanager.webapp.address</name>
  <value>master:18088</value>
</property>
<property>
  <name>yarn.resourcemanager.resource-tracker.address</name>
  <value>master:18025</value>
</property>
<property>
  <name>yarn.resourcemanager.admin.address</name>
  <value>master:18141</value>
</property>
<property>
  <name>yarn.nodemanager.aux-services</name>
  <value>mapreduce_shuffle</value>
</property>
</configuration>

cp mapred-site.xml.template mapred-site.xml
vim mapred-site.xml

<configuration>

<property>
  <name>mapreduce.framework.name</name>
  <value>yarn</value>
</property>

</configuration>

vim /usr/local/hbase-1.3.1/conf/hbase-site.xml

<property>

<name>hbase.rootdir</name>

<!-- 对应于hdfs中配置 micmiu.com -->

<value>hdfs://localhost:9000/hbase</value>

</property>

<property>

<name>hbase.cluster.distributed</name>

<value>true</value>

</property>

vim /usr/local/hbase-1.3.1/conf/hbase-env.sh

export JAVA_HOME=/usr/local/jdk1.8.0 #jdk如果已有就不用添加
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export SCALA_HOME=/usr/local/scala-2.12.4
export PATH=$PATH:$SCALA_HOME/bin
export HADOOP_HOME=/usr/local/hadoop-2.8.2
export PATH=$PATH:$HADOOP_HOME/bin
export HBASE_HOME=/usr/local/hbase-1.3.1
export PATH=$PATH:$HBASE_HOME/bin
export SPARK_HOME=/usr/local/spark-2.2.0-bin-hadoop2.7
export PATH=$PATH:$SPARK_HOME/bin
export HBASE_MANAGES_ZK=true

mv /usr/local/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh.template /usr/local/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh

mv /usr/local/spark-2.2.0-bin-hadoop2.7/conf/spark-defaults.conf.template /usr/local/spark-2.2.0-bin-hadoop2.7/conf/spark-defaults.conf

mkdir -p /disk/spark

vim /usr/local/spark-2.2.0-bin-hadoop2.7/conf/spark-env.sh

export JAVA_HOME=/usr/local/jdk1.8.0
export SCALA_HOME=/usr/local/scala-2.12.4
export HADOOP_HOME=/usr/local/hadoop-2.8.2
export HBASE_HOME=/usr/local/hbase-1.3.1
export SPARK_HOME=/usr/local/spark-2.2.0-bin-hadoop2.7
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_LOCAL_DIRS=/disk/spark
export SPARK_DAEMON_MEMORY=256m
export SPARK_HISTORY_OPTS="$SPARK_HISTORY_OPTS -Dspark.history.fs.logDirectory=/tmp/spark -Dspark.history.ui.port=18082"
export STANDALONE_SPARK_MASTER_HOST=localhost

vim /usr/local/spark-2.2.0-bin-hadoop2.7/conf/spark-defaults.conf

spark.master=spark://localhost:7077
spark.eventLog.dir=/dask/spark/applicationHistory

spark.eventLog.enabled=true

spark.yarn.historyServer.address=localhost:18082

初始化环境

格式化namenode
hdfs namenode -format

启动服务

启动hdfs

sh /usr/local/hadoop-2.8.2/sbin/start-dfs.sh

启动hbase

sh /usr/local/hbase-1.3.1/bin/start-hbase.sh

启动spark

sh /usr/local/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh

7 设置开机启动(这个不太好使)

su - root -c “sh /usr/local/hadoop-2.8.2/sbin/start-dfs.sh”

su - root -c “sh /usr/local/hbase-1.3.1/bin/start-hbase.sh”

su - root -c “sh /usr/local/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh”

运行Hbase shell时报错：

hbase shell
hbase(main):006:0> list
TABLE
ERROR: Can’t get master address from ZooKeeper; znode data == null

Here is some help for this command:
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

解决办法:

重新启动,stop-hbase.sh->start-hbase.sh 即解决