学习笔记:从0开始学习大数据-12. spark安装部署

为了教学方便，考虑ALL IN ONE，一台虚拟机构建整个实训环境，因此是伪分布式搭建spark
 环境：

　　hadoop2.6.0-cdh5.15.1

　　jdk1.8

　　centos7 64位

1. 安装scala环境

版本是scala-2.12.7，官网下载地址http://www.scala-lang.org/download/

scala-2.12.7.tgz 
 
tar -zxvf scala-2.12.7.tgz

nano /etc/profile

export SCALA_HOME=/home/linbin/software/scala-2.12.7
export PATH=$PATH:$SCALA_HOME/bin

[root@centos7 bin]# scala -version
Scala code runner version 2.12.7 -- Copyright 2002-2018, LAMP/EPFL and Lightbend, Inc.

2. 下载spark
wget http://archive.cloudera.com/cdh5/cdh/5/spark1.6.0-cdh5.15.1.tar.gz

3.解压 tar -zxvf spark1.6.0-cdh5.15.1.tar.gz

4.进入spark/conf目录，把 spark-env.sh.template文件重命名为 spark-env.sh，并且在文件末尾加上如下配置
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.191.b12-0.el7_5.x86_64
export SCALA_HOME=/home/linbin/software/scala-2.12.7
export SPARK_MASTER_IP=centos7
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/home/linbin/software/hadoop-2.6.0-cdh5.15.1/etc/hadoop
export SPARK_DIST_CLASSPATH=$(/home/linbin/software/hadoop-2.6.0-cdh5.15.1/bin/hadoop classpath)

5. slaves.template文件重命名为slaves，文件的默认内容是localhost，把localhost删除，并添加本机名字centos7

6. 提示log4j类未发现，在其它组件下找到并复制到
sparkXX/lib/slf4j-api-1.7.5.jar

7. worker启动提示如下错误  
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ObjectMapper
解决办法：
把 /home/linbin/software/hadoop-2.6.0-cdh5.15.1/share/hadoop/mapreduce1/lib/目录下的 
jackson-databind-2.2.3.jar 
jackson-core-2.2.3.jar 
jackson-annotations-2.2.3.jar
 复制到/home/linbin/software/hadoop-2.6.0-cdh5.15.1/share/hadoop/common/lib 目录

8.启动  

start-dfs.sh

start-yarn.sh
sparkXX/sbin/start-all.sh    //spark 的start-all.sh  和 hdfs 的 start-all.sh  不要搞混了

看到master和Worker 说明spark启动成功
9.启动spark-shell
出现 java.lang.NoClassDefFoundError: parquet/hadoop/ParquetOutputCommitter
解决：
在sqoop目录找到
sqoop2-1.99.5-cdh5.15.1/server/webapps/sqoop/WEB-INF/lib/parquet-hadoop-1.5.0-cdh5.15.1.jar
复制到/home/linbin/software/hadoop-2.6.0-cdh5.15.1/share/hadoop/common/lib 目录

最后，成功启动

10.通过http监控spark  master 和worker

 

有价值的参考文章：
https://www.jianshu.com/p/a0c38dc46b89   搭建Spark所遇过的坑