有个笑话是这样的,
程序员小明说,我的女朋友,我就得让她给我买好菜,烧好饭,洗好衣服,叠好被子......
然后运营的妹子轻轻得问了一句,那是不是首先得有一个女朋友?#¥$^%@.....
没错,同样的道理,要Spark实践,我们首先得有个Spark,哪怕他是个单点的。
参考官网:http://spark.apache.org/docs/latest/index.html
Spark单点安装流程:
环境:Cetos 7
Java:1.8 (需要自己在系统内安装好,设置好环境变量,这里不再展开。
1. 检查JAVA环境是否安装好,如果JAVA_HOME目录没有东西,那就得先弄好Java环境
[root@FSB-Centos-213 spark-2.2.1-bin-hadoop2.7]# echo $JAVA_HOME
/usr/local/java/jdk1.8.0_131
2.进入某个指定的目录,下载spark包,版本挑个最新的就可以
cd /data1/software/spark
wget http://mirrors.shu.edu.cn/apache/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
3. 下载完成之后,解压包
tar xvf spark-2.2.1-bin-hadoop2.7.tgz
4. 解压之后,进入spark的目录,可以看到该目录下有以下的文件和目录,如红色字体所示,这些目录的命名都很常规,基本是不言而喻的,比如bin,肯定是存命令的,conf肯定存配置的,这里主要关注一下README.md文件,这个文件一般都是使用手册,如果英文还可以的话,直接看这个文件就可以安装了,而不用整天去百度一下。
[root@FSB-Centos-213 spark]# cd spark-2.2.1-bin-hadoop2.7
[root@FSB-Centos-213 spark-2.2.1-bin-hadoop2.7]# ls
bin conf data examples jars LICENSE licenses NOTICE python R README.md RELEASE sbin yarn
5. 启动一个spark-shell,由于spark是scala写的,所以,肯定会支持scala shell,当然他也支持python,由于作者对scala语言不是非常熟悉,所以这里用python shell,这个shell启动之后,spark的但个节点也随之启动了。
[root@FSB-Centos-213 spark-2.2.1-bin-hadoop2.7]# bin/pyspark
Python 2.6.6 (r266:84292, Nov 22 2013, 12:16:22)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/02/28 11:08:58 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/02/28 11:08:59 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
/data1/software/spark/spark-2.2.1-bin-hadoop2.7/python/pyspark/context.py:195: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0")
18/02/28 11:09:06 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/02/28 11:09:07 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
18/02/28 11:09:07 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.2.1
/_/
Using Python version 2.6.6 (r266:84292, Nov 22 2013 12:16:22)
SparkSession available as 'spark'.
>>>
6. 在系统另外一个shell(不是python shell,就是普通的linux的shell)看一下进程,明显spark已经启动
[root@FSB-Centos-213 spark-2.2.1-bin-hadoop2.7]#
[root@FSB-Centos-213 spark-2.2.1-bin-hadoop2.7]#
[root@FSB-Centos-213 spark-2.2.1-bin-hadoop2.7]# ps axu|grep spark
root 7662 0.0 0.0 106080 1440 pts/2 T 10:39 0:00 bash bin/spark-shell
root 8652 14.0 2.7 4721436 447816 pts/0 Sl+ 11:08 0:16 /usr/local/java/jdk1.8.0_131/bin/java -cp /data1/software/spark/spark-2.2.1-bin-hadoop2.7
7. 退出spark的python shell的时候,如果需要将spark进程也杀了, 那就用 quit() 方法,如果直接CTRL + C 去暴力退出python shell,进程一直留在后台
>>> quit()
[root@FSB-Centos-213 spark-2.2.1-bin-hadoop2.7]#
整个安装过程基本就完成了,总体而言,安装个spark,比小明同学有个女朋友不知道简单了多少倍。