spark安装与使用(Standalone模式)
安装环境: Ubuntu sever版 ,java ,scala,
一:在linux下安装java环境(自行安装jdk)
二:安装Scala2.9.3
$ tar -zxf scala-2.9.3.tgz
$ sudo mv scala-2.9.3 /usr/lib
$ sudo vim /etc/profile
# add the following lines at the end
export SCALA_HOME=/usr/lib/scala-2.9.3
export PATH=$PATH:$SCALA_HOME/bin
# save and exit vim
#make the bash profile take effect immediately
source /etc/profile
# test
$ scala -version
三:安装spark
从官网下载最新版本的spark,截止目前最新版的是1.5.1.下载地址:http://spark.apache.org/downloads.html
记住选择预编译好的文件下载,选择Pre-build for Hadoop 2.6 and later,下载的文件为spark-1.5.1-bin-hadoop2.6.tgz
解压
$ tar -zxf spark-1.5.1-bin-hadoop2.6.tgz
设置SPARK_EXAMPLES_JAR 环境变量
$ vim ~/.bashrc
# add the following lines at the end
export SPARK_EXAMPLES_JAR=$HOME/spark-0.7.2/examples/target/scala-2.9.3/spark-examples_2.9.3-0.7.2.jar
# save and exit vim
#make the bash profile take effect immediately
$ source /etc/profile
这一步其实最关键,很不幸的是,官方文档和网上的博客,都没有提及这一点。我是偶然看到了这两篇帖子,Running SparkPi, Null pointer exception when running ./run spark.examples.SparkPi local,才补上了这一步,之前死活都无法运行SparkPi。
(可选)设置 SPARK_HOME环境变量,并将SPARK_HOME/bin加入PATH
$ vim ~/.bashrc
# add the following lines at the end
export SPARK_HOME=$HOME/spark-0.7.2
export PATH=$PATH:$SPARK_HOME/bin
# save and exit vim
#make the bash profile take effect immediately
$ source /etc/profile
后来安装以上两步感觉没用,但还是照做了。spark和hadoop是一样的,解压即可使用。
单机运行spark
四:Spark配置
配置Spark环境变量
cd $SPARK_HOME/conf
cp spark-env.sh.template spark-env.sh
vi spark-env.sh 添加以下内容:
<code class="prettyprint" style="box-sizing: border-box; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Courier New', monospace, serif;font-size:undefined; padding: 0px; color: inherit; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; margin: 0px; border: 0px; outline: 0px; vertical-align: baseline; position: relative; overflow: auto; word-break: break-word; max-height: 600px; display: block; background-color: rgb(247, 247, 247);"><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> JAVA_HOME</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="str" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">/usr/</span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">local</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">java</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">1.7</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">0</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> HADOOP_HOME</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="str" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">/opt/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">hadoop</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">2.3</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">0</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">cdh5</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">0.0</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> HADOOP_CONF_DIR</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="str" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">/etc/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">hadoop</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">conf </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SCALA_HOME</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="str" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">/usr/</span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">local</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">scala</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">2.11</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">4</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_HOME</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="str" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">/home/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">lxw1234</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">spark</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">1.3</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">1</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">bin</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">hadoop2</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">3</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_MASTER_IP</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">127.0</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">0.1</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_MASTER_PORT</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">7077</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_MASTER_WEBUI_PORT</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">8099</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_WORKER_CORES</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">3</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="com" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: gray; background-color: transparent;">//每个Worker使用的CPU核数</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_WORKER_INSTANCES</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">1</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="com" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: gray; background-color: transparent;">//每个Slave中启动几个Worker实例</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_WORKER_MEMORY</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">10G</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="com" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: gray; background-color: transparent;">//每个Worker使用多大的内存</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_WORKER_WEBUI_PORT</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">8081</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="com" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: gray; background-color: transparent;">//Worker的WebUI端口号</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_EXECUTOR_CORES</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">1</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="com" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: gray; background-color: transparent;">//每个Executor使用使用的核数</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_EXECUTOR_MEMORY</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">1G</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="com" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: gray; background-color: transparent;">//每个Executor使用的内存</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_CLASSPATH</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="str" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">/opt/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">hadoop</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">lzo</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">current</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">hadoop</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">lzo</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">jar </span><span class="com" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: gray; background-color: transparent;">//由于要用到lzo,因此需要配置</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> SPARK_CLASSPATH</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">$SPARK_CLASSPATH</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">:</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">$CLASSPATH </span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">export</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> LD_LIBRARY_PATH</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">=</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">$</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">{</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">LD_LIBRARY_PATH</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">}:</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">$HADOOP_HOME</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">lib</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="kwd" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(0, 0, 139); background-color: transparent;">native</span></code>
- 配置Slave
cp slaves.template slaves
vi slaves 添加以下内容:
localhost
五、配置免密码ssh登陆
因为Master和Slave处于一台机器,因此配置本机到本机的免密码ssh登陆,如有其他Slave,都需要配置Master到Slave的无密码ssh登陆。
<code class="prettyprint" style="box-sizing: border-box; font-family: Consolas, Menlo, Monaco, 'Lucida Console', 'Liberation Mono', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Courier New', monospace, serif;font-size:undefined; padding: 0px; color: inherit; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; margin: 0px; border: 0px; outline: 0px; vertical-align: baseline; position: relative; overflow: auto; word-break: break-word; max-height: 600px; display: block; background-color: rgb(247, 247, 247);"><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">cd </span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">~/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> ssh</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">-</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">keygen </span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">(一路回车)</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> cd </span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">ssh</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">/</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> cat id_rsa</span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">.</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">pub </span><span class="pun" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;">>></span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> authorized_keys chmod </span><span class="lit" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; color: rgb(128, 0, 0); background-color: transparent;">600</span><span class="pln" style="box-sizing: border-box; margin: 0px; padding: 0px; font-family: 'Microsoft YaHei'; border: 0px; outline: 0px; font-size: 12px; vertical-align: baseline; background-color: transparent;"> authorized_keys</span></code>
六、启动Spark Master
cd $SPARK_HOME/sbin/
./start-master.sh
启动日志位于 $SPARK_HOME/logs/目录下,正常启动的日志如下:
15/06/05 14:54:16 INFO server.AbstractConnector: Started SelectChannelConnector@localhost:6066
15/06/05 14:54:16 INFO util.Utils: Successfully started service on port 6066.
15/06/05 14:54:16 INFO rest.StandaloneRestServer: Started REST server for submitting applications on port 6066
15/06/05 14:54:16 INFO master.Master: Starting Spark master at spark://127.0.0.1:7077
15/06/05 14:54:16 INFO master.Master: Running Spark version 1.3.1
15/06/05 14:54:16 INFO server.Server: jetty-8.y.z-SNAPSHOT
15/06/05 14:54:16 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:8099
15/06/05 14:54:16 INFO util.Utils: Successfully started service ‘MasterUI’ on port 8099.
15/06/05 14:54:16 INFO ui.MasterWebUI: Started MasterWebUI at http://127.1.1.1:8099
15/06/05 14:54:16 INFO master.Master: I have been elected leader! New state: ALIVE
七、启动Spark Slave
cd $SPARK_HOME/sbin/
./start-slaves.sh
会根据$SPARK_HOME/conf/slaves文件中配置的主机,逐个ssh过去,启动Spark Worker
成功启动后,在WebUI界面上可以看到,已经有Worker注册上来了,如图:
在浏览器输入:http://192.168.1.84:8080/ (前面为master的ip地址)
八、简单小实例(统计文件中出现最多的50个单词)
var srcFile = sc.textFile("/usr/local/kern.log")
var a = srcFile.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_)
a.map(word=>(word._2,word._1)).sortByKey(false).map(word=>(word._2,word._1)).take(50).foreach(println)
结果打印在终端:
在4040端口可查看job的情况 http://192.168.1.84:4040/jobs/
八、Spark Java programming (Spark and Spark Streaming)
2:Spark Streaming, 读取kafka数据做单词统计。
注意几点:
由于在pom.xml中指明了入口类,因此不用加--class,如果没有指明,在命令中要用--class 指明入口。