Spark源码学习——在Linux环境下使用IDEA看Spark源码
本篇文章主要解决的问题
1.Spark在Linux下实验环境的搭建
一、Spark源码阅读环境的准备
本文介绍的是Centos下的各项配置方法
下面列举搭建此环境需要的各个组件:
- JDK的安装(JDK1.7)
- Scala的安装
- SBT的安装
- Git的安装
- IDEA的安装
最后根据安装的环境,导入源码,进行源码的学习,整片文章会讲的非常细致,适合新手学习参考。
二、JDK的安装
1.在安装CentOS时,系统会自动安装jdk,需要先把它卸载掉,装上自定义的jdk 1.7。
卸载步骤如下:
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># rpm -qa|grep jdk</span> java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-openjdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.6</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.50</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.11</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>.el6_3.x86_64 java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-openjdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.3</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4.1</span>.el6_3.x86_64 [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># rpm -qa|grep gcj</span> java-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>-gcj-compat-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.4</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">40</span>jpp.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">115</span> libgcj-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4.1</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">48</span>.el5 [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove java java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64</span> [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove java java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64</span> [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove java java-1.4.2-gcj-compat-1.4.2.0-40jpp.115 </span> [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@localhost</span> ~]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># yum -y remove libgcj-4.1.2-48.el5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li></ul>
操作完成后,输入java -version后会显示无法查询到Java版本,说明卸载成功。
2.首先从官网下载我们需要的JDK版本,为JDK1.7,
下载链接为:JDK1.7版本下载链接
会涉及到版本选择的问题,我们需要查询一下当前系统的版本,
在命令行输入输入
Uname -a 或 more /proc/version
在内核版本后面会有一个X86_64就是表示64位系统啦
否则就为32位系统,然后根据系统的版本下载相应的JDK。我这边下载的是jdk-7u45-linux-x64.tar.gz。
3.进行JDK的安装
解压源码包
通过终端在/usr/local目录下新建java文件夹,命令行:
<code class="hljs perl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sudo <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">mkdir</span> /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/java</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
然后将下载到压缩包拷贝到java文件夹中,命令行:
进入jdk源码包所在目录
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">cp</span> jdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>u45-linux-x64<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span> /usr/local/java</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
然后进入java目录,命令行:
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">cd</span> /usr/local/java</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
解压压缩包,命令行:
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sudo tar xvf jdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>u45-linux-x64<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
然后可以把压缩包删除,命令行:
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">sudo rm jdk-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7</span>u45-linux-x64<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
设置JDK的环境变量
这里采用全局设置方法,就是修改etc/profile,它是是所有用户的共用的环境变量
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">sudo</span> gedit /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
没有图形界面的请用vi打开。
打开之后在末尾添加
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JAVA_HOME=/usr/local/java/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7.0</span>_79 <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> JRE_HOME=/usr/local/java/jdk1.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">7.0</span>_79/jre <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> CLASSPATH=.:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/lib/dt.jar:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/lib/tools.jar:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JRE_HOME</span>/lib:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$CLASSPATH</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$JAVA_HOME</span>/bin:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li></ul>
请记住,在上述添加过程中,等号两侧不要加入空格,不然会出现“不是有效的标识符”,因为source /etc/profile 时不能识别多余到空格,会理解为是路径一部分。
然后保存
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">source</span> /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
使profile生效
最后在终端输入java -verson,检验是否安装成功。
成功则显示
java version “1.7.0_79”
Java(TM) SE Runtime Environment (build 1.7.0_79-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
至此,JDK已经安装完成。
三、Scala的安装
1.scala下载
首先下载安装包scala下载链接
我下载的是2.10.6版本。
2.解压
解压下载的安装包,并把它移动到指定目录。
<code class="hljs vala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#tar -zxf scala-2.10.6.tgz</span> <span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">#sudo mv scala-2.10.6 /usr/local</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
3.配置环境变量
<code class="hljs vala has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;"># sudo gedit/etc/profile</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
在最后加
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> SCALA_HOME=/usr/local/scala-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.10</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">6</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">export</span> PATH=<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span>:<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SCALA_HOME</span>/bin</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
使配置文件立即生效
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">source</span> /etc/profile</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
4.测试
输入scala -version进行测试。
显示:Scala code runner version 2.10.6…安装成功
四、SBT的安装
1.下载sbt的安装包
安装包地址:sbt安装包地址
2.建立目录,解压文件到所建立目录
<code class="hljs perl has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">$ sudo <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">mkdir</span> /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/sbt $ sudo tar zxvf sbt-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">13.5</span>.tgz -C /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
3.建立启动sbt的脚本文件
/选定一个位置,建立启动sbt的脚本文本文件,如/usr/local/sbt/ 目录下面新建文件名为sbt的文本文件/
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>cd /usr/local/sbt/ <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>vim sbt</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
在sbt文本文件中添加
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">BT_OPTS<span class="hljs-subst" style="color: rgb(0, 0, 0); box-sizing: border-box;">=</span><span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"</span> java <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$SBT_OPTS</span> <span class="hljs-attribute" style="box-sizing: border-box;">-jar</span> /opt/scala/sbt/bin/sbt<span class="hljs-attribute" style="box-sizing: border-box;">-launch</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>jar <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"$@"</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>
然后按esc键 输入 :wq 保存退出,注意路径需要能够正确的定位到解压的sbt文件包中的sbt-launch.jar文件即可
/×修改sbt文件权限×/
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>chmod u+x sbt </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
4.配置PATH环境变量,保证在控制台中可以使用sbt命令
$ vim ~/.bashrc
/在文件尾部添加如下代码后,保存退出/
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">export <span class="hljs-constant" style="box-sizing: border-box;">PATH</span>=<span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/opt/scala</span><span class="hljs-regexp" style="color: rgb(0, 136, 0); box-sizing: border-box;">/sbt/</span><span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$PATH</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
使配置文件立刻生效
<code class="hljs bash has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">$ <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">source</span> ~/.bashrc</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
5.测试sbt是否安装成功
输入sbt -version
第一次执行时,会下载一些文件包,然后才能正常使用,要确保联网了,安装成功后显示如下
[info] Set current project to sbt (in build file:/opt/scala/sbt/)
[info] 0.13.5
至此,sbt已经安装成功。
五、Git的安装
新版本的centos能够通过命令行直接安装
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yum install git</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
但是这种方法虽然简单,但是一般仓库里的版本更新不及时,比如 CentOS 仓库中的 git 最新版是1.7.1,但是 git 官方已经到2.x 的版本了。对于想要获取最新git的系统,只能下rpm包或者用源码。
步骤如下:
1.下载git的安装包
下载安装包:git安装包
2.解压源文件
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">tar <span class="hljs-attribute" style="box-sizing: border-box;">-zxvf</span> git<span class="hljs-attribute" style="box-sizing: border-box;">-latest</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>gz 或者 xz <span class="hljs-attribute" style="box-sizing: border-box;">-d</span> git<span class="hljs-attribute" style="box-sizing: border-box;">-latest</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>xz tar <span class="hljs-attribute" style="box-sizing: border-box;">-xvf</span> git<span class="hljs-attribute" style="box-sizing: border-box;">-latest</span><span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>tar</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
3.编译安装
进入git的目录下,依次运行以下命令
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>autoconf <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>./configure <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>make <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>make install</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>
可能会遇到无法识别autoconf这个命令,则需要先安装下
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yum -y install autoconf</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
在执行make操作时如果报错:缺少gcc,则通过yum安装gcc
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>yum -y install gcc</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
重新执行make操作,还报错,缺少zlib.h。我们可以查看是否存在zlib.h
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>whereis zlib</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
如果没有,则需要安装zlib,这儿不再详细讲述,安装完zlib后,重新执行上面的命令。
4.验证是否成功
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">[root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@sl</span> git-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># whereis git</span> <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">git:</span>/usr/local/bin/git [root<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@sl</span> git-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>]<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># git -version</span> git version <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1.7</span>.<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span> 至此,git已经安装完成。 </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li></ul>
六、IDEA的安装
1.下载安装包
首先需要从官网下载IntelliJ IDEA的安装包。
下载地址为:IDEA下载链接
2.安装
将下载的安装包拷贝到需要安装的位置,
然后进行解压,解压的过程其实就是安装的过程,
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">$ sudo tar -xvf ideaIU-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11.1</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.1</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.tar</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.gz</span> </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
安装比较简单,至此已经安装完成。
七、在IDEA中导入源码
从 Github 导入 Spark 工程
打开IntelliJ IDEA 后,在菜单栏中选择 VCS→Check out from Version Control→Git,之后在 Git Repository URL 中填入 Spark 项目的地址,并指定好本地路径。
<code class="hljs avrasm has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-label" style="box-sizing: border-box;">https:</span>//github<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.com</span>/apache/spark</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
点击该窗口中的的 Clone 后,开始从 Github 中 clone 该项目,该过程试你网速而定,大概需要3-10分钟。
编译 Spark
当 clone 完毕后,IntelliJ IDEA 会自动提示你该项目有对应的 pom.xml 文件,是否打开。这里直接选择 Open 该 pom.xml 文件,然后系统会自动解析项目的相关依赖,该步骤也会因你的网络和系统相关环境,所需时间不同。
该步骤完成之后,请手动编辑 Spark 根目录下的 pom.xml 文件,找到其中指定 java 版本的那一行(java.version),根据你的系统环境,如果使用的是 jdk1.7 ,那么或许需要你将其值改成1.7(默认是1.6)。
之后打开 shell 终端,在命令行中进入刚才导入的 spark 项目根目录,执行
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;"><span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>sbt/sbt assembly</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
该编译命令将全部采用默认的配置来编译 Spark,若想指定相关组件的版本,可以查看 Spark 官网中的 Build-Spark(http://spark.apache.org/docs/latest/building-spark.html),查看所有常用的编译选项。
至此,如果能够看完整篇文章,应该能够完成整个环境的安装。