安装Scala语言和Spark

安装scala 2.11.6

1. 解压tar xvf scala-2.11.6.tgz
2. 迁移目录
   1. sudo mv scala-2.11.6 /usr/local/scala
3. 修改环境变量
   1. sudo vim ~/.bashrc

export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin

4. 使环境变量生效
   1. source ~/.bashrc
5. 启动scala看是否生效
   1. scala

安装Spark2.0.2

1. 解压
   1. tar zxf spark-2.0.2-bin-hadoop2.6.tgz
2. 迁移目录
   1. sudo mv spark-2.0.2-bin-hadoop2.6 /usr/local/spark/
3. 修改环境变量
   1. sudo vim ~/.bashrc

export SPARK_HOME=/usr/local/spark
export PATH=$PATH:$SPARK_HOME/bin

4. 使环境变量生效
   1. source ~/.bashrc
5. 启动spark看是否生效
   1. pyspark
6. 设置pysdpark显示信息, 只显示警告信息
   1. cd /usr/local/spark/conf
   2. cp log4j.properties.template log4j.properties
   3. sudo vim log4j.properties
      1.log4j.rootCategory=INFO, console 改为 log4j.rootCategory=WARN, console