阿里云搭建大数据平台（5）：scala和spark安装和配置

一、scala

1.下载scala-2.11.8.tgz

tar -zxvf /opt/softwares/scala-2.11.8.tgz -C ./

2.修改/etc/profile

sudo vi /etc/profile

添加：

export SCALA_HOME=/opt/modules/scala-2.11.8
export PATH=$SCALA_HOME/bin:$PATH

3.即时生效

source /etc/profile

二、spark

1.安装配置

（1）解压缩spark

tar -zxvf spark-2.1.0-bin-hadoop2.6.tgz -C /opt/modules/

（2）复制配置文件/opt/modules/spark-2.1.0-bin-hadoop2.6/conf

cp spark-env.sh.template spark-env.sh

（3）修改配置文件 spark-env.sh

JAVA_HOME=/opt/modules/jdk1.8.0_151
SCALA_HOME=/opt/modules/scala-2.11.8

HADOOP_CONF_DIR=/opt/modules/hadoop-2.6.0-cdh5.15.0/etc/hadoop
SPARK_LOCAL_IP=hadoop

（4）启动hdfs（一定，否则出现问题！）

（5）修改 slaves 文件

cp slaves.template slaves

添加：【这个是内网ip】

hadoop

（6）重命名log4j.properties

cp log4j.properties.template log4j.properties

2.测试

（1）启动服务【master/worker】

sbin/start-all.sh

    结果：jps

        [hadoop@hadoop spark-2.1.0-bin-hadoop2.6]$ jps
        3744 NameNode
        8084 Jps
        8036 Worker
        7975 Master
        4008 SecondaryNameNode
        3865 DataNode

（2）启动自带测试案例

./bin/run-example SparkPi 50
        【结果：测试成功！】

（3）启动客户端/opt/modules/spark-2.1.0-bin-hadoop2.6/下面

./bin/spark-shell

结果：

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/11/02 00:14:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/11/02 00:15:25 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/11/02 00:15:26 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
18/11/02 00:15:40 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://172.19.199.187:4040
Spark context available as 'sc' (master = local[*], app id = local-1541088862914).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0
      /_/
         
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_151)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

（5）查看端口

http://hadoop:4040

备注：hadoop此处为公网ip

(成功~）