scala spark 安装（local）

scala 正常解压 修改vi /etc/profile 配置环境变量
source /etc/profile
即可正常运行（命令：scala）

spark 正常解压 修改 vi /etc/profile 配置环境变量
【选配SPARK_CONF_DIR=$SPARK_HOME/conf】
source /etc/profile
1.
mv spark-env.sh.template spark-env.sh
vi spark-env.sh

export JAVA_HOME=/root/software/jdk1.8.0_221
export SCALA_HOME=/root/software/scala
export SPARK_HOME=/root/software/spark
export HADOOP_HOME=/root/software/hadoop
export HADOOP_CONF_DIR=/root/software/hadoop/etc/hadoop
export SPARK_MASTER_IP=hadoop103
wxport SPARK_EXECUTOR_MEMORY=1G

mv slaves.template slaves
vi slaves
看一下是localhost 不用动

spark-shell 启动spark

3.wordCount
val wordCount=sc.parallelize(Array(“hello world”,“hello java”,“hello scala”,“scala is very funny”))

【
val wordCount=sc.parallelize(Array(“hello world”,“hello java”,“hello scala”,“scala is very funny”，3)) 可指定分区数
wordCount.getNumPartitions 查看分区数量
】

wordCount.flatMap(x=>x.split(" ")).map(x=>(x,1)).reduceByKey(_+_).collect

wordCount.flatMap(x=>x.split(" ")).map(x=>(x,1)).groupBykey().map(x=>(x._1,x._2.size).collect