spark起步

字数统计案列：（以Scala为基础编程语言）

object wordCount {
  def main(args: Array[String]): Unit = {
    //做出sc
    val conf=new SparkConf()
    conf.setMaster("local")
    conf.setAppName("wordCount")
    val sc=new SparkContext(conf)
    //2.操作sc,做出RDD
//    val lines:RDD[String]=sc.textFile(args(0))
    val localPath = "c:/kms10.log"
    val linuxPath="hdfs://hadoop131:8020/input/cc.txt"
    val lines: RDD[String]=sc.textFile(linuxPath)
    //操作RDD count
    val resCount:Long =lines.count()
    println(resCount)
    //字数统计
  val words:RDD[String]=lines.flatMap(_.split(","))
    val wordAndOne: RDD[(String, Int)]=words.map((_,1))
    val wordAndCount: RDD[(String, Int)]=wordAndOne.reduceByKey( _ + _)
    wordAndCount.saveAsTextFile( "hdfs://hadoop131:8020/out")
    //    wordAndCount.saveAsTextFile( ":/out")
    sc.stop()
  }
}

相关地址以自己的虚拟机地址为准

想关依赖：（pom文件）

<dependencies>
    <dependency>
        <groupId>org.scala-lang</groupId>
        <artifactId>scala-library</artifactId>
        <version>2.11.8</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.8.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <version>2.2.0</version>
    </dependency>
</dependencies>

打包配置

<build>
        <pluginManagement>
            <plugins>
                <!-- 编译scala的插件 -->
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.2.2</version>
                </plugin>
                <!-- 编译java的插件 -->
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>3.5.1</version>
                </plugin>
            </plugins>
        </pluginManagement>
        <plugins>
            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>compile</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <!-- 打jar插件 -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <filters>
                                <filter>
                                    <artifact>*:*</artifact>
                                    <excludes>
                                        <exclude>META-INF/*.SF</exclude>
                                        <exclude>META-INF/*.DSA</exclude>
                                        <exclude>META-INF/*.RSA</exclude>
                                    </excludes>
                                </filter>
                            </filters>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

启动spark编辑的命令：（将相应的配置换成自己的集群配置）

./spark-shell --master spark://hadoop131:7077 --executor-memory 500m --total-executor-cores 1

打包上传：（打的jar地址以自己的地址为准）

./spark-submit --master spark://hadoop131:7077 --class wordCount /opt/rh/original-bb-1.0-SNAPSHOT.jar hdfs://hadoop131:8020/input/cc.txt hdfs://hadoop131:8020/out1