spark-streaming例子程序

开发spark-streaming从服务器端口实时接收数据进行worldcount；

环境搭建

idea+maven 其pom文件如下：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.yzztech.shushuo</groupId>
    <artifactId>myspark</artifactId>
    <version>1.0-SNAPSHOT</version>


    <repositories>
        <repository>
            <id>cloudera</id>
            <url>http://repository.cloudera.com/artifactory/cloudera-repos</url>
        </repository>
    </repositories>

    <properties>
        <spark.version>1.6.0-cdh5.14.0</spark.version>
        <scala.version>2.10</scala.version>
    </properties>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-sql -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.10</artifactId>
            <version>${spark.version}</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-mllib -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-mllib_2.10</artifactId>
            <version>${spark.version}</version>
            <scope>runtime</scope>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-streaming-kafka_2.10</artifactId>
            <version>1.6.0-cdh5.14.0</version>
        </dependency>







    </dependencies>



</project>

spark版本为1.6；

idea安装略过；

代码如下

val sparkConf = new SparkConf().setAppName("NetworkWordCount")
    val ssc = new StreamingContext(sparkConf, Seconds(5))
    Logger.getLogger("org").setLevel(Level.ERROR)

    // Create a socket stream on target ip:port and count the
    // words in input stream of \n delimited text (eg. generated by 'nc')
    // Note that no duplication in storage level only for running locally.
    // Replication necessary in distributed scenario for fault tolerance.
    val lines = ssc.socketTextStream(args(0), args(1).toInt)
    val words = lines.flatMap(_.split(" "))
    val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)

    
    wordCounts.print()

    ssc.start()
    ssc.awaitTermination()

提交命令

spark-submit --class StreamingTest /home/ubuntu/lixin_test/myspark.jar

//提交任务前发送端口消息
nc -lk 9999

插曲

上面提交命令后默认不是cluster模式也就意味着driver在本地，可以看到输出；但是如果指定cluster模式则不然

 spark-submit --class StreamingTest \
    --master yarn \
    --deploy-mode cluster \
    --driver-memory 1g \
    --executor-memory  1g \
    --executor-cores 2 \
    --driver-cores 2 \
    --num-executors 3 \
    /home/ubuntu/myspark.jar \
    localhost 9999

用以上命令提交则看不到输出，但是任务是正产在running的，如果想看输出的话可以将结果保存到hdfs

wordCounts.saveAsTextFiles("/spark/spark-streaming-out")