SparkStreaming实验错误

转载 2015年11月18日 16:17:00

转自:http://blog.csdn.net/stark_summer/article/details/49251709

NetworkWordCount代码

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">/*
* Licensed <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> the Apache Software Foundation (ASF) under one <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">or</span> more
* contributor license agreements. See the NOTICE file distributed <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span>
* this work <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> additional information regarding copyright ownership.
* The ASF licenses this file <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> You under the Apache License, Version <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2.0</span>
* (the <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"License"</span>); you may not <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">use</span> this file except <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> compliance <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span>
* the License. You may obtain a copy <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">of</span> the License at
*
* http:<span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//www.apache.org/licenses/LICENSE-2.0</span>
*
* Unless required by applicable law <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">or</span> agreed <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> writing, software
* distributed under the License is distributed on an <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"AS IS"</span> BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">or</span> implied.
* See the License <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> the specific language governing permissions <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span>
* limitations under the License.
*/
package org.apache.spark.examples.streaming
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
/**
* Counts words <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">in</span> UTF8 encoded, '\n' delimited text received from the network every second.
*
* Usage: NetworkWordCount <hostname> <port>
* <hostname> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <port> describe the TCP server that Spark Streaming would connect <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> receive data.
*
* To run this on your local machine, you need <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">to</span> first run a Netcat server
* `$ nc -lk <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>`
* <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">then</span> run the example
* `$ bin/run-example org.apache.spark.examples.streaming.NetworkWordCount localhost <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>`
*/
object NetworkWordCount {
    def main(args: Array[String]) {
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (args.length < <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>) {
        System.err.println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Usage: NetworkWordCount <hostname> <port>"</span>)
        System.exit(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)
    }
    StreamingExamples.setStreamingLogLevels()
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Create the context with a 1 second batch size</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// 创建SparkConf实例</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"NetworkWordCount"</span>)

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">///创建Spark Streaming Context,每隔1秒钟处理一批数据,那么这一秒收集的数据存放在哪,如何将收集的数据推送出去?是生产者主动推出去还是消费者每隔1秒钟来拉取一次数据</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> ssc = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StreamingContext(sparkConf, Seconds(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>))

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Create a socket stream on target ip:port and count the</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// words in input stream of \n delimited text (eg. generated by 'nc')</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Note that no duplication in storage level only for running locally.</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Replication necessary in distributed scenario for fault tolerance.</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> lines = ssc.socketTextStream(args(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), args(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>).toInt, StorageLevel.MEMORY_AND_DISK_SER)
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//flatMap是把将每一行使用空格做分解,那么words对应的数据结构是怎么样的?</span>
        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">///words是个集合,每个集合元素依然是个集合,这个集合存放单词</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> words = lines.flatMap(_.split(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">" "</span>))
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> wordCounts = words.map(x => (x, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)).reduceByKey(_ + _)
    wordCounts.print()
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//启动计算作业</span>
    ssc.start()

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//等待结束,什么时候结束作业,即触发什么条件会让作业执行结束</span>
    ssc.awaitTermination()
   }
}</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li></ul>

运行NetworkWordCount

  • 启动NetCat命令

    <code class="hljs ruby has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">[hadoop<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">@hadoop</span> ~]<span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$ </span>nc -lk <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>
    <span class="hljs-constant" style="box-sizing: border-box;">This</span> is a book </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li></ul>

    启动后,可以在后面输入文本,比如This is a book

  • 在另外一个终端,Spark提交NetworkWordCount任务

    <code class="hljs avrasm has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">./bin/run-example streaming<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.NetworkWordCount</span> localhost <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

启动后,发现每隔一秒钟,打印一行日志,例如

<code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">   <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953009000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953010000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953011000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">12</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953012000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">13</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953013000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">14</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953014000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953015000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">16</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953016000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">17</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953017000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">18</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953018000</span> ms

   <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"> ///Added jobs是个什么概念?不就是一个Job吗</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">19</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953019000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953020000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">21</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953021000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">22</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953022000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">23</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953023000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">24</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953024000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">25</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953025000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">26</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953026000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">27</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953027000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">10</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">28</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420953028000</span> ms</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li></ul>
  • 在nc -lk运行的终端,输入文本,发现Spark作业没有将文本输出 
    不过通过nc -lk输入一行文本后,控制台会显示如下信息

    <code class="hljs livecodeserver has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.MemoryStore: ensureFreeSpace(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span>) called <span class="hljs-operator" style="box-sizing: border-box;">with</span> curMem=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">91198</span>, maxMem=<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">280248975</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.MemoryStore: Block input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span> stored <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">as</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">bytes</span> <span class="hljs-operator" style="box-sizing: border-box;">in</span> memory (estimated size <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11.0</span> B, free <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">267.2</span> MB)
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO scheduler.JobScheduler: Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> <span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">time</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954148000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.BlockManagerInfo: Added input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span> <span class="hljs-operator" style="box-sizing: border-box;">in</span> memory <span class="hljs-command" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">on</span> <span class="hljs-title" style="box-sizing: border-box;">localhost</span>:<span class="hljs-title" style="box-sizing: border-box;">57786</span> (<span class="hljs-title" style="box-sizing: border-box;">size</span>: <span class="hljs-title" style="box-sizing: border-box;">11</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> <span class="hljs-title" style="box-sizing: border-box;">B</span>, <span class="hljs-title" style="box-sizing: border-box;">free</span>: <span class="hljs-title" style="box-sizing: border-box;">267</span><span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.2</span> <span class="hljs-title" style="box-sizing: border-box;">MB</span>)</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO storage.BlockManagerMaster: Updated info <span class="hljs-operator" style="box-sizing: border-box;">of</span> block input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">29</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">08</span> INFO receiver.BlockGenerator: Pushed block input-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420954147800</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li></ul>
  • 关闭nc命令的执行,此时,Spark报错,提示9999端口连接不上

    <code class="hljs avrasm has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">///相对于数据源而言,Spark是Receiver,所以Spark Streaming有Receiver这个模块
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">36</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Stopped receiver <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">37</span> INFO scheduler<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.JobScheduler</span>: Added jobs for time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420952977000</span> ms
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO scheduler<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.JobScheduler</span>: Added jobs for time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1420952978000</span> ms
    ///重提开始Receiver
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Starting receiver again
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Starting receiver
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Called receiver onStart
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO scheduler<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverTracker</span>: Registered receiver for stream <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> from akka://sparkDriver
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Receiver started again
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> INFO dstream<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocketReceiver</span>: Connecting to localhost:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>
    <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">15</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">01</span>/<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">11</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> WARN receiver<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.ReceiverSupervisorImpl</span>: Restarting receiver with delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span> ms: Error connecting to localhost:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>
    <span class="hljs-label" style="box-sizing: border-box;">java.net.ConnectException:</span> Connection refused
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.PlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.socketConnect</span>(Native Method)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AbstractPlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.doConnect</span>(AbstractPlainSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">339</span>)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AbstractPlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connectToAddress</span>(AbstractPlainSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">200</span>)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.AbstractPlainSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(AbstractPlainSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">182</span>)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocksSocketImpl</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(SocksSocketImpl<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">392</span>)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">579</span>)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.connect</span>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">528</span>)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span>.<init>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">425</span>)
        at java<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.net</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.Socket</span>.<init>(Socket<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.java</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">208</span>)
        at org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.streaming</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.dstream</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocketReceiver</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.receive</span>(SocketInputDStream<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.scala</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">71</span>)
        at org<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.apache</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.spark</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.streaming</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.dstream</span><span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.SocketReceiver</span>$$anon$2<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.run</span>(SocketInputDStream<span class="hljs-preprocessor" style="color: rgb(68, 68, 68); box-sizing: border-box;">.scala</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">57</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li></ul>

关于接不到消息

下面的代码收不到消息

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SparkStreamingExample"</span>).setMaster(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"local"</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

而下面的代码则能收到消息

<code class="hljs fsharp has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SparkStreamingExample"</span>).setMaster(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"local[2]"</span>)</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

原因来自于http://spark.apache.org/docs/latest/streaming-programming-guide.html: 
When running a Spark Streaming program locally, do not use “local” or “local[1]” as the master URL. Either of these means that only one thread will be used for running tasks locally. If you are using a input DStream based on a receiver (e.g. sockets, Kafka, Flume, etc.), then the single thread will be used to run the receiver, leaving no thread for processing the received data. Hence, when running locally, always use “local[n]” as the master URL where n > number of receivers to run (see Spark Properties for information on how to set the master). 
Extending the logic to running on a cluster, the number of cores allocated to the Spark Streaming application must be more than the number of receivers. Otherwise the system will receive data, but not be able to process them.


关于Receiver的线程数

上面提到,有一个线程用于运行Receiver,实际中,可能需要多个线程运行Receiver: 
Receiving data over the network (like Kafka, Flume, socket, etc.) requires the data to deserialized and stored in Spark. If the data receiving becomes a bottleneck in the system, then consider parallelizing the data receiving. Note that each input DStream creates a single receiver (running on a worker machine) that receives a single stream of data. Receiving multiple data streams can therefore be achieved by creating multiple input DStreams and configuring them to receive different partitions of the data stream from the source(s). For example, a single Kafka input DStream receiving two topics of data can be split into two Kafka input streams, each receiving only one topic. This would run two receivers on two workers, thus allowing data to be received in parallel, and increasing overall throughput. These multiple DStream can be unioned together to create a single DStream. Then the transformations that was being applied on the single input DStream can applied on the unified stream. This is done as follows

<code class="hljs r has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">val numStreams = <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">5</span>
val kafkaStreams = (<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span> to numStreams).map { i => KafkaUtils.createStream(<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">...</span>) }
val unifiedStream = streamingContext.union(kafkaStreams)
unifiedStream.print()</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li></ul>

关于Spark内部处理数据的并行数

Cluster resources can be under-utilized if the number of parallel tasks used in any stage of the computation is not high enough. For example, for distributed reduce operations like reduceByKey and reduceByKeyAndWindow, the default number of parallel tasks is controlled by thespark.default.parallelism configuration property. You can pass the level of parallelism as an argument (see PairDStreamFunctions documentation), or set the spark.default.parallelism configuration property to change the default.

关于谁先启动:

如果SparkStreaming先启动而9999端口后启动,会发生什么样的情况? 
首先,SparkStreaming每次到了时间点都会提示连接失败,也就是说,每次操作都是尝试新建连接。 
1. 实际上,当SparkStreaming每次时间间隔到了进行提交作业时,都会首先start Receiver,本次Job运行完成时再stop Receiver,如下是一个时间点在没有开启9999的一个情况

<code class="hljs python has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;">[hadoop<span class="hljs-decorator" style="color: rgb(0, 102, 102); box-sizing: border-box;">@hadoop ~]$ cat spark.log | grep "2015-09-20 00:50:04"</span>
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Slicing <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411402000</span> ms to <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms (aligned to <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411402000</span> ms <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">and</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms)
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Added jobs <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting job: foreachRDD at LogAnalyzerStreamingSQL.scala:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">44</span>
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Job <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">238</span> finished: foreachRDD at LogAnalyzerStreamingSQL.scala:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">44</span>, took <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.000036</span> s
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting job streaming job <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> job set of time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Finished job streaming job <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">.0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> job set of time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Total delay: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.012</span> s <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms (execution: <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0.000</span> s)
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">711</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> persistence list
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">711</span>
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">704</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> persistence list
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">704</span>
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">703</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> persistence list
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing RDD <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">703</span>
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Removing blocks of RDD BlockRDD[<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">703</span>] at socketTextStream at LogAnalyzerStreamingSQL.scala:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">38</span> of time <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411404000</span> ms
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Deleting batches ArrayBuffer()
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Deleting batches ArrayBuffer(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1424411400000</span> ms)
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting receiver again
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Starting receiver
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Called receiver onStart
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Registered receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> stream <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span> <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">from</span> akka://sparkDriver
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Receiver started again
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>
[WARN ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logWarning:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">92</span>] Restarting receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span> ms: Error connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Stopping receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> message: Restarting receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span>ms: Error connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>: java.net.ConnectException: Connection refused
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Called receiver onStop
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Deregistering receiver <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>
[ERROR] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logError:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">75</span>] Deregistered receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">for</span> stream <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>: Restarting receiver <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">with</span> delay <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2000</span>ms: Error connecting to hadoop.master:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span> - java.net.ConnectException: Connection refused
[INFO ] [<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2015</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">09</span>-<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">20</span> <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">00</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">50</span>:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">04</span>] [Logging$<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">class</span>:</span>logInfo:<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">59</span>] Stopped receiver <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li></ul>

上面的日志有Starting receiver和Stopped Receiver的输出,同时,也有java.net.Connection refused的异常

  1. 虽然9999端口没有监听上,但是此时由于提交了spark streaming application,那么spark streaming application已经开始正常运行了,即上面的wordcount例子,words这个RDD为空,下面的代码更能说明问题:
<code class="hljs scala has-numbering" style="display: block; padding: 0px; background-color: transparent; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-top-left-radius: 0px; border-top-right-radius: 0px; border-bottom-right-radius: 0px; border-bottom-left-radius: 0px; word-wrap: normal; background-position: initial initial; background-repeat: initial initial;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">package</span> spark.examples.databricks.reference.apps.loganalysis

<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.spark.sql.SQLContext
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.spark.{SparkContext, SparkConf}
<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> org.apache.spark.streaming.{StreamingContext, Duration}

<span class="hljs-javadoc" style="color: rgb(136, 0, 0); box-sizing: border-box;">/**
 * The LogAnalyzerStreamingSQL is similar to LogAnalyzerStreaming, except
 * it computes stats using Spark SQL.
 *
 * To feed the new lines of some logfile into a socket for streaming,
 * Run this command:
 *   % tail -f [YOUR_LOG_FILE] | nc -lk 9999
 *
 * If you don't have a live log file that is being written to,
 * you can add test lines using this command:
 *   % cat ../../data/apache.access.log >> [YOUR_LOG_FILE]
 *
 * Example command to run:
 * % spark-submit
 *   --class "com.databricks.apps.logs.chapter1.LogAnalyzerStreaming"
 *   --master local[4]
 *   target/scala-2.10/spark-logs-analyzer_2.10-1.0.jar
 */</span>
<span class="hljs-class" style="box-sizing: border-box;"><span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">object</span> <span class="hljs-title" style="box-sizing: border-box; color: rgb(102, 0, 102);">LogAnalyzerStreamingSQL</span> {</span>
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> WINDOW_LENGTH = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Duration(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">4</span> * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)
  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> SLIDE_INTERVAL = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> Duration(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span> * <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)

  <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">def</span> main(args: Array[String]) {
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sparkConf = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkConf().setAppName(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Log Analyzer Streaming SQL in Scala"</span>)
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sc = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SparkContext(sparkConf)

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> sqlContext = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> SQLContext(sc)
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">import</span> sqlContext.createSchemaRDD

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> streamingContext = <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">new</span> StreamingContext(sc, SLIDE_INTERVAL)

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//首先应该监听上localhost:9999端口,如果不监听会发生什么情况?</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//下面的逻辑  println("No access com.databricks.app.logs received in this time interval")已经执行了</span>
    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//同时日志中报出Connection Refused错误</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> logLinesDStream = streamingContext.socketTextStream(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"localhost"</span>, <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">9999</span>)

    <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">//转换成DStream[ApacheAccessLog]</span>
    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> accessLogsDStream = logLinesDStream.map(ApacheAccessLog.parseLogLine).cache()

    <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> windowDStream = accessLogsDStream.window(WINDOW_LENGTH, SLIDE_INTERVAL)

    windowDStream.foreachRDD(accessLogs => {
      <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">if</span> (accessLogs.count() == <span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) {
        println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"No access com.databricks.app.logs received in this time interval"</span>)
      } <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">else</span> {
        accessLogs.registerTempTable(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"TBL_ACCESS_LOG"</span>)

        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Calculate statistics based on the content size.</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> contentSizeStats = sqlContext
          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT SUM(contentSize), COUNT(*), MIN(contentSize), MAX(contentSize) FROM TBL_ACCESS_LOG"</span>)
          .first()
        println(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"Content Size Avg: %s, Min: %s, Max: %s"</span>.format(
          contentSizeStats.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>) / contentSizeStats.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>),
          contentSizeStats(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">2</span>),
          contentSizeStats(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">3</span>)))

        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Compute Response Code to Count.</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> responseCodeToCount = sqlContext
          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT responseCode, COUNT(*) FROM TBL_ACCESS_LOG GROUP BY responseCode"</span>)
          .map(row => (row.getInt(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), row.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)))
          .take(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1000</span>)
        println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"""Response code counts: ${responseCodeToCount.mkString("[", ",", "]")}"""</span>)

        <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">// Any IPAddress that has accessed the server more than 10 times.</span>
        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> ipAddresses =sqlContext
          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT ipAddress, COUNT(*) AS total FROM TBL_ACCESS_LOG GROUP BY ipAddress HAVING total > 10"</span>)
          .map(row => row.getString(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>))
          .take(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">100</span>)
        println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"""IPAddresses > 10 times: ${ipAddresses.mkString("[", ",", "]")}"""</span>)

        <span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">val</span> topEndpoints = sqlContext
          .sql(<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"SELECT endpoint, COUNT(*) AS total FROM TBL_ACCESS_LOG GROUP BY endpoint ORDER BY total DESC LIMIT 10"</span>)
          .map(row => (row.getString(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">0</span>), row.getLong(<span class="hljs-number" style="color: rgb(0, 102, 102); box-sizing: border-box;">1</span>)))
          .collect()
        println(s<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"""Top Endpoints: ${topEndpoints.mkString("[", ",", "]")}"""</span>)
      }
    })

    streamingContext.start()
    streamingContext.awaitTermination()
  }
}
</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li><li style="box-sizing: border-box; padding: 0px 5px;">19</li><li style="box-sizing: border-box; padding: 0px 5px;">20</li><li style="box-sizing: border-box; padding: 0px 5px;">21</li><li style="box-sizing: border-box; padding: 0px 5px;">22</li><li style="box-sizing: border-box; padding: 0px 5px;">23</li><li style="box-sizing: border-box; padding: 0px 5px;">24</li><li style="box-sizing: border-box; padding: 0px 5px;">25</li><li style="box-sizing: border-box; padding: 0px 5px;">26</li><li style="box-sizing: border-box; padding: 0px 5px;">27</li><li style="box-sizing: border-box; padding: 0px 5px;">28</li><li style="box-sizing: border-box; padding: 0px 5px;">29</li><li style="box-sizing: border-box; padding: 0px 5px;">30</li><li style="box-sizing: border-box; padding: 0px 5px;">31</li><li style="box-sizing: border-box; padding: 0px 5px;">32</li><li style="box-sizing: border-box; padding: 0px 5px;">33</li><li style="box-sizing: border-box; padding: 0px 5px;">34</li><li style="box-sizing: border-box; padding: 0px 5px;">35</li><li style="box-sizing: border-box; padding: 0px 5px;">36</li><li style="box-sizing: border-box; padding: 0px 5px;">37</li><li style="box-sizing: border-box; padding: 0px 5px;">38</li><li style="box-sizing: border-box; padding: 0px 5px;">39</li><li style="box-sizing: border-box; padding: 0px 5px;">40</li><li style="box-sizing: border-box; padding: 0px 5px;">41</li><li style="box-sizing: border-box; padding: 0px 5px;">42</li><li style="box-sizing: border-box; padding: 0px 5px;">43</li><li style="box-sizing: border-box; padding: 0px 5px;">44</li><li style="box-sizing: border-box; padding: 0px 5px;">45</li><li style="box-sizing: border-box; padding: 0px 5px;">46</li><li style="box-sizing: border-box; padding: 0px 5px;">47</li><li style="box-sizing: border-box; padding: 0px 5px;">48</li><li style="box-sizing: border-box; padding: 0px 5px;">49</li><li style="box-sizing: border-box; padding: 0px 5px;">50</li><li style="box-sizing: border-box; padding: 0px 5px;">51</li><li style="box-sizing: border-box; padding: 0px 5px;">52</li><li style="box-sizing: border-box; padding: 0px 5px;">53</li><li style="box-sizing: border-box; padding: 0px 5px;">54</li><li style="box-sizing: border-box; padding: 0px 5px;">55</li><li style="box-sizing: border-box; padding: 0px 5px;">56</li><li style="box-sizing: border-box; padding: 0px 5px;">57</li><li style="box-sizing: border-box; padding: 0px 5px;">58</li><li style="box-sizing: border-box; padding: 0px 5px;">59</li><li style="box-sizing: border-box; padding: 0px 5px;">60</li><li style="box-sizing: border-box; padding: 0px 5px;">61</li><li style="box-sizing: border-box; padding: 0px 5px;">62</li><li style="box-sizing: border-box; padding: 0px 5px;">63</li><li style="box-sizing: border-box; padding: 0px 5px;">64</li><li style="box-sizing: border-box; padding: 0px 5px;">65</li><li style="box-sizing: border-box; padding: 0px 5px;">66</li><li style="box-sizing: border-box; padding: 0px 5px;">67</li><li style="box-sizing: border-box; padding: 0px 5px;">68</li><li style="box-sizing: border-box; padding: 0px 5px;">69</li><li style="box-sizing: border-box; padding: 0px 5px;">70</li><li style="box-sizing: border-box; padding: 0px 5px;">71</li><li style="box-sizing: border-box; padding: 0px 5px;">72</li><li style="box-sizing: border-box; padding: 0px 5px;">73</li><li style="box-sizing: border-box; padding: 0px 5px;">74</li><li style="box-sizing: border-box; padding: 0px 5px;">75</li><li style="box-sizing: border-box; padding: 0px 5px;">76</li><li style="box-sizing: border-box; padding: 0px 5px;">77</li><li style="box-sizing: border-box; padding: 0px 5px;">78</li><li style="box-sizing: border-box; padding: 0px 5px;">79</li><li style="box-sizing: border-box; padding: 0px 5px;">80</li><li style="box-sizing: border-box; padding: 0px 5px;">81</li><li style="box-sizing: border-box; padding: 0px 5px;">82</li><li style="box-sizing: border-box; padding: 0px 5px;">83</li><li style="box-sizing: border-box; padding: 0px 5px;">84</li><li style="box-sizing: border-box; padding: 0px 5px;">85</li><li style="box-sizing: border-box; padding: 0px 5px;">86</li><li style="box-sizing: border-box; padding: 0px 5px;">87</li><li style="box-sizing: border-box; padding: 0px 5px;">88</li><li style="box-sizing: border-box; padding: 0px 5px;">89</li></ul>

虽然9999端口没有开启监听,但是spark streaming application的console,定时的输出No access com.databricks.app.logs received in this time interval

相关文章推荐

spark streaming的NetworkWordCount实例理解

/** Licensed to the Apache Software Foundation (ASF) under one or more* contributor license agreemen...

spark streaming的NetworkWordCount实例理解

NetworkWordCount代码/* * Licensed to the Apache Software Foundation (ASF) under one or more * contribu...

第7课:Spark Streaming源码解读之JobScheduler内幕实现和深度思考

本期内容: 1. JobScheduler内幕实现 2. JobScheduler深度思考   所有工作的关键都是jobScheduler SparkStreaming至少要设置两条线程是因为一条用于...

spark的checkpoint

====================================================================================================...

解密SparkStreaming另类实验及SparkStreaming本质解析(第一篇)

本期亮点: 通过SparkStreaming在线另类实验瞬间理解SparkStreaming运行本质 SparkStreaming背景介绍 当今社会处于一个大数据的时代,而SparkS...

Nature综述:NGS实验中的错误来自哪里?如何减少?

尽管测序技术在不断发展,但不论是哪种测序平台,测序过程中都不可避免地存在一些错误。《Nature ReviewGenetics》近日发表了一篇很有意思的综述,谈论了NGS实验中错误的来源,以及如何利用...

疑问 实验 错误 C++篇

1.       class Book{ private: public: void print(int m){ cout } }; class AudioBook: public Book{ pu...
  • fofu33
  • fofu33
  • 2013年08月28日 11:14
  • 591

WenGoat实验之Inptoper Error Handling(错误处理不当)- 2016.01.09

Fail Open Authentication Scheme(打开认证失败方案),有时候我们在服务端接受客户端数据的时候并没有判断这个参数是否存在就对其直接使用,这是很不安全的,因为往往会由于参数的...

Http 401错误重现实验及解决办法

根据http协议对状态码的定义,401代表未经授权的访问。在IIS中在401的基础上定义了一系列的子状态码来代表各种不同情况下未经授权访问的具体原因。 401.1 - Logon failed....

[Embedded] [FL2440] ADS实验五: 按键与蜂鸣器: 修改了光盘配的代码的一句错误, 并优化了当中 的计数值

1.5-key-bell\init.s: AREA |DATA|,CODE,READONLY ENTRY ldr r13, =0x1000 ; Set sp IMPORT BellMa...
  • Dooit
  • Dooit
  • 2011年08月05日 17:56
  • 973
内容举报
返回顶部
收藏助手
不良信息举报
您举报文章:SparkStreaming实验错误
举报原因:
原因补充:

(最多只允许输入30个字)