关闭

第58课:使用Java和Scala在IDE中开发DataFrame实战学习笔记

标签: SparkSQL
1210人阅读 评论(0) 收藏 举报
分类:

第58课:使用JavaScalaIDE中开发DataFrame实战学习笔记

本期内容:

1. 使用Java开发DataFrame实战

2.  使用Scala开发DataFrame实战

 

Spark1.3开始大多数SparkSQL都基于DataFrame编程。因为DataFrame高效且功能强大。

可以把SparkSQl作为一个分布式查询引擎。SparkSQL一般都和Hive一起使用。

使用Java开发Spark应用程序的原因:

1. 企业生产环境下大多以Java为核心

2. Java更傻瓜,更易理解

 

SQLContext中有HiveContext子类。

 

Spark官网建议任何时候都用HiveContext。一般情况下都可以直接用HiveContext而不需要使用SQLContext

./spark-submit --file,可以通过指定hive-site.xml配置,这给我们一个启发:如果有一些特别的配置,可以这样指定。但如果指定其他文件,会覆盖默认参数hive-site.xml(注意:SparkConf目录下也有hive-site.xml文件)

spark-submit --class com.dt.spark.sql.DataFrameOps --master spark://slq1:7077 /home/richard/spark-1.6.0/SparkApps/wordCount.jar

 

配置Hive的不同数据来源。如果不配置的话Spark会自动找Spark中的Hive配置信息。

在哪台机上提交就需要在哪台机上安装Hive

 

Java代码如下:

package com.dt.spark.SparkApps;

 

import org.apache.spark.SparkConf;

import org.apache.spark.api.java.JavaSparkContext;

import org.apache.spark.sql.DataFrame;

import org.apache.spark.sql.SQLContext;

 

public class DataFrameOps {

public static void main(String[] args){

//创建SparkConf用于读取系统配置信息并设置当前应用程序名称

SparkConf conf = new SparkConf().setAppName("DataFrameOps");

//创建JavaSparkContext对象实例作为整个Driver核心基石

JavaSparkContext sc = new JavaSparkContext(conf);

//创建SQLContext上下文对象用于SQL分析

SQLContext sqlContext = new SQLContext(sc);

//创建DataFrame,可以简单认为DataFrame是一张表

//DataFrame可以来源于多种格式

//如JSON,可以直接创建DataFrame

//SQLContext只支持SQL一种方言,用HiveContext就可以支持多种

//方言(默认HQL,可以设置)。

DataFrame df = sqlContext.read().json("HDFS://slq1:9000/user/people.json");

//select * from table

df.show();

//describe table;

df.printSchema();

df.select("name").show();

// select name age+1 from table;

df.select(df.col("name"),df.col("age").plus(10)).show();

//select * from table where age > 20;

df.filter(df.col("age").gt(20)).show();

//select count(1) from table group by age;

df.groupBy(df.col("age")).count().show();

}

}

 

下面编写scala代码:

package com.dt.spark.SparkApps.sql

 

import org.apache.spark.SparkContext

import org.apache.spark.SparkConf

import org.apache.spark.sql.SQLContext

 

// 使用Java的方式开发实战对DataFrame的操作

object DataFrameOps {

  def main(args: Array[String]): Unit = {

    val conf = new SparkConf()

    conf.setAppName("DataFrameOps")

    conf.setMaster("spark://slq1:7077")

    val sc = new SparkContext

    val sqlContext = new SQLContext(sc)

    val df = sqlContext.read.json("hdfs://slq1:9000/user/data/SparkResources/people.json")

    df.show()

    df.printSchema()

    df.select("name").show()

    df.select(df("name"),df("age") + 10).show()

    df.filter(df("age") > 10).show()

    df.groupBy("age").count.show()    

  }

}

导出为jar包:

点击eclipse菜单File->export,选择destinationJAR file后点击Next

 

选择SparkAppsScala项目,选择导出路径后点击Finish

 

将生成的SparkAppsScala.jar文件放入虚拟机中。

编写shell执行:

/home/richard/spark-1.6.0/bin/spark-submit --class com.dt.spark.SparkApps.sql.DataFrameOps --master spark://slq1:7077 /home/richard/slq/spark/SparkAppsScala.jar

生产环境下,执行shell一般是这样的:

/home/richard/spark-1.6.0/bin/spark-submit --class com.dt.spark.SparkApps.sql.DataFrameOps --files //home/richard/spark-1.6.0/conf/hive-site.xml --driver-class-path /home/richard/hive-1.2.1/lib/mysql-connector-java-5.1.32-bin.jar --master spark://slq1:7077 /home/richard/slq/spark/SparkAppsScala.jar

 

执行结果如下:

 

[richard@slq1 spark]$ ./SparkAppsScala.sh

16/03/27 07:36:38 INFO spark.SparkContext: Running Spark version 1.6.0

16/03/27 07:36:45 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

16/03/27 07:36:49 INFO spark.SecurityManager: Changing view acls to: richard

16/03/27 07:36:49 INFO spark.SecurityManager: Changing modify acls to: richard

16/03/27 07:36:50 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(richard); users with modify permissions: Set(richard)

16/03/27 07:37:00 INFO util.Utils: Successfully started service 'sparkDriver' on port 34946.

16/03/27 07:37:05 INFO slf4j.Slf4jLogger: Slf4jLogger started

16/03/27 07:37:06 INFO Remoting: Starting remoting

16/03/27 07:37:09 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.1.121:46548]

16/03/27 07:37:10 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 46548.

16/03/27 07:37:10 INFO spark.SparkEnv: Registering MapOutputTracker

16/03/27 07:37:11 INFO spark.SparkEnv: Registering BlockManagerMaster

16/03/27 07:37:11 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-967b95a2-650b-4485-8052-cd9158865d38

16/03/27 07:37:11 INFO storage.MemoryStore: MemoryStore started with capacity 517.4 MB

16/03/27 07:37:13 INFO spark.SparkEnv: Registering OutputCommitCoordinator

16/03/27 07:37:17 INFO server.Server: jetty-8.y.z-SNAPSHOT

16/03/27 07:37:18 WARN component.AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: 地址已在使用

java.net.BindException: 地址已在使用

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:437)

at sun.nio.ch.Net.bind(Net.java:429)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)

at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)

at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)

at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)

at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)

at org.spark-project.jetty.server.Server.doStart(Server.java:293)

at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)

at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:252)

at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)

at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)

at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1964)

at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)

at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1955)

at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)

at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)

at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)

at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)

at scala.Option.foreach(Option.scala:236)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:481)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:123)

at com.dt.spark.SparkApps.sql.DaraFrameOps$.main(DaraFrameOps.scala:13)

at com.dt.spark.SparkApps.sql.DaraFrameOps.main(DaraFrameOps.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

16/03/27 07:37:18 WARN component.AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@49bf29c6: java.net.BindException: 地址已在使用

java.net.BindException: 地址已在使用

at sun.nio.ch.Net.bind0(Native Method)

at sun.nio.ch.Net.bind(Net.java:437)

at sun.nio.ch.Net.bind(Net.java:429)

at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)

at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)

at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)

at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)

at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)

at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)

at org.spark-project.jetty.server.Server.doStart(Server.java:293)

at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)

at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:252)

at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)

at org.apache.spark.ui.JettyUtils$$anonfun$5.apply(JettyUtils.scala:262)

at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1964)

at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)

at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1955)

at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:262)

at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)

at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)

at org.apache.spark.SparkContext$$anonfun$13.apply(SparkContext.scala:481)

at scala.Option.foreach(Option.scala:236)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:481)

at org.apache.spark.SparkContext.<init>(SparkContext.scala:123)

at com.dt.spark.SparkApps.sql.DaraFrameOps$.main(DaraFrameOps.scala:13)

at com.dt.spark.SparkApps.sql.DaraFrameOps.main(DaraFrameOps.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)

at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}

16/03/27 07:37:18 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}

16/03/27 07:37:18 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.

16/03/27 07:37:18 INFO server.Server: jetty-8.y.z-SNAPSHOT

16/03/27 07:37:19 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041

16/03/27 07:37:19 INFO util.Utils: Successfully started service 'SparkUI' on port 4041.

16/03/27 07:37:19 INFO ui.SparkUI: Started SparkUI at http://192.168.1.121:4041

16/03/27 07:37:19 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-1f459855-2e46-4caa-9b7f-897e3926749c/httpd-6a314baa-db64-40c8-8a36-52541d75c8f7

16/03/27 07:37:19 INFO spark.HttpServer: Starting HTTP Server

16/03/27 07:37:20 INFO server.Server: jetty-8.y.z-SNAPSHOT

16/03/27 07:37:20 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:48259

16/03/27 07:37:20 INFO util.Utils: Successfully started service 'HTTP file server' on port 48259.

16/03/27 07:37:20 INFO spark.SparkContext: Added JAR file:/home/richard/slq/spark/SparkAppsScala.jar at http://192.168.1.121:48259/jars/SparkAppsScala.jar with timestamp 1459035440616

16/03/27 07:37:23 INFO client.AppClient$ClientEndpoint: Connecting to master spark://slq1:7077...

16/03/27 07:37:30 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160327073729-0000

16/03/27 07:37:30 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44773.

16/03/27 07:37:30 INFO netty.NettyBlockTransferService: Server created on 44773

16/03/27 07:37:30 INFO storage.BlockManagerMaster: Trying to register BlockManager

16/03/27 07:37:31 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.1.121:44773 with 517.4 MB RAM, BlockManagerId(driver, 192.168.1.121, 44773)

16/03/27 07:37:31 INFO storage.BlockManagerMaster: Registered BlockManager

16/03/27 07:37:32 INFO client.AppClient$ClientEndpoint: Executor added: app-20160327073729-0000/0 on worker-20160327012804-192.168.1.121-47541 (192.168.1.121:47541) with 1 cores

16/03/27 07:37:32 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160327073729-0000/0 on hostPort 192.168.1.121:47541 with 1 cores, 1024.0 MB RAM

16/03/27 07:37:32 INFO client.AppClient$ClientEndpoint: Executor added: app-20160327073729-0000/1 on worker-20160327012200-192.168.1.123-59271 (192.168.1.123:59271) with 1 cores

16/03/27 07:37:32 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160327073729-0000/1 on hostPort 192.168.1.123:59271 with 1 cores, 1024.0 MB RAM

16/03/27 07:37:32 INFO client.AppClient$ClientEndpoint: Executor added: app-20160327073729-0000/2 on worker-20160327012159-192.168.1.122-49405 (192.168.1.122:49405) with 1 cores

16/03/27 07:37:32 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160327073729-0000/2 on hostPort 192.168.1.122:49405 with 1 cores, 1024.0 MB RAM

16/03/27 07:37:35 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160327073729-0000/2 is now RUNNING

16/03/27 07:37:35 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160327073729-0000/1 is now RUNNING

16/03/27 07:37:44 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160327073729-0000/0 is now RUNNING

16/03/27 07:37:45 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0

16/03/27 07:38:14 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slq2:49520) with ID 2

16/03/27 07:38:15 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slq3:36997) with ID 1

16/03/27 07:38:16 INFO storage.BlockManagerMasterEndpoint: Registering block manager slq2:42224 with 517.4 MB RAM, BlockManagerId(2, slq2, 42224)

16/03/27 07:38:16 INFO storage.BlockManagerMasterEndpoint: Registering block manager slq3:47358 with 517.4 MB RAM, BlockManagerId(1, slq3, 47358)

16/03/27 07:38:27 INFO json.JSONRelation: Listing hdfs://slq1:9000/user/data/SparkResources/people.json on driver

16/03/27 07:38:45 INFO storage.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 212.1 KB, free 212.1 KB)

16/03/27 07:38:47 INFO storage.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 19.9 KB, free 232.0 KB)

16/03/27 07:38:47 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.121:44773 (size: 19.9 KB, free: 517.4 MB)

16/03/27 07:38:47 INFO spark.SparkContext: Created broadcast 0 from json at DaraFrameOps.scala:15

16/03/27 07:38:52 INFO mapred.FileInputFormat: Total input paths to process : 1

16/03/27 07:38:55 INFO spark.SparkContext: Starting job: json at DaraFrameOps.scala:15

16/03/27 07:38:56 INFO scheduler.DAGScheduler: Got job 0 (json at DaraFrameOps.scala:15) with 2 output partitions

16/03/27 07:38:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (json at DaraFrameOps.scala:15)

16/03/27 07:38:56 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:38:56 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:38:56 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[3] at json at DaraFrameOps.scala:15), which has no missing parents

16/03/27 07:38:58 INFO storage.MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.3 KB, free 236.3 KB)

16/03/27 07:38:58 INFO storage.MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.4 KB, free 238.7 KB)

16/03/27 07:38:58 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.1.121:44773 (size: 2.4 KB, free: 517.4 MB)

16/03/27 07:38:58 INFO spark.SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006

16/03/27 07:38:58 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[3] at json at DaraFrameOps.scala:15)

16/03/27 07:38:58 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

16/03/27 07:39:00 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, slq3, partition 0,NODE_LOCAL, 2215 bytes)

16/03/27 07:39:00 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, slq2, partition 1,NODE_LOCAL, 2215 bytes)

16/03/27 07:39:09 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on slq3:47358 (size: 2.4 KB, free: 517.4 MB)

16/03/27 07:39:09 INFO storage.BlockManagerInfo: Added broadcast_1_piece0 in memory on slq2:42224 (size: 2.4 KB, free: 517.4 MB)

16/03/27 07:39:12 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slq2:42224 (size: 19.9 KB, free: 517.4 MB)

16/03/27 07:39:12 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on slq3:47358 (size: 19.9 KB, free: 517.4 MB)

16/03/27 07:39:36 INFO cluster.SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (slq1:57687) with ID 0

16/03/27 07:39:40 INFO storage.BlockManagerMasterEndpoint: Registering block manager slq1:46119 with 517.4 MB RAM, BlockManagerId(0, slq1, 46119)

16/03/27 07:40:02 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 61683 ms on slq2 (1/2)

16/03/27 07:40:02 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 63035 ms on slq3 (2/2)

16/03/27 07:40:02 INFO scheduler.DAGScheduler: ResultStage 0 (json at DaraFrameOps.scala:15) finished in 63.352 s

16/03/27 07:40:02 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool

16/03/27 07:40:06 INFO scheduler.DAGScheduler: Job 0 finished: json at DaraFrameOps.scala:15, took 71.273053 s

16/03/27 07:40:12 INFO storage.MemoryStore: Block broadcast_2 stored as values in memory (estimated size 62.4 KB, free 301.2 KB)

16/03/27 07:40:13 INFO storage.MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 19.7 KB, free 320.9 KB)

16/03/27 07:40:13 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.1.121:44773 (size: 19.7 KB, free: 517.4 MB)

16/03/27 07:40:13 INFO spark.SparkContext: Created broadcast 2 from show at DaraFrameOps.scala:16

16/03/27 07:40:14 INFO storage.MemoryStore: Block broadcast_3 stored as values in memory (estimated size 212.1 KB, free 533.0 KB)

16/03/27 07:40:14 INFO storage.MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 19.9 KB, free 552.9 KB)

16/03/27 07:40:14 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.1.121:44773 (size: 19.9 KB, free: 517.3 MB)

16/03/27 07:40:14 INFO spark.SparkContext: Created broadcast 3 from show at DaraFrameOps.scala:16

16/03/27 07:40:16 INFO mapred.FileInputFormat: Total input paths to process : 1

16/03/27 07:40:16 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:16

16/03/27 07:40:16 INFO scheduler.DAGScheduler: Got job 1 (show at DaraFrameOps.scala:16) with 1 output partitions

16/03/27 07:40:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 1 (show at DaraFrameOps.scala:16)

16/03/27 07:40:16 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:40:16 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:40:16 INFO scheduler.DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[9] at show at DaraFrameOps.scala:16), which has no missing parents

16/03/27 07:40:16 INFO storage.MemoryStore: Block broadcast_4 stored as values in memory (estimated size 5.7 KB, free 558.5 KB)

16/03/27 07:40:16 INFO storage.MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 3.2 KB, free 561.8 KB)

16/03/27 07:40:16 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.1.121:44773 (size: 3.2 KB, free: 517.3 MB)

16/03/27 07:40:16 INFO spark.SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1006

16/03/27 07:40:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[9] at show at DaraFrameOps.scala:16)

16/03/27 07:40:16 INFO scheduler.TaskSchedulerImpl: Adding task set 1.0 with 1 tasks

16/03/27 07:40:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, slq1, partition 0,NODE_LOCAL, 2215 bytes)

16/03/27 07:40:22 INFO storage.BlockManagerInfo: Added broadcast_4_piece0 in memory on slq1:46119 (size: 3.2 KB, free: 517.4 MB)

16/03/27 07:40:35 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in memory on slq1:46119 (size: 19.9 KB, free: 517.4 MB)

16/03/27 07:40:56 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 39566 ms on slq1 (1/1)

16/03/27 07:40:56 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool

16/03/27 07:40:56 INFO scheduler.DAGScheduler: ResultStage 1 (show at DaraFrameOps.scala:16) finished in 39.571 s

16/03/27 07:40:56 INFO scheduler.DAGScheduler: Job 1 finished: show at DaraFrameOps.scala:16, took 39.812207 s

16/03/27 07:40:56 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:16

16/03/27 07:40:56 INFO scheduler.DAGScheduler: Got job 2 (show at DaraFrameOps.scala:16) with 1 output partitions

16/03/27 07:40:56 INFO scheduler.DAGScheduler: Final stage: ResultStage 2 (show at DaraFrameOps.scala:16)

16/03/27 07:40:56 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:40:56 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:40:56 INFO scheduler.DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[9] at show at DaraFrameOps.scala:16), which has no missing parents

16/03/27 07:40:56 INFO storage.MemoryStore: Block broadcast_5 stored as values in memory (estimated size 5.7 KB, free 567.4 KB)

16/03/27 07:40:56 INFO storage.MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 3.2 KB, free 570.7 KB)

16/03/27 07:40:56 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.1.121:44773 (size: 3.2 KB, free: 517.3 MB)

16/03/27 07:40:56 INFO spark.SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1006

16/03/27 07:40:56 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[9] at show at DaraFrameOps.scala:16)

16/03/27 07:40:56 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 tasks

16/03/27 07:40:57 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 (TID 3, slq1, partition 1,NODE_LOCAL, 2215 bytes)

16/03/27 07:40:57 INFO storage.BlockManagerInfo: Added broadcast_5_piece0 in memory on slq1:46119 (size: 3.2 KB, free: 517.4 MB)

16/03/27 07:40:57 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 (TID 3) in 909 ms on slq1 (1/1)

16/03/27 07:40:57 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool

16/03/27 07:40:57 INFO scheduler.DAGScheduler: ResultStage 2 (show at DaraFrameOps.scala:16) finished in 0.917 s

16/03/27 07:40:57 INFO scheduler.DAGScheduler: Job 2 finished: show at DaraFrameOps.scala:16, took 1.228109 s

+----+-------+

| age|   name|

+----+-------+

|null|Michael|

|  30|   Andy|

|  19| Justin|

+----+-------+

 

root

 |-- age: long (nullable = true)

 |-- name: string (nullable = true)

 

16/03/27 07:40:59 INFO storage.MemoryStore: Block broadcast_6 stored as values in memory (estimated size 211.3 KB, free 782.0 KB)

16/03/27 07:41:00 INFO storage.MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 19.7 KB, free 801.7 KB)

16/03/27 07:41:00 INFO storage.BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.1.121:44773 (size: 19.7 KB, free: 517.3 MB)

16/03/27 07:41:00 INFO spark.SparkContext: Created broadcast 6 from show at DaraFrameOps.scala:18

16/03/27 07:41:00 INFO storage.MemoryStore: Block broadcast_7 stored as values in memory (estimated size 212.1 KB, free 1013.9 KB)

16/03/27 07:41:01 INFO storage.MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 19.9 KB, free 1033.7 KB)

16/03/27 07:41:01 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.1.121:44773 (size: 19.9 KB, free: 517.3 MB)

16/03/27 07:41:01 INFO spark.SparkContext: Created broadcast 7 from show at DaraFrameOps.scala:18

16/03/27 07:41:01 INFO mapred.FileInputFormat: Total input paths to process : 1

16/03/27 07:41:01 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:18

16/03/27 07:41:01 INFO scheduler.DAGScheduler: Got job 3 (show at DaraFrameOps.scala:18) with 1 output partitions

16/03/27 07:41:01 INFO scheduler.DAGScheduler: Final stage: ResultStage 3 (show at DaraFrameOps.scala:18)

16/03/27 07:41:01 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:41:01 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:41:01 INFO scheduler.DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[15] at show at DaraFrameOps.scala:18), which has no missing parents

16/03/27 07:41:02 INFO storage.MemoryStore: Block broadcast_8 stored as values in memory (estimated size 5.7 KB, free 1039.4 KB)

16/03/27 07:41:02 INFO storage.MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 3.3 KB, free 1042.7 KB)

16/03/27 07:41:02 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.1.121:44773 (size: 3.3 KB, free: 517.3 MB)

16/03/27 07:41:02 INFO spark.SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:02 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[15] at show at DaraFrameOps.scala:18)

16/03/27 07:41:02 INFO scheduler.TaskSchedulerImpl: Adding task set 3.0 with 1 tasks

16/03/27 07:41:02 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 3.0 (TID 4, slq1, partition 0,NODE_LOCAL, 2215 bytes)

16/03/27 07:41:02 INFO storage.BlockManagerInfo: Added broadcast_8_piece0 in memory on slq1:46119 (size: 3.3 KB, free: 517.4 MB)

16/03/27 07:41:02 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on slq1:46119 (size: 19.9 KB, free: 517.4 MB)

16/03/27 07:41:04 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 3.0 (TID 4) in 2034 ms on slq1 (1/1)

16/03/27 07:41:04 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool

16/03/27 07:41:04 INFO scheduler.DAGScheduler: ResultStage 3 (show at DaraFrameOps.scala:18) finished in 2.038 s

16/03/27 07:41:04 INFO scheduler.DAGScheduler: Job 3 finished: show at DaraFrameOps.scala:18, took 2.269697 s

16/03/27 07:41:04 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:18

16/03/27 07:41:04 INFO scheduler.DAGScheduler: Got job 4 (show at DaraFrameOps.scala:18) with 1 output partitions

16/03/27 07:41:04 INFO scheduler.DAGScheduler: Final stage: ResultStage 4 (show at DaraFrameOps.scala:18)

16/03/27 07:41:04 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:41:04 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:41:04 INFO scheduler.DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[15] at show at DaraFrameOps.scala:18), which has no missing parents

16/03/27 07:41:04 INFO storage.MemoryStore: Block broadcast_9 stored as values in memory (estimated size 5.7 KB, free 1048.3 KB)

16/03/27 07:41:04 INFO storage.MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 3.3 KB, free 1051.6 KB)

16/03/27 07:41:04 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on 192.168.1.121:44773 (size: 3.3 KB, free: 517.3 MB)

16/03/27 07:41:04 INFO spark.SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:04 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[15] at show at DaraFrameOps.scala:18)

16/03/27 07:41:04 INFO scheduler.TaskSchedulerImpl: Adding task set 4.0 with 1 tasks

16/03/27 07:41:04 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 4.0 (TID 5, slq3, partition 1,NODE_LOCAL, 2215 bytes)

16/03/27 07:41:04 INFO storage.BlockManagerInfo: Added broadcast_9_piece0 in memory on slq3:47358 (size: 3.3 KB, free: 517.4 MB)

16/03/27 07:41:06 INFO storage.BlockManagerInfo: Added broadcast_7_piece0 in memory on slq3:47358 (size: 19.9 KB, free: 517.4 MB)

16/03/27 07:41:09 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 4.0 (TID 5) in 4781 ms on slq3 (1/1)

16/03/27 07:41:09 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool

16/03/27 07:41:09 INFO scheduler.DAGScheduler: ResultStage 4 (show at DaraFrameOps.scala:18) finished in 4.828 s

16/03/27 07:41:09 INFO scheduler.DAGScheduler: Job 4 finished: show at DaraFrameOps.scala:18, took 5.025773 s

+-------+

|   name|

+-------+

|Michael|

|   Andy|

| Justin|

+-------+

 

16/03/27 07:41:11 INFO storage.MemoryStore: Block broadcast_10 stored as values in memory (estimated size 211.3 KB, free 1262.9 KB)

16/03/27 07:41:11 INFO storage.MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 19.7 KB, free 1282.6 KB)

16/03/27 07:41:11 INFO storage.BlockManagerInfo: Added broadcast_10_piece0 in memory on 192.168.1.121:44773 (size: 19.7 KB, free: 517.3 MB)

16/03/27 07:41:11 INFO spark.SparkContext: Created broadcast 10 from show at DaraFrameOps.scala:19

16/03/27 07:41:12 INFO storage.MemoryStore: Block broadcast_11 stored as values in memory (estimated size 212.1 KB, free 1494.8 KB)

16/03/27 07:41:12 INFO storage.MemoryStore: Block broadcast_11_piece0 stored as bytes in memory (estimated size 19.9 KB, free 1514.6 KB)

16/03/27 07:41:12 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on 192.168.1.121:44773 (size: 19.9 KB, free: 517.3 MB)

16/03/27 07:41:12 INFO spark.SparkContext: Created broadcast 11 from show at DaraFrameOps.scala:19

16/03/27 07:41:13 INFO mapred.FileInputFormat: Total input paths to process : 1

16/03/27 07:41:13 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:19

16/03/27 07:41:13 INFO scheduler.DAGScheduler: Got job 5 (show at DaraFrameOps.scala:19) with 1 output partitions

16/03/27 07:41:13 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 (show at DaraFrameOps.scala:19)

16/03/27 07:41:13 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:41:13 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:41:13 INFO scheduler.DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[22] at show at DaraFrameOps.scala:19), which has no missing parents

16/03/27 07:41:14 INFO storage.MemoryStore: Block broadcast_12 stored as values in memory (estimated size 7.8 KB, free 1522.5 KB)

16/03/27 07:41:14 INFO storage.MemoryStore: Block broadcast_12_piece0 stored as bytes in memory (estimated size 4.2 KB, free 1526.7 KB)

16/03/27 07:41:14 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on 192.168.1.121:44773 (size: 4.2 KB, free: 517.3 MB)

16/03/27 07:41:14 INFO spark.SparkContext: Created broadcast 12 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:14 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[22] at show at DaraFrameOps.scala:19)

16/03/27 07:41:14 INFO scheduler.TaskSchedulerImpl: Adding task set 5.0 with 1 tasks

16/03/27 07:41:14 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 5.0 (TID 6, slq3, partition 0,NODE_LOCAL, 2215 bytes)

16/03/27 07:41:14 INFO storage.BlockManagerInfo: Added broadcast_12_piece0 in memory on slq3:47358 (size: 4.2 KB, free: 517.4 MB)

16/03/27 07:41:15 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on slq3:47358 (size: 19.9 KB, free: 517.3 MB)

16/03/27 07:41:16 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 5.0 (TID 6) in 1961 ms on slq3 (1/1)

16/03/27 07:41:16 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool

16/03/27 07:41:16 INFO scheduler.DAGScheduler: ResultStage 5 (show at DaraFrameOps.scala:19) finished in 2.015 s

16/03/27 07:41:16 INFO scheduler.DAGScheduler: Job 5 finished: show at DaraFrameOps.scala:19, took 2.249874 s

16/03/27 07:41:16 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:19

16/03/27 07:41:16 INFO scheduler.DAGScheduler: Got job 6 (show at DaraFrameOps.scala:19) with 1 output partitions

16/03/27 07:41:16 INFO scheduler.DAGScheduler: Final stage: ResultStage 6 (show at DaraFrameOps.scala:19)

16/03/27 07:41:16 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:41:16 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:41:16 INFO scheduler.DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[22] at show at DaraFrameOps.scala:19), which has no missing parents

16/03/27 07:41:16 INFO storage.MemoryStore: Block broadcast_13 stored as values in memory (estimated size 7.8 KB, free 1534.5 KB)

16/03/27 07:41:16 INFO storage.MemoryStore: Block broadcast_13_piece0 stored as bytes in memory (estimated size 4.2 KB, free 1538.7 KB)

16/03/27 07:41:16 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on 192.168.1.121:44773 (size: 4.2 KB, free: 517.2 MB)

16/03/27 07:41:16 INFO spark.SparkContext: Created broadcast 13 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:16 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[22] at show at DaraFrameOps.scala:19)

16/03/27 07:41:16 INFO scheduler.TaskSchedulerImpl: Adding task set 6.0 with 1 tasks

16/03/27 07:41:16 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 6.0 (TID 7, slq2, partition 1,NODE_LOCAL, 2215 bytes)

16/03/27 07:41:16 INFO storage.BlockManagerInfo: Added broadcast_13_piece0 in memory on slq2:42224 (size: 4.2 KB, free: 517.4 MB)

16/03/27 07:41:17 INFO storage.BlockManagerInfo: Added broadcast_11_piece0 in memory on slq2:42224 (size: 19.9 KB, free: 517.4 MB)

16/03/27 07:41:20 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 6.0 (TID 7) in 4251 ms on slq2 (1/1)

16/03/27 07:41:20 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool

16/03/27 07:41:20 INFO scheduler.DAGScheduler: ResultStage 6 (show at DaraFrameOps.scala:19) finished in 4.268 s

16/03/27 07:41:20 INFO scheduler.DAGScheduler: Job 6 finished: show at DaraFrameOps.scala:19, took 4.491418 s

+-------+----------+

|   name|(age + 10)|

+-------+----------+

|Michael|      null|

|   Andy|        40|

| Justin|        29|

+-------+----------+

 

16/03/27 07:41:21 INFO storage.MemoryStore: Block broadcast_14 stored as values in memory (estimated size 211.3 KB, free 1750.0 KB)

16/03/27 07:41:21 INFO storage.MemoryStore: Block broadcast_14_piece0 stored as bytes in memory (estimated size 19.7 KB, free 1769.7 KB)

16/03/27 07:41:21 INFO storage.BlockManagerInfo: Added broadcast_14_piece0 in memory on 192.168.1.121:44773 (size: 19.7 KB, free: 517.2 MB)

16/03/27 07:41:21 INFO spark.SparkContext: Created broadcast 14 from show at DaraFrameOps.scala:20

16/03/27 07:41:21 INFO storage.MemoryStore: Block broadcast_15 stored as values in memory (estimated size 212.1 KB, free 1981.9 KB)

16/03/27 07:41:22 INFO storage.MemoryStore: Block broadcast_15_piece0 stored as bytes in memory (estimated size 19.9 KB, free 2001.7 KB)

16/03/27 07:41:22 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on 192.168.1.121:44773 (size: 19.9 KB, free: 517.2 MB)

16/03/27 07:41:22 INFO spark.SparkContext: Created broadcast 15 from show at DaraFrameOps.scala:20

16/03/27 07:41:22 INFO mapred.FileInputFormat: Total input paths to process : 1

16/03/27 07:41:23 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:20

16/03/27 07:41:23 INFO scheduler.DAGScheduler: Got job 7 (show at DaraFrameOps.scala:20) with 1 output partitions

16/03/27 07:41:23 INFO scheduler.DAGScheduler: Final stage: ResultStage 7 (show at DaraFrameOps.scala:20)

16/03/27 07:41:23 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:41:23 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:41:23 INFO scheduler.DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[29] at show at DaraFrameOps.scala:20), which has no missing parents

16/03/27 07:41:23 INFO storage.MemoryStore: Block broadcast_16 stored as values in memory (estimated size 7.7 KB, free 2009.4 KB)

16/03/27 07:41:23 INFO storage.MemoryStore: Block broadcast_16_piece0 stored as bytes in memory (estimated size 4.1 KB, free 2013.5 KB)

16/03/27 07:41:23 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on 192.168.1.121:44773 (size: 4.1 KB, free: 517.2 MB)

16/03/27 07:41:23 INFO spark.SparkContext: Created broadcast 16 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:23 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[29] at show at DaraFrameOps.scala:20)

16/03/27 07:41:23 INFO scheduler.TaskSchedulerImpl: Adding task set 7.0 with 1 tasks

16/03/27 07:41:23 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 7.0 (TID 8, slq3, partition 0,NODE_LOCAL, 2215 bytes)

16/03/27 07:41:23 INFO storage.BlockManagerInfo: Added broadcast_16_piece0 in memory on slq3:47358 (size: 4.1 KB, free: 517.3 MB)

16/03/27 07:41:23 INFO storage.BlockManagerInfo: Added broadcast_15_piece0 in memory on slq3:47358 (size: 19.9 KB, free: 517.3 MB)

16/03/27 07:41:24 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 7.0 (TID 8) in 1306 ms on slq3 (1/1)

16/03/27 07:41:24 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool

16/03/27 07:41:24 INFO scheduler.DAGScheduler: ResultStage 7 (show at DaraFrameOps.scala:20) finished in 1.312 s

16/03/27 07:41:24 INFO scheduler.DAGScheduler: Job 7 finished: show at DaraFrameOps.scala:20, took 1.586639 s

16/03/27 07:41:24 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:20

16/03/27 07:41:24 INFO scheduler.DAGScheduler: Got job 8 (show at DaraFrameOps.scala:20) with 1 output partitions

16/03/27 07:41:24 INFO scheduler.DAGScheduler: Final stage: ResultStage 8 (show at DaraFrameOps.scala:20)

16/03/27 07:41:24 INFO scheduler.DAGScheduler: Parents of final stage: List()

16/03/27 07:41:24 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:41:24 INFO scheduler.DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[29] at show at DaraFrameOps.scala:20), which has no missing parents

16/03/27 07:41:24 INFO storage.MemoryStore: Block broadcast_17 stored as values in memory (estimated size 7.7 KB, free 2021.2 KB)

16/03/27 07:41:24 INFO storage.MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 4.1 KB, free 2025.3 KB)

16/03/27 07:41:24 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on 192.168.1.121:44773 (size: 4.1 KB, free: 517.2 MB)

16/03/27 07:41:24 INFO spark.SparkContext: Created broadcast 17 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:24 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[29] at show at DaraFrameOps.scala:20)

16/03/27 07:41:24 INFO scheduler.TaskSchedulerImpl: Adding task set 8.0 with 1 tasks

16/03/27 07:41:24 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 8.0 (TID 9, slq3, partition 1,NODE_LOCAL, 2215 bytes)

16/03/27 07:41:25 INFO storage.BlockManagerInfo: Added broadcast_17_piece0 in memory on slq3:47358 (size: 4.1 KB, free: 517.3 MB)

16/03/27 07:41:25 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 8.0 (TID 9) in 717 ms on slq3 (1/1)

16/03/27 07:41:25 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool

16/03/27 07:41:25 INFO scheduler.DAGScheduler: ResultStage 8 (show at DaraFrameOps.scala:20) finished in 0.720 s

16/03/27 07:41:25 INFO scheduler.DAGScheduler: Job 8 finished: show at DaraFrameOps.scala:20, took 0.961406 s

+---+------+

|age|  name|

+---+------+

| 30|  Andy|

| 19|Justin|

+---+------+

 

16/03/27 07:41:26 INFO storage.MemoryStore: Block broadcast_18 stored as values in memory (estimated size 211.3 KB, free 2.2 MB)

16/03/27 07:41:27 INFO storage.MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 19.7 KB, free 2.2 MB)

16/03/27 07:41:27 INFO storage.BlockManagerInfo: Added broadcast_18_piece0 in memory on 192.168.1.121:44773 (size: 19.7 KB, free: 517.2 MB)

16/03/27 07:41:27 INFO spark.SparkContext: Created broadcast 18 from show at DaraFrameOps.scala:21

16/03/27 07:41:27 INFO storage.MemoryStore: Block broadcast_19 stored as values in memory (estimated size 212.1 KB, free 2.4 MB)

16/03/27 07:41:27 INFO storage.MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 19.9 KB, free 2.4 MB)

16/03/27 07:41:27 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on 192.168.1.121:44773 (size: 19.9 KB, free: 517.2 MB)

16/03/27 07:41:27 INFO spark.SparkContext: Created broadcast 19 from show at DaraFrameOps.scala:21

16/03/27 07:41:28 INFO mapred.FileInputFormat: Total input paths to process : 1

16/03/27 07:41:29 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:21

16/03/27 07:41:29 INFO scheduler.DAGScheduler: Registering RDD 35 (show at DaraFrameOps.scala:21)

16/03/27 07:41:29 INFO scheduler.DAGScheduler: Got job 9 (show at DaraFrameOps.scala:21) with 1 output partitions

16/03/27 07:41:29 INFO scheduler.DAGScheduler: Final stage: ResultStage 10 (show at DaraFrameOps.scala:21)

16/03/27 07:41:29 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 9)

16/03/27 07:41:29 INFO scheduler.DAGScheduler: Missing parents: List(ShuffleMapStage 9)

16/03/27 07:41:29 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 9 (MapPartitionsRDD[35] at show at DaraFrameOps.scala:21), which has no missing parents

16/03/27 07:41:29 INFO storage.MemoryStore: Block broadcast_20 stored as values in memory (estimated size 10.7 KB, free 2.4 MB)

16/03/27 07:41:29 INFO storage.MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 5.5 KB, free 2.4 MB)

16/03/27 07:41:29 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on 192.168.1.121:44773 (size: 5.5 KB, free: 517.2 MB)

16/03/27 07:41:29 INFO spark.SparkContext: Created broadcast 20 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:30 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 9 (MapPartitionsRDD[35] at show at DaraFrameOps.scala:21)

16/03/27 07:41:30 INFO scheduler.TaskSchedulerImpl: Adding task set 9.0 with 2 tasks

16/03/27 07:41:30 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 9.0 (TID 10, slq1, partition 0,NODE_LOCAL, 2204 bytes)

16/03/27 07:41:30 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 9.0 (TID 11, slq2, partition 1,NODE_LOCAL, 2204 bytes)

16/03/27 07:41:30 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on slq2:42224 (size: 5.5 KB, free: 517.4 MB)

16/03/27 07:41:31 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on slq2:42224 (size: 19.9 KB, free: 517.3 MB)

16/03/27 07:41:31 INFO storage.BlockManagerInfo: Added broadcast_20_piece0 in memory on slq1:46119 (size: 5.5 KB, free: 517.4 MB)

16/03/27 07:41:33 INFO storage.BlockManagerInfo: Added broadcast_19_piece0 in memory on slq1:46119 (size: 19.9 KB, free: 517.3 MB)

16/03/27 07:41:37 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 9.0 (TID 11) in 7259 ms on slq2 (1/2)

16/03/27 07:41:43 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 9.0 (TID 10) in 12974 ms on slq1 (2/2)

16/03/27 07:41:43 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool

16/03/27 07:41:43 INFO scheduler.DAGScheduler: ShuffleMapStage 9 (show at DaraFrameOps.scala:21) finished in 12.988 s

16/03/27 07:41:43 INFO scheduler.DAGScheduler: looking for newly runnable stages

16/03/27 07:41:43 INFO scheduler.DAGScheduler: running: Set()

16/03/27 07:41:43 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 10)

16/03/27 07:41:43 INFO scheduler.DAGScheduler: failed: Set()

16/03/27 07:41:43 INFO scheduler.DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[39] at show at DaraFrameOps.scala:21), which has no missing parents

16/03/27 07:41:43 INFO storage.MemoryStore: Block broadcast_21 stored as values in memory (estimated size 12.8 KB, free 2.5 MB)

16/03/27 07:41:43 INFO storage.MemoryStore: Block broadcast_21_piece0 stored as bytes in memory (estimated size 6.8 KB, free 2.5 MB)

16/03/27 07:41:43 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on 192.168.1.121:44773 (size: 6.8 KB, free: 517.2 MB)

16/03/27 07:41:43 INFO spark.SparkContext: Created broadcast 21 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:43 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from ResultStage 10 (MapPartitionsRDD[39] at show at DaraFrameOps.scala:21)

16/03/27 07:41:43 INFO scheduler.TaskSchedulerImpl: Adding task set 10.0 with 1 tasks

16/03/27 07:41:43 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 10.0 (TID 12, slq2, partition 0,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:44 INFO storage.BlockManagerInfo: Added broadcast_21_piece0 in memory on slq2:42224 (size: 6.8 KB, free: 517.3 MB)

16/03/27 07:41:45 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to slq2:49520

16/03/27 07:41:45 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 164 bytes

16/03/27 07:41:46 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 10.0 (TID 12) in 3256 ms on slq2 (1/1)

16/03/27 07:41:46 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool

16/03/27 07:41:46 INFO scheduler.DAGScheduler: ResultStage 10 (show at DaraFrameOps.scala:21) finished in 3.264 s

16/03/27 07:41:46 INFO scheduler.DAGScheduler: Job 9 finished: show at DaraFrameOps.scala:21, took 17.401051 s

16/03/27 07:41:47 INFO spark.SparkContext: Starting job: show at DaraFrameOps.scala:21

16/03/27 07:41:47 INFO scheduler.DAGScheduler: Got job 10 (show at DaraFrameOps.scala:21) with 199 output partitions

16/03/27 07:41:47 INFO scheduler.DAGScheduler: Final stage: ResultStage 12 (show at DaraFrameOps.scala:21)

16/03/27 07:41:47 INFO scheduler.DAGScheduler: Parents of final stage: List(ShuffleMapStage 11)

16/03/27 07:41:47 INFO scheduler.DAGScheduler: Missing parents: List()

16/03/27 07:41:47 INFO scheduler.DAGScheduler: Submitting ResultStage 12 (MapPartitionsRDD[39] at show at DaraFrameOps.scala:21), which has no missing parents

16/03/27 07:41:48 INFO storage.MemoryStore: Block broadcast_22 stored as values in memory (estimated size 12.8 KB, free 2.5 MB)

16/03/27 07:41:48 INFO storage.MemoryStore: Block broadcast_22_piece0 stored as bytes in memory (estimated size 6.8 KB, free 2.5 MB)

16/03/27 07:41:48 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on 192.168.1.121:44773 (size: 6.8 KB, free: 517.1 MB)

16/03/27 07:41:48 INFO spark.SparkContext: Created broadcast 22 from broadcast at DAGScheduler.scala:1006

16/03/27 07:41:48 INFO scheduler.DAGScheduler: Submitting 199 missing tasks from ResultStage 12 (MapPartitionsRDD[39] at show at DaraFrameOps.scala:21)

16/03/27 07:41:48 INFO scheduler.TaskSchedulerImpl: Adding task set 12.0 with 199 tasks

16/03/27 07:41:48 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 12.0 (TID 13, slq2, partition 1,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:48 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 12.0 (TID 14, slq1, partition 2,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:49 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on slq2:42224 (size: 6.8 KB, free: 517.3 MB)

16/03/27 07:41:49 INFO storage.BlockManagerInfo: Added broadcast_22_piece0 in memory on slq1:46119 (size: 6.8 KB, free: 517.3 MB)

16/03/27 07:41:49 INFO scheduler.TaskSetManager: Starting task 2.0 in stage 12.0 (TID 15, slq2, partition 3,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:49 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 13) in 909 ms on slq2 (1/199)

16/03/27 07:41:49 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 12.0 (TID 16, slq2, partition 4,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:49 INFO scheduler.TaskSetManager: Finished task 2.0 in stage 12.0 (TID 15) in 250 ms on slq2 (2/199)

16/03/27 07:41:50 INFO scheduler.TaskSetManager: Starting task 4.0 in stage 12.0 (TID 17, slq2, partition 5,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:50 INFO scheduler.TaskSetManager: Finished task 3.0 in stage 12.0 (TID 16) in 387 ms on slq2 (3/199)

16/03/27 07:41:50 INFO scheduler.TaskSetManager: Starting task 5.0 in stage 12.0 (TID 18, slq2, partition 6,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:50 INFO scheduler.TaskSetManager: Finished task 4.0 in stage 12.0 (TID 17) in 292 ms on slq2 (4/199)

16/03/27 07:41:50 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to slq1:57687

16/03/27 07:41:50 INFO scheduler.TaskSetManager: Starting task 6.0 in stage 12.0 (TID 19, slq2, partition 7,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:50 INFO scheduler.TaskSetManager: Finished task 5.0 in stage 12.0 (TID 18) in 282 ms on slq2 (5/199)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Starting task 7.0 in stage 12.0 (TID 20, slq2, partition 8,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Finished task 6.0 in stage 12.0 (TID 19) in 182 ms on slq2 (6/199)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Starting task 8.0 in stage 12.0 (TID 21, slq2, partition 9,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Finished task 7.0 in stage 12.0 (TID 20) in 142 ms on slq2 (7/199)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Starting task 9.0 in stage 12.0 (TID 22, slq2, partition 10,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Finished task 8.0 in stage 12.0 (TID 21) in 211 ms on slq2 (8/199)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Starting task 10.0 in stage 12.0 (TID 23, slq2, partition 11,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Finished task 9.0 in stage 12.0 (TID 22) in 202 ms on slq2 (9/199)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Starting task 11.0 in stage 12.0 (TID 24, slq2, partition 12,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Finished task 10.0 in stage 12.0 (TID 23) in 218 ms on slq2 (10/199)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Starting task 12.0 in stage 12.0 (TID 25, slq2, partition 13,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:51 INFO scheduler.TaskSetManager: Finished task 11.0 in stage 12.0 (TID 24) in 189 ms on slq2 (11/199)

16/03/27 07:41:52 INFO scheduler.TaskSetManager: Starting task 13.0 in stage 12.0 (TID 26, slq2, partition 14,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:52 INFO scheduler.TaskSetManager: Finished task 12.0 in stage 12.0 (TID 25) in 466 ms on slq2 (12/199)

16/03/27 07:41:52 INFO scheduler.TaskSetManager: Starting task 14.0 in stage 12.0 (TID 27, slq2, partition 15,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:52 INFO scheduler.TaskSetManager: Finished task 13.0 in stage 12.0 (TID 26) in 308 ms on slq2 (13/199)

16/03/27 07:41:52 INFO scheduler.TaskSetManager: Starting task 15.0 in stage 12.0 (TID 28, slq2, partition 16,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:52 INFO scheduler.TaskSetManager: Finished task 14.0 in stage 12.0 (TID 27) in 253 ms on slq2 (14/199)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Starting task 16.0 in stage 12.0 (TID 29, slq2, partition 17,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Finished task 15.0 in stage 12.0 (TID 28) in 383 ms on slq2 (15/199)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Starting task 17.0 in stage 12.0 (TID 30, slq2, partition 18,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Finished task 16.0 in stage 12.0 (TID 29) in 226 ms on slq2 (16/199)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Starting task 18.0 in stage 12.0 (TID 31, slq1, partition 19,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Finished task 1.0 in stage 12.0 (TID 14) in 4803 ms on slq1 (17/199)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Starting task 19.0 in stage 12.0 (TID 32, slq2, partition 20,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:53 INFO scheduler.TaskSetManager: Finished task 17.0 in stage 12.0 (TID 30) in 298 ms on slq2 (18/199)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Starting task 20.0 in stage 12.0 (TID 33, slq2, partition 21,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Finished task 19.0 in stage 12.0 (TID 32) in 294 ms on slq2 (19/199)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Starting task 21.0 in stage 12.0 (TID 34, slq1, partition 22,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Finished task 18.0 in stage 12.0 (TID 31) in 659 ms on slq1 (20/199)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Starting task 22.0 in stage 12.0 (TID 35, slq2, partition 23,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Finished task 20.0 in stage 12.0 (TID 33) in 264 ms on slq2 (21/199)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Starting task 23.0 in stage 12.0 (TID 36, slq2, partition 24,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Finished task 22.0 in stage 12.0 (TID 35) in 246 ms on slq2 (22/199)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Starting task 24.0 in stage 12.0 (TID 37, slq1, partition 25,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Finished task 21.0 in stage 12.0 (TID 34) in 633 ms on slq1 (23/199)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Starting task 25.0 in stage 12.0 (TID 38, slq2, partition 26,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:54 INFO scheduler.TaskSetManager: Finished task 23.0 in stage 12.0 (TID 36) in 349 ms on slq2 (24/199)

16/03/27 07:41:55 INFO scheduler.TaskSetManager: Starting task 26.0 in stage 12.0 (TID 39, slq2, partition 27,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:55 INFO scheduler.TaskSetManager: Finished task 25.0 in stage 12.0 (TID 38) in 321 ms on slq2 (25/199)

16/03/27 07:41:55 INFO scheduler.TaskSetManager: Starting task 27.0 in stage 12.0 (TID 40, slq1, partition 28,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:55 INFO scheduler.TaskSetManager: Finished task 24.0 in stage 12.0 (TID 37) in 759 ms on slq1 (26/199)

16/03/27 07:41:55 INFO scheduler.TaskSetManager: Starting task 28.0 in stage 12.0 (TID 41, slq2, partition 29,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:55 INFO scheduler.TaskSetManager: Finished task 26.0 in stage 12.0 (TID 39) in 629 ms on slq2 (27/199)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Starting task 29.0 in stage 12.0 (TID 42, slq1, partition 30,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Finished task 27.0 in stage 12.0 (TID 40) in 797 ms on slq1 (28/199)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Starting task 30.0 in stage 12.0 (TID 43, slq2, partition 31,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Finished task 28.0 in stage 12.0 (TID 41) in 540 ms on slq2 (29/199)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Starting task 31.0 in stage 12.0 (TID 44, slq2, partition 32,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Finished task 30.0 in stage 12.0 (TID 43) in 244 ms on slq2 (30/199)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Starting task 32.0 in stage 12.0 (TID 45, slq2, partition 33,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:56 INFO scheduler.TaskSetManager: Finished task 31.0 in stage 12.0 (TID 44) in 262 ms on slq2 (31/199)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Starting task 33.0 in stage 12.0 (TID 46, slq1, partition 34,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Finished task 29.0 in stage 12.0 (TID 42) in 616 ms on slq1 (32/199)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Starting task 34.0 in stage 12.0 (TID 47, slq2, partition 35,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Finished task 32.0 in stage 12.0 (TID 45) in 452 ms on slq2 (33/199)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Starting task 35.0 in stage 12.0 (TID 48, slq2, partition 36,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Finished task 34.0 in stage 12.0 (TID 47) in 423 ms on slq2 (34/199)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Starting task 36.0 in stage 12.0 (TID 49, slq1, partition 37,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:57 INFO scheduler.TaskSetManager: Finished task 33.0 in stage 12.0 (TID 46) in 925 ms on slq1 (35/199)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Starting task 37.0 in stage 12.0 (TID 50, slq2, partition 38,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Finished task 35.0 in stage 12.0 (TID 48) in 311 ms on slq2 (36/199)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Starting task 38.0 in stage 12.0 (TID 51, slq2, partition 39,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Finished task 37.0 in stage 12.0 (TID 50) in 238 ms on slq2 (37/199)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Starting task 39.0 in stage 12.0 (TID 52, slq1, partition 40,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Finished task 36.0 in stage 12.0 (TID 49) in 535 ms on slq1 (38/199)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Starting task 40.0 in stage 12.0 (TID 53, slq2, partition 41,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:58 INFO scheduler.TaskSetManager: Finished task 38.0 in stage 12.0 (TID 51) in 408 ms on slq2 (39/199)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Starting task 41.0 in stage 12.0 (TID 54, slq2, partition 42,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Finished task 40.0 in stage 12.0 (TID 53) in 429 ms on slq2 (40/199)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Starting task 42.0 in stage 12.0 (TID 55, slq1, partition 43,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Finished task 39.0 in stage 12.0 (TID 52) in 817 ms on slq1 (41/199)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Starting task 43.0 in stage 12.0 (TID 56, slq2, partition 44,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Finished task 41.0 in stage 12.0 (TID 54) in 337 ms on slq2 (42/199)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Starting task 44.0 in stage 12.0 (TID 57, slq1, partition 45,NODE_LOCAL, 2059 bytes)

16/03/27 07:41:59 INFO scheduler.TaskSetManager: Finished task 42.0 in stage 12.0 (TID 55) in 726 ms on slq1 (43/199)

16/03/27 07:42:00 INFO scheduler.TaskSetManager: Starting task 45.0 in stage 12.0 (TID 58, slq2, partition 46,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:00 INFO scheduler.TaskSetManager: Finished task 43.0 in stage 12.0 (TID 56) in 546 ms on slq2 (44/199)

16/03/27 07:42:00 INFO scheduler.TaskSetManager: Starting task 46.0 in stage 12.0 (TID 59, slq2, partition 47,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:00 INFO scheduler.TaskSetManager: Finished task 45.0 in stage 12.0 (TID 58) in 471 ms on slq2 (45/199)

16/03/27 07:42:00 INFO scheduler.TaskSetManager: Starting task 47.0 in stage 12.0 (TID 60, slq1, partition 48,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:00 INFO scheduler.TaskSetManager: Finished task 44.0 in stage 12.0 (TID 57) in 779 ms on slq1 (46/199)

16/03/27 07:42:00 INFO scheduler.TaskSetManager: Starting task 48.0 in stage 12.0 (TID 61, slq2, partition 49,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Finished task 46.0 in stage 12.0 (TID 59) in 317 ms on slq2 (47/199)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Starting task 49.0 in stage 12.0 (TID 62, slq2, partition 50,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Finished task 48.0 in stage 12.0 (TID 61) in 398 ms on slq2 (48/199)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Starting task 50.0 in stage 12.0 (TID 63, slq2, partition 51,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Finished task 49.0 in stage 12.0 (TID 62) in 214 ms on slq2 (49/199)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Starting task 51.0 in stage 12.0 (TID 64, slq1, partition 52,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Finished task 47.0 in stage 12.0 (TID 60) in 851 ms on slq1 (50/199)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Starting task 52.0 in stage 12.0 (TID 65, slq2, partition 53,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:01 INFO scheduler.TaskSetManager: Finished task 50.0 in stage 12.0 (TID 63) in 277 ms on slq2 (51/199)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Starting task 53.0 in stage 12.0 (TID 66, slq2, partition 54,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Finished task 52.0 in stage 12.0 (TID 65) in 371 ms on slq2 (52/199)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Starting task 54.0 in stage 12.0 (TID 67, slq1, partition 55,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Finished task 51.0 in stage 12.0 (TID 64) in 646 ms on slq1 (53/199)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Starting task 55.0 in stage 12.0 (TID 68, slq2, partition 56,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Finished task 53.0 in stage 12.0 (TID 66) in 367 ms on slq2 (54/199)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Starting task 56.0 in stage 12.0 (TID 69, slq2, partition 57,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Finished task 55.0 in stage 12.0 (TID 68) in 313 ms on slq2 (55/199)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Starting task 57.0 in stage 12.0 (TID 70, slq2, partition 58,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:02 INFO scheduler.TaskSetManager: Finished task 56.0 in stage 12.0 (TID 69) in 177 ms on slq2 (56/199)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Starting task 58.0 in stage 12.0 (TID 71, slq2, partition 59,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Finished task 57.0 in stage 12.0 (TID 70) in 275 ms on slq2 (57/199)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Starting task 59.0 in stage 12.0 (TID 72, slq1, partition 60,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Finished task 54.0 in stage 12.0 (TID 67) in 984 ms on slq1 (58/199)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Starting task 60.0 in stage 12.0 (TID 73, slq2, partition 61,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Finished task 58.0 in stage 12.0 (TID 71) in 527 ms on slq2 (59/199)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Starting task 61.0 in stage 12.0 (TID 74, slq1, partition 62,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Finished task 59.0 in stage 12.0 (TID 72) in 547 ms on slq1 (60/199)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Starting task 62.0 in stage 12.0 (TID 75, slq2, partition 63,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:03 INFO scheduler.TaskSetManager: Finished task 60.0 in stage 12.0 (TID 73) in 301 ms on slq2 (61/199)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Starting task 63.0 in stage 12.0 (TID 76, slq2, partition 64,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Finished task 62.0 in stage 12.0 (TID 75) in 295 ms on slq2 (62/199)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Starting task 64.0 in stage 12.0 (TID 77, slq1, partition 65,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Finished task 61.0 in stage 12.0 (TID 74) in 638 ms on slq1 (63/199)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Starting task 65.0 in stage 12.0 (TID 78, slq2, partition 66,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Finished task 63.0 in stage 12.0 (TID 76) in 303 ms on slq2 (64/199)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Starting task 66.0 in stage 12.0 (TID 79, slq1, partition 67,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Finished task 64.0 in stage 12.0 (TID 77) in 489 ms on slq1 (65/199)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Starting task 67.0 in stage 12.0 (TID 80, slq2, partition 68,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:04 INFO scheduler.TaskSetManager: Finished task 65.0 in stage 12.0 (TID 78) in 415 ms on slq2 (66/199)

16/03/27 07:42:05 INFO scheduler.TaskSetManager: Starting task 68.0 in stage 12.0 (TID 81, slq2, partition 69,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:05 INFO scheduler.TaskSetManager: Finished task 67.0 in stage 12.0 (TID 80) in 334 ms on slq2 (67/199)

16/03/27 07:42:05 INFO scheduler.TaskSetManager: Starting task 69.0 in stage 12.0 (TID 82, slq2, partition 70,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:05 INFO scheduler.TaskSetManager: Finished task 68.0 in stage 12.0 (TID 81) in 394 ms on slq2 (68/199)

16/03/27 07:42:05 INFO scheduler.TaskSetManager: Starting task 70.0 in stage 12.0 (TID 83, slq2, partition 71,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:05 INFO scheduler.TaskSetManager: Finished task 69.0 in stage 12.0 (TID 82) in 237 ms on slq2 (69/199)

16/03/27 07:42:05 INFO scheduler.TaskSetManager: Starting task 71.0 in stage 12.0 (TID 84, slq1, partition 72,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:06 INFO scheduler.TaskSetManager: Finished task 66.0 in stage 12.0 (TID 79) in 1170 ms on slq1 (70/199)

16/03/27 07:42:06 INFO scheduler.TaskSetManager: Starting task 72.0 in stage 12.0 (TID 85, slq2, partition 73,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:06 INFO scheduler.TaskSetManager: Finished task 70.0 in stage 12.0 (TID 83) in 502 ms on slq2 (71/199)

16/03/27 07:42:06 INFO scheduler.TaskSetManager: Starting task 73.0 in stage 12.0 (TID 86, slq1, partition 74,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:06 INFO scheduler.TaskSetManager: Finished task 71.0 in stage 12.0 (TID 84) in 657 ms on slq1 (72/199)

16/03/27 07:42:06 INFO scheduler.TaskSetManager: Starting task 74.0 in stage 12.0 (TID 87, slq2, partition 75,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:06 INFO scheduler.TaskSetManager: Finished task 72.0 in stage 12.0 (TID 85) in 381 ms on slq2 (73/199)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Starting task 75.0 in stage 12.0 (TID 88, slq1, partition 76,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Starting task 76.0 in stage 12.0 (TID 89, slq2, partition 77,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Finished task 73.0 in stage 12.0 (TID 86) in 839 ms on slq1 (74/199)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Finished task 74.0 in stage 12.0 (TID 87) in 710 ms on slq2 (75/199)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Starting task 77.0 in stage 12.0 (TID 90, slq2, partition 78,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Finished task 76.0 in stage 12.0 (TID 89) in 473 ms on slq2 (76/199)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Starting task 78.0 in stage 12.0 (TID 91, slq1, partition 79,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:07 INFO scheduler.TaskSetManager: Finished task 75.0 in stage 12.0 (TID 88) in 795 ms on slq1 (77/199)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Starting task 79.0 in stage 12.0 (TID 92, slq2, partition 80,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Finished task 77.0 in stage 12.0 (TID 90) in 317 ms on slq2 (78/199)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Starting task 80.0 in stage 12.0 (TID 93, slq2, partition 81,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Finished task 79.0 in stage 12.0 (TID 92) in 300 ms on slq2 (79/199)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Starting task 81.0 in stage 12.0 (TID 94, slq1, partition 82,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Finished task 78.0 in stage 12.0 (TID 91) in 469 ms on slq1 (80/199)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Starting task 82.0 in stage 12.0 (TID 95, slq2, partition 83,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Finished task 80.0 in stage 12.0 (TID 93) in 326 ms on slq2 (81/199)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Starting task 83.0 in stage 12.0 (TID 96, slq2, partition 84,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:08 INFO scheduler.TaskSetManager: Finished task 82.0 in stage 12.0 (TID 95) in 209 ms on slq2 (82/199)

16/03/27 07:42:09 INFO scheduler.TaskSetManager: Starting task 84.0 in stage 12.0 (TID 97, slq2, partition 85,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:09 INFO scheduler.TaskSetManager: Finished task 83.0 in stage 12.0 (TID 96) in 281 ms on slq2 (83/199)

16/03/27 07:42:10 INFO scheduler.TaskSetManager: Starting task 85.0 in stage 12.0 (TID 98, slq2, partition 86,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:10 INFO scheduler.TaskSetManager: Finished task 84.0 in stage 12.0 (TID 97) in 966 ms on slq2 (84/199)

16/03/27 07:42:10 INFO scheduler.TaskSetManager: Starting task 86.0 in stage 12.0 (TID 99, slq1, partition 87,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:10 INFO scheduler.TaskSetManager: Finished task 81.0 in stage 12.0 (TID 94) in 2378 ms on slq1 (85/199)

16/03/27 07:42:10 INFO scheduler.TaskSetManager: Starting task 87.0 in stage 12.0 (TID 100, slq2, partition 88,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:10 INFO scheduler.TaskSetManager: Finished task 85.0 in stage 12.0 (TID 98) in 775 ms on slq2 (86/199)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Starting task 88.0 in stage 12.0 (TID 101, slq2, partition 89,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Finished task 87.0 in stage 12.0 (TID 100) in 295 ms on slq2 (87/199)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Starting task 89.0 in stage 12.0 (TID 102, slq2, partition 90,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Finished task 88.0 in stage 12.0 (TID 101) in 234 ms on slq2 (88/199)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Starting task 90.0 in stage 12.0 (TID 103, slq1, partition 91,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Finished task 86.0 in stage 12.0 (TID 99) in 765 ms on slq1 (89/199)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Starting task 91.0 in stage 12.0 (TID 104, slq2, partition 92,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:11 INFO scheduler.TaskSetManager: Finished task 89.0 in stage 12.0 (TID 102) in 369 ms on slq2 (90/199)

16/03/27 07:42:12 INFO scheduler.TaskSetManager: Starting task 92.0 in stage 12.0 (TID 105, slq2, partition 93,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:12 INFO scheduler.TaskSetManager: Finished task 91.0 in stage 12.0 (TID 104) in 762 ms on slq2 (91/199)

16/03/27 07:42:12 INFO scheduler.TaskSetManager: Starting task 93.0 in stage 12.0 (TID 106, slq1, partition 94,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:12 INFO scheduler.TaskSetManager: Finished task 90.0 in stage 12.0 (TID 103) in 1333 ms on slq1 (92/199)

16/03/27 07:42:12 INFO scheduler.TaskSetManager: Starting task 94.0 in stage 12.0 (TID 107, slq2, partition 95,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:12 INFO scheduler.TaskSetManager: Finished task 92.0 in stage 12.0 (TID 105) in 390 ms on slq2 (93/199)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Starting task 95.0 in stage 12.0 (TID 108, slq2, partition 96,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Finished task 94.0 in stage 12.0 (TID 107) in 203 ms on slq2 (94/199)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Starting task 96.0 in stage 12.0 (TID 109, slq1, partition 97,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Finished task 93.0 in stage 12.0 (TID 106) in 421 ms on slq1 (95/199)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Starting task 97.0 in stage 12.0 (TID 110, slq2, partition 98,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Finished task 95.0 in stage 12.0 (TID 108) in 271 ms on slq2 (96/199)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Starting task 98.0 in stage 12.0 (TID 111, slq2, partition 99,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Finished task 97.0 in stage 12.0 (TID 110) in 250 ms on slq2 (97/199)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Starting task 99.0 in stage 12.0 (TID 112, slq2, partition 100,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Finished task 98.0 in stage 12.0 (TID 111) in 179 ms on slq2 (98/199)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Starting task 100.0 in stage 12.0 (TID 113, slq1, partition 101,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:13 INFO scheduler.TaskSetManager: Finished task 96.0 in stage 12.0 (TID 109) in 747 ms on slq1 (99/199)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Starting task 101.0 in stage 12.0 (TID 114, slq2, partition 102,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Finished task 99.0 in stage 12.0 (TID 112) in 377 ms on slq2 (100/199)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Starting task 102.0 in stage 12.0 (TID 115, slq2, partition 103,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Finished task 101.0 in stage 12.0 (TID 114) in 188 ms on slq2 (101/199)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Starting task 103.0 in stage 12.0 (TID 116, slq2, partition 104,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Finished task 102.0 in stage 12.0 (TID 115) in 180 ms on slq2 (102/199)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Starting task 104.0 in stage 12.0 (TID 117, slq1, partition 105,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Finished task 100.0 in stage 12.0 (TID 113) in 584 ms on slq1 (103/199)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Starting task 105.0 in stage 12.0 (TID 118, slq2, partition 106,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Finished task 103.0 in stage 12.0 (TID 116) in 175 ms on slq2 (104/199)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Starting task 106.0 in stage 12.0 (TID 119, slq2, partition 107,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:14 INFO scheduler.TaskSetManager: Finished task 105.0 in stage 12.0 (TID 118) in 208 ms on slq2 (105/199)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Starting task 107.0 in stage 12.0 (TID 120, slq2, partition 108,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Starting task 108.0 in stage 12.0 (TID 121, slq1, partition 109,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Finished task 106.0 in stage 12.0 (TID 119) in 324 ms on slq2 (106/199)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Finished task 104.0 in stage 12.0 (TID 117) in 603 ms on slq1 (107/199)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Starting task 109.0 in stage 12.0 (TID 122, slq2, partition 110,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Finished task 107.0 in stage 12.0 (TID 120) in 396 ms on slq2 (108/199)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Starting task 110.0 in stage 12.0 (TID 123, slq1, partition 111,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:15 INFO scheduler.TaskSetManager: Finished task 108.0 in stage 12.0 (TID 121) in 699 ms on slq1 (109/199)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Starting task 111.0 in stage 12.0 (TID 124, slq2, partition 112,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Finished task 109.0 in stage 12.0 (TID 122) in 773 ms on slq2 (110/199)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Starting task 112.0 in stage 12.0 (TID 125, slq2, partition 113,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Finished task 111.0 in stage 12.0 (TID 124) in 280 ms on slq2 (111/199)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Starting task 113.0 in stage 12.0 (TID 126, slq1, partition 114,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Finished task 110.0 in stage 12.0 (TID 123) in 782 ms on slq1 (112/199)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Starting task 114.0 in stage 12.0 (TID 127, slq2, partition 115,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Finished task 112.0 in stage 12.0 (TID 125) in 285 ms on slq2 (113/199)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Starting task 115.0 in stage 12.0 (TID 128, slq2, partition 116,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:16 INFO scheduler.TaskSetManager: Finished task 114.0 in stage 12.0 (TID 127) in 193 ms on slq2 (114/199)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Starting task 116.0 in stage 12.0 (TID 129, slq2, partition 117,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Finished task 115.0 in stage 12.0 (TID 128) in 156 ms on slq2 (115/199)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Starting task 117.0 in stage 12.0 (TID 130, slq1, partition 118,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Finished task 113.0 in stage 12.0 (TID 126) in 624 ms on slq1 (116/199)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Starting task 118.0 in stage 12.0 (TID 131, slq2, partition 119,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Finished task 116.0 in stage 12.0 (TID 129) in 272 ms on slq2 (117/199)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Starting task 119.0 in stage 12.0 (TID 132, slq2, partition 120,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Finished task 118.0 in stage 12.0 (TID 131) in 186 ms on slq2 (118/199)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Starting task 120.0 in stage 12.0 (TID 133, slq1, partition 121,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Finished task 117.0 in stage 12.0 (TID 130) in 327 ms on slq1 (119/199)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Starting task 121.0 in stage 12.0 (TID 134, slq2, partition 122,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Finished task 119.0 in stage 12.0 (TID 132) in 182 ms on slq2 (120/199)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Starting task 122.0 in stage 12.0 (TID 135, slq2, partition 123,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:17 INFO scheduler.TaskSetManager: Finished task 121.0 in stage 12.0 (TID 134) in 295 ms on slq2 (121/199)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Starting task 123.0 in stage 12.0 (TID 136, slq1, partition 124,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Starting task 124.0 in stage 12.0 (TID 137, slq2, partition 125,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Finished task 122.0 in stage 12.0 (TID 135) in 398 ms on slq2 (122/199)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Finished task 120.0 in stage 12.0 (TID 133) in 844 ms on slq1 (123/199)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Starting task 125.0 in stage 12.0 (TID 138, slq1, partition 126,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Finished task 123.0 in stage 12.0 (TID 136) in 515 ms on slq1 (124/199)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Starting task 126.0 in stage 12.0 (TID 139, slq2, partition 127,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Finished task 124.0 in stage 12.0 (TID 137) in 418 ms on slq2 (125/199)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Starting task 127.0 in stage 12.0 (TID 140, slq2, partition 128,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:18 INFO scheduler.TaskSetManager: Finished task 126.0 in stage 12.0 (TID 139) in 234 ms on slq2 (126/199)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Starting task 128.0 in stage 12.0 (TID 141, slq2, partition 129,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Finished task 127.0 in stage 12.0 (TID 140) in 179 ms on slq2 (127/199)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Starting task 129.0 in stage 12.0 (TID 142, slq1, partition 130,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Finished task 125.0 in stage 12.0 (TID 138) in 562 ms on slq1 (128/199)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Starting task 130.0 in stage 12.0 (TID 143, slq2, partition 131,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Finished task 128.0 in stage 12.0 (TID 141) in 542 ms on slq2 (129/199)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Starting task 131.0 in stage 12.0 (TID 144, slq2, partition 132,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:19 INFO scheduler.TaskSetManager: Finished task 130.0 in stage 12.0 (TID 143) in 231 ms on slq2 (130/199)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Starting task 132.0 in stage 12.0 (TID 145, slq1, partition 133,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Finished task 129.0 in stage 12.0 (TID 142) in 875 ms on slq1 (131/199)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Starting task 133.0 in stage 12.0 (TID 146, slq2, partition 134,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Finished task 131.0 in stage 12.0 (TID 144) in 244 ms on slq2 (132/199)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Starting task 134.0 in stage 12.0 (TID 147, slq2, partition 135,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Finished task 133.0 in stage 12.0 (TID 146) in 234 ms on slq2 (133/199)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Starting task 135.0 in stage 12.0 (TID 148, slq1, partition 136,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Finished task 132.0 in stage 12.0 (TID 145) in 450 ms on slq1 (134/199)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Starting task 136.0 in stage 12.0 (TID 149, slq2, partition 137,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Finished task 134.0 in stage 12.0 (TID 147) in 341 ms on slq2 (135/199)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Starting task 137.0 in stage 12.0 (TID 150, slq2, partition 138,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:20 INFO scheduler.TaskSetManager: Finished task 136.0 in stage 12.0 (TID 149) in 284 ms on slq2 (136/199)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Starting task 138.0 in stage 12.0 (TID 151, slq1, partition 139,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Finished task 135.0 in stage 12.0 (TID 148) in 730 ms on slq1 (137/199)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Starting task 139.0 in stage 12.0 (TID 152, slq2, partition 140,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Finished task 137.0 in stage 12.0 (TID 150) in 534 ms on slq2 (138/199)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Starting task 140.0 in stage 12.0 (TID 153, slq2, partition 141,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Starting task 141.0 in stage 12.0 (TID 154, slq1, partition 142,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Finished task 139.0 in stage 12.0 (TID 152) in 402 ms on slq2 (139/199)

16/03/27 07:42:21 INFO scheduler.TaskSetManager: Finished task 138.0 in stage 12.0 (TID 151) in 655 ms on slq1 (140/199)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Starting task 142.0 in stage 12.0 (TID 155, slq2, partition 143,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Finished task 140.0 in stage 12.0 (TID 153) in 302 ms on slq2 (141/199)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Starting task 143.0 in stage 12.0 (TID 156, slq2, partition 144,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Finished task 142.0 in stage 12.0 (TID 155) in 249 ms on slq2 (142/199)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Starting task 144.0 in stage 12.0 (TID 157, slq1, partition 145,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Finished task 141.0 in stage 12.0 (TID 154) in 707 ms on slq1 (143/199)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Starting task 145.0 in stage 12.0 (TID 158, slq2, partition 146,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Finished task 143.0 in stage 12.0 (TID 156) in 233 ms on slq2 (144/199)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Starting task 146.0 in stage 12.0 (TID 159, slq2, partition 147,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Finished task 145.0 in stage 12.0 (TID 158) in 251 ms on slq2 (145/199)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Starting task 147.0 in stage 12.0 (TID 160, slq1, partition 148,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:22 INFO scheduler.TaskSetManager: Finished task 144.0 in stage 12.0 (TID 157) in 396 ms on slq1 (146/199)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Starting task 148.0 in stage 12.0 (TID 161, slq2, partition 149,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Finished task 146.0 in stage 12.0 (TID 159) in 193 ms on slq2 (147/199)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Starting task 149.0 in stage 12.0 (TID 162, slq2, partition 150,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Finished task 148.0 in stage 12.0 (TID 161) in 255 ms on slq2 (148/199)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Starting task 150.0 in stage 12.0 (TID 163, slq1, partition 151,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Finished task 147.0 in stage 12.0 (TID 160) in 652 ms on slq1 (149/199)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Starting task 151.0 in stage 12.0 (TID 164, slq2, partition 152,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Finished task 149.0 in stage 12.0 (TID 162) in 410 ms on slq2 (150/199)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Starting task 152.0 in stage 12.0 (TID 165, slq2, partition 153,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:23 INFO scheduler.TaskSetManager: Finished task 151.0 in stage 12.0 (TID 164) in 228 ms on slq2 (151/199)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Starting task 153.0 in stage 12.0 (TID 166, slq1, partition 154,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Finished task 150.0 in stage 12.0 (TID 163) in 416 ms on slq1 (152/199)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Starting task 154.0 in stage 12.0 (TID 167, slq2, partition 155,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Finished task 152.0 in stage 12.0 (TID 165) in 251 ms on slq2 (153/199)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Starting task 155.0 in stage 12.0 (TID 168, slq2, partition 156,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Finished task 154.0 in stage 12.0 (TID 167) in 332 ms on slq2 (154/199)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Starting task 156.0 in stage 12.0 (TID 169, slq1, partition 157,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Finished task 153.0 in stage 12.0 (TID 166) in 561 ms on slq1 (155/199)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Starting task 157.0 in stage 12.0 (TID 170, slq2, partition 158,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:24 INFO scheduler.TaskSetManager: Finished task 155.0 in stage 12.0 (TID 168) in 314 ms on slq2 (156/199)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Starting task 158.0 in stage 12.0 (TID 171, slq2, partition 159,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Finished task 157.0 in stage 12.0 (TID 170) in 220 ms on slq2 (157/199)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Starting task 159.0 in stage 12.0 (TID 172, slq1, partition 160,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Finished task 156.0 in stage 12.0 (TID 169) in 597 ms on slq1 (158/199)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Starting task 160.0 in stage 12.0 (TID 173, slq2, partition 161,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Finished task 158.0 in stage 12.0 (TID 171) in 210 ms on slq2 (159/199)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Starting task 161.0 in stage 12.0 (TID 174, slq2, partition 162,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Finished task 160.0 in stage 12.0 (TID 173) in 378 ms on slq2 (160/199)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Starting task 162.0 in stage 12.0 (TID 175, slq1, partition 163,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Finished task 159.0 in stage 12.0 (TID 172) in 500 ms on slq1 (161/199)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Starting task 163.0 in stage 12.0 (TID 176, slq2, partition 164,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:25 INFO scheduler.TaskSetManager: Finished task 161.0 in stage 12.0 (TID 174) in 245 ms on slq2 (162/199)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Starting task 164.0 in stage 12.0 (TID 177, slq2, partition 165,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Finished task 163.0 in stage 12.0 (TID 176) in 331 ms on slq2 (163/199)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Starting task 165.0 in stage 12.0 (TID 178, slq2, partition 166,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Finished task 164.0 in stage 12.0 (TID 177) in 247 ms on slq2 (164/199)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Starting task 166.0 in stage 12.0 (TID 179, slq1, partition 167,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Finished task 162.0 in stage 12.0 (TID 175) in 926 ms on slq1 (165/199)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Starting task 167.0 in stage 12.0 (TID 180, slq2, partition 168,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:26 INFO scheduler.TaskSetManager: Finished task 165.0 in stage 12.0 (TID 178) in 398 ms on slq2 (166/199)

16/03/27 07:42:27 INFO scheduler.TaskSetManager: Starting task 168.0 in stage 12.0 (TID 181, slq2, partition 169,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:27 INFO scheduler.TaskSetManager: Finished task 167.0 in stage 12.0 (TID 180) in 784 ms on slq2 (167/199)

16/03/27 07:42:27 INFO scheduler.TaskSetManager: Starting task 169.0 in stage 12.0 (TID 182, slq1, partition 170,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:27 INFO scheduler.TaskSetManager: Finished task 166.0 in stage 12.0 (TID 179) in 1098 ms on slq1 (168/199)

16/03/27 07:42:28 INFO scheduler.TaskSetManager: Starting task 170.0 in stage 12.0 (TID 183, slq1, partition 171,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:28 INFO scheduler.TaskSetManager: Finished task 169.0 in stage 12.0 (TID 182) in 887 ms on slq1 (169/199)

16/03/27 07:42:29 INFO scheduler.TaskSetManager: Starting task 171.0 in stage 12.0 (TID 184, slq1, partition 172,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:29 INFO scheduler.TaskSetManager: Finished task 170.0 in stage 12.0 (TID 183) in 621 ms on slq1 (170/199)

16/03/27 07:42:29 INFO scheduler.TaskSetManager: Starting task 172.0 in stage 12.0 (TID 185, slq1, partition 173,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:29 INFO scheduler.TaskSetManager: Finished task 171.0 in stage 12.0 (TID 184) in 577 ms on slq1 (171/199)

16/03/27 07:42:30 INFO scheduler.TaskSetManager: Starting task 173.0 in stage 12.0 (TID 186, slq1, partition 174,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:30 INFO scheduler.TaskSetManager: Finished task 172.0 in stage 12.0 (TID 185) in 647 ms on slq1 (172/199)

16/03/27 07:42:30 INFO scheduler.TaskSetManager: Starting task 174.0 in stage 12.0 (TID 187, slq2, partition 175,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:30 INFO scheduler.TaskSetManager: Finished task 168.0 in stage 12.0 (TID 181) in 3368 ms on slq2 (173/199)

16/03/27 07:42:30 INFO scheduler.TaskSetManager: Starting task 175.0 in stage 12.0 (TID 188, slq1, partition 176,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:30 INFO scheduler.TaskSetManager: Finished task 173.0 in stage 12.0 (TID 186) in 621 ms on slq1 (174/199)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Starting task 176.0 in stage 12.0 (TID 189, slq2, partition 177,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Finished task 174.0 in stage 12.0 (TID 187) in 303 ms on slq2 (175/199)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Starting task 177.0 in stage 12.0 (TID 190, slq2, partition 178,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Finished task 176.0 in stage 12.0 (TID 189) in 313 ms on slq2 (176/199)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Starting task 178.0 in stage 12.0 (TID 191, slq1, partition 179,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Finished task 175.0 in stage 12.0 (TID 188) in 536 ms on slq1 (177/199)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Starting task 179.0 in stage 12.0 (TID 192, slq2, partition 180,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:31 INFO scheduler.TaskSetManager: Finished task 177.0 in stage 12.0 (TID 190) in 233 ms on slq2 (178/199)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Starting task 180.0 in stage 12.0 (TID 193, slq1, partition 181,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Starting task 181.0 in stage 12.0 (TID 194, slq2, partition 182,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Finished task 178.0 in stage 12.0 (TID 191) in 494 ms on slq1 (179/199)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Finished task 179.0 in stage 12.0 (TID 192) in 382 ms on slq2 (180/199)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Starting task 182.0 in stage 12.0 (TID 195, slq2, partition 183,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Finished task 181.0 in stage 12.0 (TID 194) in 413 ms on slq2 (181/199)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Starting task 183.0 in stage 12.0 (TID 196, slq2, partition 184,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Finished task 182.0 in stage 12.0 (TID 195) in 119 ms on slq2 (182/199)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Starting task 184.0 in stage 12.0 (TID 197, slq1, partition 185,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Finished task 180.0 in stage 12.0 (TID 193) in 550 ms on slq1 (183/199)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Starting task 185.0 in stage 12.0 (TID 198, slq2, partition 186,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Finished task 183.0 in stage 12.0 (TID 196) in 242 ms on slq2 (184/199)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Starting task 186.0 in stage 12.0 (TID 199, slq2, partition 187,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:32 INFO scheduler.TaskSetManager: Finished task 185.0 in stage 12.0 (TID 198) in 134 ms on slq2 (185/199)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Starting task 187.0 in stage 12.0 (TID 200, slq1, partition 188,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Finished task 184.0 in stage 12.0 (TID 197) in 514 ms on slq1 (186/199)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Starting task 188.0 in stage 12.0 (TID 201, slq2, partition 189,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Finished task 186.0 in stage 12.0 (TID 199) in 257 ms on slq2 (187/199)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Starting task 189.0 in stage 12.0 (TID 202, slq2, partition 190,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Finished task 188.0 in stage 12.0 (TID 201) in 422 ms on slq2 (188/199)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Starting task 190.0 in stage 12.0 (TID 203, slq2, partition 191,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:33 INFO scheduler.TaskSetManager: Finished task 189.0 in stage 12.0 (TID 202) in 307 ms on slq2 (189/199)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Starting task 191.0 in stage 12.0 (TID 204, slq2, partition 192,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Finished task 190.0 in stage 12.0 (TID 203) in 210 ms on slq2 (190/199)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Starting task 192.0 in stage 12.0 (TID 205, slq2, partition 193,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Finished task 191.0 in stage 12.0 (TID 204) in 148 ms on slq2 (191/199)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Starting task 193.0 in stage 12.0 (TID 206, slq2, partition 194,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Finished task 192.0 in stage 12.0 (TID 205) in 312 ms on slq2 (192/199)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Starting task 194.0 in stage 12.0 (TID 207, slq2, partition 195,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Finished task 193.0 in stage 12.0 (TID 206) in 193 ms on slq2 (193/199)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Starting task 195.0 in stage 12.0 (TID 208, slq2, partition 196,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:34 INFO scheduler.TaskSetManager: Finished task 194.0 in stage 12.0 (TID 207) in 150 ms on slq2 (194/199)

16/03/27 07:42:35 INFO scheduler.TaskSetManager: Starting task 196.0 in stage 12.0 (TID 209, slq2, partition 197,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:35 INFO scheduler.TaskSetManager: Finished task 195.0 in stage 12.0 (TID 208) in 377 ms on slq2 (195/199)

16/03/27 07:42:35 INFO scheduler.TaskSetManager: Starting task 197.0 in stage 12.0 (TID 210, slq2, partition 198,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:35 INFO scheduler.TaskSetManager: Finished task 196.0 in stage 12.0 (TID 209) in 491 ms on slq2 (196/199)

16/03/27 07:42:35 INFO scheduler.TaskSetManager: Starting task 198.0 in stage 12.0 (TID 211, slq2, partition 199,NODE_LOCAL, 2059 bytes)

16/03/27 07:42:35 INFO scheduler.TaskSetManager: Finished task 197.0 in stage 12.0 (TID 210) in 199 ms on slq2 (197/199)

16/03/27 07:42:36 INFO scheduler.TaskSetManager: Finished task 198.0 in stage 12.0 (TID 211) in 255 ms on slq2 (198/199)

16/03/27 07:42:39 INFO scheduler.TaskSetManager: Finished task 187.0 in stage 12.0 (TID 200) in 6900 ms on slq1 (199/199)

16/03/27 07:42:39 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 12.0, whose tasks have all completed, from pool

16/03/27 07:42:39 INFO scheduler.DAGScheduler: ResultStage 12 (show at DaraFrameOps.scala:21) finished in 51.197 s

16/03/27 07:42:39 INFO scheduler.DAGScheduler: Job 10 finished: show at DaraFrameOps.scala:21, took 52.798009 s

+----+-----+

| age|count|

+----+-----+

|null|    1|

|  19|    1|

|  30|    1|

+----+-----+

 

16/03/27 07:42:40 INFO spark.SparkContext: Invoking stop() from shutdown hook

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null}

16/03/27 07:42:40 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null}

16/03/27 07:42:40 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.1.121:4041

16/03/27 07:42:41 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors

16/03/27 07:42:41 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down

16/03/27 07:42:42 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!

16/03/27 07:42:43 INFO storage.MemoryStore: MemoryStore cleared

16/03/27 07:42:43 INFO storage.BlockManager: BlockManager stopped

16/03/27 07:42:44 INFO storage.BlockManagerMaster: BlockManagerMaster stopped

16/03/27 07:42:44 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

16/03/27 07:42:44 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.

16/03/27 07:42:44 INFO spark.SparkContext: Successfully stopped SparkContext

16/03/27 07:42:44 INFO util.ShutdownHookManager: Shutdown hook called

16/03/27 07:42:44 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-1f459855-2e46-4caa-9b7f-897e3926749c/httpd-6a314baa-db64-40c8-8a36-52541d75c8f7

16/03/27 07:42:45 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-1f459855-2e46-4caa-9b7f-897e3926749c

16/03/27 07:42:45 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.

[richard@slq1 spark]$

 

 

 

 

以上内容是王家林老师DT大数据梦工厂《 IMF传奇行动》第57课的学习笔记。
王家林老师是Spark、Flink、DockerAndroid技术中国区布道师。Spark亚太研究院院长和首席专家,DT大数据梦工厂创始人,Android软硬整合源码级专家,英语发音魔术师,健身狂热爱好者。

微信公众账号:DT_Spark

联系邮箱18610086859@126.com 

电话:18610086859

QQ:1740415547

微信号:18610086859  

新浪微博:ilovepains

 

 

 

 

 

 

0
0

查看评论
* 以上用户言论只代表其个人观点,不代表CSDN网站的观点或立场
    个人资料
    • 访问:80138次
    • 积分:1917
    • 等级:
    • 排名:千里之外
    • 原创:116篇
    • 转载:1篇
    • 译文:0篇
    • 评论:4条
    文章分类
    最新评论