scala 提交任务

maven添加依赖

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>jydata</groupId>
<artifactId>scala_from_scratch</artifactId>
<version>1.0-SNAPSHOT</version>
<dependencies>
    <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.12</artifactId>
        <version>3.0.0</version>
    </dependency>

</dependencies>


</project>

scala代码

import scala.math.random

import org.apache.spark._

/** Computes an approximation to pi */
object Pi {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://hackintosh:7077")
    val spark = new SparkContext(conf)
    val slices = if (args.length > 0) args(0).toInt else 2
    val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow
    val count = spark.parallelize(1 until n, slices).map { i =>
      val x = random * 2 - 1
      val y = random * 2 - 1
      if (x*x + y*y < 1) 1 else 0
    }.reduce(_ + _)
    println("Pi is roughly " + 4.0 * count / n)
    spark.stop()
  }
}
// scalastyle:on println

idea打包

执行spark-submit

首先要启动spark集群，单机也可以

$SPARK+HOME/bin/start-all.sh

localhost:8080查看

andrew@1hackintosh:~/IdeaProjects/scala_from_scratch#   spark-submit --name SparkPi  \
> --class org.apache.spark.examples.SparkPi \
> --master spark://Master:7077 \
> --executor-memory 512M \
> --total-executor-cores 1 target/scala_from_scratch-1.0-SNAPSHOT.jar


WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.0/libexec/jars/spark-unsafe_2.12-3.0.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/08/31 23:59:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Error: Failed to load class org.apache.spark.examples.SparkPi.
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
andrew@101hackintosh:~/IdeaProjects/scala_from_scratch#   spark-submit --name SparkPi  \
--class Pi \                               
--master spark://Master:7077 \
--executor-memory 512M \
--total-executor-cores 1 target/scala_from_scratch-1.0-SNAPSHOT.jar
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/usr/local/Cellar/apache-spark/3.0.0/libexec/jars/spark-unsafe_2.12-3.0.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/08/31 23:59:46 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
20/08/31 23:59:46 INFO SparkContext: Running Spark version 3.0.0
20/08/31 23:59:46 INFO ResourceUtils: ==============================================================
20/08/31 23:59:46 INFO ResourceUtils: Resources for spark.driver:

20/08/31 23:59:46 INFO ResourceUtils: ==============================================================
20/08/31 23:59:46 INFO SparkContext: Submitted application: Spark Pi
20/08/31 23:59:46 INFO SecurityManager: Changing view acls to: andrew
20/08/31 23:59:46 INFO SecurityManager: Changing modify acls to: andrew
20/08/31 23:59:46 INFO SecurityManager: Changing view acls groups to: 
20/08/31 23:59:46 INFO SecurityManager: Changing modify acls groups to: 
20/08/31 23:59:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(andrew); groups with view permissions: Set(); users  with modify permissions: Set(andrew); groups with modify permissions: Set()
20/08/31 23:59:46 INFO Utils: Successfully started service 'sparkDriver' on port 58738.
20/08/31 23:59:46 INFO SparkEnv: Registering MapOutputTracker
20/08/31 23:59:46 INFO SparkEnv: Registering BlockManagerMaster
20/08/31 23:59:46 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/08/31 23:59:46 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/08/31 23:59:46 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
20/08/31 23:59:46 INFO DiskBlockManager: Created local directory at /private/var/folders/s8/_5116jwj03l5sdj__s1jv4pr0000gn/T/blockmgr-3cb05a1e-0814-4a62-a6cb-c6512a18d0ae
20/08/31 23:59:46 INFO MemoryStore: MemoryStore started with capacity 434.4 MiB
20/08/31 23:59:46 INFO SparkEnv: Registering OutputCommitCoordinator
20/08/31 23:59:46 INFO Utils: Successfully started service 'SparkUI' on port 4040.
20/08/31 23:59:46 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://imac-pro:4040
20/08/31 23:59:46 INFO SparkContext: Added JAR file:/Users/andrew/IdeaProjects/scala_from_scratch/target/scala_from_scratch-1.0-SNAPSHOT.jar at spark://imac-pro:58738/jars/scala_from_scratch-1.0-SNAPSHOT.jar with timestamp 1598889586911
20/08/31 23:59:47 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://hackintosh:7077...
20/08/31 23:59:47 INFO TransportClientFactory: Successfully created connection to hackintosh/192.168.28.94:7077 after 26 ms (0 ms spent in bootstraps)
20/08/31 23:59:47 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20200831235947-0003
20/08/31 23:59:47 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20200831235947-0003/0 on worker-20200831234212-192.168.28.94-51410 (192.168.28.94:51410) with 1 core(s)
20/08/31 23:59:47 INFO StandaloneSchedulerBackend: Granted executor ID app-20200831235947-0003/0 on hostPort 192.168.28.94:51410 with 1 core(s), 512.0 MiB RAM
20/08/31 23:59:47 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 58740.
20/08/31 23:59:47 INFO NettyBlockTransferService: Server created on imac-pro:58740
20/08/31 23:59:47 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/08/31 23:59:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, imac-pro, 58740, None)
20/08/31 23:59:47 INFO BlockManagerMaterEndpoint: Registering block manager imac-pro:58740 with 434.4 MiB RAM, BlockManagerId(driver, imac-pro, 58740, None)
20/08/31 23:59:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, imac-pro, 58740, None)
20/08/31 23:59:47 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, imac-pro, 58740, None)
20/08/31 23:59:47 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20200831235947-0003/0 is now RUNNING
20/08/31 23:59:47 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
20/08/31 23:59:47 INFO SparkContext: Starting job: reduce at Pi.scala:16
20/08/31 23:59:47 INFO DAGScheduler: Got job 0 (reduce at Pi.scala:16) with 2 output partitions
20/08/31 23:59:47 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at Pi.scala:16)
20/08/31 23:59:47 INFO DAGScheduler: Parents of final stage: List()
20/08/31 23:59:47 INFO DAGScheduler: Missing parents: List()
20/08/31 23:59:47 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at Pi.scala:12), which has no missing parents
20/08/31 23:59:47 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.1 KiB, free 434.4 MiB)
20/08/31 23:59:48 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1789.0 B, free 434.4 MiB)
20/08/31 23:59:48 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on imac-pro:58740 (size: 1789.0 B, free: 434.4 MiB)
20/08/31 23:59:48 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1200
20/08/31 23:59:48 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at Pi.scala:12) (first 15 tasks are for partitions Vector(0, 1))
20/08/31 23:59:48 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
20/08/31 23:59:48 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 512, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
20/08/31 23:59:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (192.168.28.94:58755) with ID 0
20/08/31 23:59:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.28.94:58760 with 127.2 MiB RAM, BlockManagerId(0, 192.168.28.94, 58760, None)
20/08/31 23:59:49 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.28.94, executor 0, partition 0, PROCESS_LOCAL, 7397 bytes)
20/08/31 23:59:49 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.28.94:58760 (size: 1789.0 B, free: 127.2 MiB)
20/08/31 23:59:49 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.28.94, executor 0, partition 1, PROCESS_LOCAL, 7397 bytes)
20/08/31 23:59:49 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 638 ms on 192.168.28.94 (executor 0) (1/2)
20/08/31 23:59:49 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 21 ms on 192.168.28.94 (executor 0) (2/2)
20/08/31 23:59:49 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
20/08/31 23:59:49 INFO DAGScheduler: ResultStage 0 (reduce at Pi.scala:16) finished in 2.250 s
20/08/31 23:59:49 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
20/08/31 23:59:49 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
20/08/31 23:59:49 INFO DAGScheduler: Job 0 finished: reduce at Pi.scala:16, took 2.295795 s
Pi is roughly 3.14112
20/08/31 23:59:50 INFO SparkUI: Stopped Spark web UI at http://imac-pro:4040
20/08/31 23:59:50 INFO StandaloneSchedulerBackend: Shutting down all executors
20/08/31 23:59:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
20/08/31 23:59:50 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/08/31 23:59:50 INFO MemoryStore: MemoryStore cleared
20/08/31 23:59:50 INFO BlockManager: BlockManager stopped
20/08/31 23:59:50 INFO BlockManagerMaster: BlockManagerMaster stopped
20/08/31 23:59:50 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/08/31 23:59:50 INFO SparkContext: Successfully stopped SparkContext
20/08/31 23:59:50 INFO ShutdownHookManager: Shutdown hook called
20/08/31 23:59:50 INFO ShutdownHookManager: Deleting directory /private/var/folders/s8/_5116jwj03l5sdj__s1jv4pr0000gn/T/spark-549743d9-66c4-4f1e-a59f-388de7884996
20/08/31 23:59:50 INFO ShutdownHookManager: Deleting directory /private/var/folders/s8/_5116jwj03l5sdj__s1jv4pr0000gn/T/spark-a10dde87-0271-402c-b56f-d797764381ca
andrew@hackintosh:~/IdeaProjects/scala_from_scratch#

ji