java调用spark,Spark - 使用参数从java调用Spark jar

I would like to call spark jar from java (to run spark process on yarn), and try to use this link code.

It looks fit in my case, but I need to pass hashmap and some java values to spakr jar. Is it able to pass java object to spark jar?

And is java side able to know how mush spark jar process or is it done? if so, how?

解决方案

I think you misunderstood the content given in data-algorithms

There are 2 ways to submit job

1) Spark-submit like below example from shell script

cat run_secondarysorting.sh

#!/bin/bash

export JAVA_HOME=/usr/java/jdk7

export SPARK_HOME=/home/hadoop/spark-1.1.0

export SPARK_MASTER=spark://myserver100:7077

BOOK_HOME=/home/mp/data-algorithms-book

APP_JAR=$BOOK_HOME/dist/data_algorithms_book.jar

INPUT=/home/hadoop/testspark/timeseries.txt

# Run on a Spark standalone cluster

prog=org.dataalgorithms.chap01.spark.SparkSecondarySort

$SPARK_HOME/bin/spark-submit \

--class $prog \

--master $SPARK_MASTER \

--executor-memory 2G \

--total-executor-cores 20 \

$APP_JAR

2) From Yarn Client which was described in the link.

Usage of Yarn Client is

want to submit Spark jobs from Java code (such as Java servlets or

other Java code such as REST servers).

When you are calling this yarn client... then you need to call as method in your rest-service or servlet etc... (i.e. through web) in which you can also pass parameters like HashMap or any java object kind...

For demo purpose he has written standalone client (with public static void main)

Hope you understood..