Apache Spark’s Hidden REST API

Apache Spark™ is a fast and general engine for large-scale data processing. It’s really popular as it’s very fast, easy to use and has a great documentation.

It even comes with handy script to submit an application spark-submit 

./bin/spark-submit \
    --class org.apache.spark.examples.SparkPi  \
    --master spark://my-cluster:7077 \
    --deploy-mode cluster \
    /path/to/examples.jar 1000

spark-submit is great if you need to submit applications from command line but what if you need to submit applications from other applications. Let’s say you have a quartz-scheduler based application and you want to periodically submit applications to perform some batch processing.

One option would be to install spark binaries next to your quartz-scheduler and fork processes from your app to submit application. But first of all it’s hacky and second it’s not clear how to check the status of submitted application or how to cancel it.

The second would be to use an open source project called spark-jobserver which tries to solve the problem by providing REST API to submit applications. But this means you have to add additional statefull service to your tech stack, maintain it and take care of availability.

It turns out there is a third surprisingly easy option which is not documented. Spark has a hidden REST API which handles application submission, status checking and cancellation.

Below is the Spark’s hidden REST API.

Submit an Application

	curl -X POST http://spark-cluster-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data'{
	"action" : "CreateSubmissionRequest",
	"appArgs" : [ "myAppArgument1" ],
	"appResource" : "file:/myfilepath/spark-job-1.0.jar",
	"clientSparkVersion" : "1.5.0",
	"environmentVariables" : {
	"SPARK_ENV_LOADED" : "1"
	},
	"mainClass" : "com.mycompany.MyJob",
	"sparkProperties" : {
	"spark.jars" : "file:/myfilepath/spark-job-1.0.jar",
	"spark.driver.supervise" : "false",
	"spark.app.name" : "MyJob",
	"spark.eventLog.enabled": "true",
	"spark.submit.deployMode" : "cluster",
	"spark.master" : "spark://spark-cluster-ip:6066"
	}
	}'

Submission Response

	{
	"action" :"CreateSubmissionResponse",
	"message" :"Driver successfully submitted as driver-20151008145126-0000",
	"serverSparkVersion" :"1.5.0",
	"submissionId" :"driver-20151008145126-0000",
	"success" :true
	}

Get the status of a submitted application

curl http://spark-cluster-ip:6066/v1/submissions/status/driver-20151008145126-0000

Status Response

	{
	"action" :"SubmissionStatusResponse",
	"driverState" :"FINISHED",
	"serverSparkVersion" :"1.5.0",
	"submissionId" :"driver-20151008145126-0000",
	"success" :true,
	"workerHostPort" :"192.168.3.153:46894",
	"workerId" :"worker-20151007093409-192.168.3.153-46894"
	}

Kill submitted application

curl -X POST http://spark-cluster-ip:6066/v1/submissions/kill/driver-20151008145126-0000

Kill Response

	{
	"action" :"KillSubmissionResponse",
	"message" :"Kill request for driver-20151008145126-0000 submitted",
	"serverSparkVersion" :"1.5.0",
	"submissionId" :"driver-20151008145126-0000",
	"success" :true
	}

As you can see there is a very easy way to solve the problem at hand.

There's also another hidden UI Json API: http://[master-node]:[master-ui-port]/json/ which exposes all information available on the master UI in JSON format.