I have a query regarding creating multiple spark sessions in one JVM. I have read that creating multiple contexts is not recommended in earlier versions of Spark. Is it true with the SparkSession in Spark 2.0 as well.
I am thinking of making a call to a web service or a servlet from the UI, and the service creates a spark session, performs some operation and returns the result. This will result in a spark session being created for every request from the client side. Is this practice recommended ?
Say I have a method something like :
public void runSpark() throws Exception {
SparkSession spark = SparkSession
.builder()
.master("spark://")
.appName("JavaWordCount")
.getOrCreate();
and so on....
If I put this method in a web service , will there be any JVM issues ? As such I am able invoke this method multiple times from a main method.But not sure if this is good practice.
解决方案
It is not supported and won't be. SPARK-2243 is resolved as Won't Fix.
If you need multiple contexts there are different projects which can help you (Mist, Livy).