spark - Running on Cluster

- package spark app (maven)

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.3</version>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
</plugin>

dependency conflict: Shading allows you to make a second copy of the conflicting package under a different namespace and rewrites your application’s code to use the renamed version.

- run driver in client or one of worker node

- scheduling spark apps

queue with priority and capacity defined. submit job to queue.

Fair Scheduler + queue management