Apache Spark Technical Notes

1. Load data file from local file system:

"Local Mode" --- No Network IO

The absolute path of all files (identical) should be the same on all nodes (including the master node, even when no slaves is running on the same machine with the master node).

Example:

text = sc.textFile("/a/b/ttt.txt")

2. For a typical application running on a Apache spark cluster, each node is responsible for processing a small piece of the overall computation in a parallel style (Map-Reduce). In contrast, each node in a Apache storm cluster is responsible for its own processing/computation work, the relation between each node is sequential instead of parallel.