[spark-src] 1-overview

what is

  "Apache Spark™ is a fast and general engine for large-scale data processing....Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk." stated in apache spark 

 

  in despite of it's real a fact or not, i think certain key concepts/components to support these points of view:

a.use Resilient Distributed Datasets(RDD) program modeling largely differs from common ideas,eg. mapreduce.spark uses many optimized algorithms(e.g. iterative,localization etc) spread workload to across many workers in cluster.specially in reuse of data computation.

  RDD:A resilient distributed dataset (RDD) is a read-only col- lection of objects partitioned across a set of machines that can be rebuilt if a partition is lost.[1]

 

b.uses memory as far as possible.most of the intermediate results from spark retains in memory other than disks,so it's  needles suffer from the io problem and serial-deserial cases.

  in fact we use many tools to do similar stuffs ,like memocache,redis..

c.emphasizes the parallism concept.

d.degrades the jvm supervior responsibilities.eg. use one executor to hold on certain tasks instead of one container per task in yarn.

 

architecture

  (the core component is  as a platform for other components)

 

 

usages of spark

1.iterative alogrithms.eg. machine learning,clustering..

2.interactive analystics. eg. query a ton of data loaded from disk to memory to reduce the latency of io

3.batch process

 

program language 

  most of the source code are writing with scala( i think many functions,ideas are inspirated from scala;),but u can also write with java,python in it

 

flex integrations

  many popular frameworks are supported by spark,e.g. hadoop,hbase,mesos etc

 

ref:

[1] some papers 

[spark-src]-source reading