1、 Resilient Distributed Datasets(RDDs)
Immutable,partitioned collections of objects
不可变,对象分区
Created through parallel transformations(map,filter,groupBy,join…) on data in stable storage
在固定存储上的数据创建并行转换
Can be cached for efficient reuse
数据cache在内存中为再次重用
2、 RDD Fault Tolerance
故障容忍
RDDs maintain lineage information that can be used to reconstruct lost partitions
RDDs维护血统信息用来重建丢失的分区
3、 Aggregations on many keys w/ same WHERE clause
同样where条件,多键值聚集,比Hive快40倍。原因是:
Not re-reading unused columns or filtered records
不用预读不用到的列和 过滤记录
Avoiding repeated decompression
避免重复的解压
In-memory storage of deserialized objects
串行对象存放在内存存储中
4、 Runs on Apache Mesos to share resources with Hadoop & other apps
运行在Mesos,可以和hadoop等其他应用共享资源
Can read from any Hadoop input source (e.g. HDFS)
可以从任何hadoop读取资源,比如hdfs
No changes to Scala compiler
原生scala
5.Spark scheduler(spark调度)
Dryad-like DAGs
类似DAG调度
Pipelines functions within a stage
Stage内部通过管道函数传输
Cache-aware work reuse & locality
缓存感知工作的重用和本地化
Partitioning-aware to avoid shuffles
分区感知避免shuffles