Apache Spark 之 入门笔记

Spark 是什么 ## [草稿 - 待整理]

Spark is a java virtual machine(JVM) based distributed data processing engine that scales. And it is fast compare to other data processing framework
Spark的内核是一套通用的执行引擎。Spark平台基于该引擎的其他功能包括：
Spark Core is the general execution engine for the Spark platform that other functionality is built atop
Mesos文献

• 高速内存计算能力
  - in-memory computing capabilities deliver speed
• 广泛的用例支持
  - general execution model supports wide variety of use cases
• 易于开发
  - ease of development – native APIs in Java, Scala, Python (+ SQL, Clojure, R)

Situation	Trade-off	Tech
Cheap Memory	Recompute	RDD
Cheap Storage	Replicate	DFS
Cheap Network	Reference	URI

RDD: Resilient Distributed Dataset
DFS:Distributed File System

Spark 优势

• 比Hadoop MapReduce快100倍
• 利用了工作节点的主要内存进行处理
• 防止了磁盘不必要的I/O运行
• 在应用程序层串联任务的能力
• 最小化磁盘书写数量

Spark 效率

• 使用DAG(Directed Acyclic Graph) 有向无环图

---

Spark 安装

Scala, Python, R –> REPL

—————————————————————————–

Spark 集成

- Databricks Cloud
-
- Spark + Hadoop + Hbase + etc.

- Spark + PyData
- Kafka + Spark + Cassandra
- Spark + ElasticSearch
- Spark + Play
- Spark +Mesos
-