分布式基础架构以及好文推荐

• 大规模数据操作

  • Flink
    • A distributive streaming data engine. With pipeline mechanism, it enables the batch process and stream process.
    • Flink offers the iterative execution.
  • Spark
    • A general-purpose distributed data processing engine. (JAVA, Scala, R)
    • Financial streaming data analysis
    • Handle rapid query, analyze and transform data at scale.
    • ETL, SQL batch jobs across large data sets.
  • Hadoop & Hive
    • A distributed data analysis framework
    • A stable data parsing and storing environment
    • Hadoop is an ideal choice to handle cluster computation. HDFS functions as a hardware of this computation system, while MapReduce is the central controller.
    • Hive is an ETL tool that enables the transformation from SQL lines to MapReduce tasks to execute with.

• Papers:
  亚马逊、netflix和linkedln等大型互联网公司正在使用微服务架构模式在云中部署大型应用程序，隔离每一组可以独立开发、测试、部署、扩展、操作和升级的小型服务。然而，除了获得灵活性、独立开发和可扩展性之外，基础设施成本是采用这种模式的公司必须解决的一个主要问题。

  • Amazon :
    • Infrastructure cost comparison
    • Microservice-based Architecture for the NRDC
  • Google
    • GFS
    • GFS vs HDFS