Distributed system design

1. ACP in 12 years
   1. http://blog.csdn.net/chen77716/article/details/30635543
      
2. Memcached
   1. Distributed in-memory key-value store for small chunks of arbitrary data
   2. Membase, later changed to CouchDB
   3. Caveat:
      1. Volatile due to eviction, full, service crashing
      2. Too many items may cause thrashing
      3. Non-transactional
3. UDP vs TCP
   1. UDP is connection-less protocol and faster than TCP, especially when occasional loss is not a big issue such as video stream, VoIP. UDP is very good for pub-sub used in Tibco. Good for heartbeat too. Packages may arrive out of order. Package based
   2. TCP has very huge overhead for handshaking and is bytestream based, therefore merging them back is a big overhead. Point-to-point connection has to be maintained
   3. http://stackoverflow.com/questions/1099672/when-is-it-appropriate-to-use-udp-instead-of-tcp
4. Scatter-gather IO
   1. readv, writev
   2. facebook uses it cut down CPU by 50%
5. Tinyurl / google url shortener
   
   1. Normally they don't expire as long as the company is up and running
   2. Bitly
      
6. Zookeeper
   
   1. http://highscalability.com/zookeeper-reliable-scalable-distributed-coordination-system
   2. More down to the earth explanation: https://www.altamiracorp.com/blog/employee-posts/distributed-coordination-with-zookeeper-part-1-introduction
      1. System characteristics
      2. Curator and Exhibitor from Netflix
         
   3. Powered by ZK including akka and kafka
   4. Might be influenced by Google Chubby (Slides, PDF), https://www.igvita.com/2010/04/30/distributed-coordination-with-zookeeper/
      
   5. Weak points: 1) Unable to capture events happening after old watcher is fired and new watcher is setup, 2) ZK server itself could die and when that happens applications need to find a new/backup ZK server
      
7. Kafka
   1. Benchmarking 2 Million Writes Per Second
   2. Kernel coding is more efficient, such as sendfile, which cannot be done on Windows?
   3. About log (lots of interesting reference reading in the end)
      
8. Paxos
   1. Paxos made simple
   2. Explain by examples
   3. Seems more accurate with actual number
   4. Use Paxos to create replicated log: https://www.youtube.com/watch?v=JEpsBg0AO6o
9. Spark
   1. Check if this is the right solution to this design problem (57. linkedin interview (including design questions))
      
10. Cassandra
    1. Actual discussion about how to design Amazon view history, (57. amazon design discussion (cookie or server side storage)). Also check how cookie is used.
       
11. Spanner
    1. Relational DB is not dead
       1. http://radar.oreilly.com/2012/10/google-spanner-relational-database.html?
       2. But in original video, http://new.livestream.com/accounts/1545775/osdi12/videos/4646642, the K/V part is in place but other relational related stuff not yet
       3. http://highscalability.com/blog/2012/9/24/google-spanners-most-surprising-revelation-nosql-is-out-and.html
          
12. Some basic background info (53. understanding basic advantages of hadoop nosqldb)
13. Rate limiter
    1. Similar to this one (54. dropbox interview (basically a design problem))
    2. (56. google rate limiter design)
       
14. Gossip protocol in Cassandra
    
15. Design a map-like ADT that has add: O(1)  deletion: O(1)  lookup: O(1)  clear:O(1)  iterate: O(number of elements)
    
    1. (54. pure storage interview (including design discussion about a high efficient map))
       
16. Spark, reef
    1. (54. learn spark and reed directly over hadoop (with lots of good discussion))
       
17. Find out top K topics in the past second, minute, hour, day
    
    1. stream processing (spark or kafka+storm)
       
    2. (56. Find out top K topics in the past second minute hour day)
       
    3. http://eugenedvorkin.com/implementing-top-10-most-popular-articles-in-real-time-with-storm-and-mongodb/
    4. http://www.michael-noll.com/blog/2013/01/18/implementing-real-time-trending-topics-in-storm/
    5. https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples/streaming
       
18. How to improve your system design skill
    1. (54. how to improve your system design skills)
19. How to improve your distributed system design capability
    1. (56. how to improve your distributed system design capability)
       
20. Collection of docs to read
    1. One copy: http://blog.csdn.net/supermig/article/details/23434277
    2. Might be kind of old
       
21. Fei Dong's article: http://www.zhihu.com/question/25908682/answer/31637652
    
22. How to design distributed LRU cache, hash table (consistent hashing maybe?)
    
23. Real world examples:
    1. Generate list of your friends latest posts in social service
       1. http://new.livestream.com/accounts/1545775/osdi12/videos/4646642
          
24. RESERVED