- 博客(4)
- 收藏
- 关注
RSS订阅原创 How-to: resolve spark streaming "Not enough space to cache input-0-* in memory! "
Error:15/09/22 21:52:51 WARN storage.MemoryStore: Failed to reserve initial memory threshold of 1024.0 KB for computing block input-0-1442929965600 in memory.15/09/22 21:52:51 WARN storage.MemoryS
2015-09-24 16:53:14
4856
原创 How-to: use spark to suport query across mysql tables and hbase tables
It wil be good for data analyst that he just run "big sql" to process tables from mysql, hbase or something else. And one more importance thing is the performance thing. We should avoid running "big s
2015-09-18 18:04:40
820
原创 How-to: write own Kafka Partitioner based on requirement
Kafka's default partitioner is based on hash first element which is generated by spliting log via tab. In our usage, this is not a normal case. Our logs are normal log which the first element should b
2015-09-14 11:12:31
560
转载 Spark On YARN内存分配
http://blog.javachen.com/2015/06/09/memory-in-spark-on-yarn.html非常好的文章，推荐。本文主要了解Spark On YARN部署模式下的内存分配情况，因为没有深入研究Spark的源代码，所以只能根据日志去看相关的源代码，从而了解“为什么会这样，为什么会那样”。说明按照Spark应用程序中的driver分布方式不同
2015-09-02 19:04:30
1935
空空如也
空空如也
TA创建的收藏夹 TA关注的收藏夹
TA关注的人