2011年03月_macyang

原创 Why Server Density, foursquare, SourceForge and guardiancouk choose MongoDB?

Why Server Density, foursquare, SourceForge and guardiancouk choose MongoDB? 1) Very easy to install. 2) PHP module available. 3) Very easy replication,including master-mastersupport. In testing this

2011-03-31 20:41:00 642

原创指导你学习MongoDB

文章来源：http://code.google.com/p/netbeams/wiki/MongoDBShardsClusterAndMapReduce可以用它来指导你学习MongoDB.IntroductionThis documents the use of mongoDB as the persistence layer for the collected data from NetBEAMS. It is divided into sections of setup and the CRUD

2011-03-31 19:45:00 578

原创 How we found the rudest cities in the world – Analytics @ foursquare

文章来源： http://engineering.foursquare.com/2011/02/28/how-we-found-the-rudest-cities-in-the-world-analytics-foursquare/ 推荐阅读：Foursquare outage post mortem With over 400 million check-ins in the last year, it’s safe to say t

2011-03-31 15:57:00 724

原创 MongoDB move chunk and balancing

1) Move Chunk: db.runCommand( { moveChunk : "test.blog.posts" , find : { author : "eliot" } , to : "shard1" } ) Parameters: * moveChunk: a full collection name

2011-03-30 17:05:00 1723

原创 Working With Stored JavaScript in MongoDB

文章来源：http://dirolf.com/2010/04/05/stored-javascript-in-mongodb-and-pymongo.htmlJavaScript is a very important language in the MongoDB ecosystem. Not only is the shell JavaScript, but the server features a built-in JS interpreter as well. The embedded i

2011-03-30 09:47:00 1774

原创 Napkin math for MongoDB performance

文章来源：http://rickosborne.org/blog/2010/02/napkin-math-for-mongodb-performance/As we all know, there are lies, damned lies, and statistics. What I’m about to present shouldn’t even qualify as statistics—it’s just a bunch of damned lies. I’m not set up

2011-03-29 23:04:00 942

原创 Mongodb: Array and Join

下面是关于mongodb 1.8 如何返回array部分elements和如何解决Join的讨论：1) How can I return a specific element in an array?(http://groups.google.com/group/mongodb-user/browse_thread/thread/878df8b7805035b3)Current status (目前不支持从array中只返回部分元素):This is currently not possible. Mo

2011-03-29 09:37:00 2004

原创 Redis几个认识误区（Tim总结）

Tim的这篇文章之前就看过，但研究过redis以后，细细品味一下，体会了很多当初没有意识到的东西。 文章来源： http://timyang.net/data/redis-misunderstanding/ 前几天微博发生了一起大的系统故障，很多技术的朋友都比较关心，其中的原因不会超出James Hamilton在On Designing and Deploying Internet-Scale Service(1)概括的那几个范围，J

2011-03-28 10:32:00 961

原创 Excessive Disk Space

You may notice that for a given set of data the MongoDB datafiles in /data/db are larger than the data set inserted into the database. There are several reasons for this.local.* files and replication The replication oplog is preallocate

2011-03-23 12:23:00 640

原创 HBase File Locality in HDFS

罪过啊，之前的几篇翻墙文章已经全部都转过来了，但是这篇却给忘记了。 文章的大意就是hbase是否会保证RegionServer所管理的数据在本地就可以拿到，或者到最近的地方就可以拿到。 文章来源：http://www.larsgeorge.com/2010/05/hbase-file-locality-in-hdfs.html （要翻墙） 中文版翻译：http://www.spnguru.com/?p=42

2011-03-23 00:06:00 2140

原创 MongoDB Schema Design

从关系型数据库转型到文档型存储，最容易犯的一个错误也许就是延用了老的表设计思维来对待新的结构。MongoDB 文档阅读笔记 —— 优雅的 NoSQLSchema DesignSchema Design with MongoDB - April 27MongoDB Schema Design

2011-03-21 15:08:00 1317

原创 Redis为persistent付出2倍的memory，值得不？

下面是redis作者写的一篇文章，里面阐述了为什么redis不使用compact方法去合并aof文件！ 文章来源：A few key problems in Redis persistenceSaturday, 02 October 10 推荐先阅读： http://redis.io/topics/persistence Redis: the strength is the data model, and the

2011-03-17 23:10:00 2814

原创 MongoDB MapReduce实现的group问题

用MapReduce写的group速度不行啊！！1）来源MongoDB权威指南The price of using MapReduce is speed: group is not particularly speedy, butMapReduce is slower and is not supposed to be used in “real time.” You runMapReduce as a background job, it creates a collection of res

2011-03-17 18:56:00 4725

原创从mysql导入数据到mongodb的方法

目前我知道的几种从mysql导入数据到mongodb的方法，如果发现新方法会继续添加。 1）自己写一个程序，从mysql select数据，然后调用insert，插入数据到mongodb中。 2）通过mysql工具将数据导出为csv/json格式的文件，然后使用mongodb自带的mongoimport导入数据。 （当数据量非常大的时候，可以pre-spliting + multi-mongoimport加快导入速度） <b

2011-03-17 00:18:00 14367 1

原创 Mongodb --- Manual sharding

最近在google group看到一个关于manual sharding的讨论，虽然暂时还没亲自去实践一下，但是觉得办法可行，大家都知道google group是要翻墙的，所以贴在这里方便查看.Zer0提出的问题：-----------------------------Sorry for my English I 've read all the documents at home page and search many other sites but I still can not c

2011-03-16 23:42:00 1673

原创 Translate SQL to MongoDB MapReduce

I keep hearing people complaining that MapReduce is not as easy as SQL. But there are others saying SQL is not easy to grok. I’ll keep myself away from this possible flame war and just point you out to this ☞ SQL to MongoDB translation PDF put toge

2011-03-14 19:39:00 2104

原创 MongoDB sharding: understand it first

1. what is shard?2. When to Shard?1）磁盘空间不够的时候2）单个mongod处理不过来client发送的请求3）想让更多的数据存储在内存中3. Incrementing Shard Keys Versus Random Shard Keys?选择递增的shard key是不利于写操作的，如选择timestamp，因为这样会导致所以的写请求全部发送给某个shard，造成这个shard负载很重，而其他shard无所事事。另外也不要选择shard key对

2011-03-13 22:56:00 1214

原创 MongoDB Sharding: A Detailed Overview and 15 Minute High Speed Read

Scaling is a key feature of MongoDB. And even though manual sharding is supported by most databases, MongoDB supports the concept of autosharding. This 15 minute high speed post provides a detailed overview of autosharding in MongoDB and, speci

2011-03-13 19:32:00 1007

原创 How to Choose a MongoDB Shard Key

话说这边文章写的还是不错的，当你要使用Mongodb Sharding模式的时候，选择一个好的shard key是多么重要的一件事情，它将影响balancer接下来要为你需要移动多少chunk，为客户端的请求选择几个shard等等。文章来源：http://techojito.posterous.com/how-to-choose-a-mongodb-shard-keyHow to Choose a MongoDB Shard KeyOne of the benefits of MongoDB is it

2011-03-12 23:44:00 2024

原创关于学习redis的一点想法

1) 简单试用了一下redis，通过网页版命令行测试了常见的redis命令，给自己一个直观的感受，redis都能做哪些事情。2) 因为redis是一个data structure server，所以需要了解每种structure提供了哪些功能（命令），根据自己实际的情况选择合适的redis data structure。3) 深入了解一下redis设计方面的知识，主要通过官方document，这有助于你理解redis架构方面的东西，看它提供了哪些feature（如replication, snaps

2011-03-10 22:40:00 814

原创 MongoDB auto-sharding

下面主要是学习mongodb auto-sharding需要看的一些文档，我会继续在这里更新好的auto-sharding文章！ 1）Sharding Introduction http://www.mongodb.org/display/DOCS/Sharding+Introduction 2）Choosing a Shard Key http://www.mongodb.org/display/DOCS/Choosing

2011-03-10 13:05:00 706

原创 MongoDB Performance for more data than memory

文章来源：http://www.colinhowe.co.uk/2011/02/23/mongodb-performance-for-data-bigger-than-memor/目前有几点需要和作者确认：测试环境是否千兆直连，用的是磁盘还是RAID?另外可以参考基于RAID 10的MongoDB性能测试http://www.cnblogs.com/lovecindywang/archive/2011/03/02/1969324.htmlI've recently been having a play ar

2011-03-09 11:33:00 1146

原创 Write Performance: HBase VS Cassandra with consistency level ALL

文章来源：http://www.quora.com/How-does-HBase-write-performance-differ-from-write-performance-in-Cassandra-with-consistency-level-ALL While setting the a write consistency level of ALL with a read level of ONE in Cassandra provides a strong

2011-03-07 22:39:00 7292

原创 How does HBase perform load balancing?

MauMau提出下面这样一个问题（hbase的版本应该是0.20.xx）：[Q1] Load balancingDoes HBase move regions to a newly added region server (logically, not physically on storage) immediately? If not immediately, what timing?On what criteria does the master unassign and assign regions

2011-03-07 14:13:00 1061

原创 gnuplot:让你的数据可视化

Gnuplot is a portable command-line driven graphing utility for Linux, OS/2, MS Windows, OSX, VMS,and many other platforms. The source code is copyrighted but freely distributed (i.e., you don't have topay for it). It was originally created to allow sci

2011-03-06 22:30:00 1654

原创 HBase Region的路由、分配与拆分

打算从下面三个部分仔细了解一下HBase Region的三种操作：Region的路由问题：比如当你要读一个key/value的时候，你要确定它存储在哪台Region Server上面，这就涉及到region的路由问题，有一篇文章“HBase中的Client如何路由到正确的RegionServer”已经非常详细的介绍了这部门内容，这里面就不多说了，有兴趣进入链接仔细阅读以下。表中最主要的Family：info，info里面包含三个Column：regioninfo, server, serverstartco

2011-03-05 00:11:00 6729

原创 Is there a limit to the number of columns in an HBase row?

Quora上一个哥们提出下面这个问题，之前其实读过的不过没怎么在意comments，而且自己还亲自测试了一下这个问题确定当一个row很大的时候不会发生拆分的。今天又看了一遍，其实这里面已经包含了很多信息：I am wondering if I should have lot of rows or lot of columns in a row? which is better if I have to index them as wellTodd主要从三个方面说明不推荐将大量的column塞到一

2011-03-04 09:28:00 1413

原创 MongoDB: Index Size and Memory & Possible performance impact Options

这个是google group mongodb user小组的一则讨论，提问者首先提出如下的问题：Hi, on our main mongo box, we have 4G memory (wohooo). Our data size is 1G and index size is 5.6G. This box is 1 node of a 3 node replica set. We have 1 collection and a f

2011-03-03 19:03:00 1420

原创 MongoDB Monitoring: Keep in it RAM

文章来源：http://blog.boxedice.com/2010/12/13/mongodb-monitoring-keep-in-it-ram/ 大概的意思就是使用Mongodb的时候要保证索引所占用的空间小于物理内存空间，如果索引所占用的空间太大就想办法考虑使用sharding吧（不然读索引的时候就可能需要从disk读，岂不是更慢！），另外如果索引大小+数据大小大于了物理内存，读性能也好不到哪去，我用Mongodb提供的一个工具mongostat查看page fa

2011-03-03 18:41:00 1354

原创 NoSQL Solution: Evaluation and Comparison: MongoDB vs Redis, Tokyo Cabinet, and Berkeley DB [CHART]

文章来源： http://perfectmarket.com/blog/not_only_nosql_review_solution_evaluation_guide_chart 在原文下面，redis的作者也加了自己的comments，可以仔细查看一下！ You may think this is yet another blog on NoSQL (Not Only SQL) hype. Yes, it is. But if at thi

2011-03-02 20:20:00 1692

原创 Thoughts on Redis

本文翻墙来自于：Thoughts on Redis Like many people already doing so, I’ve been digging into Redis and other NoSQL products. Unlike all those anti-SQL fanboys, I have 15+ years of experience in RDBMS, and I know the relational algebra has a l

2011-03-02 13:12:00 1341

原创 HBase Memstore Flush

我之前一直以为HBase 只会对那些达到了阈值的Memstore进行flush，虽然知道有个配置参数hbase.hregion.memstore.flush.size是控制这个阈值的，但是没想到level是HRegion，但是所有这些理解在读了Visualizing HBase Flushes And Compactions给出的测试结果后才发现当初自己理解错了，于是google一下，发现apache jire上有这样一个ticket，提出的问题如下：Today, the flush decision is

2011-03-01 22:58:00 4648

Mac Track