MongoDB Simple Query suddenly become very slow.
Case
We have a collection for processed data from Mongo Map Reduce. Which aggregated by DAY, MONTH, YEAR. We clean(Delete) DAY data once a while to prevent the DB from getting too big.
We found that the query performance getting slow over time. Ends up causing user experience issue. We start to see what is happening. DB size is around 10G and the count is 1 million to 2 million.
Debug
During this time we are doing Mongo upgrade (from 3.0 to 3.4) So in test env. We have branch new 3.4 mongo but prod we have 3.0.
What we do is
1 we export prod data and put it in test DB. Run the same query see
what is happening.
We see there is no performance issue on the test.
2 We compare storage size and index size. (PS when we move data from
prod to test, it automatically reindex, we only move data)
We found that totalIndexSize and storage size has a big difference.
That becomes very interesting. What happened on the prod server over time? Or it is something about mongo version.
3 I start looking at release note and change log of MongoDB. But
couldn’t find a big complaint about the old version’s performance. But I
still set up the same version and config (master and slave) in the
test.
But I still couldn’t replicate the issue.
4 Then we start to think about the index difference.
I am guessing reindex the index will solve the problem. So we do the reindex(also change storage engine to wiredTiger)Solved.
WiredTiger vs MMAPv1
WiredTiger | MMAPv1 | |
---|---|---|
Write Performance | Excellent (No need padding) | Good (Need padding) |
Concurrency Control | Document Level | Collection Level |
Read Performance | Excellent | Excellent |
Compression Support | Yes | No |
Write Performance | WiredTiger Cache & File System Cache | Only Journal |
Platform Availability | Linux, Windows, MacOS, Solaris(x86) | Linux, Windows, MacOS |
Document level locking
Wired tiger: No document level locking but have Concurency Control. (Pros)
(https://docs.mongodb.com/manual/core/wiredtiger/#document-level-concurrency)
MMAPv1: Multi read and single write lock.
Padding
Wired tiger: No need padding. (B-tree data type)
MMAPv1: Each doc has disk size :32KB, 64KB, …, 2MB or multiple 2MB as long as it is less than 16MB. Padding = disk - real size of Doc. (liner storage)