MongoDB performance issue over time. MMAPV1 / WiredTiger. ReIndex.

Case

We have a collection for processed data from Mongo Map Reduce. Which aggregated by DAY, MONTH, YEAR. We clean(Delete) DAY data once a while to prevent the DB from getting too big.

We found that the query performance getting slow over time. Ends up causing user experience issue. We start to see what is happening. DB size is around 10G and the count is 1 million to 2 million.

Debug

During this time we are doing Mongo upgrade (from 3.0 to 3.4) So in test env. We have branch new 3.4 mongo but prod we have 3.0.

What we do is

1 we export prod data and put it in test DB. Run the same query see
what is happening.
We see there is no performance issue on the test.

2 We compare storage size and index size. (PS when we move data from
prod to test, it automatically reindex, we only move data)
We found that totalIndexSize and storage size has a big difference.

That becomes very interesting. What happened on the prod server over time? Or it is something about mongo version.

3 I start looking at release note and change log of MongoDB. But
couldn’t find a big complaint about the old version’s performance. But I
still set up the same version and config (master and slave) in the
test.
But I still couldn’t replicate the issue.

4 Then we start to think about the index difference.
I am guessing reindex the index will solve the problem. So we do the reindex(also change storage engine to wiredTiger)

Solved.

WiredTiger vs MMAPv1

	WiredTiger	MMAPv1
Write Performance	Excellent (No need padding)	Good (Need padding)
Concurrency Control	Document Level	Collection Level
Read Performance	Excellent	Excellent
Compression Support	Yes	No
Write Performance	WiredTiger Cache & File System Cache	Only Journal
Platform Availability	Linux, Windows, MacOS, Solaris(x86)	Linux, Windows, MacOS

Document level locking

Wired tiger: No document level locking but have Concurency Control. (Pros)
(https://docs.mongodb.com/manual/core/wiredtiger/#document-level-concurrency)

MMAPv1: Multi read and single write lock.

Padding

Wired tiger: No need padding. (B-tree data type)
MMAPv1: Each doc has disk size :32KB, 64KB, …, 2MB or multiple 2MB as long as it is less than 16MB. Padding = disk - real size of Doc. (liner storage)