Understanding MongoDB Memory Consumption with WiredTiger

Operating MongoDB under significant write pressure, particularly when utilizing the WiredTiger storage engine, frequently presents challenges related to escalating memory consumption and subsequent Out-Of-Memory (OOM) failures. WiredTiger is designed to leverage available RAM extensively for its cache, enhancing performance by keeping frequently accessed data in memory. However, without proper configuration, this can lead to scenarios where the database process exhausts system memory, causing it to crash.

Consider a scenario involving high-throughput data ingestion: an application initiates over 1000 data inserts per second using a connection pool of 500 threads. When MongoDB is deployed within a Docker container, with only the default WiredTiger cache limits active, continuous operation for several hours can result in OOM termination. Analysis of MongoDB logs often reveals critical errors such as:

{"t":{"$date":"2020-11-10T13:11:57.935+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"thread1010","msg":"WiredTiger error","attr":{"error":12,"message":"[1605013917:935682][1:0x7fe214b9b700], file:index-145-6498808884659112531.wt, eviction-server: __posix_file_write, 615: /data/db/index-145-6498808884659112531.wt: handle-write: pwrite: failed to write 12288 bytes at offset 90112: Cannot allocate memory"}}
{"t":{"$date":"2020-11-10T13:11:57.935+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"thread1010","msg":"WiredTiger error","attr":{"error":12,"message":"[1605013917:935887][1:0x7fe214b9b700], eviction-server: __wt_evict_thread_run, 327: cache eviction thread error: Cannot allocate memory"}}
{"t":{"$date":"2020-11-10T13:11:57.935+00:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"thread1010","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1605013917:935926][1:0x7fe214b9b700], eviction-server: __wt_evict_thread_run, 327: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2020-11-10T13:11:57.935+00:00"},"s":"F", "c":"-", "id":23089, "ctx":"thread1010","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":446}}
{"t":{"$date":"2020-11-10T13:11:57.936+00:00"},"s":"F", "c":"-", "id":23090, "ctx":"thread1010","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2020-11-10T13:11:57.936+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"thread1010","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

These log messages conclusively point to memory allocation failures, specifically within WiredTiger's eviction processes, leading to a critical panic and termination.

WiredTiger Memory Management Principles

At its core, WiredTiger manages memory through a combination of an internal cache and an eviction mechanism. It employs a token-based concurrency control system to regulate simultaneous operations. The eviction threads are responsible for reclaiming memory from the cache when it reaches predefined thresholds, pushing dirty pages to disk.

From WiredTiger's perspective, continuous memory growth signifies an imbalance where the rate of data entering the cache (e.g., from new inserts) exceeds the rate at which the eviction process can free up memory. This imbalance can stem from two primary factors:

1. Insufficient Concurrency Tokens: If the number of concurrent write operations (controlled by tokens) is too low, new write requests may queue up, consuming additional memory resources while waiting for execution slots.
2. Ineffective Eviction: The eviction threads might not be able to process and clean the cache quickly enough. This could be due to a low number of eviction threads, or more commonly, the system being overwhelmed by write traffic, making it difficult for eviction to keep pace.

Effectively addressing this problem often involves a trade-off: higher CPU utilization to manage memory more aggressively, there by preventing OOM conditions.

Configuration Strategies to Mitigate OOM

The solution primarily involves adjusting WiredTiger's cache size and its concurrency settings. While specific eviction strategy parameters are not prominently exposed in recent MongoDB documentation (suggesting default behavior are often optimized), careful tuning of the cache and transaction concurrency is critical.

1. WiredTiger Cache Size

The wiredTigerCacheSizeGB parameter defines the maximum size of the WiredTiger internal cache in gigabytes. It's crucial to set this value appropriately based on the system's total RAM. Typically, it's recommended to allocate about 50% of available RAM, or 25% if other processes intensely share memory, but never exceeding total physical memory. Configuring this explicitly ensures that WiredTiger does not consume all available system memory, leaving room for the operating system and MongoDB's other memory needs.

2. Concurrent Write Transactions

The wiredTigerConcurrentWriteTransactions parameter controls the maximum number of concurrent write tickets (operations) that WiredTiger can handle. Increasing this value can improve throughput under heavy write loads by allowing more operations to proceed in parallel. However, setting this too high without suffficient CPU resources can lead to contention and degrade performance. It's important to note that this parameter must be set as a string literal (enclosed in double quotes), even if the value is numeric, when using setParameter in the MongoDB shell.

db.adminCommand({
    setParameter: 1, 
    wiredTigerConcurrentWriteTransactions: "1500" // Example value
});

Example Docker Deployment with Tuned Parameters

When deploying MongoDB in a Docker container, these parameters can be passed directly. For instance, to set the WiredTiger cache size to 2.4 GB and allow 1500 concurrent write transactions:

docker run --name mongodb \
    --cpus 1 \
    -m 4G \
    -v /alidata/MongoData:/data/db \
    -p 27017:27017 \
    -d mongo:4.4.1 \
    --wiredTigerCacheSizeGB 2.4 \
    --setParameter wiredTigerConcurrentWriteTransactions=1500

In this example:

• --cpus 1: Limits the container to 1 CPU core, which can be adjusted based on available resources and workload.
• -m 4G: Sets the container's memory limit to 4 GB.
• --wiredTigerCacheSizeGB 2.4: Allocates 2.4 GB for the WiredTiger cache. This value should be carefully chosen relative to the container's total memory (-m 4G) and the expected operating system overhead.
• --setParameter wiredTigerConcurrentWriteTransactions=1500: Increases the number of concurrent write operations to 1500.

Monitoring and Verifying the Solution

After applying these configuration changes, it's essential to monitor the MongoDB instance to confirm the effectiveness of the adjustments. The mongostat command-line utility is invaluable for this purpose. Key metrics to observe include:

• qrw (Queued Reads/Writes): This column shows the number of operations currently queued for reads and writes. Ideally, these values should remain low (e.g., 0-10) to indicate that the database is processing requests efficiently without significant backlog. Consistently high qrw values suggest that concurrency settings might still be insufficient or that the system is overallocation.
• Memory Usage: Observe the system's overall memory consumption. With WiredTiger, it's common for memory usage to hover around 80% of the allocated cache size. WiredTiger's eviction process is designed to become more aggressive as the cache approaches this threshold. The goal is to ensure memory usage stabilizes below system limits and OOM events are avoided.

By effectively tuning the WiredTiger cache size and concurrent write transaction limits, it's possible to prevent OOM issues. The observed outcome typically involves stable memory usage, often fluctuating around 80% of the configured cache limit, without crashes. This often comes at the cost of increased CPU utilization, as the database engine works harder to manage its cache and process concurrent operations.