Introduction

Like most database systems, MongoDB relies on several log types to track different aspects of its operation. Understanding these logs is essential for monitoring, troubleshooting, and ensuring data integrity. MongoDB primarily uses four kinds of logs: System logs, Journal logs, the Oplog (replication log), and slow query logs. This article explores each type in detail.

System Log

The system log records critical lifecycle events of a MongoDB instance, including startup and shutdown sequences, as well as any runtime anomalies or error conditions.

Configuring the system log is straightforward. When launching mongod, specify the logpath option:

mongod --logpath=/var/log/mongodb/system.log --logappend

MongoDB continuously appends entries to the file designated by logpath.

Journal Log

Journaling is a vital mechanism that ensures data durability in the event of unexpected shutdowns, such as power failures or system crashes. It provides additional reliability through a write-ahead redo log. When journaling is enabled, MongoDB creates a journal entry for each write operation, recording details like the specific disk address and bytes that will be modified. If the server stops abruptly, these journal files can be replayed during startup to reapply any writes that had not yet been flushed to the persistent data files.

When using the WiredTiger storage engine, an in-memory buffer temporarily holds journal records until they are synchronized to disk. The sync occurs based on several conditions:

• Starting with MongoDB 3.2, buffered journal data is flushed to disk every 50 milliseconds.
• A write operation with the write concern { j: true } forces an immediate synchronization of the journal data.
• Because each journal file is capped at 100 MB, WiredTiger automatically creates a new file and syncs the previous one every time approximately 100 MB of journal data accumulates.

When any of these conditions is met, MongoDB commits the pending writes to the journal, meaning updates are not necessarily written to disk one by one. Under default settings, this process guarantees that, in a crash scenario, at most 50 milliseconds of write operations are at risk of being lost.

Data files are flushed to disk every 60 seconds by default. Therefore, journal files only need to retain roughly 60 seconds' worth of write data. To manage this, the journal subsystem pre-allocates empty files in the journal directory located within the dbPath. These files initially appear with names like _j.0, _j.1.

Over time, you will see active journal files such as _j.6217 or _j.6218. The numerical suffix grows as the database runs. After a clean shutdown, journal file are deleted because they are no longer required for recovery.

In WiredTiger, data writes first land in a memory cache and are flushed to disk every 60 seconds. Similarly, journal entries are first written to an in-memory buffer and flushed every 50 milliseconds. Even with out journaling, WiredTiger can restore data from the last checkpoint (similar to a snapshot). However, recovering changes made after the last checkpoint still requires the journal.

If a process crashes or is forcefully terminated (e.g., via kill -9), mongod replays the journal automatically on restart, often logging extensive validation messages during this phase.

The exact mechanisms differ slightly for the deprecated MMAPv1 engine, which flushes journals every 100 milliseconds and uses separate "private views" and "shared views" for journal and data files respectively. Since MongoDB 3.2, WiredTiger is the default and recommended storage engine.

One important behavioral note: if client write operations outpace the rate at which journal data can be committed to disk, mongod actively throttles writes until the sync completes. This is the only scenario where MongoDB intentionally limits write throughput.

Capped Collection Overview

Before discussing the next two log types, it is helpful to understand a capped collection. Unlike regular collections that grow dynamically, a capped collection has a fixed size and behaves like a circular buffer. When the allocated space is full, the oldest documents are automatically removed to make room for new insertions.

To create a capped collection:

db.createCollection("activity_records", { capped: true, size: 100000, max: 100 })

This command creates a collection limited to 100,000 bytes and a maximum document count of 100. Whichever limit is reached first triggers the eviction of the earliest documents. Once created, a capped collection cannot be altered, though a regular collection can be converted into one:

db.runCommand({ convertToCapped: "log_data", size: 10000 })

Capped collections support natural ordering, which reflects the insertion order on disk. To retrieve documents from newest to oldest:

db.activity_records.find().sort({ $natural: -1 })

Oplog (Replication Log)

The oplog (operations log) is the backbone of MongoDB replication. It is a special capped collection stored in the local database on every member of a replica set. The primary node records every write operation in its oplog, and secondary nodes continuously read from this collection to replicate the changes.

A single mongod instance maintains one oplog shared across all databases. Every data-modifying operation (insert, update, delete) is captured there.

Each secondary node also maintains its own oplog, recording the operations it has applied. This design allows any secondary to serve as a sync source for other members.

As illustrated in the replication flow diagram below, a secondary node fetches operations from its current sync source, applies them to its data set, and then records those operations in its own oplog. If an operation fails (which usually indicates data corruption or inconsistency with the primary), the secondary stops replicating from that source.

The oplog preserves all write operations sequentially. Ideally, the oplog of each member is identical to that of the primary, although a small replication lag may exist.

If a secondary node goes offline and restarts, it automatically resumes replication from the last operation recorded in its oplog. Because the replication process writes an entry to its own oplog after applying the operation, it is possible for the same operation to be re-executed. MongoDB is designed to handle this: replaying an oplog entry multiple times produces the same result as executing it once (idempotency).

Since the oplog is a fixed-size collection, it retains only a finite number of operations. The growth rate of the oplog generally mirrors the system's write throughput: processing 1 KB of writes per minute on the primary typically adds about 1 KB of oplog entries per minute. However, operations that affect many documents produce an oplog entry for each modified document. For example, running db.students.remove() on 100,000 records generates 100,000 distinct oplog entries. Heavy batch operations can therefore fill the oplog rapidly, potentially causing replication issues if secondaries fall too far behind.

Slow Query Log

MongoDB uses the database profiler to identify operations that exceed a specified execution time threshold. Profiling data is stored in a capped collection named system.profile within each database. While the profiler offers detailed insights into slow operations, enabling it incurs a performance overhead, so it is typically activated for targeted diagnostics rather than running continuously.

Profiling is off by default. Use the setProfilingLevel command in the shell to configure it:

db.setProfilingLevel(level, slowThresholdMs)
// level: 0 = off, 1 = log slow operations, 2 = log all operations

• Level 0 disables profiling.
• Level 1 records only operations that are slower than a default threshold (usually 100 milliseconds).
• Level 2 captures every operation.

To customize the "slow" threshold, for example, to log all operations exceeding 500 milliseconds:

db.setProfilingLevel(1, 500)

If the system.profile collection does not exist when profiling is enabled, MongoDB automatically creates it as a small capped collection. For longer diagnostic sessions, you might need a larger collection. To adjust this, disable the profiler, drop and recreate system.profile with the desired size, and then re-enable profiling.

Run db.system.profile.stats() to inspect the maximum capacity and current usage of the collection.

Summary

MongoDB's logging infrastructure—comprising system logs, journal logs, the oplog, and the profiler-based slow query log—provides comprehensive visibility into database health, data integrity, replication state, and query performance. Configuring and interpreting these logs appropriately is a fundamental skill for anyone managing MongoDB deployments.