Defining Memory Fragmentation

When Redis deallocates memory, the freed space is not immediately returned to the operating system. Instead, it remains under the control of the underlying memory allocator. A critical issue arises when these released blocks are non-contiguous. While technically free, these scattered fragments cannot be allocated for larger incoming data requests, resulting in wasted space. This unusable free space is known as memory fragmentation.

Measuring Fragmentation in Redis

Redis provides diagnostic tools to evaluate memory usage and fragmentation levels. The following command retrieves detailed memory metrics:

redis-cli --stat memory

Sample output:

# Memory
dataset_bytes:5930032
dataset_bytes_human:5.66M
resident_set_size:8359936
resident_set_size_human:7.97M
...
fragmentation_metric:1.32

dataset_bytes indicates the actual bytes consumed by Redis data structures. resident_set_size represents the total memory the OS has allocated to the Redis process, which includes allocated but unused fragmented space.

The fragmentation ratio is calculated as:

fragmentation_metric = resident_set_size / dataset_bytes

Interpreting the ratio:

• Between 1.0 and 1.5: Acceptable overhead. Fragmentation is inevitable to some degree, and this range does not typically require intervention.
• Greater than 1.5: Severe fragmentation exceeding 50%. Corrective measures should be taken to reclaim wasted memory.
• Less than 1.0: Redis is utilizing less memory than the OS has allocated, indicating that swapping to disk has occurred. This severely degrades performance and demands immediate resolution.

Root Causes of Fragmentation

Memory Allocator Design

Redis defaults to the jemalloc memory allocator, which does not allocate exact byte counts upon request. Instead, it rounds up allocations to the nearest power of two (e.g., 8, 16, 32 bytes, or 4KB, 8KB). If an operation requires 20 bytes, jemalloc will assign a 32-byte block. The leftover 12 bytes become internal fragmentation, an unavoidable byproduct of this strategy. The trade-off is improved performance: pre-allocating larger blocks reduces the frequency of subsequent system-level memory requests if the data grows slightly.

Data Mutation and Deletion Patterns

Objects stored in Redis vary significantly in size. Continuous write, update, and delete operations leave gaps of differing sizes across the memory space.

Consider three contiguous 4KB blocks—Block X, Block Y, and Block Z. If Block Y is erased, the system has 8KB of occupied space and 4KB of free space. However, if a new 5KB object needs to be stored, the 4KB gap is insufficient. The 4KB fragment becomes temporarily unusable, representing external fragmentation.

Mitigating Memory Fragmentation

Instance Restart

Restarting the Redis instance forces the OS to reclaim all memory, eliminating fragmentation. However, this approach carries significant risks. Without RDB or AOF persistence enabled, all data will be lost. Even with persistence, the recovery process induces downtime, making restarts an unappealing first choice.

Active Defragmentation

Introduced in Redis 4.0, active defragmentation handles fragmentation gracefully without downtime. The core mechanism involves relocating existing data in memory to consolidate scattered free blocks into contiguous usable space.

Using the previous example, if the data in Block Z is shifted next to Block X, the previously separated gaps merge into a single 8KB continuous free block, easily accommodating the 5KB allocation request.

To enable this feature dynamically:

CONFIG SET enable_auto_defrag true

Because the defragmentation process executes on Redis's main thread, it can block client requests and impact throughput. Therefore, it should not run continuously. Redis provides granular control parameters to dictate when and how aggressively defragmentation occurs:

# Enable automatic defragmentation
# enable_auto_defrag false

# Minimum fragment size to trigger defrag
# auto_defrag_min_waste_bytes 100mb

# Minimum fragmentation percentage to start defrag
# auto_defrag_trigger_percent 10

# Fragmentation percentage for maximum defrag effort
# auto_defrag_max_effort_percent 100

# CPU time percentage for defrag at minimum threshold
# auto_defrag_cpu_min 1

# CPU time percentage for defrag at maximum threshold
# auto_defrag_cpu_max 25

# Max dict fields scanned per cycle
# auto_defrag_scan_limit 1000

Due to the potential performance penalties, it is generally advised to leave these configurations at their default values unless fragmentation is definitively impacting system stability.

A Practical Analogy

Consider a cinema with rows of paired seats. Two separate individuals book one seat each in different rows. Although two empty seats remain, a couple wishing to sit together cannot book them. The isolated seats represent fragmented, unusable memory. Asking other patrons to shuffle seats to create adjacent vacancies directly mirrors the active defragmentation process.