Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Redis Storage Architecture: Core Data Types and Underlying Implementations

Tech 1

Redis commands are case-insensitive, while key identifiers remain case-sensitive.

Application-Level Types

Strings A Redis string can store text, integers, or floating-point numbers (for example, 100.01 is handled as a six-byte sequence). Common patterns include standard key-value caching, atomic increment operations (INCR) for optimistic locking workflows, and conditional assignment (SETNX) to implement distributed locks.

Lists Lists maintain insertion order and allow duplicate elements. Accessing records near the head or tail is extremely fast. A single list can hold up to 2^32 − 1 entries. Typical use cases include stack and queue abstractions, user activity feeds, product queues, and comment threads.

Sets Sets enforce uniqueness without maintaining order. The maximum cardinality is 2^32 − 1. They are ideal for membership tracking (for example, follow relationships) and random sampling via SPOP, such as prize-draw systems.

Sorted Sets (ZSet) Elements are unique and ordered by an associated floating-point score; scores themselves may repeat. Because the structure supports range queries by ranking, its widely used for leaderboards and scoring systems.

Hashes Hashes map string fields to string values. Each hash can store up to 2^32 − 1 field-value pairs. They are commonly used to cache object properties or mimic rows from relational tables.

Bitmaps Bitmaps operate at the bit level. A key identifies the subject (for example, a user), and bit offsets represent positions (for example, calendar days). This representation is extremely compact. Patterns include monthly attendance tracking, daily active-user matrices, and real-time online status indicators.

Geospatial (Geo) Introduced in Redis 3.2, the Geo API stores and queries geographic coordinates. It relies on Z-order curves, Base32 encoding, and Geohash to map two-dimensional latitude/longitude pairs into a single 52-bit integer, which is then stored inside a sorted set. This design allows range queries and distance calculations to reuse sorted-set semantics. Common scenarios include storing POI coordinates, calculating distances, and discovering nearby users.

Streams Added in Redis 5.0, Streams provide an append-only log capable of acting as a durable message queue. They support auto-generated IDs, blocking or non-blocking reads, consumer groups, pending-entry lists, and observability commands.

Server Internals

Database Layer Redis logically partitions data into databases. During startup, the server initializes sixteen databases by default, held in an array inside the server state. Each client maintains a pointer to its selected database.

typedef struct db_instance {
    int idx;
    long avg_ttl;
    dict *store;
    dict *expire_map;
    dict *blocking;
    dict *ready;
    dict *watched;
} db_instance;

Object Wrapper Every value is wrapped in a typed object. The header contains a 4-bit logical type, a 4-bit physical encoding, a pointer to the underlying data, a reference count, and a 24-bit LRU/LFU field.

typedef struct redis_obj {
    unsigned obj_type : 4;
    unsigned encode_fmt : 4;
    void *data;
    int refcount;
    unsigned lru_clock : 24;
} redis_obj;

The TYPE command inspects the logical type; OBJECT ENCODING reveals the physical representation.

127.0.0.1:6379> set session abc
OK
127.0.0.1:6379> type session
string
127.0.0.1:6379> set hits 1000
OK
127.0.0.1:6379> object encoding hits
"int"

Primitive Data Structures

Simple Dynamic String (SDS) Redis does not use C strings directly. Instead, it wraps payloads in SDS containers.

Legacy layout (3.x):

struct sds_legacy {
    uint32_t len;
    uint32_t free;
    char buf[];
};

The buf array is implicitly null-terminated, but that trailing byte is not counted in len, allowing reuse of some C string utilities while preserving binary safety.

Modern layout (6.x+) Redis selects among multiple header sizes to minimize overhead:

#define HDR_5  0
#define HDR_8  1
#define HDR_16 2
#define HDR_32 3
#define HDR_64 4

struct __attribute__((packed)) sds_hdr_8 {
    uint8_t len;
    uint8_t alloc;
    uint8_t flags;
    char buf[];
};
/* Similar structures exist for 16-, 32-, and 64-bit length fields. */

Benefits

  • O(1) length queries: No full traversal required.
  • Binary safety: Embedded \0 characters are allowed because length is explicit.
  • Overflow prevention: Append operations verify available space. If insufficient, SDS grows automatically: if the target size is under 1 MiB, capacity doubles; otherwise it expands by exactly 1 MiB. The implementation may switch header types during this growth.
  • Memory efficiency: Typed headers avoid over-allocating metadata for small strings. The packed attribute disables compiler alignment padding, ensuring the header size is exactly the sum of its fields.

Linked Chains When a List object does not qualify for compact encoding, Redis falls back to a doubly linked chain.

typedef struct chain_node {
    struct chain_node *back;
    struct chain_node *fwd;
    void *payload;
} chain_node;

typedef struct chain {
    chain_node *head;
    chain_node *tail;
    unsigned long count;
    void *(*clone)(void *);
    void (*release)(void *);
    int (*compare)(void *, void *);
} chain;

Pros include O(1) head, tail, count, and neighbor access, plus generic storage via void*. Cons include poor CPU cache locality due to scattered allocations and high per-node overhead. To mitigate this, Redis uses compact encodings for small lists.

Hash Tables The dictionary type uses two hash tables to support incremental resizing.

typedef struct hash_tbl {
    bucket **table;
    unsigned long size;
    unsigned long mask;
    unsigned long used;
} hash_tbl;

typedef struct bucket {
    void *key;
    union {
        void *ptr;
        uint64_t u64;
        int64_t s64;
        double dbl;
    } value;
    struct bucket *next;
} bucket;

Collision handling: Chaining links buckets with identical hashes into singly linked lists.

Rehashing: Redis triggers rehash when the load factor (used / size) reaches 1 during normal operation, or 5 to force immediate resolution. Rather than blocking the server, Redis performs incremental rehash:

  1. Allocate a second table, typically twice the size of the first.
  2. Set a migration cursor to 0.
  3. During every read, write, or delete, migrate all entries in the current bucket from the old table to the new one, then advance the cursor.
  4. Once complete, release the old table and promote the new one.

During incremental rehash, new writes go exclusively to the new table; reads probe both.

ZipList ZipList is a compact, contiguous-memory structure used for small Lists, Hashes, and Sorted Sets.

Layout:

  • total_bytes: overall size.
  • tail_offset: distance to the last entry.
  • entry_count: number of entries.
  • end_marker: 0xFF.

Each entry carries:

  • prevlen: length of the previous entry (1 byte if < 254, else 5).
  • encoding: datatype and payload size.
  • payload: actual data.

Because entries store the previous entry’s size, inserting a large entry can enlarge the next entry’s prevlen, potentially triggering a cascading update across subsequent nodes and causing repeated reallocations. This makes ZipList unsuitable for large collections.

QuickList QuickList mitigates ZipList cascading by chaining multiple ZipLists as nodes.

typedef struct ql_node {
    struct ql_node *prev;
    struct ql_node *next;
    unsigned char *zl;
    unsigned int sz;
    unsigned int entries : 16;
} ql_node;

typedef struct quick_list {
    ql_node *head;
    ql_node *tail;
    unsigned long total_entries;
    unsigned long node_count;
} quick_list;

Insertion attempts to fit data into an existing node’s ZipList before allocating a new node. Configuration directives limit individual node size (list-max-ziplist-size) and enable compression of interior nodes (list-compress-depth). Because nodes still use ZipList internally, cascading updates are only reduced, not eliminated.

ListPack Redis 5.0 introduced ListPack to replace ZipList. Entries no longer store the previous entry’s length; instead, each entry records only its own total size. Consequently, inserting or modifying an entry never alters neighboring metadata, completely avoiding cascading updates while retaining compact memory layout.

Entry layout:

  • encoding: format tag for integers or strings of various lengths.
  • payload: stored data.
  • length: total bytes occupied by encoding + payload.

Skip List (ZSet) Sorted sets combine a skip list for range operations and a hash table for O(1) score lookups by member.

typedef struct rank_node {
    struct {
        struct rank_node *ahead;
        unsigned int gap;
    } tier[];
    struct rank_node *rear;
    double weight;
    sds member;
} rank_node;

typedef struct rank_list {
    rank_node *header;
    rank_node *tail;
    unsigned long length;
    int max_tiers;
} rank_list;

Level generation: Each new node’s height is randomized via a coin-flip loop (probability ~0.25 per level) capped at 64. The expected height distribution yields O(log N) average traversal cost.

Advantages: Index tiers store only pointers and span counters, not full objects, so the memory overhead is modest relative to the accelerated search.

Integer Set Small sets composed solely of integers use a flat array.

typedef struct int_group {
    uint32_t codec;
    uint32_t length;
    int8_t payload[];
} int_group;

The payload array’s true width depends on codec (INT16, INT32, or INT64). If a newly inserted integer exceeds the current width, the structure upgrades in-place: it expands the array and widens existing elements. Upgrades are permanent (no downgrade), but they prevent wasting memory on 64-bit storage when all values fit in 16 bits.

Stream A Stream is logically composed of messages, producers, consumers, and consumer groups. Messages carry auto-generated IDs and support acknowledgment tracking, group-exclusive delivery, and blocking reads.

Encoding Matrix

Redis selects physical encodings dynamically based on value size and content.

  • String: int for integral values, embstr for strings up to 44 bytes, and raw for larger strings.

    127.0.0.1:6379> set counter 999
    OK
    127.0.0.1:6379> object encoding counter
    "int"
    127.0.0.1:6379> set tag xxxxx
    OK
    127.0.0.1:6379> object encoding tag
    "embstr"
    
  • List: quicklist.

  • Hash: ziplist or listpack when entry counts and values are small; otherwise hashtable. Thresholds are controlled by hash-max-ziplist-entries and hash-max-ziplist-value.

  • Set: intset when all elements are integers inside the 64-bit signed range and the set is small; otherwise hashtable.

  • Sorted Set: ziplist or listpack for compact data; otherwise a skiplist paired with a dictionary.

Tags: Redis

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.