Persistent Dictionary Basics

The shelve module provides a dictionary-like interface for storing and retrieving Python objects persistently. When a shelf is opened, the underlying library creates files (such as .dir and .dat) on the disk to manage the stored entries.

A shelf object, created via shelve.open(), behaves like a standard dictionary. You can assign values to keys using db['key'] = value, retrieve them with db['key'], and delete entries using del db['key'].

Core Operations and Mutable Object Pitfalls

The following snippet demonstrates fundamental shelf operations and highlights a critical behavior regarding mutable objects:

import shelve

store = shelve.open('persistent_db')  # Opens or creates the persistent storage file

store['item_id'] = payload             # Assigns data to a key; overwrites if the key exists
retrieved = store['item_id']           # Fetches a copy of the data; raises KeyError if missing
del store['item_id']                   # Removes the key and its associated data
exists = 'item_id' in store            # Evaluates to True if the key is present
all_keys = list(store.keys())          # Generates a list of all stored keys

# Warning: Modifying mutable objects directly requires caution
store['records'] = [0, 1, 2]           # Initial assignment works correctly
store['records'].append(3)             # Fails silently! The value remains [0, 1, 2]

# Correct approach without writeback:
cache = store['records']               # Retrieve a copy of the list
cache.append(5)                        # Mutate the extracted copy
store['records'] = cache               # Reassign the modified copy back to the shelf

store.close()                          # Save changes and close the file handle

If shelve.open() is invoked without writeback=True, direct in-place mutations (like append()) on retrieved objects will not be persisted. The object fetched from the shelf is a copy; altering it does not update the shelf unless the modified object is explicitly reassigned to the key.

Storing and Iterating Various Data Types

A shelf can persist multiple Python data types seamlessly:

import shelve

text_data = "hello"
seq_data = [10, 20, 30]
coords_data = (500, 600)
profile_data = {'username': 'admin', 'level': 5}

with shelve.open('storage_test') as db:
    db['text_entry'] = text_data
    db['seq_entry'] = seq_data
    db['coords_entry'] = coords_data
    db['profile_entry'] = profile_data
    db['literal_entry'] = "direct value"

Reading back the stored entries is straightforward using standard dictionary methods:

with shelve.open('storage_test') as db:
    print(list(db.keys()))        # Outputs all keys
    print(list(db.values()))      # Outputs all values
    
    for k, v in db.items():       # Iterates through key-value pairs
        print(f"Key: {k}")
        print(f"Value: {v}")

    is_present = 'text_entry' in db
    print(f"Key exists: {is_present}")

Handling Mutations with Writeback

To avoid the manual reassignment step when modifying nested or mutable structures, the writeback=True parameter can be passed to shelve.open(). This ensures that any accessed entries are cached in memory, and all modifications are written back to the disk upon closing.

import shelve

user_record = {'username': 'admin', 'level': 5}

with shelve.open('storage_test') as db:
    db['user_profile'] = user_record  # Store the dictionary

print("--- Without Writeback ---")
with shelve.open('storage_test') as db:
    print(db['user_profile'])
    db['user_profile']['username'] = 'guest'  # Modification fails to persist
    print(db['user_profile']['username'])      # Still outputs 'admin'

print("--- With Writeback ---")
with shelve.open('storage_test', writeback=True) as db:
    print(db['user_profile'])
    db['user_profile']['username'] = 'guest'  # Modification succeeds
    print(db['user_profile']['username'])      # Outputs 'guest'

While writeback=True simplifies the syntax for mutating stored objects, it increases memory consumption since all accessed entries are held in the cache. Furthermore, the close() operation becomes slower because it must synchronize all cached changes back to the disk.