Iterator Fundamentals and Generator Concepts

Python provides multiple ways to traverse data structures. Two primary approaches exist: iteration-based access and index-based access.

Iteration-Based vs Index-Based Access

Iteration-Based Access

• Retrieves values without relying on indices
• Single-pass traversal — each element can only be accessed once
• Works seamlessly with any iterable object

Index-Based Access

• Allows repeated access to the same element through indices
• Requires the container to support random access (lists, tuples, strings)
• Provides more flexibility but demands more memory overhead

Generators: Custom Iterators in Python

Creating large lists upfront consumes significant memory. For instance, a list containing one million elements occupies substantial space, even if you only need the first few values. Python addresses this through generators — objects that produce items on-demand using a lazy evaluation approach.

A generator is essentially a custom iterator that computes values dynamically during iteration rather than storing everything in memory.

The yield Keyword

When a function contains the yield keyword, it transforms from a regular function into a generator. Calling such a function returns a generator object enstead of executing the function body immediately.

def data_stream():
    print('Starting data generation')
    yield 10
    print('Continuing after first yield')
    yield 20
    print('Continuing after second yield')
    yield 30

# Create generator object
processor = data_stream()

# Advance to first yield point
next(processor)  # Output: Starting data generation
                  # Returns: 10

# Advance to second yield point
next(processor)  # Output: Continuing after first yield
                  # Returns: 20

# Advance to third yield point
next(processor)  # Output: Continuing after second yield
                  # Returns: 30

Each call to next() resumes execution from where the previous yield paused, continuing until the next yield is encountered.

Generator Expressions

Generators can also be created using expression syntax, similar to list comprehensions but memory-efficient:

# Generator expression - produces values on demand
square_generator = (x ** 2 for x in range(5))

for value in square_generator:
    print(value)
# Output: 0, 1, 4, 9, 16

Sending Values to Generators

The send() method allows passing values back into a generator:

def multiplier(factor):
    while True:
        received = yield
        print(f'Received: {received}, Computed: {received * factor}')

calc = multiplier(5)
calc.send(None)  # Initialize the generator (equivalent to next(calc))
calc.send(10)    # Output: Received: 10, Computed: 50
calc.send(7)     # Output: Received: 7, Computed: 35

Building a Custom Range Function

Generators can replicate built-in functionality like range():

def custom_range(begin, end=None, step=1):
    if end is None:
        end = begin
        begin = 0
    
    current = begin
    while current < end:
        yield current
        current += step

# Usage examples
for i in custom_range(5):
    print(i)  # 0, 1, 2, 3, 4

for i in custom_range(2, 8):
    print(i)  # 2, 3, 4, 5, 6, 7

for i in custom_range(0, 10, 2):
    print(i)  # 0, 2, 4, 6, 8

yield vs return: Key Differences

Aspect	yield	return
Execution	Pauses function state, allowing resume	Terminates function completely
Return Value	Supports multiple values as tuples	Supports single or multiple values
Type	Converts function to generator	Standard function termination
Iteration	Works with next() and for loops	Ends iteration immediately