Python generators

Python generators are functions that use yield to produce a sequence of values lazily, one at a time. Instead of computing all values and returning a list, generators suspend execution at each yield, resuming when the next value is requested. They're memory-efficient for large sequences and enable elegant state machines.

Use generators for processing large datasets, infinite sequences, or building pipelines where intermediate results don't need to exist simultaneously.

This example shows generators for lazy evaluation and state machines.

# run: python3 generators.py
# description: generators for lazy sequences and state machines
 
# Simple generator
def count_up_to(n):
    i = 0
    while i < n:
        yield i
        i += 1
 
print("Counting:")
for num in count_up_to(5):
    print(f"  {num}")
 
# Generator as pipeline
def squares(numbers):
    for n in numbers:
        yield n * n
 
def evens_only(numbers):
    for n in numbers:
        if n % 2 == 0:
            yield n
 
# Compose generators
numbers = range(10)
result = list(evens_only(squares(numbers)))
print(f"\nEven squares: {result}")
 
# Generator state machine
def reader(filename):
    with open(filename, 'w') as f:
        f.write("line1\nline2\nline3\n")
 
    with open(filename) as f:
        for line in f:
            yield line.strip()
 
import tempfile
import os
with tempfile.TemporaryDirectory() as tmpdir:
    filepath = os.path.join(tmpdir, "test.txt")
    print("\nReading file:")
    for line in reader(filepath):
        print(f"  {line}")
 
# Generator expression (like list comprehension but lazy)
gen = (x * 2 for x in range(1000000))
print(f"\nGenerator type: {type(gen)}")
print(f"First few: {[next(gen) for _ in range(3)]}")
 
# Infinite generator
def infinite_sequence(start=0):
    n = start
    while True:
        yield n
        n += 1
 
gen = infinite_sequence(100)
print(f"Infinite: {[next(gen) for _ in range(5)]}")

Generator functions:

• def func(): yield value: function that returns a generator object
• Each yield suspends the function; next next() resumes it
• return value (Python 3.3+) returns value from StopIteration exception

Generator expressions:

• (x for x in iterable): like list comprehension but lazy
• Syntax: same as list comp but parentheses instead of brackets
• More memory-efficient; compute on demand

Lazy evaluation:

• Values computed only when requested
• Can represent infinite sequences
• Chains efficiently: generator of generators

Pipeline composition:

• Combine generators for data processing
• Each stage produces values on demand
• No intermediate lists allocated

Generator methods:

• next(gen): get next value; raises StopIteration when done
• gen.send(value): resume with value; rarely used
• gen.throw(exception): throw exception into generator
• gen.close(): stop generator

Bidirectional communication:

• value = (yield received_value): receive and send values
• Advanced pattern for coroutines (use async/await instead)

Common uses:

• Reading large files line-by-line
• Processing infinite streams
• Building data pipelines
• Implementing state machines
• Fibonacci, primes, other sequences

Comparison with list comprehension:

• List: [f(x) for x in range(1000000)]: creates million-item list immediately
• Generator: (f(x) for x in range(1000000)): computes on demand, constant memory