Python · Syntax · Advanced

Generators and itertools

10 tasks

Functions that yield values one at a time instead of building a full list. Covers `yield`, `yield from`, generator expressions, and itertools for efficient pipelines.

Quick topic start and explanations before exercises (exercises below):

yield, generator objects, lazy evaluation, and memory

#
**What makes a generator different?** A regular function computes and returns all results at once — the caller waits while the whole list is built in memory. A generator function uses `yield` to produce values *one at a time*, pausing execution between each one. The caller gets a value, does something with it, then asks for the next one: ```python def count_up(n): i = 0 while i < n: yield i # pause here, hand value to caller i += 1 # resume here when next() is called gen = count_up(3) # creates a generator object — runs nothing yet print(next(gen)) # 0 — runs until first yield print(next(gen)) # 1 — resumes, runs until second yield print(next(gen)) # 2 print(next(gen)) # StopIteration — generator is exhausted ``` The key insight: calling `count_up(3)` does *not* run any of the function body. It creates a generator object that remembers where to resume. Execution starts only on the first `next()` call. **Generator state machine** A generator is a suspended function. Python keeps its entire local scope — local variables, the instruction pointer — alive between `yield` calls. This means a generator can hold state without a class: ```python def running_total(numbers): total = 0 for n in numbers: total += n yield total # total persists between yields list(running_total([1, 2, 3, 4])) # [1, 3, 6, 10] ``` **Memory advantage** A generator produces one value at a time and discards it immediately. This means it uses O(1) memory regardless of the sequence length: ```python import sys big_list = [x * x for x in range(1_000_000)] # builds full list — ~8 MB big_gen = (x * x for x in range(1_000_000)) # generator expression — ~200 bytes print(sys.getsizeof(big_list)) # 8448728 print(sys.getsizeof(big_gen)) # 208 ``` Generator expressions use `()` instead of `[]`. They work anywhere an iterable is expected — `sum()`, `max()`, `join()`, `for` loops, etc.: ```python total = sum(x * x for x in range(1_000_000)) # no [] needed inside sum() ``` **When NOT to use generators** Generators are exhausted after one pass — you can't re-iterate or index into them. If you need to iterate the sequence multiple times, or look up items by index, materialise it into a list: `result = list(my_gen())`. Also, generators don't support `len()`. If you need the count and the values, collect into a list first.

yield from, send(), throw(), close(), and infinite generators

#
**`yield from` — delegating to sub-generators** `yield from iterable` yields every item from another iterable without a manual loop. It also correctly propagates `send()`, `throw()`, and `StopIteration` through the delegation chain — something a manual `for` loop cannot do: ```python def flatten(nested): for item in nested: if isinstance(item, list): yield from flatten(item) # delegate recursively else: yield item print(list(flatten([1, [2, [3, 4]], 5]))) # [1, 2, 3, 4, 5] # Without yield from, you'd need: def flatten_manual(nested): for item in nested: if isinstance(item, list): for sub in flatten_manual(item): # extra loop, loses send/throw yield sub else: yield item ``` **`.send()` — two-way communication** A generator can also *receive* values. `.send(value)` resumes the generator and makes `yield` evaluate to the sent value. The first call must be `.send(None)` or `next()` to advance to the first `yield`: ```python def accumulator(): total = 0 while True: value = yield total # yield sends total out, receives next value if value is None: break total += value acc = accumulator() next(acc) # advance to first yield — yields 0 acc.send(10) # total = 10, yields 10 acc.send(5) # total = 15, yields 15 acc.send(None) # exits loop, StopIteration ``` **`.throw()` and `.close()`** `.throw(exc)` injects an exception at the current `yield` point. `.close()` injects `GeneratorExit`, giving the generator a chance to clean up in a `try/finally` block: ```python def managed_gen(): try: while True: yield finally: print('generator cleaned up') # always runs on .close() g = managed_gen() next(g) g.close() # generator cleaned up ``` **Infinite generators** Generators shine when the sequence has no natural end. Since values are lazy, an infinite generator is perfectly safe as long as you control how many items you take from it: ```python def fibonacci(): a, b = 0, 1 while True: yield a a, b = b, a + b def take(n, gen): for _ in range(n): yield next(gen) print(list(take(8, fibonacci()))) # [0, 1, 1, 2, 3, 5, 8, 13] # itertools.islice is the standard library version of take: import itertools print(list(itertools.islice(fibonacci(), 8))) # [0, 1, 1, 2, 3, 5, 8, 13] ```

itertools reference, pipeline patterns, and generator vs list guide

#
**itertools — the generator toolkit** The `itertools` module provides lazy building blocks. All functions return iterators, never lists. They compose naturally to build data pipelines: | Function | What it does | Example | |---|---|---| | `chain(a, b, ...)` | Iterate sequences one after another | `chain('AB', 'CD')` → A B C D | | `islice(it, n)` | Take first n items (lazy slicing) | `islice(fib(), 5)` → first 5 Fibonacci | | `takewhile(pred, it)` | Yield while predicate is True | `takewhile(lambda x: x<5, count())` | | `dropwhile(pred, it)` | Skip while True, then yield rest | `dropwhile(lambda x: x<3, [1,2,3,4])` → 3 4 | | `groupby(it, key)` | Group *consecutive* equal-key items | sort first, then groupby | | `product(a, b)` | Cartesian product — all combinations | `product('AB', [1,2])` → A1 A2 B1 B2 | | `combinations(it, r)` | r-length combos, no repeats | `combinations('ABC', 2)` → AB AC BC | | `permutations(it, r)` | All orderings of r items | `permutations('AB', 2)` → AB BA | | `cycle(it)` | Repeat sequence forever | `cycle([1,2,3])` → 1 2 3 1 2 3 ... | | `repeat(val, n)` | Repeat single value n times | `repeat(0, 5)` → 0 0 0 0 0 | | `accumulate(it, fn)` | Running reduction | `accumulate([1,2,3,4], operator.add)` → 1 3 6 10 | **Building pipelines** The real power of itertools comes from chaining functions together. Each step processes one element at a time — no intermediate lists: ```python import itertools, operator # Find the first 5 even squares above 50 squares = (x * x for x in itertools.count(1)) # 1, 4, 9, 16, ... above_50 = itertools.dropwhile(lambda x: x <= 50, squares) # 64, 81, ... even_above_50 = (x for x in above_50 if x % 2 == 0) # 64, 100, ... result = list(itertools.islice(even_above_50, 5)) print(result) # [64, 100, 196, 256, 400] # Group words by first letter (must be sorted first!) words = ['apple', 'avocado', 'banana', 'blueberry', 'cherry'] for letter, group in itertools.groupby(words, key=lambda w: w[0]): print(letter, list(group)) # a ['apple', 'avocado'] # b ['banana', 'blueberry'] # c ['cherry'] ``` **Decision guide: generator vs list** Use a generator when: - You iterate once and discard the values - The sequence is very large or infinite - You need to start producing results before the sequence is complete - You're building a pipeline where each step filters or transforms Use a list when: - You need to iterate more than once - You need `len()`, indexing, or slicing - You're passing the result to code that expects a sequence - The sequence is small and you want the debugging convenience of seeing all values at once

Exercises:

01

Fibonacci generator

#
Write a generator function `fibonacci()` that yields Fibonacci numbers indefinitely (0, 1, 1, 2, 3, 5, 8, ...). Use `itertools.islice` to get the first `n` values from it.
import itertools

def fibonacci():
    pass


first10 = list(itertools.islice(fibonacci(), 10))
print(first10)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
Solution 
import itertools

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b


first10 = list(itertools.islice(fibonacci(), 10))
print(first10)  # [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
02

Custom range generator

#
Write a generator function `frange(start, stop, step)` that yields float values from `start` up to (but not including) `stop` with the given `step`. It should work like `range()` but support float steps.
def frange(start, stop, step):
    pass


print(list(frange(0, 1, 0.25)))   # [0, 0.25, 0.5, 0.75]
print(list(frange(1, 2, 0.3)))    # [1, 1.3, 1.6, 1.9]
Solution 
def frange(start, stop, step):
    current = start
    while current < stop:
        yield current
        current += step


print(list(frange(0, 1, 0.25)))   # [0, 0.25, 0.5, 0.75]
print(list(frange(1, 2, 0.3)))    # [1, 1.3, 1.6000000000000001, 1.9]
03

Flatten nested lists with yield from

#
Write a generator `flatten(nested)` that recursively flattens a nested list of any depth and yields individual items. Use `yield from` for recursion. Items that are not lists should be yielded as-is.
def flatten(nested):
    pass


data = [1, [2, 3], [4, [5, 6]], [[7], 8]]
print(list(flatten(data)))  # [1, 2, 3, 4, 5, 6, 7, 8]
Solution 
def flatten(nested):
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item


data = [1, [2, 3], [4, [5, 6]], [[7], 8]]
print(list(flatten(data)))  # [1, 2, 3, 4, 5, 6, 7, 8]
04

Running average with send()

#
Write a coroutine generator `running_average()` that computes a running average. Each call to `.send(value)` should add `value` to the total and `yield` the current average. Prime the generator with `next()` before sending values.
def running_average():
    pass


avg = running_average()
next(avg)          # prime the coroutine
print(avg.send(10))  # 10.0
print(avg.send(20))  # 15.0
print(avg.send(30))  # 20.0
Solution 
def running_average():
    total = 0
    count = 0
    value = yield  # prime point — first next() pauses here
    while True:
        total += value
        count += 1
        value = yield total / count


avg = running_average()
next(avg)            # advance to first yield
print(avg.send(10))  # 10.0
print(avg.send(20))  # 15.0
print(avg.send(30))  # 20.0
05

Sliding window generator

#
Write a generator `sliding_window(iterable, n)` that yields tuples of `n` consecutive elements from the iterable, sliding one step at a time. For example, `sliding_window([1,2,3,4,5], 3)` yields `(1,2,3)`, `(2,3,4)`, `(3,4,5)`.
from collections import deque

def sliding_window(iterable, n):
    pass


print(list(sliding_window([1, 2, 3, 4, 5], 3)))
# [(1, 2, 3), (2, 3, 4), (3, 4, 5)]
Solution 
from collections import deque

def sliding_window(iterable, n):
    window = deque(maxlen=n)
    for item in iterable:
        window.append(item)
        if len(window) == n:
            yield tuple(window)


print(list(sliding_window([1, 2, 3, 4, 5], 3)))
# [(1, 2, 3), (2, 3, 4), (3, 4, 5)]
06

take and drop generators

#
Write two generator functions: - `take(n, iterable)` — yields the first `n` items from the iterable. - `drop(n, iterable)` — skips the first `n` items and yields the rest. Both should work lazily (do not convert to list internally).
def take(n, iterable):
    pass

def drop(n, iterable):
    pass


nums = range(10)
print(list(take(3, nums)))   # [0, 1, 2]
print(list(drop(7, nums)))   # [7, 8, 9]
Solution 
def take(n, iterable):
    for i, item in enumerate(iterable):
        if i >= n:
            break
        yield item

def drop(n, iterable):
    for i, item in enumerate(iterable):
        if i >= n:
            yield item


nums = range(10)
print(list(take(3, nums)))   # [0, 1, 2]
print(list(drop(7, nums)))   # [7, 8, 9]

# Or with itertools:
import itertools
list(itertools.islice(nums, 3))       # take(3)
list(itertools.islice(nums, 7, None)) # drop(7)
07

Chunk generator

#
Write a generator `chunks(iterable, n)` that splits the iterable into consecutive chunks of size `n`. The last chunk may be smaller if the iterable length is not divisible by `n`. Yield each chunk as a list.
def chunks(iterable, n):
    pass


print(list(chunks(range(10), 3)))
# [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]
Solution 
def chunks(iterable, n):
    chunk = []
    for item in iterable:
        chunk.append(item)
        if len(chunk) == n:
            yield chunk
            chunk = []
    if chunk:  # yield the last partial chunk
        yield chunk


print(list(chunks(range(10), 3)))
# [[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]

# Python 3.12+: itertools.batched(iterable, n)
08

Prime number sieve generator

#
Write a generator `primes()` that yields prime numbers indefinitely using a simple trial-division approach: for each candidate number, check divisibility only by previously found primes. Use `itertools.islice` to get the first 10 primes.
import itertools

def primes():
    pass


print(list(itertools.islice(primes(), 10)))
# [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
Solution 
import itertools

def primes():
    found = []
    candidate = 2
    while True:
        if all(candidate % p != 0 for p in found):
            found.append(candidate)
            yield candidate
        candidate += 1


print(list(itertools.islice(primes(), 10)))
# [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]
09

itertools pipeline

#
Using `itertools.chain`, `itertools.filterfalse`, and a generator expression, build a pipeline that: 1. Chains two lists together: `[1, 2, 3]` and `[4, 5, 6]`. 2. Filters out even numbers. 3. Squares the remaining numbers. Print the result as a list.
import itertools

a = [1, 2, 3]
b = [4, 5, 6]

# Build the pipeline using itertools.chain, itertools.filterfalse,
# and a generator expression for squaring.
result = []
print(result)  # [1, 9, 25]
Solution 
import itertools

a = [1, 2, 3]
b = [4, 5, 6]

chained = itertools.chain(a, b)                      # 1 2 3 4 5 6
odds = itertools.filterfalse(lambda x: x % 2 == 0, chained)  # 1 3 5
result = list(x * x for x in odds)                  # 1 9 25
print(result)  # [1, 9, 25]
10

Group words by first letter with itertools.groupby

#
Given a sorted list of words, use `itertools.groupby` to group them by their first letter and print each group on one line: `'a: apple, avocado'`. The list is already sorted alphabetically.
import itertools

words = ['apple', 'avocado', 'banana', 'blueberry', 'cherry', 'coconut']

# Use itertools.groupby to group by first letter
# Expected output:
# a: apple, avocado
# b: banana, blueberry
# c: cherry, coconut
Solution 
import itertools

words = ['apple', 'avocado', 'banana', 'blueberry', 'cherry', 'coconut']

for letter, group in itertools.groupby(words, key=lambda w: w[0]):
    print(f'{letter}: {", ".join(group)}')