Generators in Python: yield, Iterators and List and Dict Comprehensions

A generator in Python is an object that yields values one at a time, only when they are requested, without holding the whole sequence in memory. There are two ways to create one: a function with the yield keyword, or a generator expression in round brackets. Alongside them live the list, dict and set comprehensions — a short notation for building a finished collection in one line. Let us look at how yield works, how a generator differs from an iterator, and how to avoid turning a comprehension into an unreadable line.

Iterators: what the for loop runs on

To understand generators, you first need to understand iterators. An iterable is anything you can walk through with a loop: a list, a string, a dictionary, a file. An iterator is an object that remembers its current position and, on a call to next(), gives the next element; when the elements run out, it raises StopIteration.

nums = [10, 20, 30]
it = iter(nums)      # get an iterator from the list
print(next(it))      # 10
print(next(it))      # 20
print(next(it))      # 30
next(it)             # StopIteration

A for loop does exactly this: it calls iter(), then next() until StopIteration. You can write your own iterator as a class with the __iter__ and __next__ methods:

class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value

print(list(Countdown(3)))  # [3, 2, 1]

This works, but it is a lot of boilerplate. Generators do the same thing in three lines.

What a generator in Python is and how yield works

A generator function is an ordinary function that contains yield in its body. Calling it does not execute the code; it returns a generator object. Each next() runs the body up to the nearest yield, hands out the value and freezes the function together with its local variables. The next next() continues from exactly the same place.

def countdown(start):
    while start > 0:
        yield start
        start -= 1

gen = countdown(3)
print(gen)          # <generator object countdown at 0x...>
print(next(gen))    # 3
print(list(gen))    # [2, 1] — continued from where it stopped

When the function reaches its end (or a return), the generator raises StopIteration by itself, and the for loop finishes neatly. Hence the main difference between yield and return: return ends the function and returns one value, while yield suspends it and can hand out as many values as needed.

Generator and iterator: the differences

Any generator is an iterator: it has __iter__ and __next__. But not every iterator is a generator. The difference lies in how it is created and in its capabilities.

They share one important property: you can go through them only once. A second for over an exhausted generator yields nothing — you have to create a new one.

An infinite generator and reading large files

A generator does not have to end. An infinite sequence is safe as long as you take a limited number of elements from it:

from itertools import islice

def ids():
    n = 1
    while True:
        yield f"ID-{n:04d}"
        n += 1

print(list(islice(ids(), 3)))  # ['ID-0001', 'ID-0002', 'ID-0003']

The most practical scenario is processing a large file line by line. Generators can be chained into a pipeline, and at any moment only one line is in memory rather than the whole file:

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

def only_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

for line in only_errors(read_lines("app.log")):
    print(line)

yield from and send

yield from passes on all the values of another iterable, saving a nested loop:

def flatten(groups):
    for g in groups:
        yield from g

print(list(flatten([[1, 2], [3], [4, 5]])))  # [1, 2, 3, 4, 5]

The send() method sends a value into the generator: it becomes the result of the yield expression. Before the first send, the generator must be "started" with next(). For example, this is how a running average is written:

def averager():
    total, count = 0, 0
    avg = None
    while True:
        value = yield avg
        total += value
        count += 1
        avg = total / count

a = averager()
next(a)            # run to the first yield
print(a.send(10))  # 10.0
print(a.send(20))  # 15.0

In application code send is rarely used, but it is often asked about in job interviews.

Is range a generator?

No. range is a lazy sequence: it also does not store all the numbers, but it supports indexing, len(), the in check and can be traversed any number of times. A generator can do none of this.

r = range(5)
print(r[2], len(r), 3 in r)  # 2 5 True
print(list(r), list(r))      # [0, 1, 2, 3, 4] both times

List comprehensions

A list comprehension has the form [expression for x in collection]. It creates a new list and does not change the original. It reads from left to right: first "what to put in", then "where to take it from".

amounts = [100, 250, 400]

# regular loop
with_markup = []
for a in amounts:
    with_markup.append(a * 1.2)

# the same with a comprehension
with_markup = [a * 1.2 for a in amounts]
print(with_markup)  # [120.0, 300.0, 480.0]

Despite the name "list comprehension", the result is an ordinary finished list, not a generator: all elements are computed at once.

A list comprehension with if and if else

This is where people most often get confused. A filter condition goes at the end and discards elements. A choice condition (the ternary operator) goes at the start and turns each element into one of two values, discarding nothing.

amounts = [100, 250, 400]

big = [a for a in amounts if a > 200]                   # [250, 400]
labels = ["many" if a > 200 else "few" for a in amounts]
# ['few', 'many', 'many']

# filter and choice together
odd_marks = ["even" if n % 2 == 0 else "odd" for n in range(6) if n > 2]
# ['odd', 'even', 'odd']

The form [a for a in amounts if a > 200 else 0] is a syntax error: else belongs only to a condition at the start.

Nested comprehensions and two-dimensional lists

A double for is written in the same order as nested loops, from top to bottom. This helps avoid mixing up the order when flattening a list of lists:

groups = [[1, 2], [3], [4, 5]]
flat = [x for g in groups for x in g]
# equivalent to:
# for g in groups:
#     for x in g:
#         flat.append(x)
print(flat)  # [1, 2, 3, 4, 5]

A two-dimensional list (a matrix) is created with a comprehension inside a comprehension. The classic mistake is multiplying a list: the rows end up being the same object.

bad = [[0] * 3] * 2
bad[0][0] = 1
print(bad)   # [[1, 0, 0], [1, 0, 0]] — both rows changed

grid = [[0] * 3 for _ in range(2)]
grid[0][0] = 1
print(grid)  # [[1, 0, 0], [0, 0, 0]]

# transposition
m = [[1, 2, 3], [4, 5, 6]]
print([[row[i] for row in m] for i in range(3)])  # [[1, 4], [2, 5], [3, 6]]

Dict and set comprehensions

The same technique works for other collections. Curly braces with a colon build a dictionary (a dict comprehension), without a colon — a set (a set comprehension).

spent = [("food", 500), ("taxi", 300), ("food", 200)]

categories = {cat for cat, _ in spent}
print(categories)  # {'food', 'taxi'} — the order of a set is not guaranteed

last = {cat: amount for cat, amount in spent}
print(last)        # {'food': 200, 'taxi': 300}

squares = {n: n ** 2 for n in range(4)}  # {0: 0, 1: 1, 2: 4, 3: 9}
swapped = {v: k for k, v in squares.items()}

Note last: when a key repeats, the last value wins, and the 500 is lost. If you need to add values up, a comprehension is not suitable — write a loop or use collections.Counter / defaultdict.

What about a tuple comprehension?

Python has no separate "tuple comprehension". Round brackets give a generator expression, and a tuple is obtained by passing it to tuple():

t = tuple(n * 2 for n in range(3))  # (0, 2, 4)

Generator expressions: produce rather than store

If you replace the square brackets with round ones, you get a generator expression. It computes nothing in advance and yields values one at a time. For a million elements the difference in memory is huge:

import sys

as_list = [n for n in range(1_000_000)]
as_gen = (n for n in range(1_000_000))
print(sys.getsizeof(as_list))  # about 8 MB
print(sys.getsizeof(as_gen))   # a couple of hundred bytes

Generator expressions most often live inside aggregate functions. If the generator is the only argument, the extra brackets can be omitted. any() and all(), moreover, stop at the first answer and do not iterate over the rest.

spent = [("food", 500), ("taxi", 300), ("food", 200)]

total = sum(a for _, a in spent)            # 1000
has_big = any(a > 400 for _, a in spent)    # True, does not look past 500
biggest = max(spent, key=lambda p: p[1])    # ('food', 500)

The selection rule is simple: if you need the result several times, indexing or len(), take a list. If you need to pass through once and compute something, take a generator.

Scope: the variable does not leak

A question that even experienced developers trip over:

x = 10
squares = [x for x in range(3)]
print(x)  # 10

The loop variable inside a comprehension lives in its own scope and does not overwrite the outer one (this is so in Python 3; in Python 2 it was different). In an ordinary for loop, after it x would equal 2. Since version 3.12 (PEP 709) comprehensions are inlined into the surrounding function and run faster, but the visibility rule has stayed the same.

Unpacking with * and ** in comprehensions: new in Python 3.15 (PEP 798)

The unpacking asterisk has long been allowed in a function call and in a list literal, but inside a comprehension it was a SyntaxError. In Python 3.15 the restriction was lifted: *element means "insert all of its elements here". It works in lists, sets and generators, including asynchronous ones.

# Python 3.15+
groups = [[1, 2], [3], [4, 5]]
flat = [*g for g in groups]        # [1, 2, 3, 4, 5]
uniq = {*g for g in groups}        # a set
lazy = (*g for g in groups)        # a lazy generator

The double asterisk does the same for dictionaries. This is convenient for layered settings "defaults → file → flags": each next layer overrides the keys of the previous one.

# Python 3.15+
layers = [{"debug": False, "port": 80}, {"port": 8080}, {"debug": True}]
config = {**layer for layer in layers}
# {'debug': True, 'port': 8080}

Two caveats from the course. The asterisk cannot be combined with a conditional expression without brackets: [*w if w else 0 for w in weeks] is a syntax error, and the condition has to be wrapped in brackets. And such code will not run on 3.14 or earlier: if a library supports older versions, keep the nested for or itertools.chain.from_iterable.

Asynchronous generators

If an async def function contains yield, you get an asynchronous generator. It is iterated with async for, and inside it you can wait for the network or the disk with await. There are asynchronous comprehensions too.

import asyncio

async def ticks(n):
    for i in range(n):
        await asyncio.sleep(0.1)
        yield i

async def main():
    result = [i async for i in ticks(3)]
    print(result)  # [0, 1, 2]

asyncio.run(main())

Common mistakes

• A comprehension three storeys high. Two fors, two ifs and a ternary in one line will be unreadable a week later. If it does not fit in a line, write an ordinary loop: PEP 8 values readability above brevity.
• A second pass over a generator. A second sum(gen) returns 0, because the generator is already exhausted.
• A comprehension for a side effect. [print(x) for x in xs] builds an unneeded list of None. For actions, use a for loop.
• Expecting a list from a generator expression. (x for x in xs)[0] is a TypeError: a generator has no indexes.

Generator exercises for practice

1. Write a generator fib() that yields Fibonacci numbers endlessly, and take the first 10 with islice.
2. In one line, get a dictionary "word → length" for words longer than three letters.
3. Calculate the sum of the squares of even numbers up to a million without creating a list.
4. Write a generator chunks(seq, n) that yields pieces of a list of n elements.

Summary

An iterator hands out elements one at a time through next(), and a generator is the simplest way to write one: a function with yield or an expression in round brackets. List, dict and set comprehensions build a finished collection in one line; the if filter goes at the end, the if else choice at the start. If you need to save memory or process a stream of data once, take a generator; if you need the result many times, take a list. Comprehensions, generators and the new unpacking from Python 3.15 are practised hands-on in the course, using an expense tracker as the example.