# Generators in Python: yield, Iterators and List and Dict Comprehensions

URL: https://skillaria.top/en/guides/python/python-generators-yield-comprehensions

A generator in Python is an object that yields values one at a time, on request, without keeping the whole sequence in memory; it is created either by a function with the yield keyword or by a generator expression in round brackets. List, dict and set comprehensions are a short notation for building a finished collection in one line: [x * 2 for x in xs], {k: v for k, v in pairs}, {x for x in xs}.

A generator in Python is an object that yields values one at a time, only when they are requested, without holding the whole sequence in memory. There are two ways to create one: a function with the `yield` keyword, or a generator expression in round brackets. Alongside them live the list, dict and set comprehensions — a short notation for building a finished collection in one line. Let us look at how `yield` works, how a generator differs from an iterator, and how to avoid turning a comprehension into an unreadable line.

## Iterators: what the for loop runs on

To understand generators, you first need to understand iterators. An iterable is anything you can walk through with a loop: a list, a string, a dictionary, a file. An iterator is an object that remembers its current position and, on a call to `next()`, gives the next element; when the elements run out, it raises `StopIteration`.

```
nums = [10, 20, 30]
it = iter(nums)      # get an iterator from the list
print(next(it))      # 10
print(next(it))      # 20
print(next(it))      # 30
next(it)             # StopIteration
```

A `for` loop does exactly this: it calls `iter()`, then `next()` until `StopIteration`. You can write your own iterator as a class with the `__iter__` and `__next__` methods:

```
class Countdown:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current  0:
        yield start
        start -= 1

gen = countdown(3)
print(gen)          #
print(next(gen))    # 3
print(list(gen))    # [2, 1] — continued from where it stopped
```

When the function reaches its end (or a `return`), the generator raises `StopIteration` by itself, and the `for` loop finishes neatly. Hence the main difference between yield and return: return ends the function and returns one value, while yield suspends it and can hand out as many values as needed.

### Generator and iterator: the differences

Any generator is an iterator: it has `__iter__` and `__next__`. But not every iterator is a generator. The difference lies in how it is created and in its capabilities.

|  | Iterator (class) | Generator |
| --- | --- | --- |

| How to create | a class with `__iter__` and `__next__` | a function with `yield` or `(x for x in ...)` |

| State | managed by hand in attributes | kept automatically in local variables |

| StopIteration | you raise it yourself | raised automatically when the function exits |

| Extras | whatever you write | `send()`, `throw()`, `close()` |

They share one important property: you can go through them only once. A second `for` over an exhausted generator yields nothing — you have to create a new one.

### An infinite generator and reading large files

A generator does not have to end. An infinite sequence is safe as long as you take a limited number of elements from it:

```
from itertools import islice

def ids():
    n = 1
    while True:
        yield f"ID-{n:04d}"
        n += 1

print(list(islice(ids(), 3)))  # ['ID-0001', 'ID-0002', 'ID-0003']
```

The most practical scenario is processing a large file line by line. Generators can be chained into a pipeline, and at any moment only one line is in memory rather than the whole file:

```
def read_lines(path):
    with open(path, encoding="utf-8") as f:
        for line in f:
            yield line.rstrip("\n")

def only_errors(lines):
    for line in lines:
        if "ERROR" in line:
            yield line

for line in only_errors(read_lines("app.log")):
    print(line)
```

### yield from and send

`yield from` passes on all the values of another iterable, saving a nested loop:

```
def flatten(groups):
    for g in groups:
        yield from g

print(list(flatten([[1, 2], [3], [4, 5]])))  # [1, 2, 3, 4, 5]
```

The `send()` method sends a value into the generator: it becomes the result of the `yield` expression. Before the first `send`, the generator must be "started" with `next()`. For example, this is how a running average is written:

```
def averager():
    total, count = 0, 0
    avg = None
    while True:
        value = yield avg
        total += value
        count += 1
        avg = total / count

a = averager()
next(a)            # run to the first yield
print(a.send(10))  # 10.0
print(a.send(20))  # 15.0
```

In application code `send` is rarely used, but it is often asked about in job interviews.

### Is range a generator?

No. `range` is a lazy sequence: it also does not store all the numbers, but it supports indexing, `len()`, the `in` check and can be traversed any number of times. A generator can do none of this.

```
r = range(5)
print(r[2], len(r), 3 in r)  # 2 5 True
print(list(r), list(r))      # [0, 1, 2, 3, 4] both times
```

## List comprehensions

A list comprehension has the form `[expression for x in collection]`. It creates a new list and does not change the original. It reads from left to right: first "what to put in", then "where to take it from".

```
amounts = [100, 250, 400]

# regular loop
with_markup = []
for a in amounts:
    with_markup.append(a * 1.2)

# the same with a comprehension
with_markup = [a * 1.2 for a in amounts]
print(with_markup)  # [120.0, 300.0, 480.0]
```

Despite the name "list comprehension", the result is an ordinary finished list, not a generator: all elements are computed at once.

### A list comprehension with if and if else

This is where people most often get confused. A filter condition goes at the end and discards elements. A choice condition (the ternary operator) goes at the start and turns each element into one of two values, discarding nothing.

```
amounts = [100, 250, 400]

big = [a for a in amounts if a > 200]                   # [250, 400]
labels = ["many" if a > 200 else "few" for a in amounts]
# ['few', 'many', 'many']

# filter and choice together
odd_marks = ["even" if n % 2 == 0 else "odd" for n in range(6) if n > 2]
# ['odd', 'even', 'odd']
```

The form `[a for a in amounts if a > 200 else 0]` is a syntax error: `else` belongs only to a condition at the start.

### Nested comprehensions and two-dimensional lists

A double `for` is written in the same order as nested loops, from top to bottom. This helps avoid mixing up the order when flattening a list of lists:

```
groups = [[1, 2], [3], [4, 5]]
flat = [x for g in groups for x in g]
# equivalent to:
# for g in groups:
#     for x in g:
#         flat.append(x)
print(flat)  # [1, 2, 3, 4, 5]
```

A two-dimensional list (a matrix) is created with a comprehension inside a comprehension. The classic mistake is multiplying a list: the rows end up being the same object.

```
bad = [[0] * 3] * 2
bad[0][0] = 1
print(bad)   # [[1, 0, 0], [1, 0, 0]] — both rows changed

grid = [[0] * 3 for _ in range(2)]
grid[0][0] = 1
print(grid)  # [[1, 0, 0], [0, 0, 0]]

# transposition
m = [[1, 2, 3], [4, 5, 6]]
print([[row[i] for row in m] for i in range(3)])  # [[1, 4], [2, 5], [3, 6]]
```

## Dict and set comprehensions

The same technique works for other collections. Curly braces with a colon build a dictionary (a dict comprehension), without a colon — a set (a set comprehension).

```
spent = [("food", 500), ("taxi", 300), ("food", 200)]

categories = {cat for cat, _ in spent}
print(categories)  # {'food', 'taxi'} — the order of a set is not guaranteed

last = {cat: amount for cat, amount in spent}
print(last)        # {'food': 200, 'taxi': 300}

squares = {n: n ** 2 for n in range(4)}  # {0: 0, 1: 1, 2: 4, 3: 9}
swapped = {v: k for k, v in squares.items()}
```

Note `last`: when a key repeats, the last value wins, and the 500 is lost. If you need to add values up, a comprehension is not suitable — write a loop or use `collections.Counter` / `defaultdict`.

### What about a tuple comprehension?

Python has no separate "tuple comprehension". Round brackets give a generator expression, and a tuple is obtained by passing it to `tuple()`:

```
t = tuple(n * 2 for n in range(3))  # (0, 2, 4)
```

## Generator expressions: produce rather than store

If you replace the square brackets with round ones, you get a generator expression. It computes nothing in advance and yields values one at a time. For a million elements the difference in memory is huge:

```
import sys

as_list = [n for n in range(1_000_000)]
as_gen = (n for n in range(1_000_000))
print(sys.getsizeof(as_list))  # about 8 MB
print(sys.getsizeof(as_gen))   # a couple of hundred bytes
```

Generator expressions most often live inside aggregate functions. If the generator is the only argument, the extra brackets can be omitted. `any()` and `all()`, moreover, stop at the first answer and do not iterate over the rest.

```
spent = [("food", 500), ("taxi", 300), ("food", 200)]

total = sum(a for _, a in spent)            # 1000
has_big = any(a > 400 for _, a in spent)    # True, does not look past 500
biggest = max(spent, key=lambda p: p[1])    # ('food', 500)
```

The selection rule is simple: if you need the result several times, indexing or `len()`, take a list. If you need to pass through once and compute something, take a generator.

## Scope: the variable does not leak

A question that even experienced developers trip over:

```
x = 10
squares = [x for x in range(3)]
print(x)  # 10
```

The loop variable inside a comprehension lives in its own scope and does not overwrite the outer one (this is so in Python 3; in Python 2 it was different). In an ordinary `for` loop, after it `x` would equal 2. Since version 3.12 (PEP 709) comprehensions are inlined into the surrounding function and run faster, but the visibility rule has stayed the same.

## Unpacking with * and ** in comprehensions: new in Python 3.15 (PEP 798)

The unpacking asterisk has long been allowed in a function call and in a list literal, but inside a comprehension it was a `SyntaxError`. In Python 3.15 the restriction was lifted: `*element` means "insert all of its elements here". It works in lists, sets and generators, including asynchronous ones.

```
# Python 3.15+
groups = [[1, 2], [3], [4, 5]]
flat = [*g for g in groups]        # [1, 2, 3, 4, 5]
uniq = {*g for g in groups}        # a set
lazy = (*g for g in groups)        # a lazy generator
```

The double asterisk does the same for dictionaries. This is convenient for layered settings "defaults → file → flags": each next layer overrides the keys of the previous one.

```
# Python 3.15+
layers = [{"debug": False, "port": 80}, {"port": 8080}, {"debug": True}]
config = {**layer for layer in layers}
# {'debug': True, 'port': 8080}
```

Two caveats from the course. The asterisk cannot be combined with a conditional expression without brackets: `[*w if w else 0 for w in weeks]` is a syntax error, and the condition has to be wrapped in brackets. And such code will not run on 3.14 or earlier: if a library supports older versions, keep the nested `for` or `itertools.chain.from_iterable`.

## Asynchronous generators

If an `async def` function contains `yield`, you get an asynchronous generator. It is iterated with `async for`, and inside it you can wait for the network or the disk with `await`. There are asynchronous comprehensions too.

```
import asyncio

async def ticks(n):
    for i in range(n):
        await asyncio.sleep(0.1)
        yield i

async def main():
    result = [i async for i in ticks(3)]
    print(result)  # [0, 1, 2]

asyncio.run(main())
```

## Common mistakes

- **A comprehension three storeys high.** Two `for`s, two `if`s and a ternary in one line will be unreadable a week later. If it does not fit in a line, write an ordinary loop: PEP 8 values readability above brevity.
- **A second pass over a generator.** A second `sum(gen)` returns 0, because the generator is already exhausted.
- **A comprehension for a side effect.** `[print(x) for x in xs]` builds an unneeded list of `None`. For actions, use a `for` loop.
- **Expecting a list from a generator expression.** `(x for x in xs)[0]` is a `TypeError`: a generator has no indexes.

## Generator exercises for practice

1. Write a generator `fib()` that yields Fibonacci numbers endlessly, and take the first 10 with `islice`.
2. In one line, get a dictionary "word → length" for words longer than three letters.
3. Calculate the sum of the squares of even numbers up to a million without creating a list.
4. Write a generator `chunks(seq, n)` that yields pieces of a list of `n` elements.

## Summary

An iterator hands out elements one at a time through `next()`, and a generator is the simplest way to write one: a function with yield or an expression in round brackets. List, dict and set comprehensions build a finished collection in one line; the `if` filter goes at the end, the `if else` choice at the start. If you need to save memory or process a stream of data once, take a generator; if you need the result many times, take a list. Comprehensions, generators and the new unpacking from Python 3.15 are practised hands-on in the course, using an expense tracker as the example.

## FAQ

**Can I use return in a generator?**

Yes. A return without a value simply ends the generator, while a return with a value stores it in the value attribute of the StopIteration exception; the yield from expression receives that value.

**How do I turn a generator into a list?**

Pass it to list(): list(gen). Note that the generator is exhausted after this, and all the values end up in memory.

**What is faster: a list comprehension or a loop with append?**

A comprehension is usually a little faster, because it does not call the append method on each iteration. But choose by readability; the difference is rarely decisive.

**How can I tell that an object is a generator?**

Use inspect.isgenerator(obj) for an object or inspect.isgeneratorfunction(f) for a function; to check that an object is an iterator, use isinstance(obj, collections.abc.Iterator).

**How is a generator expression different from a list comprehension?**

Square brackets build the whole list in memory at once; round brackets create a generator that computes elements on request and can be traversed only once.


---
Source: Skillaria.Top (https://skillaria.top)