Applied Mathematics & Statistics / Year 5 · Study lab
Readable code, practised until it feels obvious.
17 lessons and 3 practicals in one app, with worked examples you can run, charts built from real computation, interactive explorers and a reading list for every topic.
English · Merged into one app 2026-11-02 · What’s new
/
Search titles, objectives, examples, chart types and reading.
Start with the two foundations and their practicals, then follow the textbook chapters in order. Each page keeps the original lesson text and adds a visual lab, more worked examples and further reading.
+
Chart studio
Two samples. The full set is on the charts page.
Where the gaps are
Filled square = missing value; rows are the seven students
Read down a column to see how much each strategy has to repair. Hover a cell for the student and column.Data table
age
height_cm
weight_kg
score
Alice
0
0
0
0
Bob
1
0
1
0
Charlie
0
0
0
1
David
1
1
0
0
Eva
0
0
0
0
Frank
0
0
0
1
Grace
1
0
0
0
Steps to find the last item
Binary search against a linear scan
On a log-log chart the linear scan is a straight diagonal. Binary search takes about log2(n) steps, so a million items need only 20.Data table
Names, functions, objects, modules and reliable habits
3 extra examples1 chart5 sources
Learning objectives
Distinguish variables, functions, classes, objects and methods.
Organize reusable code into modules and packages.
Apply the supplied PEP 8 conventions consistently.
Diagnose a type error and handle invalid input deliberately.
Core explanation
For a data scientist, a Python program is a chain of transformations on data. A variable gives a name to an object, a function gives a name to a reusable computation, and a class groups related state and operations. A method is a function accessed through a class or object; an instance method receives the object as self. These ideas reduce how much of an analysis you must remember at once.
More precisely, assignment binds a name to an object. It does not necessarily copy that object. If two names refer to the same list, changing the list through one name is visible through the other. An object is an instance of a class. Its attributes hold state, and methods implement behavior. A class is useful when state and behavior naturally belong together; a simple numerical calculation often needs only a function.
A Python source file is a module. A regular package groups modules in a directory with __init__.py. The source's package diagram is an organizational example: modules can also contain standalone functions and constants, and classes are optional. Namespace packages are another package form, so an initializer is not a universal requirement.
The review recommends snake_case for variables and functions, PascalCase for classes, uppercase names for constants, four spaces per indentation level, readable spacing, organized imports, and blank lines between definitions. Constants remain a convention in Python; uppercase letters do not prevent reassignment. Group standard-library imports separately from third-party imports.
Readability helps someone inspect meaning; maintainability helps them change that meaning safely. A built-in such as sum often states intent more directly than an explicit loop, but both algorithms still visit every element. Debugging starts with the traceback, the failing operation and the types of its operands. Catch a specific expected exception such as ValueError when converting user text to an integer, and allow unrelated defects to remain visible.
Key formulas and contracts
t = p × q
t is total price, p is unit price, and q is quantity. Units must agree.
S = ∑ᵢ₌₁ⁿ xᵢ
S is the sum, n is the number of values, i is an index, and xᵢ is the ith value. A loop and sum both take O(n) time; O describes asymptotic growth.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Choosing the right built-in: time to find the last item
Median microseconds per lookup, measured while this page was built
Measured with time.perf_counter on Python 3.10.12; exact values change from machine to machine, the shapes do not. A list scan grows with n, a set lookup stays flat, bisect on sorted data grows very slowly.Data table
series
x
y
note
item in list
100
0.2445
item in list
1000
2.291
item in list
10000
25.48
item in list
100000
256.8
item in set
100
0.0283
item in set
1000
0.0372
item in set
10000
0.0313
item in set
100000
0.0317
bisect on sorted list
100
0.0703
bisect on sorted list
1000
0.0874
bisect on sorted list
10000
0.1018
bisect on sorted list
100000
0.128
Worked example
Task: repair a calculation and reuse it.
"20" + 5 combines a string and an integer, so Python raises TypeError.
Parse the text with int("20"); adding 5 now gives 25.
Put product calculations in calculate_total(price, quantity). The calls (10, 2), (20, 3), (15, 4) produce 20, 60, 60.
Place reusable functions in data_tools/analysis.py and import them from the runnable TP01 script.
The linked TP executes these examples and includes the actual output.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Two names, one list: aliasing versus copying
Assignment binds a name to an object. It does not copy the object. Watch which names see the change.
scores = [85, 90, 75]
alias = scores # a second name for the same list
snapshot = scores.copy() # a new list holding the same items
scores.append(88)
print("scores :", scores)
print("alias :", alias)
print("snapshot:", snapshot)
print("alias is scores:", alias is scores, "| snapshot is scores:", snapshot is scores)
point = (3, 4)
try:
point[0] = 9
except TypeError as error:
print("tuple is immutable ->", error)
scores : [85, 90, 75, 88]
alias : [85, 90, 75, 88]
snapshot: [85, 90, 75]
alias is scores: True | snapshot is scores: False
tuple is immutable -> 'tuple' object does not support item assignment
Try it. Replace the list with a tuple of the same numbers. Which of the first three lines now fails, and why?
Parse at the boundary and catch only the error you expect
Turn raw text into a trusted value once, close to where it enters. Name the exact failure instead of hiding every bug.
def parse_quantity(text):
"""Return a positive whole number, or raise ValueError saying why not."""
try:
value = int(text)
except ValueError:
raise ValueError(f"quantity must be a whole number, got {text!r}") from None
if value <= 0:
raise ValueError(f"quantity must be positive, got {value}")
return value
for raw in ["3", " 12 ", "0", "-2", "3.5", "abc", ""]:
try:
print(f"{raw!r:>7} -> {parse_quantity(raw)}")
except ValueError as error:
print(f"{raw!r:>7} -> rejected: {error}")
'3' -> 3
' 12 ' -> 12
'0' -> rejected: quantity must be positive, got 0
'-2' -> rejected: quantity must be positive, got -2
'3.5' -> rejected: quantity must be a whole number, got '3.5'
'abc' -> rejected: quantity must be a whole number, got 'abc'
'' -> rejected: quantity must be a whole number, got ''
Try it. Add a maximum of 100 and decide whether 100 itself is allowed. Write the rule in the error message.
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Practical exercise / TP01
Python Foundations and Clean Coding
Reproduce and explain the supplied completed review
3 extra examples1 chart4 sources
The supplied TP01 file is already a completed explanatory review. No separate numbered assignment or official answer key was supplied. The study tasks below restate its sections without claiming they were unanswered original questions.
Part 1 · Explain the Python concepts
Worked solution
Part 1 · the eight Python concepts
What you need. The difference between a definition (class, function) and a thing created from it (object, return value); files vs folders (module vs package). See L01.
1
Sort the eight words into three levels. Names: variable, constant. Code you can call: function, method. Types and things: class, object. Files: module (one .py file), package (a folder of modules).
Why: grouping first makes the differences obvious: most confusion comes from mixing levels (e.g. module vs class).
2
Variable vs constant. Both are names bound to objects. A constant is only a convention (UPPERCASE, e.g. MAX_SCORE); Python still allows reassignment.
Why: Python has no constant keyword, so the name style is the only signal to the reader.
3
Class vs object.Student is the definition; Student("CHHO Sengmeng") is an object (instance). Check: type(student).__name__ == "Student".
Why: one class can make many objects, each with its own attribute values.
4
Function vs method.calculate_sum(5, 3) returns 8 and needs no object. student.study() is a method: Python passes student as self.
Why: a method is just a function defined inside a class that receives the object automatically.
5
Module vs package.analysis.py is a module; data_tools/ with __init__.py is a regular package containing it.
Why: packages organise modules the way folders organise files; they need not contain classes.
Figure. One possible layout of the TP01 code. A package groups modules; a module holds functions and classes; a class holds methods. Functions can live directly in a module — no class is required.
Answer. variable/constant = names (constant by convention only); function/method = callables (method gets self); class/object = type and instance; module/package = file and folder of files. The package → module → class → method picture is one possible organisation, not a required hierarchy.
✔ What it provesYou can place each concept at the right level and show it in code (calculate_sum(5, 3) == 8, type(student).__name__ == "Student").
✖ What it does not sayIt does not say every program needs classes or packages: a module of plain functions is often the cleanest design. And “constant” is not enforced by Python.
→ Where it leadsL02: classes become design patterns (Strategy, Factory, Template Method) in TP02.
Restated task: distinguish variable, function, class, object, method, constant, module and package, and demonstrate how they relate.
Concept
Answer and reasoning
Variable
A name bound to an object. Assignment permits later reference; it does not necessarily copy the object.
Function
A reusable callable. calculate_sum(5, 3) returns 8.
Class
A definition of a type. Student defines attributes and an instance method.
Object
An instance such as Student("CHHO Sengmeng"); type(student).__name__ is Student.
Method
A callable on the class or object. student.study() receives student as self.
Constant
A name intended to remain fixed. PI and MAX_SCORE use the source's uppercase convention; Python permits reassignment.
Module
A Python file containing definitions. data_tools/analysis.py contains calculate_sum and calculate_total.
Package
A grouping of modules. The example uses a regular package data_tools with __init__.py. Namespace packages can omit that file.
Packages need not contain classes, and functions can exist directly at module level. The source's package → module → class → method diagram is an example of organization, rather than a mandatory hierarchy.
class Student:
def __init__(self, name):
self.name = name
def study(self):
return f"{self.name} is studying."
"""Reusable analysis module."""
def calculate_sum(a, b):
return a + b
def calculate_total(price, quantity):
return price * quantity
Part 2 · Apply the four programming practices
Worked solution
Part 2 · readability, maintainability, efficiency, error handling
What you need. What each practice protects: the reader (readability), the next change (maintainability), the running time (efficiency), the user (error handling).
1
Readability. Rename a, b, c to price, quantity, total_price. With 10 and 3 the result is still 30.
Why: good names carry meaning so the reader does not need comments; the behaviour must not change.
2
Maintainability. Move the repeated price * quantity into calculate_total; the three uses give 20, 60, 60.
Why: one definition means one place to fix when the rule changes (e.g. adding tax).
3
Efficiency.sum([10, 20, 30, 40]) == 100 replaces a manual loop.
Why: it states the intent directly. Both are O(n): shorter code is not automatically faster code.
4
Error handling.parse_age("20") → 20; parse_age("hello") catches ValueError and returns a clear message.
Why: catch only the error you expect; a bare except: would also hide real bugs.
Answer. Each practice is shown with a before/after that keeps the same results (30; 20/60/60; 100; 20 or a message).
✔ What it provesThe four practices can be applied without changing results, and each has a measurable check.
✖ What it does not sayIt does not say that sum is asymptotically faster than a loop, or that returning a message string is the best error design (raising a custom exception is often cleaner).
→ Where it leadsL03–L08 (names, comments, layout) and L16 (tests that prove behaviour did not change).
Restated task: explain readability, maintainability, efficiency and error handling using the supplied examples.
Readability: replace a, b, c with price, quantity, total_price. With price = 10 and quantity = 3, the total is 30.
Maintainability: put repeated multiplication in calculate_total. The three examples produce 20, 60 and 60. A future change to the calculation has one definition to inspect.
Efficiency: sum([10, 20, 30, 40]) gives 100. It communicates the reduction directly. Both sum and the explicit accumulation loop take linear time; concise syntax alone does not prove a complexity improvement.
Error handling: parsing "20" succeeds with 20; parsing "hello" raises ValueError, which our function converts to the supplied explanatory message. Other unexpected exceptions are not hidden.
def parse_age(text):
try:
return int(text)
except ValueError:
return "Please enter a number."
Part 3 · Apply every style convention
Worked solution
Part 3 · PEP 8 conventions on the Calc example
What you need. PEP 8 naming styles: snake_case (variables, functions), PascalCase (classes), UPPER_CASE (constants); spacing and line-length rules.
1
Identify what Calc is. It is a function, so it should be snake_case and say what it returns: calculate_sum.
Why: PascalCase signals “class” to every Python reader; using it for a function misleads.
2
Fix spacing. Spaces around operators (a + b), after commas, four-space indentation.
Why: consistent layout lets the eye find structure without reading every character.
3
Check behaviour is unchanged. Old and new versions both return 8 for (5, 3).
Why: a style change must be a pure refactoring: same inputs, same outputs.
4
Apply the rest of the checklist (imports at top in three groups, ≤ 79-character lines, two blank lines between top-level definitions, comments that explain why).
Answer.def calculate_sum(a, b): return a + b — new name and spacing, same result (8 for 5 and 3).
✔ What it provesA style review is a refactoring you can verify by comparing outputs.
✖ What it does not sayPEP 8 is a guide, not a law: a project may agree on 88 or 100 characters (e.g. Black uses 88). Style does not fix logic errors.
→ Where it leadsAutomatic tools: black, ruff, flake8 enforce these rules for you.
Restated task: explain the review's PEP 8 checklist and improve its Calc example.
Convention
Application
Meaningful names
calculate_sum describes the result better than Calc.
Variables and functions
Use snake_case: student_name, calculate_total.
Classes
Use PascalCase: Student, StudentProfile.
Constants
Use UPPERCASE: MAX_SCORE.
Indentation
Use four spaces inside a function or conditional block.
Operators
Write total = price * quantity.
Commas
Write [70, 80, 90], and calculate_total(10, 2).
Imports
Place them near the top; group standard library, third-party and local imports separately.
Long lines
Wrap an argument list inside parentheses. PEP 8's normal code guideline is 79 characters, subject to the project's agreed conventions.
Blank lines
Use two blank lines between top-level definitions, and conceptual groups inside functions.
Comments
Explain a non-obvious constraint or reason; avoid narrating an obvious addition.
def calculate_sum(a, b):
return a + b
The correction changes the name and spacing without changing the addition. For inputs 5 and 3, both implementations return 8.
Debugging task · Repair the mixed-type addition
Worked solution
Debugging · "20" + 5
What you need. Python is dynamically but strongly typed: it does not silently convert between str and int.
1
Read the error.student_age + 5 raises TypeError: the left operand is str, the right is int.
Why: for str, + means concatenation, which only accepts another string.
2
Decide the intended type. An age is a number, so convert the text: int(student_age).
Why: fix the data at the boundary (where text enters) rather than everywhere it is used.
3
Check.int("20") + 5 == 25.
4
Harden. For user input, int(text) can raise ValueError (e.g. "hello"); handle it as in Part 2.
Why: the fix for one error must not create a new crash for other inputs.
Answer. Convert first: print(int(student_age) + 5) prints 25.
✔ What it provesMixed str + int always fails in Python; explicit conversion is required.
✖ What it does not sayIt does not mean “always call int()”: "20.5" needs float, and str(5) would be right if you wanted text "205".
→ Where it leadsType hints and checkers (mypy) catch such errors before running.
Restated task: explain why student_age = "20" followed by student_age + 5 fails.
The left operand is str and the right is int.
String concatenation does not add an integer, so the operation raises TypeError.
Convert known numeric text with int(student_age).
20 + 5 now produces 25. For arbitrary user input, handle the ValueError that conversion can raise.
The demonstration uses fixed inputs for a reproducible run. It needs Python 3 and only the standard library. Paths inside the script do not depend on the terminal working directory.
Variables: CHHO Sengmeng 22 85
Function: 8
Class and object: Student
Method: CHHO Sengmeng is studying.
Constants: 3.14159 100
Module: 7
Package: data_tools contains analysis.py
Readable product: 30
Reusable totals: [20, 60, 60]
Built-in sum: 100
Valid age: 20
Invalid age: Please enter a number.
Debugging fix: 25
PEP 8 refactoring: 8
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
How long are the names in the TP01 solution?
21 defined names counted in solution.py and analysis.py
Function, class, argument and assigned names, counted once per definition or assignment. Hover a bar for its range.Data table
from
up to (not included)
count
0
2
4
2
4
1
4
6
6
6
8
2
8
10
4
10
12
1
12
14
2
14
16
1
16
18
0
18
20
0
20
22
0
Comparison with the supplied answer
The outputs agree with the explicit numerical examples in TP_01.md: sums 8 and 7, multiplication totals 20/60/60, sum 100, and converted age 25. The new Student demonstration returns a string so it can be inspected and printed, while the source's method prints directly. This is a deliberate demonstration change, not an unreported output discrepancy.
The review's descriptions are refined in three places: variables bind objects, regular packages are not the only package form, and a shorter built-in reduction does not change linear asymptotic complexity. The source's relationship diagram is illustrative. No separate official solution exists to compare against.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Student as a dataclass: less code, same behavior
The TP defines Student by hand. A dataclass generates the constructor, the readable repr and value equality for you.
from dataclasses import dataclass, field
@dataclass
class Student:
name: str
age: int
scores: list = field(default_factory=list)
def study(self):
return f"{self.name} is studying."
def average(self):
if not self.scores:
raise ValueError("no scores recorded yet")
return sum(self.scores) / len(self.scores)
a = Student("CHHO Sengmeng", 22, [85, 90, 75])
b = Student("CHHO Sengmeng", 22, [85, 90, 75])
print(a)
print(a.study())
print("average:", round(a.average(), 2))
print("equal by value:", a == b, "| same object:", a is b)
try:
Student("New", 20).average()
except ValueError as error:
print("ValueError:", error)
Student(name='CHHO Sengmeng', age=22, scores=[85, 90, 75])
CHHO Sengmeng is studying.
average: 83.33
equal by value: True | same object: False
ValueError: no scores recorded yet
Try it. Why does the list field use default_factory instead of scores: list = []? Try it and read the error.
Import the supplied data_tools package directly from the course folder and test it with a table of cases, including one that reveals a missing contract.
import sys
sys.dont_write_bytecode = True # leave the course folder untouched
sys.path.insert(0, "_study/tp/TP01")
from data_tools.analysis import calculate_total
cases = [((10, 3), 30), ((0, 5), 0), ((15, 4), 60), ((2.5, 4), 10.0), ((10, 0), 0)]
for args, expected in cases:
actual = calculate_total(*args)
status = "ok " if actual == expected else "FAIL"
print(status, f"calculate_total{args} = {actual} (expected {expected})")
# The supplied function never checks its types, so text repeats instead of failing:
print("calculate_total('ab', 3) ->", calculate_total("ab", 3))
ok calculate_total(10, 3) = 30 (expected 30)
ok calculate_total(0, 5) = 0 (expected 0)
ok calculate_total(15, 4) = 60 (expected 60)
ok calculate_total(2.5, 4) = 10.0 (expected 10.0)
ok calculate_total(10, 0) = 0 (expected 0)
calculate_total('ab', 3) -> ababab
Try it. Add a validation line so that text raises TypeError. Decide where that check belongs: the function or its caller?
PEP 8 before and after, with proof the behavior did not change
Style cleanups are only safe refactorings if outputs stay identical. Prove it on sample inputs.
def Calc( a,b ):
x=a+b
return(x)
def calculate_sum(first, second):
"""Return the sum of two numbers."""
total = first + second
return total
samples = [(5, 3), (0, 0), (-2, 7), (1.5, 2.5)]
print("identical on every sample:", all(Calc(a, b) == calculate_sum(a, b) for a, b in samples))
print([calculate_sum(a, b) for a, b in samples])
identical on every sample: True
[8, 0, 5, 4.0]
Try it. Name three PEP 8 problems in Calc before reading the rewritten version.
Definitions and reuse in L01, reader-focused choices in L03, naming in L04, consistent layout in L06, and useful comments in L07.
Go deeper
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
A reusable pandas pipeline from five connected exercises
3 extra examples1 chart1 interactive6 sources
Learning objectives
Encapsulate CSV access in an object.
Use an abstract Strategy interface to exchange missing-value policies.
Explain Python decorators and preserve wrapped behavior.
Create transformations through a simple Factory.
Compose a Template Method pipeline and reason about transformation order.
Core explanation
Imagine repeating the same data-cleaning workflow for many datasets. The steps stay recognizable, but missing-value treatment and transformations change. OOP lets each component carry its configuration; design patterns describe where variation should live.
Encapsulation puts the file path and read/preview behavior inside CSVReader. Its cached DataFrame is an implementation detail. Returning copies makes the ownership contract clearer. Inheritance gives concrete strategies a shared abstract interface, and polymorphism lets DataCleaner invoke handle(df) without knowing which concrete class it received. The cleaner uses composition: it has a strategy.
The Strategy Pattern represents interchangeable policies. DropMissing removes incomplete rows, FillMean fills numerical measurements with their observed means, and FillMode uses a most frequent observed value. No one strategy is universally statistically appropriate. Dropping can change the population being analyzed, and single-value imputation reduces variation and may distort associations.
A Python decorator replaces a callable with a wrapper. With two decorators, the one written nearest the function is applied first at definition time. Use functools.wraps to preserve metadata, return the original result, and let exceptions propagate. Logging and timing can be added without editing each method body. This illustrates extension around stable behavior, although a Python function decorator is not identical to the classic object-based GoF Decorator Pattern.
The exercise's Factory maps a string such as "normalize" to an object. This is a simple factory, rather than a full Factory Method hierarchy. A new strategy can be injected into a cleaner without changing the cleaner; adding a factory option still changes its registry. Avoid presenting the Open/Closed Principle as a promise that no file will ever change.
The Template Method fixes the sequence load → clean → transform → save in DataPipeline.run(), while a subclass implements each step. Strategies and transformations are composed into that subclass. Text standardization happens before deduplication so case and surrounding whitespace can be reconciled. Numeric min-max scaling happens after missing-value treatment. IDs are preserved because their numeric storage does not make them measurements.
Key formulas and contracts
x̄ⱼ = (∑ᵢ∈Oⱼ xᵢⱼ) / |Oⱼ|
xᵢⱼ is row i, column j; Oⱼ is the set of rows observed in column j, |Oⱼ| its size, and x̄ⱼ its observed mean. It is undefined when |Oⱼ| = 0.
zᵢⱼ = (xᵢⱼ − mⱼ)/(Mⱼ − mⱼ)
zᵢⱼ is a scaled measurement; mⱼ and Mⱼ are the observed minimum and maximum of column j after cleaning. Constant observed columns are mapped to 0 by our stated policy; missing entries remain missing.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
The Strategy pattern in this lesson
classDiagram
class DataCleaner {
+strategy
+clean(df)
}
class MissingValueStrategy {
<<abstract>>
+handle(df)
}
class DropMissing
class FillMean
class FillMode
DataCleaner o--> MissingValueStrategy : uses
MissingValueStrategy <|-- DropMissing
MissingValueStrategy <|-- FillMean
MissingValueStrategy <|-- FillMode
What each strategy does to the statistics
Course CSV: 7 rows, 7 missing cells
DropMissing keeps only two complete rows, so its statistics rest on very little data. FillMean leaves the mean of the observed values unchanged but shrinks the spread.Data table
DropMissing
FillMean
FillMode
age mean
28
30.25
28.86
age std
1.41
2.41
2.97
score mean
86.5
86
82.86
Try it: pick a strategy, see the column change
Choose a column and a strategy. Filled values are amber, dropped rows are struck through.
This interactive needs JavaScript. The text and charts around it cover the same ideas.
Worked example
Task: fill Bob's age and normalize it.
The observed ages are 29, 35, 27 and 30. Their sum is 121 and their count is 4.
The mean is 121/4 = 30.25, so Bob, David and Grace each receive 30.25.
The age minimum is 27 and maximum is 35. Bob's scaled age is (30.25 − 27)/(35 − 27) = 0.40625.
id = 2 stays 2. Text becomes bob and los angeles. The complete pipeline retains all seven rows.
The provided CSV has seven missing cells. Dropping incomplete rows retains only IDs 1 and 5; mean and mode imputation each retain seven rows and remove all missing cells in this particular dataset.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Swap the missing-value strategy and watch the statistics move
The Strategy pattern lets the cleaner stay the same while the policy changes. Run all three on the course CSV and compare what each does to the data.
from abc import ABC, abstractmethod
import pandas as pd
class MissingValueStrategy(ABC):
@abstractmethod
def handle(self, df): ...
class DropMissing(MissingValueStrategy):
def handle(self, df):
return df.dropna()
class FillMean(MissingValueStrategy):
def handle(self, df):
return df.fillna(df.mean(numeric_only=True))
class FillMode(MissingValueStrategy):
def handle(self, df):
out = df.copy()
for column in out.columns:
mode = out[column].mode()
if not mode.empty:
out[column] = out[column].fillna(mode.iloc[0])
return out
df = pd.read_csv("sample_data (2).csv")
rows = []
for strategy in (DropMissing(), FillMean(), FillMode()):
cleaned = strategy.handle(df)
rows.append({"strategy": type(strategy).__name__, "rows": len(cleaned),
"age_mean": round(cleaned["age"].mean(), 2),
"age_std": round(cleaned["age"].std(), 2),
"score_mean": round(cleaned["score"].mean(), 2)})
print(pd.DataFrame(rows).to_string(index=False))
print("mean of the observed ages:", df["age"].mean())
strategy rows age_mean age_std score_mean
DropMissing 2 28.00 1.41 86.50
FillMean 7 30.25 2.41 86.00
FillMode 7 28.86 2.97 82.86
mean of the observed ages: 30.25
Try it. Which strategy keeps the mean of the observed ages unchanged? Which one shrinks the spread the most?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Practical exercise / TP02
CSV Cleaning with OOP and Design Patterns
Five exercises · Strategy, decorators, Factory and Template Method
3 extra examples4 charts1 interactive6 sources
Filename numbering defines this as TP02. The document's heading says “TP-1”; it contains five successive exercises. The seven-row CSV is used directly from the original course folder. Every generated output remains under _study.
Dataset and explicit choices
The dataset columns are id, name, age, height_cm, weight_kg, city and score. There are seven rows. Missing counts are age 3, height_cm 1, weight_kg 1 and score 2; the other columns have no missing cells.
Mean filling applies to numeric non-boolean measurement columns; id is excluded.
Mode filling applies to non-ID columns. In homogeneous ties, choose the first sorted pandas mode. An all-missing column stays missing and produces a warning.
NormalizeColumns means numeric min-max scaling, not renaming headers. IDs remain unchanged; observed constant columns map to zero.
StandardizeText strips surrounding whitespace and lowercases text while preserving missing values.
RemoveDuplicates compares entire rows, including id, and retains the first occurrence. The supplied data has no duplicate rows.
The pipeline applies mean filling, text standardization, full-row deduplication and then scaling. In a predictive model, fit preprocessing only on the training subset and reuse its parameters. This lab is a single-dataset demonstration.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
The Template Method skeleton
flowchart LR
L[load<br/>CSVReader.read] --> C[clean<br/>DataCleaner and a Strategy] --> T[transform<br/>factory-made steps, in list order] --> S[save<br/>to_csv]
Where the gaps are
Filled square = missing value; rows are the seven students
Read down a column to see how much each strategy has to repair. Hover a cell for the student and column.Data table
age
height_cm
weight_kg
score
Alice
0
0
0
0
Bob
1
0
1
0
Charlie
0
0
0
1
David
1
1
0
0
Eva
0
0
0
0
Frank
0
0
0
1
Grace
1
0
0
0
Share of the missing cells
7 missing cells in total
age has the most gaps; DropMissing would discard any row touched by any of them.Data table
part
value
age
3
height_cm
1
weight_kg
1
score
2
Try it: the three strategies on the course CSV
Same data as the TP. Dropping removes a row from every column at once.
This interactive needs JavaScript. The text and charts around it cover the same ideas.
Time recorded by the decorators
Mean and slowest call per method, from execution.log
Recorded by the @log_time decorator during the original run. Values vary by machine and by run; they illustrate the decorator, not performance.Data table
mean (ms)
slowest (ms)
CSVReader.read
1.168
1.935
DataCleaner.clean
0.606
0.902
After min-max scaling: age against score
pipeline_output.csv from the study solution
Both axes now run from 0 to 1, so the two columns can be compared directly. Eva has the smallest age and David the smallest score.Data table
series
x
y
label
student
0.25
0.588
alice
student
0.406
0.882
bob
student
1
0.647
charlie
student
0.406
0
david
student
0
0.765
eva
student
0.375
0.647
frank
student
0.406
1
grace
Exercise 1 · Build CSVReader
Worked solution
Exercise 1 · CSVReader
What you need. A class that owns one responsibility (reading one file); pd.read_csv; why returning a .copy() protects a cache.
1
Store the path, not the data, in __init__.self.file_path = Path(file_path), self._df = None.
Why: creating the object should be cheap; reading happens when asked (lazy).
2
read() loads with pd.read_csv, caches it in self._df and returns a deep copy.
Why: callers can edit their copy without corrupting the reader’s cache.
3
preview(n) validates n (non-negative int, not bool), reads if needed, prints head(n) without the index and returns it.
Why: returning the preview makes it testable; printing alone cannot be checked by code.
4
Check. The course CSV has shape (7, 7): columns id, name, age, height_cm, weight_kg, city, score.
Answer.CSVReader(path).read() returns a (7, 7) DataFrame copy; preview(3) prints and returns Alice, Bob, Charlie.
✔ What it provesEncapsulation: all file access goes through one object with a small, checked interface.
✖ What it does not sayCopying is not free: for very large files, returning copies doubles memory; a read-only view or documented ownership may be better.
→ Where it leadsExercise 5 plugs this reader into the pipeline’s load() step.
Question: create CSVReader, accept file_path in __init__, implement read() using pandas, and implement preview(n) to print the first n rows.
Reasoning: keep file access and its cached data in one object. A read returns a copy so callers cannot accidentally edit the reader's cache. preview lazily reads if needed, validates n and prints with index=False. It also returns the preview for inspection.
class CSVReader:
def __init__(self, file_path):
self.file_path = Path(file_path)
self._df = None
@log_action
@log_time
def read(self):
self._df = pd.read_csv(self.file_path)
return self._df.copy(deep=True)
def preview(self, n=5):
if not isinstance(n, int) or isinstance(n, bool) or n < 0:
raise ValueError("n must be a nonnegative integer")
if self._df is None:
self.read()
top = self._df.head(n).copy(deep=True)
print(top.to_string(index=False))
return top
Reading the supplied CSV yields shape (7, 7). The first-three-row output appears in the full execution transcript below.
Exercise 2 · Strategy for missing values
Worked solution
Exercise 2 · Strategy for missing values
What you need. Strategy pattern = one interface, several interchangeable policies. Missing counts: age 3, height_cm 1, weight_kg 1, score 2.
1
Define the interface. Abstract MissingValueStrategy.handle(df) returning a cleaned copy.
Why: the cleaner can then call one method whatever the policy.
2
DropMissing.df.dropna() keeps only complete rows: 2 of 7 (Alice, Eva).
Why: simple, but it threw away 5 rows of data — usually too costly.
3
FillMean (numeric, not id): age (29+35+27+30)/4 = 30.25; height 167; weight 70; score 86.
Why:id is a label, not a measurement: its mean means nothing.
4
FillMode: in these columns every value appears once, so all are tied modes; pandas returns them sorted and we take the first (smallest): age 27, height 160, weight 55, score 75.
Why: state the tie rule explicitly; otherwise the result depends on an accident of the data.
5
DataCleaner(strategy).clean(df) just delegates to strategy.handle(df).
Why: changing policy = passing another object, with no if/elif chain inside the cleaner.
Figure. Strategy pattern: DataCleaner only knows the interface; each concrete strategy is one policy. Numbers from the TP02 run.
FillMeanFillMode (first of tied modes)
Figure. The value each strategy writes into the gaps. Every observed value occurs once in these columns, so “the mode” is a tie and pandas returns the sorted modes — FillMode takes the first, i.e. the smallest value.
Answer. Drop → 2 rows; Mean → 7 rows, age gaps = 30.25; Mode → 7 rows, age gaps = 27. Same clean(df) call each time.
✔ What it provesThe design lets you swap the missing-value policy without touching the caller, and each policy is testable alone.
✖ What it does not sayIt does not say which policy is statistically right: mean filling shrinks variance, and a “mode” of all-unique values is meaningless. In modelling, fit the fill values on training data only.
→ Where it leadsExercise 4 uses the same idea for transformations, chosen by name through a Factory.
Question: define abstract MissingValueStrategy.handle(df), implement DropMissing, FillMean and FillMode, and let DataCleaner apply a chosen strategy.
Reasoning: the cleaner delegates through a stable interface. Selecting a different object changes the policy while the caller still uses clean(df). Each strategy owns its policy and avoids modifying the original data.
class MissingValueStrategy(ABC):
@abstractmethod
def handle(self, df):
"""Return a cleaned copy without modifying the caller's DataFrame."""
class DropMissing(MissingValueStrategy):
def handle(self, df):
return df.dropna().copy(deep=True)
class FillMean(MissingValueStrategy):
def __init__(self, exclude=("id",)):
self.exclude = set(exclude)
def handle(self, df):
result = df.copy(deep=True)
for name in result.columns:
series = result[name]
if name in self.exclude or not is_numeric_dtype(series) or is_bool_dtype(series):
continue
if series.isna().any():
mean = series.mean()
if pd.isna(mean):
LOGGER.warning("%s has no observed values; mean is undefined", name)
continue
# Mean imputation may turn integer measurements into fractions.
result[name] = series.astype(float).fillna(mean)
return result
class FillMode(MissingValueStrategy):
def __init__(self, exclude=("id",)):
self.exclude = set(exclude)
def handle(self, df):
result = df.copy(deep=True)
for name in result.columns:
if name in self.exclude or not result[name].isna().any():
continue
modes = result[name].mode(dropna=True)
if modes.empty:
LOGGER.warning("%s has no observed values; mode is undefined", name)
continue
# pandas sorts homogeneous modes: choose the first to resolve ties.
result[name] = result[name].fillna(modes.iloc[0])
return result
class DataCleaner:
def __init__(self, strategy):
self.strategy = strategy
@log_action
@log_time
def clean(self, df):
return self.strategy.handle(df)
Derive the replacements from observed values
Column
Observed sum / count
Mean
Mode tie choice
age
(29 + 35 + 27 + 30) / 4
30.25
27
height_cm
(165 + 172 + 168 + 160 + 175 + 162) / 6
167
160
weight_kg
(68 + 72 + 80 + 55 + 85 + 60) / 6
70
55
score
(85 + 90 + 75 + 88 + 92) / 5
86
75
Every observed value in each of these four columns occurs once, so all values tie as modes. Choosing the first sorted mode is deterministic but statistically arbitrary. No dominant value was identified.
Actual results: DropMissing retains Alice (ID 1) and Eva (ID 5). Both filling strategies retain seven rows with zero missing cells. Dropping loses 5/7 ≈ 71.43% of rows. Mean filling changes the empirical distribution and should not be treated as recovery of the unknown true values.
x̄ⱼ = (∑ᵢ∈Oⱼ xᵢⱼ) / |Oⱼ|
xᵢⱼ is row i, column j; Oⱼ is the set of rows observed in column j, |Oⱼ| its size, and x̄ⱼ its observed mean. It is undefined when |Oⱼ| = 0.
Exercise 3 · Logging and timing decorators
Worked solution
Exercise 3 · logging and timing decorators
What you need. A decorator is a function that takes a function and returns a wrapped function; @a @b def f means f = a(b(f)); try/finally.
1
Write log_action. Log START, call the method, log FINISH in finally; on error log FAILED and re-raise.
Why: the log must show the end even when the call fails, and the error must not be swallowed.
2
Write log_time. Take perf_counter() before, log the elapsed seconds in finally.
Why:perf_counter is a monotonic high-resolution clock, right for durations.
3
Keep metadata with @wraps(method) and return the method’s result.
Why: without wraps every method would be named “wrapper” in logs and help(); without return the data is lost.
4
Stack them on CSVReader.read and DataCleaner.clean: order is START → TIME → FINISH.
Why: the decorator nearest the function runs innermost.
Figure. @log_action above @log_time means log_action wraps log_time, which wraps the method.
Answer. Each decorated call logs START X, TIME X … seconds, FINISH X (times vary per run; see execution.log).
✔ What it provesCross-cutting concerns (logging, timing) can be added without editing the methods themselves.
✖ What it does not sayPython function decorators are not literally the GoF Decorator class pattern, and the timings measure wall-clock time of one run, not a benchmark.
→ Where it leadsReal projects use the same idea for caching (functools.lru_cache), retries and access checks.
Question: implement @log_action and @log_time; log when a method starts and finishes, measure its duration, and apply the decorators to CSVReader or DataCleaner.
Reasoning: log_action is the outer wrapper and log_time is the inner wrapper. A call records START, executes the timed method, records TIME, then records FINISH. Both use finally so timing and finish events occur even on failure. The action wrapper records a failed call and re-raises the original exception. wraps preserves the original method's metadata, and return forwards the result.
The standard logging module writes these events to stderr by default. Actual elapsed times from this run are in execution.log; they vary on later runs. These Python function decorators illustrate wrapping and extension. They are related to, but not a literal class hierarchy implementation of, the classic object Decorator Pattern.
Exercise 4 · Factory for transformations
Worked solution
Exercise 4 · Factory for transformations
What you need. Factory pattern = create the right object from a key; min–max scaling z = (x − min)/(max − min).
1
Common interfaceDataTransform.apply(df) returning a copy.
Why: the pipeline can loop over any list of transforms.
2
NormalizeColumns: min–max per numeric column except id. Example (after mean fill): age min 27, max 35 → Alice (29−27)/8 = 0.25. A constant column maps to 0.
Why: dividing by max − min = 0 would crash; the constant case must be handled.
3
StandardizeText: strip + lowercase text columns, keep missing values as missing.
Why: converting NaN to the string "nan" would invent data.
4
RemoveDuplicates: drop_duplicates(keep="first") on full rows.
5
TransformFactory.create(name) looks up a dict; unknown names raise ValueError listing the choices.
Why: adding a transform = one dict entry; a clear error beats a silent default.
Answer.TransformFactory.create("normalize") returns a NormalizeColumns; after the pipeline every scaled column lies in [0, 1] (Alice age 0.25).
✔ What it provesObject creation is centralised: callers name what they want, not which class to build.
✖ What it does not sayMin–max scaling is sensitive to outliers (one extreme value squeezes all others) and, in modelling, min/max must come from the training set only.
→ Where it leadsStandardisation (z-scores) as an alternative scaler; registries/plugins as a larger-scale factory.
Question: define DataTransform.apply(df), implement NormalizeColumns, RemoveDuplicates and StandardizeText, then create TransformFactory to select a transformation from a string.
Reasoning: the abstract interface lets the pipeline process a sequence of transformations uniformly. A small dictionary maps normalize, deduplicate and standardize_text to concrete classes. Unknown keys raise ValueError with supported choices.
class DataTransform(ABC):
@abstractmethod
def apply(self, df):
"""Return a transformed copy."""
class NormalizeColumns(DataTransform):
"""Min-max scale numerical measurements; preserve identifier columns."""
def __init__(self, exclude=("id",)):
self.exclude = set(exclude)
def apply(self, df):
result = df.copy(deep=True)
for name in result.columns:
series = result[name]
if name in self.exclude or not is_numeric_dtype(series) or is_bool_dtype(series):
continue
lower, upper = series.min(), series.max()
if pd.isna(lower):
continue
if lower == upper:
result[name] = series.astype(float).where(series.isna(), 0.0)
else:
result[name] = (series - lower) / (upper - lower)
return result
class RemoveDuplicates(DataTransform):
def apply(self, df):
# Equality uses all columns, including id; retain the first occurrence.
return df.drop_duplicates(keep="first").copy(deep=True)
class StandardizeText(DataTransform):
def apply(self, df):
result = df.copy(deep=True)
for name in result.columns:
series = result[name]
observed = series.dropna()
if not is_numeric_dtype(series) and not observed.empty and observed.map(
lambda value: isinstance(value, str)).all():
# StringDtype preserves missing values instead of making 'nan' text.
result[name] = series.astype("string").str.strip().str.lower()
return result
class TransformFactory:
TYPES = {"normalize": NormalizeColumns, "deduplicate": RemoveDuplicates,
"standardize_text": StandardizeText}
@classmethod
def create(cls, name):
try:
transform_type = cls.TYPES[name]
except KeyError as error:
choices = ", ".join(cls.TYPES)
raise ValueError(f"Unknown transform {name!r}; choose {choices}") from error
return transform_type()
zᵢⱼ = (xᵢⱼ − mⱼ)/(Mⱼ − mⱼ)
zᵢⱼ is a scaled measurement; mⱼ and Mⱼ are the observed minimum and maximum of column j after cleaning. Constant observed columns are mapped to 0 by our stated policy; missing entries remain missing.
After mean filling, age ranges from 27 to 35. Bob's 30.25 becomes (30.25 − 27)/8 = 0.40625. For Alice, height becomes (165 − 160)/15 = 0.333333… and score becomes (85 − 75)/17 = 0.588235…. The output rounds only for display; the saved CSV retains numerical precision.
The all-column duplicate policy retains records with different IDs even if other fields agree. A business-key deduplication rule would need an explicit subset. The factory here is a simple factory, rather than a separate GoF Factory Method subclass hierarchy.
Exercise 5 · Template Method pipeline
Worked solution
Exercise 5 · Template Method pipeline
What you need. Template Method = the parent class fixes the order of steps; subclasses implement the steps.
1
Abstract DataPipeline.run() calls load → clean → transform → save, passing each result to the next.
Why: the order is a rule of the process, so it lives in one place.
2
CSVDataPipeline implements the steps with the earlier parts: reader (Ex 1), cleaner + strategy (Ex 2), transforms from the factory (Ex 4).
Why: composition: the pipeline is built from small objects that each work alone.
3
Order of transforms: standardize text before deduplicating, scale after cleaning.
Why: "Alice " and "alice" are duplicates only after standardising; scaling with gaps would give NaN.
4
Safety checks: output must stay inside _study and differ from the input.
Why: a pipeline must never overwrite its own source data.
Figure. Template Method: run() fixes the skeleton; CSVDataPipeline supplies each step from Exercises 1–4.
Answer.run() produced pipeline_output.csv: 7 rows, text lower-cased, numeric columns in [0, 1].
✔ What it provesA fixed, reusable processing skeleton with replaceable parts.
✖ What it does not sayThe sample has no duplicates and no constant columns, so those branches are only shown by separate checks; and the pipeline is fitted on the whole dataset (no train/test split).
→ Where it leadsscikit-learn Pipeline / ColumnTransformer implement the same idea with proper fit/transform separation.
Question: define abstract DataPipeline with run() calling load(), clean(), transform(), save(), and implement CSVDataPipeline using the previous components.
Reasoning: run fixes the skeleton and passes each result explicitly to the next step. CSVDataPipeline supplies CSV reading, an injected missing-value strategy, a composed transformation list and saving. The output path is checked to remain inside _study, and the input file cannot be the output. Components remain individually usable and checkable.
class DataPipeline(ABC):
def run(self):
"""Template Method: every subclass follows this same sequence."""
df = self.load()
df = self.clean(df)
df = self.transform(df)
self.save(df)
return df
@abstractmethod
def load(self):
pass
@abstractmethod
def clean(self, df):
pass
@abstractmethod
def transform(self, df):
pass
@abstractmethod
def save(self, df):
pass
class CSVDataPipeline(DataPipeline):
def __init__(self, input_path, output_path, strategy, transforms):
self.reader = CSVReader(input_path)
self.cleaner = DataCleaner(strategy)
self.transforms = list(transforms)
self.output_path = Path(output_path).resolve()
# Course rule: every generated output must remain within _study.
self.output_path.relative_to(COURSE / "_study")
if self.output_path == Path(input_path).resolve():
raise ValueError("Output must differ from input")
def load(self):
return self.reader.read()
def clean(self, df):
return self.cleaner.clean(df)
def transform(self, df):
for transform in self.transforms:
df = transform.apply(df)
return df
def save(self, df):
self.output_path.parent.mkdir(parents=True, exist_ok=True)
df.to_csv(self.output_path, index=False)
Text is standardized before duplicate comparison. Numeric scaling follows cleaning. The sample has no duplicates and no constant observed measurement columns, so separate behavioral checks cover those cases.
Run the complete solution
Requires Python 3.10+ and pandas. The exact successful environment is recorded in runtime.json. The script uses paths anchored to its location, so it can run from any working directory. Use --input PATH for a different CSV.
Captured from the supplied dataset, rather than a fabricated example. Timing records are shown separately.
EXERCISE 1: first three source rows
id name age height_cm weight_kg city score
1 Alice 29.0 165.0 68.0 New York 85.0
2 Bob NaN 172.0 NaN Los Angeles 90.0
3 Charlie 35.0 168.0 72.0 Chicago NaN
EXERCISE 2: missing values before cleaning
id 0
name 0
age 3
height_cm 1
weight_kg 1
city 0
score 2
drop: 2 rows, 0 missing cells
id name age height_cm weight_kg city score
1 Alice 29.0 165.0 68.0 New York 85.0
5 Eva 27.0 160.0 55.0 New York 88.0
mean: 7 rows, 0 missing cells
id name age height_cm weight_kg city score
1 Alice 29.00 165.0 68.0 New York 85.0
2 Bob 30.25 172.0 70.0 Los Angeles 90.0
3 Charlie 35.00 168.0 72.0 Chicago 86.0
4 David 30.25 167.0 80.0 Houston 75.0
5 Eva 27.00 160.0 55.0 New York 88.0
6 Frank 30.00 175.0 85.0 Los Angeles 86.0
7 Grace 30.25 162.0 60.0 Chicago 92.0
mode: 7 rows, 0 missing cells
id name age height_cm weight_kg city score
1 Alice 29.0 165.0 68.0 New York 85.0
2 Bob 27.0 172.0 55.0 Los Angeles 90.0
3 Charlie 35.0 168.0 72.0 Chicago 75.0
4 David 27.0 160.0 80.0 Houston 75.0
5 Eva 27.0 160.0 55.0 New York 88.0
6 Frank 30.0 175.0 85.0 Los Angeles 75.0
7 Grace 27.0 162.0 60.0 Chicago 92.0
Observed means: {"age": 30.25, "height_cm": 167.0, "score": 86.0, "weight_kg": 70.0}
EXERCISES 3-5: logged pipeline with factory transforms
id name age height_cm weight_kg city score
1 alice 0.250000 0.333333 0.433333 new york 0.588235
2 bob 0.406250 0.800000 0.500000 los angeles 0.882353
3 charlie 1.000000 0.533333 0.566667 chicago 0.647059
4 david 0.406250 0.466667 0.833333 houston 0.000000
5 eva 0.000000 0.000000 0.000000 new york 0.764706
6 frank 0.375000 1.000000 1.000000 los angeles 0.647059
7 grace 0.406250 0.133333 0.166667 chicago 1.000000
Saved cleaned_drop.csv, cleaned_mean.csv, cleaned_mode.csv, pipeline_output.csv
Show actual logging and timing output
START CSVReader.read
TIME CSVReader.read 0.001935 seconds
FINISH CSVReader.read
START DataCleaner.clean
TIME DataCleaner.clean 0.000315 seconds
FINISH DataCleaner.clean
START DataCleaner.clean
TIME DataCleaner.clean 0.000487 seconds
FINISH DataCleaner.clean
START DataCleaner.clean
TIME DataCleaner.clean 0.000902 seconds
FINISH DataCleaner.clean
START CSVReader.read
TIME CSVReader.read 0.000401 seconds
FINISH CSVReader.read
START DataCleaner.clean
TIME DataCleaner.clean 0.000721 seconds
FINISH DataCleaner.clean
Show all 13 behavioral check results
test_all_missing_column_warns_and_remains_missing (test_solution.CleaningTests.test_all_missing_column_warns_and_remains_missing) ... ok
test_decorators_preserve_result_metadata_and_exception (test_solution.CleaningTests.test_decorators_preserve_result_metadata_and_exception) ... ok
test_drop_retains_only_complete_rows (test_solution.CleaningTests.test_drop_retains_only_complete_rows) ... ok
test_empty_frame_is_supported (test_solution.CleaningTests.test_empty_frame_is_supported) ... ok
test_factory_validation (test_solution.CleaningTests.test_factory_validation) ... ok
test_mean_values_and_input_immutability (test_solution.CleaningTests.test_mean_values_and_input_immutability) ... ok
test_mode_tie_rule (test_solution.CleaningTests.test_mode_tie_rule) ... ok
test_normalize_preserves_id_and_handles_constant (test_solution.CleaningTests.test_normalize_preserves_id_and_handles_constant) ... ok
test_pipeline_round_trip_and_write_boundary (test_solution.CleaningTests.test_pipeline_round_trip_and_write_boundary) ... ok
test_reader_lazy_preview_and_copy (test_solution.CleaningTests.test_reader_lazy_preview_and_copy) ... ok
test_strategy_is_abstract (test_solution.CleaningTests.test_strategy_is_abstract) ... ok
test_template_call_order (test_solution.CleaningTests.test_template_call_order) ... ok
test_text_preserves_missing_and_deduplication_order (test_solution.CleaningTests.test_text_preserves_missing_and_deduplication_order) ... ok
----------------------------------------------------------------------
Ran 13 tests in 0.017s
OK
Validation and comparison
All 13 checks passed. They verify complete-row selection, exact mean replacements, mode ties, source immutability, empty/all-missing inputs, constant-column scaling, preservation of IDs and missing text, duplicate behavior, unknown factory keys, reader ownership, decorator metadata and exceptions, Template Method call order, and CSV output round-trip.
No official TP02 solution was supplied. The completed TP01 review supports the foundational Python style, but it is not an answer key to these five exercises. The normalization, text and tie policies above are explicit implementation assumptions.
More worked examples
Two files appeared in the course folder after the first guide was built: TP02.md (a single-script solution) and TP02_OOP.md (the same exercises as notebook cells). The first example below runs the classes from TP02.md and compares them with the study solution. Nothing in either file was changed.
1
Your own TP02.md, executed and compared with the study solution
New files TP02.md and TP02_OOP.md appeared in the folder after the first guide was built. This runs the classes from TP02.md on the course CSV and compares every output column with the study solution.
import re
import tempfile
from pathlib import Path
import pandas as pd
source = Path("TP02.md").read_text(encoding="utf-8")
code = re.search(r"```python\n(.*?)```", source, re.S).group(1)
code = code.split("# Example")[0] # keep the classes, skip the demo call
namespace = {}
exec(compile(code, "TP02.md", "exec"), namespace)
with tempfile.TemporaryDirectory() as folder:
pipeline = namespace["CSVDataPipeline"](
"sample_data (2).csv", Path(folder) / "cleaned.csv", namespace["FillMean"](),
["remove_duplicates", "standardize_text", "normalize"])
result = pipeline.run()
study = pd.read_csv("_study/tp/TP02/pipeline_output.csv")
print("your TP02.md :", list(result.columns))
print("study solution:", list(study.columns))
print()
for column in result.columns:
if result[column].dtype == object or str(result[column].dtype).startswith("str"):
same = list(result[column]) == list(study[column])
else:
same = bool(((result[column] - study[column]).abs() < 1e-9).all())
print(f"{column:10s} identical: {same}")
print()
print(result.round(3).to_string(index=False))
Starting read
Time: 0.0011s
Finished read
your TP02.md : ['id', 'name', 'age', 'height_cm', 'weight_kg', 'city', 'score']
study solution: ['id', 'name', 'age', 'height_cm', 'weight_kg', 'city', 'score']
id identical: False
name identical: True
age identical: True
height_cm identical: True
weight_kg identical: True
city identical: True
score identical: True
id name age height_cm weight_kg city score
0.000 alice 0.250 0.333 0.433 new york 0.588
0.167 bob 0.406 0.800 0.500 los angeles 0.882
0.333 charlie 1.000 0.533 0.567 chicago 0.647
0.500 david 0.406 0.467 0.833 houston 0.000
0.667 eva 0.000 0.000 0.000 new york 0.765
0.833 frank 0.375 1.000 1.000 los angeles 0.647
1.000 grace 0.406 0.133 0.167 chicago 1.000
Result: six of seven columns are identical. The one difference is id. In TP02.md, NormalizeColumns selects every numeric column (select_dtypes(include='number')), so the identifier is rescaled to 0..1. The study solution passes exclude=('id',), because an identifier is a label, not a measurement. The first three lines of output (Starting read / Time / Finished read) come from the decorators in TP02.md, not from this example.
Try it. Which column differs, and which line of NormalizeColumns causes it? Compare with the exclude=('id',) option in the study solution.
The @log_time decorator wrote real elapsed times to execution.log. Parse them and summarise per method.
import re
from collections import defaultdict
timings = defaultdict(list)
with open("_study/tp/TP02/execution.log", encoding="utf-8") as handle:
for line in handle:
match = re.match(r"TIME (\S+) ([0-9.]+) seconds", line)
if match:
timings[match.group(1)].append(float(match.group(2)))
summary = {}
for method, values in timings.items():
summary[method] = {"calls": len(values), "mean_ms": sum(values) / len(values) * 1000, "max_ms": max(values) * 1000}
print(f"{method:18s} calls={len(values)} mean={summary[method]['mean_ms']:.3f} ms max={summary[method]['max_ms']:.3f} ms")
CSVReader.read calls=2 mean=1.168 ms max=1.935 ms
DataCleaner.clean calls=4 mean=0.606 ms max=0.902 ms
Elapsed times depend on the machine and on what else is running. Treat them as an illustration of what the decorator records, not as a performance result.
Try it. Run solution.py twice and compare. Which numbers change, and why is one run not a benchmark?
Added 2 Nov 2026. TP02.ipynb (saved 23 Oct, after this page was first built) solves all five exercises. Its printed results were compared with the study solution above.
Exercise
Your notebook
Check
1 · CSVReader
read() + preview(n), data kept in self.data
Same 7 × 7 table
2 · Strategy
DropMissing keeps Alice and Eva; FillMean gives age 30.25, height 167, weight 70, score 86; FillMode gives 27, 160, 55, 75
Identical to the study solution's values
3 · Decorators
@log_action outside @log_time, functools.wraps, and wrapping Reporter.summary from outside the class
Same idea; the outside wrapping is a good extra demonstration of Open/Closed
4 · Factory
Registry dict with aliases ('text', 'duplicates') and a helpful ValueError
NormalizeColumns rescales id too (same as TP02.md); the study solution leaves id alone
5 · Template Method
New 5-row dirty dataset; FillMean fills age 30 and salary 58,750, then normalize → text → dedupe gives 4 rows
Arithmetic checked: (25+30+25+40)/4 = 30 and 235,000/4 = 58,750
One thing to fix: exercise 5 says the headers are inconsistent, but StandardizeText cleans only cell values, so the output still has headers like " Full Name ". Add result.columns = result.columns.str.strip().str.lower().str.replace(" ", "_") to clean them too. Also note the mean is taken before the duplicate Alice row is removed, so that duplicate pulls the age and salary means toward Alice's values.
Go deeper
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Chapter 1 · Choose code by the time needed to understand it
2 extra examples1 chart4 sources
Learning objectives
State the book’s central readability criterion.
Distinguish concise code from code that is quick to understand.
Define the behavior a refactoring must preserve.
Apply a reader’s perspective to numerical code.
Core explanation
A program is a message to a future reader as well as instructions for a computer. The reader must identify inputs, outputs, edge cases and side effects before changing anything safely. Readability therefore includes understanding behavior, finding defects and seeing how a function fits into the rest of the program.
The book calls minimizing another person's time to understand code its fundamental theorem of readability. This is a practical design principle, not a proved numerical law or a metric that this course has measured. A short expression can take longer to decode than several clearly named steps. A useful comment can reduce understanding time even though it adds lines.
For applied mathematics, conventional symbols can be excellent inside a small derivation. Across a larger analysis, names such as sample_weights and weighted_sum preserve context that a reader otherwise has to recover. Choose a form appropriate to the reader and scope.
A readability refactoring should preserve the relevant contract. Compare results, exceptions, side effects and evaluation order where these are observable. Do not replace a slow algorithm merely because a one-line alternative looks shorter without checking its behavior. Performance and clarity often improve together, but actual constraints still matter. Stop refining when a competent teammate can follow the code comfortably.
Key formulas and contracts
f_before(x) = f_after(x), for every supported x
f_before and f_after are functions before and after a refactoring; D is the supported input domain and x an input. Equal return values are necessary but do not alone establish equality of exceptions or side effects.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Three shapes of one rule
Counted with Python's ast module
All three return identical grades for every score from 0 to 100. The counts are proxies for what a reader must hold in mind, not a measure of understanding time.Data table
dense
table
chain
source lines
1
5
8
distinct names
1
4
1
max nesting depth
3
2
3
Worked example
Task: make a mean computation easier to inspect.
def mean(values):
if not values:
raise ValueError("mean needs at least one value")
total = sum(values)
count = len(values)
return total / count
State the empty-input contract explicitly.
Name the total and count so the formula is visible.
For [2, 4, 6], total = 12, count = 3, result = 4.0.
Check the empty list separately. Raising an informative ValueError is this new example's contract; changing an existing function's exception would require deliberate agreement.
The example is newly authored in Python to explain the chapter, rather than copied from its C++ examples.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Three ways to write one rule, measured
Same behavior, three shapes. Count lines, syntax-tree nodes, nesting depth and distinct names to see what a reader has to hold in mind.
import ast
import inspect
def grade_dense(s): return "A" if s >= 90 else "B" if s >= 80 else "C" if s >= 70 else "F"
GRADE_CUTOFFS = [(90, "A"), (80, "B"), (70, "C")]
def grade_table(score):
for cutoff, letter in GRADE_CUTOFFS:
if score >= cutoff:
return letter
return "F"
def grade_chain(score):
if score >= 90:
return "A"
elif score >= 80:
return "B"
elif score >= 70:
return "C"
return "F"
def depth(node, level=0):
nested = (ast.If, ast.For, ast.While, ast.IfExp)
here = level + 1 if isinstance(node, nested) else level
return max([here] + [depth(child, here) for child in ast.iter_child_nodes(node)])
assert all(grade_dense(s) == grade_table(s) == grade_chain(s) for s in range(0, 101))
print("all three agree on every score from 0 to 100\n")
metrics = []
for func in (grade_dense, grade_table, grade_chain):
tree = ast.parse(inspect.getsource(func))
names = {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)} | {a.arg for a in ast.walk(tree) if isinstance(a, ast.arg)}
metrics.append({"version": func.__name__, "lines": len(inspect.getsource(func).strip().splitlines()),
"nodes": sum(1 for _ in ast.walk(tree)), "depth": depth(tree), "names": len(names)})
print(f"{'version':12s} {'lines':>5} {'nodes':>6} {'depth':>6} {'names':>6}")
for m in metrics:
print(f"{m['version']:12s} {m['lines']:>5} {m['nodes']:>6} {m['depth']:>6} {m['names']:>6}")
all three agree on every score from 0 to 100
version lines nodes depth names
grade_dense 1 27 3 1
grade_table 5 25 2 4
grade_chain 8 30 3 1
These counts are simple proxies. The book's criterion is the time another person needs to understand the code, which this course has not measured.
Try it. Which version would you rather debug at midnight? The numbers do not decide; they only show what each version asks of the reader.
Compare the old and new mean on 205 inputs. Equal values are not enough: the exception type is part of observable behavior.
import random
def mean_before(values):
return sum(values) / len(values)
def mean_after(values):
if not values:
raise ValueError("mean needs at least one value")
total = sum(values)
count = len(values)
return total / count
random.seed(7)
cases = [[], [5], [2, 4, 6], [1.5, 2.5], [-3, 3]]
cases += [[random.randint(-9, 9) for _ in range(random.randint(1, 6))] for _ in range(200)]
same = 0
differences = set()
for values in cases:
outcomes = []
for func in (mean_before, mean_after):
try:
outcomes.append(("value", func(values)))
except Exception as error:
outcomes.append((type(error).__name__, None))
if outcomes[0] == outcomes[1]:
same += 1
else:
differences.add((tuple(values), outcomes[0][0], outcomes[1][0]))
print(f"checked {len(cases)} inputs: {same} identical, {len(differences)} distinct difference(s)")
for values, before, after in sorted(differences):
print("differs on", list(values), "->", before, "became", after)
checked 205 inputs: 204 identical, 1 distinct difference(s)
differs on [] -> ZeroDivisionError became ValueError
Try it. Decide: is this a bug in the refactoring or an improvement? Who has to agree before the change ships?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L04
Packing Information into Names
Chapter 2 · Meaning, units, scope and conventions
2 extra examples1 chart5 sources
Learning objectives
Replace generic names with names that reveal a value’s role.
Encode important units and data state.
Choose name length according to scope.
Use naming conventions consistently without replacing precise meaning.
Core explanation
A useful name is a small explanation repeated at every use. size could mean bytes, rows or tree height. memory_bytes, row_count and tree_height remove different uncertainties. A specific verb such as download_page can also reveal an operation's cost and source more clearly than get_page.
Generic names have legitimate local uses. An index i in a two-line mathematical loop or a temporary swap variable can be clear. A value assembled from several user fields deserves user_summary, even if it is short-lived. In nested loops, role-based names reduce accidental use of the wrong index.
Prefer a name describing actual behavior over the circumstances in which you first used it. A flag that increases logging should say extra_logging, rather than run_locally. Separate independent options instead of hiding several policies behind one vague name.
Attach units or representation when confusing them would cause a bug: elapsed_ms, angle_radians, raw_text. A state-bearing name is useful information, but it does not itself validate data. A wider scope requires a name carrying more context. Familiar abbreviations may be fine; project-specific acronyms burden new readers. Follow the language's conventions for functions, classes and constants, while choosing words that remain accurate.
Key formulas and contracts
t_seconds = t_ms / 1000
t_ms is a time duration in milliseconds, t_seconds is the same duration in seconds, and 1000 is the number of milliseconds per second.
‖v‖₂ = √(∑ᵢ₌₁ⁿ vᵢ²)
v is a real vector with n components; vᵢ is component i and ‖v‖₂ is its Euclidean norm. The accumulator name sum_squares describes the inner sum.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Name length against lines alive
54 local names in the TP02 solution
Each dot is one local name or argument. Long names tend to live longer, but the pattern is loose; hover a dot for the name and function.Data table
The inputs are milliseconds, so the difference is 2250 milliseconds.
The suffix keeps that unit visible at the subtraction.
Divide once by 1000 to obtain 2.25 seconds.
A name like elapsed would make it easier to label 2250 incorrectly as seconds.
For a norm calculation, sum_squares += value visually contradicts its name; sum_squares += value * value agrees. A good name can help a reader detect a missing operation.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Units belong in the name
A bare number carries no unit. A suffix makes a wrong sum visible before the code runs.
SECONDS_PER_MINUTE = 60
def minutes_to_seconds(duration_minutes):
return duration_minutes * SECONDS_PER_MINUTE
timeout_seconds = 30
delay_minutes = 2
total_wait_seconds = timeout_seconds + minutes_to_seconds(delay_minutes)
print("total wait:", total_wait_seconds, "seconds")
# The same bug when names say nothing about units:
timeout = 30
delay = 2
print("mixed units:", timeout + delay, "(seconds plus minutes, silently wrong)")
total wait: 150 seconds
mixed units: 32 (seconds plus minutes, silently wrong)
Try it. Rename the arguments of a function you wrote recently so that every physical quantity carries its unit.
The lesson says short names suit short scopes. Test it on the TP02 solution: for every local name, measure how many lines it stays alive.
import ast
from pathlib import Path
tree = ast.parse(Path("_study/tp/TP02/solution.py").read_text(encoding="utf-8"))
def own_nodes(function):
"""Walk a function body without descending into nested functions."""
stack = list(function.body)
while stack:
node = stack.pop()
yield node
if not isinstance(node, (ast.FunctionDef, ast.Lambda)):
stack.extend(ast.iter_child_nodes(node))
records = []
for function in ast.walk(tree):
if not isinstance(function, ast.FunctionDef):
continue
declared = {a.arg for a in function.args.args + function.args.kwonlyargs}
lines = {}
for node in own_nodes(function):
if isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store):
declared.add(node.id)
for node in own_nodes(function):
if isinstance(node, ast.Name) and node.id in declared:
lines.setdefault(node.id, []).append(node.lineno)
for name, where in lines.items():
if name not in {"self", "cls"}:
records.append({"function": function.name, "name": name, "length": len(name),
"span": max(where) - min(where) + 1})
n = len(records)
mx = sum(r["length"] for r in records) / n
my = sum(r["span"] for r in records) / n
cov = sum((r["length"] - mx) * (r["span"] - my) for r in records)
sx = sum((r["length"] - mx) ** 2 for r in records) ** 0.5
sy = sum((r["span"] - my) ** 2 for r in records) ** 0.5
print(f"{n} local names; correlation between name length and lines alive: r = {cov / (sx * sy):.2f}")
print("longest-lived names:")
for r in sorted(records, key=lambda r: -r["span"])[:6]:
print(f" {r['name']:<12} in {r['function']:<14} alive {r['span']:>3} lines, {r['length']} characters")
54 local names; correlation between name length and lines alive: r = -0.02
longest-lived names:
args in main alive 18 lines, 4 characters
result in handle alive 13 lines, 6 characters
result in apply alive 13 lines, 6 characters
original in main alive 11 lines, 8 characters
name in handle alive 11 lines, 4 characters
result in handle alive 11 lines, 6 characters
One 273-line file is a tiny sample. The result illustrates how to measure the idea; it does not prove the guideline.
Try it. Is the correlation strong enough to call it a rule? What does a single code base (273 lines) not tell you?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L05
Names That Cannot Be Misconstrued
Chapter 3 · Boundaries and expectations are part of the name
2 extra examples1 chart4 sources
Learning objectives
Identify plausible wrong interpretations of an API name.
Specify inclusive and half-open boundaries.
Name boolean values in the positive form.
Avoid names that mislead callers about cost or effects.
Core explanation
Even an informative name can support two interpretations. Does filter retain matching rows or remove them? For a new API, select_rows and exclude_rows make the direction explicit. Established library names must be understood according to that library's contract; do not rename library calls merely to satisfy a slogan.
Boundary words deserve care. If ten items are allowed, MAX_ITEMS = 10 goes with a rejection condition greater than ten. Inclusive endpoints can be described with first and last. Half-open ranges use an included beginning and an excluded end. Python's existing range(start, stop) follows its documented half-open convention even though the book prefers begin/end for a new API.
Boolean names should answer a yes/no question. has_missing_values reads more clearly than missing_values, which could be a count or collection. Avoid double negations such as not disable_validation. Distinguish a request to do work from a report that work has been done.
Callers bring expectations. A name beginning with get often suggests a cheap accessor. A scan over all observations is better described by compute_mean if that cost matters. The textbook's linked-list size example is historical; the supplied text itself notes that a later C++ standard requires constant-time size. The durable lesson is to make actual cost and behavior unsurprising.
Key formulas and contracts
[a,b) = {x : a ≤ x < b}
a is the included lower endpoint; b is the excluded upper endpoint; x is an element of the interval.
count > MAX_ITEMS ⇒ reject
c is the requested item count and M is the inclusive maximum allowed count. Equality c = M is accepted.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Which integers does each convention include?
Four ways to write the bounds 2 and 4
Number of integers included: [2, 4] has 3, [2, 4) has 2, (2, 4] has 2, (2, 4) has 1. The half-open form [2, 4) has length 4 - 2 = 2, matching range(2, 4).Data table
0
1
2
3
4
5
6
[2, 4] closed
0
0
1
1
1
0
0
[2, 4) half-open
0
0
1
1
0
0
0
(2, 4] half-open left
0
0
0
1
1
0
0
(2, 4) open
0
0
0
1
0
0
0
Worked example
Task: allow at most ten rows.
MAX_ROWS = 10
def exceeds_row_limit(row_count):
return row_count > MAX_ROWS
print([exceeds_row_limit(n) for n in [9, 10, 11]])
# [False, False, True]
The phrase “at most ten” includes ten.
The predicate names the rejected condition, so True means too many rows.
Test 9, 10 and 11 to inspect the boundary.
For a row slice from index 2 to index 5, rows[2:5] includes indices 2, 3 and 4, exactly three positions.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Half-open ranges make lengths and joins trivial
With [start, end) the length is end - start and neighbouring ranges meet without a gap or an overlap. Python's range and slices use this convention.
A negative name plus a negation in the condition forces the reader to untangle a double negative.
def scheme_positive(use_ssl):
return "https" if use_ssl else "http"
def scheme_negative(disable_ssl):
return "http" if disable_ssl else "https"
for flag in (True, False):
print(f"use_ssl={flag!s:5} -> {scheme_positive(flag)}")
for flag in (True, False):
print(f"disable_ssl={flag!s:5} -> {scheme_negative(flag)} # 'not disable' means enabled")
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L06
Aesthetics and Consistent Layout
Chapter 4 · Make structure visible before the reader decodes details
2 extra examples1 chart4 sources
Learning objectives
Give similar code a consistent visual pattern.
Group lines into coherent conceptual blocks.
Maintain one meaningful order across related operations.
Use helpers when formatting alone cannot remove distracting repetition.
Core explanation
Readable layout gives the eyes a map. Repeated operations should have comparable shapes, and related lines should appear together. A reader can then recognize the structure and spend attention on the values that differ.
Whitespace can group a function into stages such as data preparation, calculation and reporting. A blank line is useful when it marks an actual conceptual transition. A blank line after every statement destroys grouping. Consistent line wrapping helps readers compare calls with similar arguments.
Choose a meaningful ordering and retain it: input-form order, processing order or alphabetical order may each work. If loading uses age, height, weight, subsequent validation should preferably present the same sequence. Inconsistency makes missing or duplicated fields harder to notice.
Column alignment can make a small table of cases easy to scan, but it can also require unnecessary whitespace changes. Respect the formatter and team conventions rather than maintaining decorative spacing at any cost. In Python, ordinary PEP 8 spacing is a sound baseline.
Sometimes the obstacle is repeated setup rather than alignment. A small helper can hide irrelevant object construction and leave input/expected-output pairs visible. This moves beyond surface formatting, but the criterion is still ease of understanding. Layout does not prove correctness; attractive code can still implement the wrong formula.
Key formulas and contracts
μ = (x₁ + ⋯ + xₙ)/n
μ denotes the arithmetic mean in this example; xᵢ is observation i and n is a positive number of observations. Formatting changes should leave this computation unchanged.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Line lengths in the TP02 solution
221 non-blank lines; 13 longer than 79 characters
Amber bars start at 80 characters, beyond the PEP 8 limit. The dashed line marks 79.Data table
For a reusable function, explicitly handle an empty sample before division or min/max.
A helper that checks a named statistic against an expected value can remove repeated fixture setup in tests, while keeping the important numbers prominent.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Align columns so the eye can scan them
Consistent widths turn a pile of strings into a table. Format specifications do the alignment so the code stays regular.
import csv
with open("sample_data (2).csv", newline="") as handle:
rows = list(csv.DictReader(handle))
header = f"{'id':>2} {'name':<8} {'city':<12} {'score':>5}"
print(header)
print("-" * len(header))
for row in rows:
print(f"{row['id']:>2} {row['name']:<8} {row['city']:<12} {row['score']:>5}")
id name city score
-------------------------------
1 Alice New York 85
2 Bob Los Angeles 90
3 Charlie Chicago NaN
4 David Houston 75
5 Eva New York 88
6 Frank Los Angeles NaN
7 Grace Chicago 92
Try it. Change the score column to show one decimal place. Which format specification changes?
A layout rule you cannot check is a wish. Count how many lines of the TP02 solution exceed the 79-character limit from PEP 8.
from pathlib import Path
LIMIT = 79
lines = Path("_study/tp/TP02/solution.py").read_text(encoding="utf-8").splitlines()
lengths = [len(text) for text in lines if text.strip()]
over = [(number, len(text)) for number, text in enumerate(lines, 1) if len(text) > LIMIT]
print(f"{len(lines)} lines, {len(lengths)} non-blank, longest = {max(lengths)} characters")
print(f"lines over {LIMIT}: {len(over)}")
for number, width in over[:5]:
print(f" line {number}: {width} characters")
273 lines, 221 non-blank, longest = 96 characters
lines over 79: 13
line 24: 80 characters
line 91: 93 characters
line 96: 88 characters
line 114: 84 characters
line 146: 93 characters
Try it. Wrap one long line with parentheses instead of a backslash. Does the meaning change?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L07
Knowing What to Comment
Chapter 5 · Preserve the context a future reader will need
2 extra examples1 chart4 sources
Learning objectives
Distinguish redundant comments from useful explanations.
Record decision rationale, constant choices and known limitations.
Document surprising effects, runtime or boundary behavior.
Use file-level and block-level comments to explain the big picture.
Improve a first draft into a precise comment.
Core explanation
A useful comment transfers information the writer knows but the reader cannot recover quickly. A future reader may see the same code yet lack the experiment that motivated a data structure, the reason for a threshold or the limitation that has not been fixed.
Start with clear names and structure. A comment that simply repeats class Account adds little. A comment that must explain an unclear name often points to a naming improvement. Yet “already derivable from code” does not automatically mean useless: a dense expression may take enough time to decode that a short explanation helps.
Record decisions with their scope. A benchmark favoring one data structure for a particular dataset does not establish that it is always faster. A tuned constant should explain what was optimized, on which evidence, and which constraints matter. The slides' image quality 0.72 is an example-specific choice, not a universal recommendation.
Document known limitations and anticipated pitfalls. Teams use work markers for unfinished tasks, known defects or temporary workarounds; the relevant information is the exact failing case and intended follow-up. A synchronous method contacting an external service needs a delay warning. A function with unexpectedly expensive inputs should describe those inputs. These are part of a caller's practical understanding.
Big-picture comments explain a file's responsibility, the direction of data flow and where to begin reading. Within a long function, a short heading can summarize a whole block. If a block has an independent responsibility, a named function may communicate it better. The source does not require comments to contain only “why”: what or how can help when they communicate useful information.
The supplied ten-slide deck and speaker script follow this chapter closely: motivation, unhelpful comments, search-depth behavior, decision rationale, work markers, surprises, high-level summaries and revision. Their logos identify the institution; the instruction-manual cartoon illustrates that explanations are valuable when the operation is difficult or risky.
Key formulas and contracts
inspect levels 0,…,d when d > 0; inspect level 0 when d ≤ 0
d is a requested search depth and level 0 is the starting node. This describes the supplied tree-search example’s contract; it is not a rule for every tree API.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Code, comments and docstrings per definition
TP02 solution, definitions that carry a comment or docstring
Docstrings and comments are a small share of most definitions. Classes include their methods, so they are tall.Data table
code
comment
docstring
log_action
12
0
1
log_time
10
0
1
FillMean
16
1
0
FillMode
14
1
0
NormalizeColumns
17
0
1
RemoveDuplicates
3
1
0
StandardizeText
10
1
0
CSVDataPipeline
20
1
0
handle
1
0
1
handle
13
1
0
handle
11
1
0
apply
1
0
1
apply
2
1
0
apply
9
1
0
run
6
0
1
__init__
8
1
0
Worked example
Task: explain a missing-value choice without repeating code.
# Use the observed-column mean for this lab demonstration.
# An all-missing column stays missing because its mean is undefined.
cleaned = FillMean().handle(df)
The code already says which strategy is called, so a comment saying “fill mean” would add little.
The first comment explains the scope of the choice: a lab demonstration.
The second states an edge case a caller needs to know.
Link to the strategy implementation for details and to TP02 for observed outputs.
In the tree example, depth 2 means the root, children and grandchildren. A name alone rarely communicates both that convention and the no-match return value.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
How much of a real file is comment, docstring and code?
Comments are only useful where they add what the code cannot say. Measure the mix function by function in the TP02 solution.
import ast
import io
import tokenize
from pathlib import Path
path = Path("_study/tp/TP02/solution.py")
text = path.read_text(encoding="utf-8")
lines = text.splitlines()
tree = ast.parse(text)
comment_lines = set()
for token in tokenize.generate_tokens(io.StringIO(text).readline):
if token.type == tokenize.COMMENT and lines[token.start[0] - 1].strip().startswith("#"):
comment_lines.add(token.start[0])
rows = []
for node in ast.walk(tree):
if not isinstance(node, (ast.FunctionDef, ast.ClassDef)):
continue
doc_lines = set()
if ast.get_docstring(node):
first = node.body[0]
doc_lines = set(range(first.lineno, first.end_lineno + 1))
span = set(range(node.lineno, node.end_lineno + 1))
blank = {n for n in span if not lines[n - 1].strip()}
comments = span & comment_lines
code = span - blank - comments - doc_lines
rows.append({"name": node.name, "code": len(code), "comment": len(comments), "docstring": len(doc_lines)})
rows = [r for r in rows if r["docstring"] or r["comment"]]
print(f"{'definition':22s} {'code':>5} {'comment':>8} {'docstring':>10}")
for r in rows:
print(f"{r['name']:22s} {r['code']:>5} {r['comment']:>8} {r['docstring']:>10}")
A docstring that states limits and is checked by doctest
Document the corner cases and the cost, then let doctest keep the examples honest.
import doctest
def moving_average(values, window):
"""Return the trailing mean of each full window.
Assumes window >= 1. Returns [] when there are fewer than `window` values.
Runs in O(len(values)) because the running sum is updated, not recomputed.
>>> moving_average([1, 2, 3, 4], 2)
[1.5, 2.5, 3.5]
>>> moving_average([5], 3)
[]
"""
if len(values) < window:
return []
running = sum(values[:window])
result = [running / window]
for index in range(window, len(values)):
running += values[index] - values[index - window]
result.append(running / window)
return result
print(doctest.testmod())
TestResults(failed=0, attempted=2)
Try it. Break the function on purpose (for example change window to window + 1) and read how doctest reports the failure.
Do not state an illustrative benchmark or constant as a general truth.
Do not use comments as a permanent substitute for a clearer API name.
Update comments when behavior changes; stale intent can mislead.
The vector swap illustration is a historical C++ example. Its lesson is to explain surprising memory behavior, not to present a universal memory-return guarantee.
Summary
Write the missing context: rationale, limitations, contracts, surprises and the larger workflow. Then remove comments that cost attention without helping.
Five self-check questions
1. What is the purpose of a good comment?
To help the reader recover useful knowledge the writer had.
2. Can a comment explaining what code does be valuable?
Yes, if it makes a difficult operation faster to understand.
3. What should accompany a tuned constant?
The reason for the value, relevant evidence and the scope of applicability.
4. What does depth 2 inspect in the supplied example?
The starting node plus two levels below it.
5. How should you overcome comment-writing hesitation?
Write the thought, read it as an outsider, and replace vague language with exact behavior.
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L08
Making Comments Precise and Compact
Chapter 6 · State exact behavior with little ambiguity
2 extra examples1 chart4 sources
Learning objectives
Remove ambiguous pronouns and vague claims.
Define function behavior at corner cases.
Choose examples that distinguish competing interpretations.
Use concise established terms and named arguments where appropriate.
Core explanation
After deciding a comment is useful, make it specific enough to guide a caller. A short comment is valuable when it preserves the information needed to use the code correctly. Brevity that removes a crucial boundary is a poor trade.
Replace pronouns when several referents are plausible. “Check whether it is too big before inserting data in the cache” could refer to the cache or the data. Name the quantity being limited. Likewise, a priority comment should say which kind of item receives higher priority.
“Count lines” is not a complete contract. Does an empty file contain zero lines? Does a final line without a newline count? A function that counts newline bytes should say exactly that. Its behavior can differ from a reader's text-editor notion of a line.
A carefully chosen example can resolve several uncertainties at once. A stripping example with repeated characters at both ends and a matching character in the middle distinguishes boundary trimming from global replacement and character sets from exact substring removal. Choose examples that reveal behavior rather than merely looking typical.
State high-level intent, particularly when the implementation could accidentally do the opposite. Use actual named arguments in Python to explain boolean and numeric options. In languages without them, a compact parameter comment may help. Established terms such as cache or canonicalize can replace a long description if the intended audience knows them; define unfamiliar terms once.
Key formulas and contracts
N(s) = ∑ᵢ₌₁ᵐ 1{sᵢ = newline}
s is a byte sequence of length m, sᵢ its ith byte, N(s) the newline-byte count, and 1{condition} is 1 when the condition holds and 0 otherwise.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Readings still consistent as examples accumulate
Four plausible meanings of 'count the lines'
Typical examples leave several readings alive. Corner cases such as an empty string and a blank line in the middle remove them quickly.Data table
Test empty input and text without a final newline.
Test carriage return plus newline to expose what is ignored.
For character trimming, "abba/a/ba".strip("ab") gives "/a/": repeated boundary characters disappear while the internal a remains.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Which example rules out which reading?
'Count the lines' can mean several things. Run four plausible implementations, take len(splitlines()) as the intended meaning, and see which examples pull the others apart.
readings = {
"count newline characters": lambda s: s.count("\n"),
"count non-empty pieces": lambda s: len([p for p in s.split("\n") if p]),
"len(splitlines())": lambda s: len(s.splitlines()),
"len(split('\\n'))": lambda s: len(s.split("\n")),
}
intended = "len(splitlines())"
inputs = ["", "a", "a\n", "a\nb", "a\n\nb\n", "\n"]
print(f"{'input':10s}" + "".join(f"{name[:12]:>14s}" for name in readings))
for text in inputs:
print(f"{text!r:10s}" + "".join(f"{func(text):>14d}" for func in readings.values()))
def survivors(chosen):
"""Readings that agree with the intended one on every chosen example."""
expected = [readings[intended](s) for s in chosen]
return [n for n, f in readings.items() if [f(s) for s in chosen] == expected]
typical = ["a\nb", "a\nb\nc", "a\nb\nc\nd"]
sharp = ["", "a\n\nb", "a\n"]
elimination = {
"typical examples": [len(survivors(typical[:k])) for k in range(0, 4)],
"chosen corner cases": [len(survivors(sharp[:k])) for k in range(0, 4)],
}
print()
for label, counts in elimination.items():
print(f"{label:20s} readings still consistent after 0, 1, 2, 3 examples: {counts}")
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L09
Making Control Flow Easy to Read
Chapter 7 · Guard clauses, natural comparisons and visible cleanup
2 extra examples1 chart5 sources
Learning objectives
Write comparisons in a natural order.
Choose a helpful order for conditional branches.
Reduce nesting with early return and continue.
Preserve cleanup and understand hidden execution paths.
Core explanation
Control flow asks the reader to remember which path is active. Each nested branch adds a condition to that mental context. Prefer a structure where a reader can dismiss finished cases and follow the main computation in one direction.
In comparisons, put the value being inspected on the left and a stable reference on the right when that reads naturally: rows_read < expected_rows. For if/else, a positive, simple or conspicuous case may deserve first position. These priorities can conflict; choose the one that communicates the situation best.
A guard clause handles a special case and returns before the main work. Within a loop, continue can skip an invalid item early. These techniques reduce indentation without requiring a mutable “done” flag. A simple conditional expression choosing two small values is fine; nested expressions that also perform calculations often become harder to read.
Resource cleanup must still happen. Use a context manager for a file or an appropriate try/finally when returning early. Do not move cleanup below a return. The source discusses C-specific cleanup jumps, but Python normally provides more structured ways to express the same guarantee.
At a larger scale, exceptions, callbacks, threads and polymorphic calls can obscure which code runs next. They are useful features, including in TP02. Keep their behavior documented and traceable instead of introducing unnecessary layers. A readable pipeline has one visible entry point and a short sequence of calls.
Key formulas and contracts
valid row ⇔ row exists ∧ score ≥ 0
r is a row, E(r) states that the row exists, s(r) is its score, and V(r) means the row is accepted. The score may only be read after existence is established.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
The guard-clause version as a flow
flowchart TD
A[order arrives] --> B{user missing or inactive?}
B -- yes --> X[return False]
B -- no --> C{cart empty?}
C -- yes --> X
C -- no --> D{everything in stock?}
D -- no --> X
D -- yes --> Y[return True]
Nested versus guard clauses
Same answers on all nine test combinations
The flat version has half the nesting and fewer branches, with identical results.Data table
nested
flat
max nesting
4
1
if statements
4
2
return statements
5
3
Worked example
Task: select nonnegative scores from optional records.
def accepted_scores(rows):
scores = []
for row in rows:
if row is None:
continue
if row["score"] < 0:
continue
scores.append(row["score"])
return scores
# [None, {"score": -1}, {"score": 0}, {"score": 3}] → [0, 3]
Skip None before accessing a field.
Skip negative scores, including no valid work inside that branch.
Append accepted values at the main indentation level.
Include zero to make the boundary visible.
This example assumes every non-None record has a numeric score. Missing keys require a separately agreed contract.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Guard clauses: same answers, flatter code
Handle the exits first and the main path reads straight down. Prove both versions agree on every combination, then compare nesting.
import ast
import inspect
from itertools import product
def can_checkout_nested(user, cart):
if user is not None:
if user["active"]:
if cart:
if all(item["stock"] > 0 for item in cart):
return True
else:
return False
else:
return False
else:
return False
else:
return False
def can_checkout_flat(user, cart):
if user is None or not user["active"]:
return False
if not cart:
return False
return all(item["stock"] > 0 for item in cart)
users = {"no user": None, "active": {"active": True}, "inactive": {"active": False}}
carts = {"empty": [], "in stock": [{"stock": 2}, {"stock": 1}], "one sold out": [{"stock": 2}, {"stock": 0}]}
print(f"{'user':10s} {'cart':13s} nested flat")
agree = True
for (uname, user), (cname, cart) in product(users.items(), carts.items()):
a, b = can_checkout_nested(user, cart), can_checkout_flat(user, cart)
agree = agree and a == b
print(f"{uname:10s} {cname:13s} {a!s:7} {b!s}")
print("identical on all 9 combinations:", agree)
def depth(node, level=0):
here = level + 1 if isinstance(node, ast.If) else level
return max([here] + [depth(child, here) for child in ast.iter_child_nodes(node)])
metrics = []
for func in (can_checkout_nested, can_checkout_flat):
tree = ast.parse(inspect.getsource(func))
metrics.append({"version": func.__name__.replace("can_checkout_", ""),
"max nesting": depth(tree),
"if statements": sum(isinstance(n, ast.If) for n in ast.walk(tree)),
"return statements": sum(isinstance(n, ast.Return) for n in ast.walk(tree))})
print()
for m in metrics:
print(m)
user cart nested flat
no user empty False False
no user in stock False False
no user one sold out False False
active empty False False
active in stock True True
active one sold out False False
inactive empty False False
inactive in stock False False
inactive one sold out False False
identical on all 9 combinations: True
{'version': 'nested', 'max nesting': 4, 'if statements': 4, 'return statements': 5}
{'version': 'flat', 'max nesting': 1, 'if statements': 2, 'return statements': 3}
Try it. Add a rule: banned users can never check out. In which version is it easier to add and to see?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L10
Breaking Down Giant Expressions
Chapter 8 · Name concepts and simplify the logic itself
2 extra examples1 chart1 interactive5 sources
Learning objectives
Introduce explaining and summary variables.
Apply De Morgan’s laws correctly.
Preserve short-circuit behavior and effects.
Derive a simple half-open interval overlap test.
Core explanation
A long expression can combine several ideas before the reader has named any of them. An explaining variable labels a difficult subexpression; a summary variable names a concept used in several places. user_owns_document lets the reader reason about permission as one meaningful fact.
Boolean algebra can reduce negation. De Morgan's laws distribute not across a logical conjunction or disjunction while exchanging and/or. On pure boolean predicates the truth values agree. In real code, preserve short-circuit evaluation when the second condition reads data that might not exist or has observable side effects.
Sometimes the best simplification changes the question. To find whether two nonempty half-open intervals overlap, describing all overlapping endpoint arrangements is cumbersome. Non-overlap has two simple cases: the first ends before or at the second's beginning, or the second ends before or at the first's beginning. Negate those cases to obtain overlap.
State the interval domain. The textbook's short overlap rule needs nonempty, correctly ordered intervals. An empty interval inside another interval otherwise gives a misleading result. For code meant to accept empty intervals, check them explicitly first. A simpler formula is valuable when its assumptions remain visible.
Repeated method chains, selectors or long literals can also be named once. Do not automatically cache an expression whose value must change later, and do not turn a side-effecting call into a precomputed boolean when its execution must remain conditional.
Key formulas and contracts
¬(A ∧ B) ⇔ ¬A ∨ ¬B; ¬(A ∨ B) ⇔ ¬A ∧ ¬B
A and B are boolean propositions; ¬ means not, ∧ means and, ∨ means or, and ⇔ means logical equivalence.
[a,b) overlaps [c,d) ⇔ a < d ∧ c < b, if a < b and c < d
a,b are the first interval endpoints and c,d the second. Each lower endpoint is included and each upper endpoint excluded. ∅ denotes the empty set.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Does [2, 5) overlap [c, d)?
Every valid half-open interval with 0 <= c < d <= 8
Grey squares are invalid (d <= c). The rule a < d and c < b gives the amber region, confirmed against brute force in the example.Data table
d=1
d=2
d=3
d=4
d=5
d=6
d=7
d=8
c = 0
0
0
1
1
1
1
1
1
c = 1
0
1
1
1
1
1
1
c = 2
1
1
1
1
1
1
c = 3
1
1
1
1
1
c = 4
1
1
1
1
c = 5
0
0
0
c = 6
0
0
c = 7
0
Try it: drag the interval ends
Move the four sliders. The rule a < d and c < b is compared with a brute-force search for a shared integer.
This interactive needs JavaScript. The text and charts around it cover the same ideas.
Worked example
Task: derive and check overlap.
Non-overlap is b <= c or d <= a.
Negate the disjunction: b > c and d > a.
Reorder comparisons to a < d and c < b.
Check [0,2) and [2,4): 0 < 4 is true but 2 < 2 is false, so they do not overlap.
Check containment [0,10) and [3,4): both inequalities hold.
def overlaps(a, b, c, d):
if a > b or c > d:
raise ValueError("interval endpoints are reversed")
if a == b or c == d:
return False
return a < d and c < b
An empty interval [3,3) is rejected as an overlap even inside [0,10).
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
De Morgan's laws, verified exhaustively
Distributing a negation flips and/or. With three booleans there are only eight cases, so check them all instead of trusting memory.
from itertools import product
def violates_policy(is_admin, has_token, is_expired):
return not (is_admin or (has_token and not is_expired))
def violates_policy_simplified(is_admin, has_token, is_expired):
return not is_admin and (not has_token or is_expired)
for combo in product([False, True], repeat=3):
a, b = violates_policy(*combo), violates_policy_simplified(*combo)
assert a == b
print(combo, "->", a)
print("identical on all 8 combinations")
Prove the interval-overlap rule, then find where it breaks
For half-open [a, b) and [c, d), 'a < d and c < b' means they overlap. Brute force confirms it for non-empty intervals and exposes the empty ones.
def overlaps(a, b, c, d):
return a < d and c < b
def share_a_point(a, b, c, d):
return any(c <= x < d for x in range(a, b))
N = 7
valid = [(a, b) for a in range(N) for b in range(a + 1, N)]
checked = mismatches = 0
for a, b in valid:
for c, d in valid:
checked += 1
mismatches += overlaps(a, b, c, d) != share_a_point(a, b, c, d)
print(f"non-empty intervals: {checked} pairs checked, {mismatches} mismatches")
empty_bad = [(a, b, c, d) for a in range(N) for b in range(N) for c in range(N) for d in range(N)
if (b <= a or d <= c) and overlaps(a, b, c, d) != share_a_point(a, b, c, d)]
print(f"with empty or reversed intervals allowed: {len(empty_bad)} mismatches, e.g. {empty_bad[0]}")
# overlap of [2, 5) with every [c, d) -- 1 means they overlap, None means d <= c (not a valid interval)
grid = [[(1 if overlaps(2, 5, c, d) else 0) if d > c else None for d in range(1, 9)] for c in range(0, 8)]
non-empty intervals: 441 pairs checked, 0 mismatches
with empty or reversed intervals allowed: 140 mismatches, e.g. (0, 2, 1, 1)
Try it. State the precondition in a docstring so the next reader does not have to rediscover it.
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L11
Variables and Readability
Chapter 9 · Fewer live names and fewer state changes
2 extra examples1 chart5 sources
Learning objectives
Remove intermediate variables that add no meaning.
Shrink scope and define values near first use.
Prefer stable values over repeatedly reassigned state.
Apply Python scoping rules accurately.
Core explanation
A reader tracks not only a variable's meaning but also where it can change. Many variables, wide scopes and repeated assignments multiply that work. The goal is not a minimum variable count; it is a manageable set of useful facts.
Keep explaining variables from the previous chapter when they name a concept. Remove a temporary that merely forwards an already clear expression once. When a search finds what it needs, returning the result immediately can remove separate found flags and intermediate result indices.
Local state is easier to inspect than global state or a class attribute shared by many methods. Pass a value explicitly when only two methods need it rather than making it visible to an entire large class. Define a variable close to its first real use. Do not delay an operation if later evaluation could change its required meaning.
Python if, for and while statements do not create separate local block scopes. A variable assigned only within a branch may be unbound if the branch does not execute. A comprehension does have its own iteration-variable scope in Python 3. A list being mutable and a name being rebound are different issues: write-once assignment alone does not make the underlying list immutable.
The textbook's JavaScript advice describes older var-style code. The lesson here is the general scope principle, not an instruction to copy historical JavaScript syntax into a modern program. Keep this study site's small shared script isolated in its own function scope.
Key formulas and contracts
return first x with P(x), else None
X is an ordered sequence of values xᵢ; P is a predicate; k is the earliest matching index. If the index set is empty, the implementation returns None.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
How long each variable in main() stays alive
First assignment to last use, TP02 solution
Each bar spans from the first line a name appears to its last; dots are the lines that mention it. Short bars mean less to remember.Data table
name
first line
last line
lines mentioning it
parser
245
247
245, 246, 247
args
247
264
247, 250, 264
reader
250
252
250, 251, 252
original
251
261
251, 254, 257, 261
strategies
255
256
255, 256
name
256
265
256, 258, 260, 265
strategy
256
257
256, 257
cleaned
257
260
257, 258, 259, 260
means
261
262
261, 262
pipeline
264
267
264, 267
result
267
268
267, 268
Worked example
Task: find the first positive observation without extra flags.
def first_positive(values):
for value in values:
if value > 0:
return value
return None
# [-2, 0, 4, 7] → 4; [-2, 0] → None
The loop's value has one local purpose.
A successful match returns immediately, eliminating found and result flags.
The final None makes the no-match case explicit.
If zero were a permitted match, the caller should test result is not None rather than truthiness.
Moving a repeated input into a function parameter can similarly remove a class attribute used only as temporary communication between methods.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Late binding and nonlocal: two scope traps
Closures capture variables, not values. The classic loop bug and its fix, then a counter that updates enclosing state on purpose.
callbacks = [lambda: i for i in range(3)]
print("late binding:", [f() for f in callbacks])
fixed = [lambda i=i: i for i in range(3)]
print("default arg :", [f() for f in fixed])
def make_counter():
count = 0
def increment():
nonlocal count
count += 1
return count
return increment
tick = make_counter()
print("counter:", tick(), tick(), tick())
Shorter lifetimes mean fewer things to remember. Measure the first assignment and last use of every local variable in main() of the TP02 solution.
import ast
from pathlib import Path
tree = ast.parse(Path("_study/tp/TP02/solution.py").read_text(encoding="utf-8"))
main = next(n for n in tree.body if isinstance(n, ast.FunctionDef) and n.name == "main")
stores, uses = {}, {}
for node in ast.walk(main):
if isinstance(node, ast.Name):
(stores if isinstance(node.ctx, ast.Store) else uses).setdefault(node.id, []).append(node.lineno)
rows = []
for name, where in stores.items():
marks = sorted(set(where + uses.get(name, [])))
rows.append({"name": name, "start": marks[0], "end": marks[-1], "marks": marks})
rows.sort(key=lambda r: r["start"])
print(f"main() runs from line {main.lineno} to {main.end_lineno}")
for r in rows:
print(f" {r['name']:<12} lines {r['start']:>3} to {r['end']:>3} (alive {r['end'] - r['start'] + 1} lines, touched {len(r['marks'])} times)")
main() runs from line 244 to 269
parser lines 245 to 247 (alive 3 lines, touched 3 times)
args lines 247 to 264 (alive 18 lines, touched 3 times)
reader lines 250 to 252 (alive 3 lines, touched 3 times)
original lines 251 to 261 (alive 11 lines, touched 4 times)
strategies lines 255 to 256 (alive 2 lines, touched 2 times)
name lines 256 to 265 (alive 10 lines, touched 4 times)
strategy lines 256 to 257 (alive 2 lines, touched 2 times)
cleaned lines 257 to 260 (alive 4 lines, touched 4 times)
means lines 261 to 262 (alive 2 lines, touched 2 times)
pipeline lines 264 to 267 (alive 4 lines, touched 2 times)
result lines 267 to 268 (alive 2 lines, touched 2 times)
Try it. Which variable is alive longest? Could its first assignment move closer to its first use?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L12
Extracting Unrelated Subproblems
Chapter 10 · Separate the analysis goal from reusable mechanics
2 extra examples1 chart4 sources
Learning objectives
Identify calculations independent of the main application goal.
Define a helper with a small explicit contract.
Use wrappers to simplify an awkward interface.
Avoid fragmenting a clear operation into unnecessary tiny functions.
Core explanation
An analysis often mixes scientific intent with routine mechanics. “Find the nearest observation” is a high-level goal; computing a distance between two vectors is a reusable subproblem. Separating the distance calculation allows the selection code to communicate its goal directly.
The textbook illustrates this with spherical distance between geographic locations. The important design move is that the distance helper does not need to know why its result is used. Similarly, CSV reading, text standardization and missing-value treatment can be developed independently of a particular report.
A helper needs clear inputs, outputs and errors. Prefer inputs that expose the actual data it needs over giving it a whole application object. Small isolated helpers are easier to inspect and check against known examples. They may be reusable, but usefulness to the current code is enough; do not build speculative general frameworks.
Wrapper functions can convert a library's awkward calling convention into a better fit for your application. That wrapper should keep important errors and effects visible. Moving code behind a nice name does not repair incorrect assumptions.
There is a cost to every extra abstraction: readers must jump to its definition. Extract a coherent responsibility, not every individual statement. The source's old cryptography illustration is about extracting interface glue; this study lesson does not reproduce it as implementation guidance.
Key formulas and contracts
d(x,y) = √(∑ⱼ₌₁ᵖ (xⱼ − yⱼ)²)
x and y are real vectors with the same p dimensions; xⱼ and yⱼ are coordinate j; d is Euclidean distance. This newly authored numerical example is not a formula for geographic distance.
i* = argminᵢ d(q,xᵢ)
q is the query vector, xᵢ a candidate observation, i a candidate index, and i* an index attaining the minimum. Tie handling must be specified.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Nearest point to the query
14 seeded random points in a 10 by 10 square
The segment joins the query to the point returned by nearest_point. The helper only needs distance(), not any knowledge of the analysis.Data table
series
x
y
label
candidate
4.7
6.6
candidate
6.7
1.4
candidate
0.1
3.7
candidate
2.7
8.1
candidate
6.9
6
candidate
5.6
6.6
candidate
1.5
4.4
candidate
1.6
9.1
candidate
0.6
8.2
candidate
0.7
6.9
candidate
3.4
4
candidate
8.4
0.2
candidate
0.6
9.2
candidate
5.1
0.9
query
5
5
nearest
4.7
6.6
Worked example
Task: find the nearest two-dimensional point.
from math import hypot
def distance(point_a, point_b):
return hypot(point_a[0] - point_b[0],
point_a[1] - point_b[1])
def nearest_point(query, candidates):
if not candidates:
return None
return min(candidates, key=lambda point: distance(query, point))
# query (0,0), candidates [(3,4), (1,1)] → (1,1)
The distances are 5 and √2 ≈ 1.414.
The minimum selects (1,1).
The helper has no knowledge of storage, reporting or the nearest-point application.
The caller explicitly handles an empty list. Python min retains the first encountered minimum, defining the tie rule here.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Wrap an awkward interface once
csv.DictReader gives strings for everything, including blanks and the text NaN. Hide the conversion in one small, tested helper.
import csv
def read_rows(path):
"""Yield one dict per CSV row: numbers become int or float, blank or 'NaN' becomes None."""
def convert(text):
if text in ("", "NaN"):
return None
for kind in (int, float):
try:
return kind(text)
except ValueError:
pass
return text
with open(path, newline="") as handle:
for row in csv.DictReader(handle):
yield {key: convert(value) for key, value in row.items()}
rows = list(read_rows("sample_data (2).csv"))
print(rows[1])
ages = [row["age"] for row in rows if row["age"] is not None]
print("observed ages:", ages, "| mean:", sum(ages) / len(ages))
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L13
One Task at a Time
Chapter 11 · Separate parsing, decisions and updates
2 extra examples1 chart4 sources
Learning objectives
List the distinct tasks interleaved in a block of code.
Separate extraction, defaults, decisions and side effects.
Derive a simple update from a numerical representation.
Recognize when stages matter even without new functions.
Core explanation
A function can have one overall purpose while still mixing several lower-level tasks. Updating a score might involve parsing a vote, removing its old contribution and adding its new contribution. If these are woven together in many branches, the reader must verify every combination.
List the tasks before deciding where to draw function boundaries. Extracting fields, choosing defaults, converting types and updating a result are often separate concerns. They can become helper functions or simply adjacent coherent blocks. The separation matters more than the number of functions.
A good representation can make the update obvious. Map an up vote to +1, a down vote to −1, and no vote to 0. Changing a vote then means subtracting the old value and adding the new one. This handles no change, reversal and removal with the same expression.
For data cleaning, loading, selecting a missing-value policy, applying transformations and saving are distinct tasks. TP02 makes those boundaries explicit. Within a transformation, distinguish finding the relevant columns from performing the numerical operation.
Separating tasks can require additional passes or temporary data. Preserve dependencies and consider costs for large datasets, but do not interleave everything merely to save a trivial pass. If a default such as 0 is valid data, avoid truthy-fallback logic that silently treats it as absent.
Key formulas and contracts
S_new = S_old − v_old + v_new
S_old and S_new are the total score before and after the change; v_old and v_new are the numerical contributions of the old and new vote, each in {−1,0,+1}.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Score after each vote change
Updated incrementally, checked against a full recount each time
Each step applies score - old + new. The example asserts that the result equals recomputing from all current votes.Data table
Changing “up” to “up” gives 10, while changing “down” to no vote gives 11.
Unexpected vote strings raise KeyError in this example, rather than silently becoming no vote.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Split a log report into parse, count and format
One loop that parses, decides, counts and prints is four jobs at once. Separate the stages and each becomes testable alone.
raw_lines = ["alice:login", "bob:login", "alice:view", "alice:logout", "bad line", "bob:view", "bob:view"]
def parse(line):
user, separator, action = line.partition(":")
return (user, action) if separator else None
def valid_events(lines):
return [event for event in map(parse, lines) if event is not None]
def count_actions(events):
counts = {}
for user, action in events:
per_user = counts.setdefault(user, {})
per_user[action] = per_user.get(action, 0) + 1
return counts
def report(counts):
return [f"{user}: " + ", ".join(f"{a}={n}" for a, n in sorted(actions.items()))
for user, actions in sorted(counts.items())]
events = valid_events(raw_lines)
print("skipped", len(raw_lines) - len(events), "malformed line(s)")
for line in report(count_actions(events)):
print(line)
Update a score incrementally and prove it matches a full recount
Derive the update from the representation: remove the old vote's value, add the new one. Check it against recomputing from scratch at every step.
VOTE_VALUE = {"up": 1, "down": -1, "": 0}
def change_score(score, old_vote, new_vote):
return score - VOTE_VALUE[old_vote] + VOTE_VALUE[new_vote]
def recompute(votes):
return sum(VOTE_VALUE[v] for v in votes.values())
changes = [("amy", "up"), ("ben", "up"), ("cai", "down"), ("amy", "down"),
("ben", ""), ("dee", "up"), ("cai", "up"), ("amy", "")]
votes, score, history = {}, 0, [0]
for user, new_vote in changes:
old_vote = votes.get(user, "")
score = change_score(score, old_vote, new_vote)
votes[user] = new_vote
assert score == recompute(votes)
history.append(score)
print(f"{user:>4}: {old_vote or '-':>4} -> {new_vote or '-':<4} score = {score}")
print("incremental and recomputed scores matched at every step")
amy: - -> up score = 1
ben: - -> up score = 2
cai: - -> down score = 1
amy: up -> down score = -1
ben: up -> - score = -2
dee: - -> up score = -1
cai: down -> up score = 1
amy: down -> - score = 2
incremental and recomputed scores matched at every step
Try it. Add a 'super-up' vote worth 2. Which single line of the update has to change?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L14
Turning Thoughts into Code
Chapter 12 · Explain the rule before translating it
2 extra examples1 chart4 sources
Learning objectives
Describe a computation in ordinary language.
Translate its key concepts into predicates and helpers.
Specify input assumptions before simplifying an algorithm.
Use an invariant to explain progress and termination.
Core explanation
Explaining an algorithm to another person forces you to identify its actual rule. If you cannot state the rule clearly, the code often contains an unresolved assumption. Write the description first, then make the code follow its meaningful nouns and verbs.
For authorization, “an administrator or the owner of an existing document may proceed” is clearer than a tree of negative checks. Define the predicate from that statement. Short-circuiting lets you avoid reading an owner from a missing document.
The source also joins three time-ordered streams. Describe the process as “align the current timestamps, output a match, then advance.” That phrase suggests an alignment helper. Inside it, find the latest current timestamp and advance any stream that is behind. Sorted order ensures a behind row cannot match the current leading time until it advances.
The algorithm needs assumptions: timestamps are ordered, the streams signal exhaustion, and the source treats timestamps as unique keys. Repeated timestamps would require a different join contract to avoid losing combinations. Each iteration either finds alignment or advances at least one stream, so it makes progress.
Knowing library vocabulary helps the translation. A join operation may already express the problem directly. “Rubber duck” explanation is useful during debugging too: speaking the inputs, intended result and actual behavior can expose the mismatch before you rewrite anything.
A is authorization; R is administrator status; E means the document exists; O means the user owns that document. Evaluate O only when E is true.
t_target = max(t₁,t₂,t₃)
t₁,t₂,t₃ are current timestamps of three non-exhausted sorted streams; t_target is the latest current time. Any stream with a smaller time must advance before a match is possible.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Steps to find the last item
Binary search against a linear scan
On a log-log chart the linear scan is a straight diagonal. Binary search takes about log2(n) steps, so a million items need only 20.Data table
series
x
y
note
linear scan
10
10
linear scan
100
100
linear scan
1000
1000
linear scan
10000
10000
linear scan
100000
100000
linear scan
1000000
1000000
binary search
10
3
binary search
100
7
binary search
1000
10
binary search
10000
13
binary search
100000
17
binary search
1000000
20
Worked example
Task: turn the authorization sentence into code.
def can_view(is_admin, user_id, document):
owns_document = (
document is not None and document["owner_id"] == user_id
)
return is_admin or owns_document
# can_view(False, 7, {"owner_id": 7}) → True
# can_view(False, 7, None) → False
# can_view(True, 7, None) → True
Name the two permitted situations.
Guard document access with an existence check.
Combine the permitted cases with or.
Test the missing-document case for both administrator and ordinary user.
This example assumes existing documents contain owner_id. If only administrators should access a missing resource, the expression agrees; a separate “not found” response belongs to the surrounding application contract.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Binary search, explained by its invariant
State what is always true before the loop condition is tested, and progress and termination follow. The asserts check the invariant on small inputs; the table counts steps on large ones.
def binary_search(sorted_values, target, check=False):
low, high = 0, len(sorted_values) # invariant: every value before low is < target,
steps = 0 # every value from high on is >= target
while low < high:
steps += 1
middle = (low + high) // 2
if sorted_values[middle] < target:
low = middle + 1
else:
high = middle
if check:
assert all(v < target for v in sorted_values[:low])
assert all(v >= target for v in sorted_values[high:])
found = low < len(sorted_values) and sorted_values[low] == target
return (low if found else None), steps
small = list(range(0, 40, 2))
assert all(binary_search(small, t, check=True)[0] == (t // 2 if t % 2 == 0 and 0 <= t < 40 else None) for t in range(-3, 45))
print("invariant held on every probe of a 20-element list\n")
table = []
print(f"{'n':>9} {'binary steps':>13} {'linear scan':>12}")
for n in [10, 100, 1000, 10_000, 100_000, 1_000_000]:
values = range(n)
position, steps = binary_search(values, n - 1)
assert position == n - 1
table.append({"n": n, "binary": steps, "linear": n})
print(f"{n:>9} {steps:>13} {n:>12}")
invariant held on every probe of a 20-element list
n binary steps linear scan
10 3 10
100 7 100
1000 10 1000
10000 13 10000
100000 17 100000
1000000 20 1000000
Try it. Change < to <= in the comparison. Which invariant breaks first, and what does the code return for duplicates?
Write the requirement as a sentence, name each clause, and the code is a transcription of the sentence.
# Rule: a student passes if the mean of their best three scores is at least 50
# and none of their scores is below 20. A student with no scores does not pass.
def best_three_mean(scores):
return sum(sorted(scores, reverse=True)[:3]) / min(3, len(scores))
def passes(scores, pass_mark=50, floor=20):
has_scores = bool(scores)
no_disaster = all(score >= floor for score in scores)
strong_enough = has_scores and best_three_mean(scores) >= pass_mark
return has_scores and no_disaster and strong_enough
for scores in [[], [60], [10, 90, 90, 90], [49, 50, 51], [20, 20, 90], [19, 99, 99]]:
print(f"{scores!s:18} passes: {passes(scores)}")
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L15
Writing Less Code
Chapter 13 · Solve the required problem with existing tools
2 extra examples2 charts5 sources
Learning objectives
Distinguish necessary behavior from speculative features.
Use built-in data structures to remove routine implementation.
Preserve ordering and type contracts when replacing code.
Compare a simple sufficient design with a more general one.
Core explanation
Every line you own creates future work: checking it, documenting it and changing it when requirements move. Before implementing a feature, establish whether the required behavior needs it. Simplifying the actual requirement can remove more complexity than clever coding.
The source's cache example notices that repeated accesses arrive in consecutive runs. A one-item cache can help that workload without a complete least-recently-used cache. This depends on the access pattern; it is not a universal replacement. Likewise, an approximate regional locator may be enough for a narrowly bounded task, but changing a scientific accuracy requirement requires explicit agreement.
Use existing containers and library operations where their contracts match. A set removes duplicates but does not provide an ordered sequence contract. If first-occurrence order matters for hashable Python values, a dictionary-based operation can communicate that requirement. The supplied textbook's set example should not be read as promising the same order as the input list.
Dead or unused code can complicate a codebase, but the course workflow has a stricter preservation rule: never remove original course files. Apply pruning principles to programs when authorized, and keep study outputs separate from sources.
Concise existing tools still need input assumptions. A short log-processing pipeline works only for the expected log format. A library call can reduce owned implementation while adding dependency constraints, so note those dependencies and their versions in runnable work.
Key formulas and contracts
unique([2,1,2]) = [2,1] under first-occurrence order
U is an order-preserving unique operation on a sequence. Equality here includes order; a set alone expresses membership rather than this sequence contract.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Source lines for three ways to deduplicate
Counted with inspect.getsource
Fewer lines is not automatically clearer, but here the dictionary version states the intent in one line. The set version is shortest yet gives up order.Data table
lines
unique_manual
8
unique_dict
2
unique_set
2
Hand-written versus standard library
Source lines for the same task
Each pair has the same observable behavior apart from the documented tie-breaking differences.Data table
hand-written
standard library
deduplicate
8
2
count items
5
2
top k
3
2
Worked example
Task: remove duplicates while retaining first occurrences.
Dictionary keys retain the first insertion position for these values.
The second 2 does not create another key.
Convert the keys to a list to match the requested result type.
This method assumes hashable values. A list of dictionaries needs a key-selection or equality policy. For DataFrames, TP02 uses drop_duplicates with an explicit all-column comparison.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
Three ways to remove duplicates
Specify the contract first: unique values, original order kept. Then see which tools satisfy it and how many lines each costs.
import inspect
def unique_manual(values):
seen = set()
result = []
for value in values:
if value not in seen:
seen.add(value)
result.append(value)
return result
def unique_dict(values):
return list(dict.fromkeys(values))
def unique_set(values):
return list(set(values))
data = [3, 1, 3, 2, 1]
loc = {}
for func in (unique_manual, unique_dict, unique_set):
loc[func.__name__] = len(inspect.getsource(func).strip().splitlines())
print(f"{func.__name__:14s} {loc[func.__name__]} lines -> {func(data)}")
print("keeps first-seen order:", [f(data) == [3, 1, 2] for f in (unique_manual, unique_dict, unique_set)])
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L16
Testing and Readability
Chapter 14 · Make the input and expected behavior easy to see
2 extra examples1 chart5 sources
Learning objectives
Write concise behavior-focused test cases.
Choose small inputs that expose boundaries.
Give failures enough information to diagnose the mismatch.
Use isolated components and deterministic inputs to simplify tests.
Core explanation
Tests communicate a contract through examples. Their readers should quickly see the situation, action and expected behavior. Large fixture setup can bury the essential difference between cases and discourage future maintenance.
Hide irrelevant object construction behind a helper while retaining the meaningful inputs in each test. Use the simplest numbers that distinguish correct from incorrect behavior. If filtering is nonnegative rather than positive, include zero; testing only negative and positive values misses the boundary.
Separate cases by behavior: sorting, removal, duplicates and empty input. This makes a failed case easier to diagnose. A clear test name can state the function, situation and expected result. An equality assertion should show actual and expected values, and a custom message can include the triggering input.
Determinism matters. Inject a known clock into time-sensitive code, use fixed examples and avoid network dependence for a small unit check. Globals and hidden mutable state create setup work and can make tests interfere. An easy-to-test design often also has clearer responsibilities.
Do not make the production code incomprehensible merely to enable a test. Choose checks appropriate to the likely failure modes. Passing a finite test set provides evidence, not a mathematical proof that every possible behavior is correct. TP02's checks include realistic CSV outcomes and edge cases that the seven-row sample does not exercise.
Key formulas and contracts
F(x) = sort_desc([xᵢ : xᵢ ≥ 0])
x is the input sequence, xᵢ is an element, F is a function that retains nonnegative values and sorts them in descending order; sort_desc denotes descending sorting. Duplicates are retained.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Which test cases catch which planted bugs?
Three mutants of clamp(), six test cases
An empty row is a test case that caught none of these bugs. Edge cases do the work here, the interior case does none.Data table
<function upper off by one at 0xfaa1ad0739a0>
<function lower off by one at 0xfaa1ad073a30>
<function wrong side at 0xfaa1ad073ac0>
inside (5 in 0..10)
0
0
0
lower edge (0 in 0..10)
0
1
0
upper edge (10 in 0..10)
1
0
0
just below (-1)
0
1
0
just above (11)
1
0
1
zero width (3 in 3..3)
0
0
0
Worked example
Task: specify sorting and filtering separately.
import unittest
def sort_nonnegative(values):
return sorted((value for value in values if value >= 0),
reverse=True)
class ScoreTests(unittest.TestCase):
def test_keeps_zero_removes_negative(self):
self.assertEqual(sort_nonnegative([0, -1]), [0])
def test_sorts_and_keeps_duplicates(self):
self.assertEqual(sort_nonnegative([1, 3, 1]), [3, 1, 1])
def test_empty_input(self):
self.assertEqual(sort_nonnegative([]), [])
The zero case distinguishes ≥ from >.
The duplicate case distinguishes a sorted list from a set.
The empty case documents a valid no-data result.
Each expected list is explicit and not computed with the same implementation as the function being tested.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
A boundary table as one readable test
Put inputs and expected outputs side by side. subTest reports every failing row instead of stopping at the first.
import io
import unittest
def clamp(value, low, high):
if low > high:
raise ValueError("low must not exceed high")
return max(low, min(value, high))
class ClampTests(unittest.TestCase):
def test_boundaries_and_interior(self):
cases = [
(5, 0, 10, 5), # inside
(0, 0, 10, 0), # lower edge
(10, 0, 10, 10), # upper edge
(-1, 0, 10, 0), # just below
(11, 0, 10, 10), # just above
(3, 3, 3, 3), # zero-width range
]
for value, low, high, expected in cases:
with self.subTest(value=value, low=low, high=high):
self.assertEqual(clamp(value, low, high), expected)
def test_reversed_bounds_are_rejected(self):
with self.assertRaises(ValueError):
clamp(1, 5, 0)
stream = io.StringIO()
suite = unittest.defaultTestLoader.loadTestsFromTestCase(ClampTests)
result = unittest.TextTestRunner(stream=stream, verbosity=0).run(suite)
print("tests run:", result.testsRun, "| failures:", len(result.failures), "| errors:", len(result.errors))
print("passed" if result.wasSuccessful() else "FAILED")
tests run: 2 | failures: 0 | errors: 0
passed
Try it. Make clamp wrong at the upper edge only. Which row in the table reports it?
Which tests catch which bugs? A small mutation check
Plant three plausible bugs and see which test cases notice. A case that kills no mutant adds little.
def clamp_correct(value, low, high):
return max(low, min(value, high))
def mutant_upper_off_by_one(value, low, high):
return max(low, min(value, high - 1))
def mutant_lower_off_by_one(value, low, high):
return min(high, max(value, low + 1))
def mutant_wrong_side(value, low, high):
return value if low <= value <= high else low
tests = {
"inside (5 in 0..10)": (5, 0, 10),
"lower edge (0 in 0..10)": (0, 0, 10),
"upper edge (10 in 0..10)": (10, 0, 10),
"just below (-1)": (-1, 0, 10),
"just above (11)": (11, 0, 10),
"zero width (3 in 3..3)": (3, 3, 3),
}
mutants = [mutant_upper_off_by_one, mutant_lower_off_by_one, mutant_wrong_side]
matrix = []
print(f"{'test case':26s}" + "".join(f"{m.__name__[7:]:>22s}" for m in mutants))
for name, args in tests.items():
row = [int(m(*args) != clamp_correct(*args)) for m in mutants]
matrix.append(row)
print(f"{name:26s}" + "".join(f"{('caught' if r else '-'):>22s}" for r in row))
caught = [any(row[i] for row in matrix) for i in range(len(mutants))]
print("every mutant caught by at least one case:", all(caught))
test case upper_off_by_one lower_off_by_one wrong_side
inside (5 in 0..10) - - -
lower edge (0 in 0..10) - caught -
upper edge (10 in 0..10) caught - -
just below (-1) - caught -
just above (11) caught - caught
zero width (3 in 3..3) - - -
every mutant caught by at least one case: True
Try it. Which test case caught nothing? Would you delete it, or does it protect something the mutants do not model?
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Study lesson / L17
Designing a Minute and Hour Counter
Chapter 15 · Exact windows, queues and time buckets
2 extra examples2 charts1 interactive4 sources
Learning objectives
Specify trailing-window semantics and a nonnegative update contract.
Compare naive, event-queue and time-bucket designs.
Maintain a running-sum invariant when old data expires.
Explain time resolution separately from count accuracy.
Identify initialization and clock assumptions in the source listing.
Core explanation
The counter accumulates nonnegative event amounts over the last 60 and 3600 seconds. “This minute” is ambiguous: it could mean the current clock minute or a trailing interval. Define the interval precisely before choosing a data structure. This lesson uses events after the cutoff and up to the current time.
A naive design retains timestamped events and scans relevant entries at each query. It is straightforward, but retained memory can grow without bound and repeated scans are expensive. Extracting a shared count-since helper improves readability without fixing those design costs.
An event-queue design keeps running minute and hour totals. New events enter the minute queue and both totals. At age 60 seconds, an event leaves the minute queue, reducing the minute total, and moves to a queue holding older hour events. At age 3600 seconds it leaves that queue and reduces the hour total. Call expiry logic on both updates and queries; otherwise a quiet interval leaves stale totals. An individual query may expire many events, so constant work is an amortized claim, rather than a guarantee for every call.
Time buckets summarize events in a fixed number of intervals. A minute counter can use 60 one-second buckets and an hour counter 60 sixty-second buckets. The three responsibilities are a bounded queue with a sum, a clockless bucket controller, and the public minute/hour wrapper. This is a direct application of separating tasks and extracting subproblems.
A bucket boundary creates uncertainty about the precise trailing cutoff. Its time width is about one sixtieth of the window in this design. That does not guarantee a relative event-count error of at most 1/60: arbitrarily many events might lie in a partial boundary bucket. Exact timestamps or stronger assumptions are needed for a count-error guarantee.
The supplied C++ listing at printed page 177 does not visibly initialize last_update_time. A runnable implementation must initialize the first bucket explicitly. It must also define behavior for backward clock readings and out-of-order events. The code below uses a nondecreasing externally supplied time for easy checking. The book's illustrative performance table also appears to overcount 100 events/s for an hour as 3.6 million; the arithmetic is 360,000. Its byte-memory figures depend on implementation and should be treated as illustrative.
Key formulas and contracts
C_W(t) = ∑ᵢ cᵢ · 1{t−W < tᵢ ≤ t}
t is query time; W is window length, 60 or 3600 seconds; tᵢ is event time; cᵢ ≥ 0 is its amount; C_W is the trailing total; 1 is an indicator. The lower cutoff is excluded.
b(t) = floor(t/Δ); W = BΔ
b(t) is the integer bucket index at time t; Δ > 0 is bucket width in seconds; B is the number of retained buckets and W the nominal window length. A boundary time uncertainty of Δ is not a uniform relative error bound on event counts.
total = ∑ₖ qₖ
T is the cached running total; qₖ is retained bucket or queue value k. Every insertion or removal must preserve this invariant.
Visual lab
Every chart is drawn from numbers that the course code or the examples below actually produce. Hover a mark for its value; open the data table under each chart for the same numbers in text.
Worst and average error against bucket count
Stream with two bursts, window of 60 seconds
More buckets shrink the error, at the cost of more counters. The error depends on the shape of the stream, not on a fixed fraction of the count.Data table
series
x
y
note
max error
1
153
max error
2
138
max error
3
131
max error
5
128
max error
6
130
max error
10
75
max error
12
124
max error
20
73
max error
30
64
max error
60
36
mean error
1
57.37
mean error
2
21.4
mean error
3
8.68
mean error
5
9.76
mean error
6
5.87
mean error
10
5.51
mean error
12
4.28
mean error
20
3.17
mean error
30
2.15
mean error
60
1.5
Exact count and the 6-bucket estimate over time
Two bursts near t = 100 and t = 220
The estimate never exceeds the exact count; it lags most just after a burst when old events are still inside the exact window.Data table
series
x
y
note
exact
60
38
exact
66
36
exact
72
32
exact
78
30
exact
84
30
exact
90
34
exact
96
36
exact
102
86
exact
108
156
exact
114
155
exact
120
153
exact
126
151
exact
132
153
exact
138
155
exact
144
158
exact
150
158
exact
156
154
exact
162
105
exact
168
33
exact
174
30
exact
180
33
exact
186
34
exact
192
36
exact
198
37
exact
204
37
exact
210
37
exact
216
39
exact
222
104
exact
228
133
exact
234
136
exact
240
142
exact
246
141
exact
252
140
exact
258
140
exact
264
139
exact
270
134
exact
276
135
exact
282
70
exact
288
48
exact
294
48
exact
300
44
6 buckets
60
28
6 buckets
66
32
6 buckets
72
27
6 buckets
78
29
6 buckets
84
28
6 buckets
90
29
6 buckets
96
34
6 buckets
102
77
6 buckets
108
152
6 buckets
114
153
6 buckets
120
149
6 buckets
126
151
6 buckets
132
151
6 buckets
138
155
6 buckets
144
152
6 buckets
150
153
6 buckets
156
154
6 buckets
162
26
6 buckets
168
29
6 buckets
174
30
6 buckets
180
29
6 buckets
186
32
6 buckets
192
30
6 buckets
198
35
6 buckets
204
31
6 buckets
210
34
6 buckets
216
37
6 buckets
222
101
6 buckets
228
133
6 buckets
234
133
6 buckets
240
135
6 buckets
246
137
6 buckets
252
133
6 buckets
258
138
6 buckets
264
133
6 buckets
270
127
6 buckets
276
131
6 buckets
282
37
6 buckets
288
47
6 buckets
294
39
6 buckets
300
40
Try it: how many buckets does the estimate need?
Change the bucket count and the current time. The stream is the same one used in the second example.
This interactive needs JavaScript. The text and charts around it cover the same ideas.
Worked example
Task: trace two exact trailing windows.
Add amount 5 at t = 0 and amount 3 at t = 30.
At t = 59, both are within the minute, so minute = 8 and hour = 8.
At t = 60, the event at 0 lies exactly on the excluded minute cutoff: minute = 3 and hour = 8.
At t = 3600, the event at 0 also expires from the hour: minute = 0 and hour = 3.
At t = 3630, the event at 30 expires too: both totals are 0.
def trailing_total(events, now, window):
return sum(amount for time, amount in events
if now - window < time <= now)
# trailing_total([(0, 5), (30, 3)], 60, 60) == 3
Design
Query/maintenance work
Retained memory
Naive event list
O(n) scan of relevant events
Unbounded without expiry
Event queues and totals
O(k) for k events expired on a call; amortized O(1) per event
O(n) recent events
B fixed buckets
Up to O(B) to advance/clear; bounded for fixed B
O(B)
n is the number of relevant events, k the number removed in that call and B the configured bucket count.
Source footnote extension: the book suggests weighting the oldest partial bucket. If f is the elapsed fraction of the current bucket, retain the current bucket, the B−1 intervening full buckets and weight the older boundary bucket by 1−f. The estimate is current + middle_sum + (1 − f) × oldest. This is unbiased only under an appropriate uniform-within-bucket expectation; a burst near the boundary can still produce a large error. It needs B+1 buckets and is not an exact counter.
The supplement implements exact windows and solves the partial-bucket estimation exercise. Its ring storage uses B+1 buckets and O(B) query work. It explicitly validates first use and nondecreasing time.
Exact trailing windows
t=59: minute=8, hour=8
t=60: minute=3, hour=8
t=3600: minute=0, hour=3
t=3630: minute=0, hour=0
Partial-bucket exercise at t=25, W=20
Exact: 50
Unweighted: 50
Weighted: 55.0
At a long idle gap: weighted count = 0
The weighted estimate assumes uniform within-bucket arrivals.
This burst example shows it can overestimate the exact count.
More worked examples
Each example runs on its own with the course folder as the working directory. The output tab is what the code actually printed when this page was built.
1
An exact trailing counter on a deque, with the work counted
Events expire from the front, so each event is added once and removed once. Count the removals to see the amortised cost.
from collections import deque
class TrailingCounter:
"""Exact total of amounts in the trailing window (now - window, now]."""
def __init__(self, window):
self.window = window
self.events = deque() # (time, amount), nondecreasing time
self.total = 0
self.removals = 0
def add(self, time, amount=1):
if self.events and time < self.events[-1][0]:
raise ValueError("time must not go backwards")
self.events.append((time, amount))
self.total += amount
self._expire(time)
def count(self, now):
self._expire(now)
return self.total
def _expire(self, now):
while self.events and self.events[0][0] <= now - self.window:
_, amount = self.events.popleft()
self.total -= amount
self.removals += 1
minute = TrailingCounter(60)
minute.add(0, 5)
minute.add(30, 3)
print("t=59 ->", minute.count(59), "| t=60 ->", minute.count(60), "| t=90 ->", minute.count(90))
busy = TrailingCounter(60)
longest = 0
for step in range(1000):
busy.add(step * 0.5)
longest = max(longest, len(busy.events))
print(f"1000 events added: longest queue = {longest}, removals so far = {busy.removals}")
print("every event is removed at most once, so total work stays proportional to events added")
t=59 -> 8 | t=60 -> 3 | t=90 -> 0
1000 events added: longest queue = 120, removals so far = 880
every event is removed at most once, so total work stays proportional to events added
Constant work per event is an amortised statement. A single query can still remove many events at once.
Try it. Why does the t=60 query drop the event at t=0? Rewrite the cutoff comparison to include it and see what changes.
Buckets trade memory for accuracy: how far off is the estimate?
Keep B counters instead of every event. Compare the bucket estimate with the exact count on a stream that contains two bursts.
import random
WINDOW = 60
rng = random.Random(5)
events = sorted([rng.uniform(0, 300) for _ in range(200)]
+ [100 + rng.uniform(0, 5) for _ in range(120)]
+ [220 + rng.uniform(0, 3) for _ in range(90)])
events = [round(e, 2) for e in events]
def exact(t):
return sum(1 for e in events if t - WINDOW < e <= t)
def estimate(t, buckets):
width = WINDOW / buckets
current = int(t // width)
return sum(1 for e in events if e <= t and int(e // width) > current - buckets)
times = list(range(WINDOW, 301))
never_over = all(estimate(t, b) <= exact(t) for t in times for b in (1, 2, 3, 5, 6, 10, 12, 20, 30, 60))
print("events:", len(events), "| estimate never exceeds the exact count:", never_over)
errors = []
print(f"{'buckets':>8} {'width (s)':>10} {'max error':>10} {'mean error':>11}")
for b in (1, 2, 3, 5, 6, 10, 12, 20, 30, 60):
diffs = [exact(t) - estimate(t, b) for t in times]
errors.append({"buckets": b, "max": max(diffs), "mean": round(sum(diffs) / len(diffs), 2)})
print(f"{b:>8} {WINDOW / b:>10.1f} {max(diffs):>10} {errors[-1]['mean']:>11.2f}")
b = 6
series_exact = [[t, exact(t)] for t in times]
series_estimate = [[t, estimate(t, b)] for t in times]
This estimate keeps the B most recent buckets, including the current partial one. It can only undercount, and its error depends on how bursty the stream is, not on a fixed fraction of the total.
Try it. Both bursts are shorter than a second bucket width at B=6. Which B first makes the worst error small, and what does that cost in counters?
Expired data must be shifted on reads as well as writes.
Initialize the first timestamp and reject or handle backward time.
One bucket of temporal uncertainty does not imply a 1/60 bound on count error.
100 events/s × 3600 s is 360,000 events; do not reuse inconsistent source arithmetic.
Summary
Choose an explicit time contract, preserve the running-sum invariant and separate exact event retention from approximate bucket summaries.
Five self-check questions
1. At t = 60, does an event at 0 remain in (t−60,t]?
No. It lies on the excluded lower boundary.
2. Why expire on a query?
Counts must decrease during periods with no new events.
3. What is the cached-sum invariant?
The cached total equals the sum of all retained queue or bucket values.
4. Why is a bucket not a uniform relative count-error guarantee?
A partial bucket can contain an arbitrarily large burst of events.
5. What initialization is missing from the supplied listing?
A visible initial value or first-use policy for last_update_time.
Go deeper
Related reading beyond the course materials. Links open in a new tab. A tick means the page title was fetched when this edition was built; the others are stable reference addresses that were not re-fetched.
Explanations and Python examples are newly written study material unless explicitly described as source examples. The source book uses several programming languages.
Practical / TP03 · new 2 Nov 2026
Unsloth, LoRA and Fine-tuning
Adapt a pretrained language model by training small adapter matrices, and know when retrieval (RAG) is the better tool
2 runnable examples2 chartsLecturer: Mr. VANN Visal
Learning objectives
Explain what Unsloth adds on top of Hugging Face training, and why its speed and memory figures are benchmark-dependent.
Check a GPU environment before installing, and load a 4-bit model with FastLanguageModel.
Use the Llama-3 chat template correctly for training (add_generation_prompt=False) and inference (True).
Read every argument of get_peft_model and compute how many parameters LoRA actually trains.
Choose between fine-tuning and RAG, or combine them, for a given problem.
Core explanation
Unsloth is an open-source Python framework for training and fine-tuning language models faster and with less GPU memory. It works with the Hugging Face ecosystem and supports families such as Llama, Mistral, Gemma and Qwen. The “70% less memory” figure from class is a benchmark result for particular models and settings, not a guarantee: real usage depends on model size, precision, context length, batch size and hardware. Faster training also does not make the model more accurate; the data and training settings decide that.
Fine-tuning continues training a pretrained model on examples for one task or domain. Updating all 8 billion weights of a model like Llama 3 8B needs a lot of memory. LoRA (Low-Rank Adaptation) avoids that: it freezes the original weights and trains two small matrices per chosen layer, called adapters. QLoRA goes one step further and stores the frozen base model in 4-bit form, which is what load_in_4bit=True does.
The TP has three code steps: load a 4-bit model, format the conversations with the model's chat template, and attach LoRA adapters. These steps only prepare the model. Weights change only when a trainer runs over a dataset, which the TP lists as the next step.
Before installing: check the environment
Create a separate virtual environment for the project.
Check that the Python version is supported by Unsloth.
Check that PyTorch can see the GPU (code below).
Pick a model and check its licence and access conditions (Llama models need approval on Hugging Face).
Prepare training examples in one consistent format.
nvidia-smi # driver and GPU visible?
python -m pip install --upgrade pip
python -m pip install unsloth # in a notebook: %pip install unsloth
If CUDA available prints False, fix the environment first. There is no single VRAM requirement: smaller models, shorter context and 4-bit loading all reduce memory.
Load the model and apply the chat template
Worked solution
Load the model and format a conversation
What you need. A chat model was trained on text with special role tokens; the tokenizer’s chat template inserts them.
1
Load with FastLanguageModel.from_pretrained(…, max_seq_length=2048, load_in_4bit=True).
Why: 4-bit base weights (QLoRA) cut memory roughly 4× vs 16-bit.
2
Attach the template: get_chat_template(tokenizer, chat_template="llama-3").
Why: the template must match the model family, or the model sees an unfamiliar format.
3
Training text: full conversation incl. the correct answer, add_generation_prompt=False.
Why: the model must see the target answer to learn it.
4
Inference text: conversation up to the question, add_generation_prompt=True.
Why: it opens the assistant turn so the model writes the answer.
Answer. Let apply_chat_template build the string; never type <|eot_id|> and friends by hand.
✔ What it provesCorrect formatting is part of the data: same template for training and inference.
✖ What it does not sayA correct template does not guarantee good answers; and 4-bit loading may slightly change outputs vs full precision.
→ Where it leadsDataset preparation (instruction/response pairs) and evaluation on held-out examples.
from unsloth import FastLanguageModel
from unsloth.chat_templates import get_chat_template
model, tokenizer = FastLanguageModel.from_pretrained(
model_name="unsloth/llama-3-8b-Instruct-bnb-4bit",
max_seq_length=2048, # longest sequence, in tokens
dtype=None, # pick the compute dtype automatically
load_in_4bit=True, # store base weights in 4 bits (QLoRA)
)
tokenizer = get_chat_template(tokenizer, chat_template="llama-3")
messages = [
{"role": "system", "content": "Explain programming clearly."},
{"role": "user", "content": "What is a Python list?"},
{"role": "assistant", "content": "A list stores an ordered collection of items."},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=False)
A chat template wraps each message in the special tokens the model was trained with, such as <|begin_of_text|>, <|start_header_id|>, <|end_header_id|> and <|eot_id|>. Let the tokenizer insert them; writing them by hand is the most common way to get a silently broken dataset.
Use
Messages you pass
add_generation_prompt
Training
Full conversation, including the correct assistant answer
False
Inference
Conversation up to the user's question
True (opens the assistant's turn)
Key formulas and contracts
W_eff = W_base + (alpha / r) · B A
W_base is the frozen d_out × d_in weight; only A and B are trained. r is the rank and α (lora_alpha) the scaling setting. B starts at zero, so training begins from exactly the base model.
trainable = r (d_in + d_out) vs d_in · d_out
Parameters per adapted matrix. For a 4096 × 4096 projection with r = 16: 131,072 instead of 16,777,216 (0.78%).
scale = alpha / r (standard), alpha / sqrt(r) (rsLoRA)
With r = 16 and α = 16 the standard scale is 1; with use_rslora=True it would be 4.
Rank of A and B. Higher rank = more capacity and more trainable parameters (linear in r).
lora_alpha=16
Scaling setting. α = r is a common starting point, not a proven optimum.
target_modules
The attention projections (q, k, v, o) and feed-forward projections (gate, up, down) that get adapters.
lora_dropout=0
No dropout on the adapter path.
bias="none"
Bias terms are not trained.
use_gradient_checkpointing="unsloth"
Saves activation memory by recomputing some values in the backward pass (a little slower per step, much less memory).
random_state=3407
Seed for reproducible adapter initialisation.
use_rslora=False
Standard α/r scaling instead of rank-stabilised α/√r.
loftq_config=None
No LoftQ initialisation.
Visual lab
Both charts are computed by example 1 below from Llama 3 8B's published shapes. Hover a mark for its value; open the data table for the same numbers.
Trainable parameters by rank
All seven target modules, 32 layers, millions
Doubling r doubles the adapter size. At r = 16, LoRA trains 41.9 M numbers, 0.52% of the 8.03 B in the base model.Data table
rank r
trainable (M)
% of base
4
10.49
0.131%
8
20.97
0.261%
16
41.94
0.522%
32
83.89
1.045%
64
167.77
2.089%
Where the adapters go, per layer (r = 16)
Trainable numbers added to each module
The three feed-forward projections take two thirds of the adapter budget because their 14,336-wide side is the largest. k and v are smaller because Llama 3 shares 8 key/value heads (1,024 wide).Data table
module
d_in × d_out
LoRA numbers
q_proj
4096 × 4096
131,072
k_proj
4096 × 1024
81,920
v_proj
4096 × 1024
81,920
o_proj
4096 × 4096
131,072
gate_proj
4096 × 14336
294,912
up_proj
4096 × 14336
294,912
down_proj
14336 × 4096
294,912
Worked example
Worked solution
Count the trainable parameters of the TP configuration
What you need. LoRA adds Weff = W + (α/r)·BA with B: d_out×r and A: r×d_in, so each adapted matrix trains r·(d_in + d_out) numbers. Llama 3 8B: hidden 4096, MLP 14336, k/v width 8×128 = 1024, 32 layers.
1
q and o (4096 → 4096): 16 × (4096 + 4096) = 131,072 each.
Why: A has r×d_in numbers and B has d_out×r numbers.
2
k and v (4096 → 1024, grouped-query attention): 16 × (4096 + 1024) = 81,920 each.
Why: only 8 key/value heads of 128, so the output width is 1024, not 4096.
One layer: 2·131,072 + 2·81,920 + 3·294,912 = 1,310,720.
5
32 layers: 1,310,720 × 32 = 41,943,040 ≈ 0.52 % of 8.03 B — the figure Unsloth prints.
Why: that is why an adapter file is tens of MB while the base model is GBs.
attention q, k, v, ofeed-forward gate, up, down
Figure. LoRA parameters per adapted matrix in one Llama 3 8B layer at r = 16: r·(d_in + d_out). One layer = 1,310,720; 32 layers = 41,943,040 (≈ 0.52 % of 8.03 B). A full 4096×4096 q matrix alone has 16,777,216 numbers.
Answer. 41,943,040 trainable parameters (≈ 42 M, 0.52 % of the model).
✔ What it provesLoRA cost grows linearly in r and in the matrix sizes; you can predict it before training.
✖ What it does not sayFewer trainable parameters do not mean less GPU memory for activations, nor equal quality to full fine-tuning; 4-bit loading (QLoRA) is a separate memory saving.
→ Where it leadsChoosing r and α, rsLoRA scaling α/√r, and merging adapters into the base weights.
Task: how many parameters does the TP's configuration train?
Each adapted d_out × d_in matrix adds r(d_in + d_out) numbers. Llama 3 8B has hidden size 4096, MLP size 14336 and 8 KV heads × 128 = 1024.
32 layers: 1,310,720 × 32 = 41,943,040, the “Trainable parameters” figure Unsloth prints for this setup, about 0.52% of 8.03 B.
The saved adapter holds only these 42 M numbers, which is why LoRA checkpoints are tens of megabytes, not gigabytes.
Fine-tuning workflow, and fine-tuning versus RAG
Worked solution
Fine-tuning or RAG?
What you need. Fine-tuning changes weights (adapters); RAG retrieves documents at question time and leaves the generator unchanged.
1
Ask what must change. Behaviour, style, output format → fine-tune. Facts from a changing document set → RAG.
Why: weights store how to answer; documents store what is currently true.
2
Ask how often it changes. New facts with fine-tuning need a new training run; with RAG you update the index.
3
Name the main risk. Fine-tuning: overfitting, no guarantee of recalling facts. RAG: fails when retrieval misses the passage.
Why: each method fails in a different place, so the checks differ.
4
Combine when needed: fine-tune the format, retrieve the facts.
Answer. Use fine-tuning for how the model answers and RAG for what facts it uses; they work well together.
✔ What it provesA decision rule based on what changes and how often.
✖ What it does not sayRetrieval ranks by similarity; it does not prove a passage is true or relevant, and fine-tuning does not make a model reliably memorise facts.
→ Where it leadsEvaluation: hallucination checks, retrieval recall@k, held-out task metrics.
Choose the pretrained model.
Prepare clean examples; keep a separate evaluation set.
Format conversations with the correct chat template.
Load the model and configure LoRA.
Train with a suitable trainer, learning rate, batch size and number of steps.
Evaluate on unseen examples and compare with the original model.
Save the adapters and the tokenizer.
RAG (Retrieval-Augmented Generation) retrieves relevant passages from an external document collection and gives them to the model as context. Retrieval ranks relevance (often by embedding similarity); it does not prove that a passage is true.
Aspect
Fine-tuning
RAG
Mechanism
Learns from training examples
Retrieves information for the current question
Weights
Updates model or adapter parameters
Generator weights usually unchanged
Good for
Behaviour, style, output format, specialised tasks
Answers grounded in a document collection
New information
Needs another training run
Update the documents and the index
Main risk
Overfitting; recall of facts not guaranteed
Fails if retrieval misses the useful passage
They combine well: fine-tune for how to answer, retrieve for what the facts are.
More worked examples
Both run with plain Python (no GPU, no downloads). The output tab is what the code printed on your computer on 2 Nov 2026.
1
Count LoRA's trainable parameters on Llama 3 8B
Per-module and per-rank counts from the model's shapes, plus the α/r and α/√r scales.
"""How many parameters does LoRA train on Llama 3 8B?
Shapes come from the model's published config (meta-llama/Meta-Llama-3-8B):
hidden size 4096, MLP size 14336, 32 layers, 8 key/value heads of size 128.
A LoRA adapter on a d_out x d_in weight adds B (d_out x r) and A (r x d_in),
so it trains r * (d_in + d_out) numbers. The base weight W stays frozen.
"""
HIDDEN, MLP, LAYERS, KV = 4096, 14336, 32, 8 * 128
# (d_in, d_out) for every module named in target_modules
SHAPES = {
"q_proj": (HIDDEN, HIDDEN), "k_proj": (HIDDEN, KV), "v_proj": (HIDDEN, KV),
"o_proj": (HIDDEN, HIDDEN), "gate_proj": (HIDDEN, MLP),
"up_proj": (HIDDEN, MLP), "down_proj": (MLP, HIDDEN),
}
TOTAL_BASE = 8_030_261_248 # published parameter count of Llama 3 8B
def lora_params(rank):
per_layer = sum(rank * (d_in + d_out) for d_in, d_out in SHAPES.values())
return per_layer * LAYERS
print("module frozen W LoRA (r=16) share")
for name, (d_in, d_out) in SHAPES.items():
full, lora = d_in * d_out, 16 * (d_in + d_out)
print(f"{name:9s} {full:>11,} {lora:>12,} {lora / full:6.2%}")
print("\nrank trainable % of 8.03B scale a/r (a=16) rsLoRA a/sqrt(r)")
for r in (4, 8, 16, 32, 64):
n = lora_params(r)
print(f"{r:>4} {n:>12,} {n / TOTAL_BASE:8.3%} {16 / r:>10.3g} {16 / r ** 0.5:>14.3g}")
Shapes from the published Llama 3 8B config; 8,030,261,248 is the published total. Files: TP03_ex1.py · output.
Try it. Change SHAPES to target only q_proj and v_proj (the original LoRA paper's choice). How many parameters remain at r = 16?
2
Merging an adapter does not change the output
A 3 × 4 weight: B = 0 reproduces the base model, and the side path equals the merged weight.
"""LoRA on a tiny 3 x 4 weight, in plain Python.
Checks three facts behind W_effective = W_base + (alpha / r) * B @ A:
1. B starts at zero, so before training the adapted model equals the base model.
2. Running the adapter as a side path gives the same output as merging it into W.
3. Only A and B would be trained: 2*(3) + 2*(4) = 14 numbers instead of 12 here,
so LoRA saves nothing on a matrix this small; it pays off when r << d.
"""
import random
random.seed(3407)
d_out, d_in, r, alpha = 3, 4, 2, 2
def matmul(X, Y):
return [[sum(X[i][k] * Y[k][j] for k in range(len(Y))) for j in range(len(Y[0]))] for i in range(len(X))]
def add(X, Y, s=1.0):
return [[X[i][j] + s * Y[i][j] for j in range(len(X[0]))] for i in range(len(X))]
def matvec(M, v):
return [round(sum(a * b for a, b in zip(row, v)), 6) for row in M]
W = [[round(random.uniform(-1, 1), 2) for _ in range(d_in)] for _ in range(d_out)]
A = [[round(random.gauss(0, 0.5), 2) for _ in range(d_in)] for _ in range(r)]
B = [[0.0] * r for _ in range(d_out)] # LoRA initialises B to zero
x = [1.0, 2.0, -1.0, 0.5]
scale = alpha / r
print("base output ", matvec(W, x))
print("adapted, B = 0 ", matvec(add(W, matmul(B, A), scale), x))
B = [[0.3, -0.1], [0.0, 0.2], [-0.4, 0.1]] # pretend training changed B
side_path = [w + scale * d for w, d in zip(matvec(W, x), matvec(matmul(B, A), x))]
merged = matvec(add(W, matmul(B, A), scale), x)
print("side path W x + s BAx", [round(v, 6) for v in side_path])
print("merged (W + s BA) x ", merged)
print("identical:", all(abs(a - b) < 1e-9 for a, b in zip(side_path, merged)))
print("trainable numbers: LoRA", r * (d_in + d_out), "vs full", d_in * d_out)
base output [-1.975, -0.81, -1.09]
adapted, B = 0 [-1.975, -0.81, -1.09]
side path W x + s BAx [-1.991, -1.111, -1.0185]
merged (W + s BA) x [-1.991, -1.111, -1.0185]
identical: True
trainable numbers: LoRA 14 vs full 12
On a tiny matrix LoRA trains more numbers than the matrix has; the saving needs r much smaller than d_in and d_out. Files: TP03_ex2.py · output.
Try it. Set r = 1. What is the smallest square size d for which 2d < d²?
Common mistakes and exam traps
Treating “70% less memory” or “2× faster” as guaranteed: they are benchmark results.
Thinking get_peft_model trains the model. It only attaches adapters; a trainer and data are still needed.
Writing Llama-3 special tokens by hand, or using the wrong template for the model family.
Using add_generation_prompt=True on training data (the answer then sits outside the assistant turn).
Saying fine-tuning “shrinks the model to one subject”. It adapts behaviour; the model size is unchanged.
Saying RAG “trains the model on your documents”. A normal RAG query changes no weights.
Changing r without changing α also changes α/r, so runs are not directly comparable.
Summary
Load a 4-bit base model, format data with the model's own chat template, and train small adapters W + (α/r)BA instead of all weights. At r = 16 on Llama 3 8B that is 41.9 M trainable numbers (0.52%). Fine-tune for behaviour, use RAG for up-to-date facts, and evaluate both.
Five self-check questions
1. Why does training start from exactly the base model?
B is initialised to zero, so BA = 0 and W_eff = W_base until the first update.
2. With r = 8 and α = 16, what is the standard LoRA scale? And with rsLoRA?
α/r = 2; α/√r = 16/2.83 ≈ 5.66.
3. How many trainable numbers does a 4096 × 4096 matrix get at r = 32?
32 × (4096 + 4096) = 262,144 (1.56% of 16,777,216).
4. You need answers that follow last week's policy changes. Fine-tune or RAG?
RAG: update the documents and the index, no retraining. Fine-tuning would need a new run for every change.
5. What does use_gradient_checkpointing="unsloth" trade?
Less activation memory in exchange for some recomputation (extra time) during backpropagation.
Parameter counts, charts, the worked example and the self-check are new study material computed from Llama 3 8B's published configuration; they are not in the TP file. This TP names the course “Advance Programming for Data Science (APDS)”.
Part 3 · Practice
Self-check bank
Chart studio
Every chart in one place
24 charts, each computed from the course files or from the extra examples
Choosing the right built-in: time to find the last item
Median microseconds per lookup, measured while this page was built
Measured with time.perf_counter on Python 3.10.12; exact values change from machine to machine, the shapes do not. A list scan grows with n, a set lookup stays flat, bisect on sorted data grows very slowly.Data table
DropMissing keeps only two complete rows, so its statistics rest on very little data. FillMean leaves the mean of the observed values unchanged but shrinks the spread.Data table
Filled square = missing value; rows are the seven students
Read down a column to see how much each strategy has to repair. Hover a cell for the student and column.Data table
age
height_cm
weight_kg
score
Alice
0
0
0
0
Bob
1
0
1
0
Charlie
0
0
0
1
David
1
1
0
0
Eva
0
0
0
0
Frank
0
0
0
1
Grace
1
0
0
0
Share of the missing cells
7 missing cells in total
age has the most gaps; DropMissing would discard any row touched by any of them.Data table
part
value
age
3
height_cm
1
weight_kg
1
score
2
Time recorded by the decorators
Mean and slowest call per method, from execution.log
Recorded by the @log_time decorator during the original run. Values vary by machine and by run; they illustrate the decorator, not performance.Data table
mean (ms)
slowest (ms)
CSVReader.read
1.168
1.935
DataCleaner.clean
0.606
0.902
After min-max scaling: age against score
pipeline_output.csv from the study solution
Both axes now run from 0 to 1, so the two columns can be compared directly. Eva has the smallest age and David the smallest score.Data table
All three return identical grades for every score from 0 to 100. The counts are proxies for what a reader must hold in mind, not a measure of understanding time.Data table
Number of integers included: [2, 4] has 3, [2, 4) has 2, (2, 4] has 2, (2, 4) has 1. The half-open form [2, 4) has length 4 - 2 = 2, matching range(2, 4).Data table
Fewer lines is not automatically clearer, but here the dictionary version states the intent in one line. The set version is shortest yet gives up order.Data table
lines
unique_manual
8
unique_dict
2
unique_set
2
Hand-written versus standard library
Source lines for the same task
Each pair has the same observable behavior apart from the documented tie-breaking differences.Data table
More buckets shrink the error, at the cost of more counters. The error depends on the shape of the stream, not on a fixed fraction of the count.Data table
series
x
y
note
max error
1
153
max error
2
138
max error
3
131
max error
5
128
max error
6
130
max error
10
75
max error
12
124
max error
20
73
max error
30
64
max error
60
36
mean error
1
57.37
mean error
2
21.4
mean error
3
8.68
mean error
5
9.76
mean error
6
5.87
mean error
10
5.51
mean error
12
4.28
mean error
20
3.17
mean error
30
2.15
mean error
60
1.5
Exact count and the 6-bucket estimate over time
Two bursts near t = 100 and t = 220
The estimate never exceeds the exact count; it lags most just after a burst when old events are still inside the exact window.Data table
series
x
y
note
exact
60
38
exact
66
36
exact
72
32
exact
78
30
exact
84
30
exact
90
34
exact
96
36
exact
102
86
exact
108
156
exact
114
155
exact
120
153
exact
126
151
exact
132
153
exact
138
155
exact
144
158
exact
150
158
exact
156
154
exact
162
105
exact
168
33
exact
174
30
exact
180
33
exact
186
34
exact
192
36
exact
198
37
exact
204
37
exact
210
37
exact
216
39
exact
222
104
exact
228
133
exact
234
136
exact
240
142
exact
246
141
exact
252
140
exact
258
140
exact
264
139
exact
270
134
exact
276
135
exact
282
70
exact
288
48
exact
294
48
exact
300
44
6 buckets
60
28
6 buckets
66
32
6 buckets
72
27
6 buckets
78
29
6 buckets
84
28
6 buckets
90
29
6 buckets
96
34
6 buckets
102
77
6 buckets
108
152
6 buckets
114
153
6 buckets
120
149
6 buckets
126
151
6 buckets
132
151
6 buckets
138
155
6 buckets
144
152
6 buckets
150
153
6 buckets
156
154
6 buckets
162
26
6 buckets
168
29
6 buckets
174
30
6 buckets
180
29
6 buckets
186
32
6 buckets
192
30
6 buckets
198
35
6 buckets
204
31
6 buckets
210
34
6 buckets
216
37
6 buckets
222
101
6 buckets
228
133
6 buckets
234
133
6 buckets
240
135
6 buckets
246
137
6 buckets
252
133
6 buckets
258
138
6 buckets
264
133
6 buckets
270
127
6 buckets
276
131
6 buckets
282
37
6 buckets
288
47
6 buckets
294
39
6 buckets
300
40
Further reading
Related sources by topic
87 entries. A tick means the page title was fetched on 2026-10-23; the rest are stable reference addresses that were not re-fetched.