Learning dashboard

From a single neuron to the transformer

An interactive study companion built on Vanny Khon's lecture notes Language Modeling: A Mathematical Perspective and on his 22-slide lecture Language Modeling (A First Look). Each built chapter runs Introduction → Intuition → Theory → Visualization → Worked example → Code → Practice → Summary.

Continue learning → Start with the First Look lesson

Your progress

0%of built material read

0 / 0practice questions correct

2 of 15sections fully built

0flashcards marked known

Progress is saved in this browser only.

How to read this site

Blocks tagged From the lesson restate the original notes or lecture slides, keeping their notation and claims. Blocks tagged Tutor addition are extra explanations, examples, code and questions written for this site. Use Beginner / Advanced at the top to switch between plain-language boxes and rigorous extensions. The core lesson content is shown in both modes.

Chapters

Chapter 0

Preface

Why these notes exist and how to use them.

≈ 5 minNo prerequisites

From the lesson

The notes cover deep-learning language models, presented as concisely as possible in mathematical terms, motivating each term along the way.
The path runs from neural networks up to transformer-based language models, the kind of model used to build ChatGPT, and ends with the author's view on why they work and some open questions.
Every theory part comes with code, written in PyTorch.
The notes grew out of a class the author teaches to friends in Cambodia with the goal of improving language models for Khmer, and were revised for an NLP course at the Institute of Technology of Cambodia.
Recommended order: read the theory part first for the concepts, then the code and implement it yourself.

Tutor addition

Prerequisites for the whole course: vectors and matrices (matrix–vector products, transposes), functions and composition, partial derivatives and the gradient (for Chapter 4 onward), basic probability (for Chapters 5–9), and enough Python to read a class definition.

Tutor addition

Two ways in. The 22-slide lecture Language Modeling: A First Look is a short tour of the whole idea, from “what is a language model?” to a small trained recurrent network. Chapter 1 starts the detailed path from a single neuron. The lecture's ideas return in Chapters 5–8, and each lesson section links to them.

Chapter 1 · Foundations

Artificial Neural Network

How a chain of neurons becomes a function built from matrices, biases and gates, and why the gates are what make depth matter.

≈ 60 minDifficulty ●●○○○Prereq: vectors, matrix product, composition

01 · Introduction

What you will learn

Tutor addition

Model one neuron as a weighted sum (a dot product) and a whole layer as a matrix–vector product $X^i = W^iX^{i-1}$.
Explain why stacking purely linear layers gives nothing new: $W^L\cdots W^1$ is again one matrix.
Use a gate (activation function) with a threshold (bias) to break that collapse: $X^{i+1}=\sigma(W^{i+1}X^i+b^{i+1})$.
State the formal definition of an $L$-layer neural network as a composition of affine maps and pointwise non-linearities.
Compare hard and soft gates (ReLU vs SiLU) and the common activations sigmoid, tanh and softplus.

In plain wordsA neural network is a recipe: multiply by a table of numbers, add a few numbers, squash, and repeat. This chapter explains where each of those three steps comes from.

02 · Intuition

A signal travelling through neurons

From the lesson

A signal $x$ (say, light entering the eye) flows to a first group of neurons $N_1^1,\dots,N_{k_1}^1$. Each processes what it receives and passes it to a deeper group $N_1^2,\dots,N_{k_2}^2$, and so on until depth $L$, where the last group $N_1^L,\dots,N_{k_L}^L$ produces an output, such as "cat" or "flower".

From the lesson

Neurons do not pass information on continuously. Between neuron A and neuron B there is a synaptic gap, and A only sends to B once it has enough signal: a gate that opens when a condition is met. The lesson's analogy is water seeping through a ceiling: surface tension holds it back until enough has gathered for gravity to pull a drop down.

Analogy checkThink of each neuron as a voter who listens to friends (inputs), trusts some more than others (weights), and only speaks up once convinced enough (threshold). The network is a chain of such committees.

03 · Mathematical theory

From weighted sums to the definition

3.1 One neuron is a dot product

From the lesson

The simplest model of neuron $N_{\jc{j}}^i$ adds its inputs. Since connections differ in strength, we use a weighted sum instead of $\sum_k x_{\kc{k}}^{i-1}$: $$x_{\jc{j}}^{i}=\sum_{\kc{k}} w_{\kc{k},\jc{j}}^{i}\,x_{\kc{k}}^{i-1}=(W_{\jc{j}}^{i})^TX^{i-1},$$ where $w_{\kc{k},\jc{j}}^{i}$ is the strength of the connection from neuron $N_{\kc{k}}^{i-1}$ in layer $i-1$ to neuron $N_{\jc{j}}^i$ in layer $i$, $W_{\jc{j}}^{i}=(w_{\kc{1},\jc{j}}^{i},\ldots,w_{\kc{k_{i-1}},\jc{j}}^{i})^T$ is neuron $\jc{j}$'s weight vector and $X^{i-1}=(x_{\kc{1}}^{i-1},\ldots,x_{\kc{k_{i-1}}}^{i-1})^T$ is the output of layer $i-1$.

Tutor addition · colour code

As in the original, teal indices $\kc{k}$ count neurons in the sending layer $i-1$ and red indices $\jc{j}$ count neurons in the receiving layer $i$. The superscript $i$ is the layer number, not a power.

3.2 A layer is a matrix

From the lesson

Stacking the weight vectors of all $k_i$ neurons of layer $i$ as rows gives the weight matrix $W^{i}=[W_{\jc{1}}^{i},\ldots,W_{\jc{k_i}}^{i}]^T$, and the layer output is $$X^{i}=\begin{bmatrix} (W_{\jc{1}}^{i})^TX^{i-1}\\ \vdots \\ (W_{\jc{k_i}}^{i})^TX^{i-1}\end{bmatrix}=\begin{bmatrix} w_{\kc{1},\jc{1}}^{i} & \cdots & w_{\kc{k_{i-1}},\jc{1}}^{i}\\ \vdots & \ddots & \vdots\\ w_{\kc{1},\jc{k_i}}^{i} & \cdots & w_{\kc{k_{i-1}},\jc{k_i}}^{i}\end{bmatrix}X^{i-1}=W^iX^{i-1}.$$ Starting from the input $X^0$ we get a dynamical system: $X^0=\text{input}$, $X^{i+1}=W^{i+1}X^i$.

Advanced · shapes$W^i\in\mathbb{R}^{k_i\times k_{i-1}}$: row $\jc{j}$ belongs to receiving neuron $\jc{j}$, column $\kc{k}$ to sending neuron $\kc{k}$. Entry $(W^i)_{\jc{j}\kc{k}}=w^i_{\kc{k},\jc{j}}$. Note the index order flips between the entry name and the matrix position; this is the most common source of shape errors.

3.3 Why purely linear layers collapse

From the lesson

There is a problem. The product of the matrices is another matrix $W$, so $X^L=WX^0$, and the whole process is just one layer with one matrix multiplication. Step through it:

Step-by-step derivation

1

$$X^L=W^LX^{L-1}$$

Definition of the last layer.

2

$$X^L=W^L\big(W^{L-1}X^{L-2}\big)$$

Substitute $X^{L-1}=W^{L-1}X^{L-2}$.

3

$$X^L=\big(W^LW^{L-1}\big)X^{L-2}$$

Matrix multiplication is associative, so the brackets can move.

4

$$X^L=\big(W^L\cdots W^1\big)X^0$$

Repeat the substitution down to the input.

5

$$X^L=W\,X^0,\qquad W:=W^L\cdots W^1\in\mathbb{R}^{k_L\times k_0}$$

One matrix. Depth added nothing: an $L$-layer linear network computes exactly what a 1-layer linear network can.

Advanced · same for affine layers (tutor addition)Biases do not rescue a network without activations: $W^2(W^1x+b^1)+b^2=(W^2W^1)x+(W^2b^1+b^2)$, which is again affine. Only a non-linear $\sigma$ between the layers prevents the collapse.

3.4 The gate: ReLU with a threshold

From the lesson

To model the gate we use the rectified linear function $\text{ReLu}:\mathbb{R}\to\mathbb{R}$, $$\text{ReLu}(x)=\begin{cases}x & x>0\\ 0 & x\le 0\end{cases}$$ It lets information through only when it is positive. Neuron $N_{\jc{j}}^i$ gets its own threshold $b_{\jc{j}}^i$ and gate $\text{ReLu}(x-b_{\jc{j}}^i)$ (a self gate). For a whole layer we collect thresholds in the bias vector $b^i=(b_{\jc{1}}^i,\ldots,b_{\jc{k_i}}^i)^T$ and the dynamics become $$\begin{cases} X^0=\text{input}\\ X^{i+1}=\text{ReLu}(W^{i+1}X^i+b^{i+1})\end{cases}$$ with ReLu applied component-wise. The gate is usually called the activation function and written $\sigma$.

Tutor addition · watch the sign

A threshold $\theta$ appears as $\text{ReLu}(x-\theta)$, while the layer formula uses $+b$. They agree when $b=-\theta$. A neuron with bias $b=-2$ fires only once its weighted input exceeds $2$. Both conventions appear in the lesson; the formal definition below uses $Wx+b$, as PyTorch does.

3.5 Linear and affine maps

From the lesson

A map $L:\mathbb{R}^n\to\mathbb{R}^m$, $L(x)=Wx$ with $W$ an $m\times n$ matrix, is a linear map. Their collection $\mathcal{L}(\mathbb{R}^n,\mathbb{R}^m)$ can be identified with the $m\times n$ matrices $M_{m\times n}$. The map $A(x)=Wx+b$ is an affine map, generalising the line $y=ax+b$.

Definition · from the lesson

A neural network is a function obtained by composing affine maps with non-linear functions. A neural network of $L$ layers is a function $NN:\mathbb{R}^n\to\mathbb{R}^m$, $$NN(x)=Af_L\circ\sigma_{L-1}\circ\cdots\circ Af_2\circ\sigma_1\circ Af_1(x),$$ where each $Af_i:\mathbb{R}^{k_{i-1}}\to\mathbb{R}^{k_i}$ is affine, $Af_i(x)=W^ix+b^i$ with $W^i\in\mathcal{L}(\mathbb{R}^{k_{i-1}},\mathbb{R}^{k_i})$, $b^i\in\mathbb{R}^{k_i}$, $k_0=n$, $k_L=m$, and each $\sigma_i$ is a non-linear function applied pointwise to the output of $Af_i$.

Read it right to left$\circ$ means "then apply". The input $x$ first goes through $Af_1$, then $\sigma_1$, then $Af_2$, and so on. The last step is $Af_L$, with no activation after it.

From the lesson · "pointwise"

For $f:\mathbb{R}\to\mathbb{R}$ and $v=(v_1,\dots,v_N)$, applying $f$ pointwise means $f(v)=(f(v_1),\dots,f(v_N))$.

3.6 Architecture, hard and soft gates

From the lesson

Picture a network as neurons ordered in layers, connections between them whose strengths are stored in $W^i$, and a threshold (bias) $b^i$ for each neuron. The number of layers, the number of neurons per layer and the choice of activation together form the network's architecture.

$\text{ReLu}(x-b)$ either sends the input forward or does not, with no middle ground: a hard gate. A soft gate is $\text{SiLu}(x-b)=(x-b)\,\text{sigmoid}(x-b)$, where $\text{sigmoid}(x)=e^x/(1+e^x)\in(0,1)$, so the input is let through partly. For large $x$, $\text{sigmoid}(x)\approx1$ and $\text{SiLu}(x)\approx x$; for very negative $x$, $\text{SiLu}(x)\approx 0$; near $0$, SiLu is not simply $0$ as ReLu is. Other common activations are $$\text{sigmoid}(x)=\frac{e^{x}}{1+e^{x}},\qquad \tanh(x)=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}},\qquad \text{softplus}(x)=\ln(1+e^{x}).$$

Advanced · useful identities (tutor addition)

$\text{sigmoid}(x)=\dfrac{1}{1+e^{-x}}$, and $\text{sigmoid}'(x)=\text{sigmoid}(x)\,(1-\text{sigmoid}(x))$.
$\tanh(x)=2\,\text{sigmoid}(2x)-1$, so tanh is a rescaled, re-centred sigmoid with range $(-1,1)$.
$\text{softplus}'(x)=\text{sigmoid}(x)$: softplus is a smooth version of ReLu, and $\text{softplus}(x)-\text{ReLu}(x)\le\ln 2$ with equality at $x=0$.
$\text{SiLu}'(x)=\text{sigmoid}(x)\big(1+x(1-\text{sigmoid}(x))\big)$. SiLu is not monotone: its minimum is about $-0.278$ at $x\approx-1.278$.
ReLu is not differentiable at $0$; in practice the derivative there is set to $0$.

04 · Interactive visualization

See the mathematics move

Forward propagation simulator

Weights coloured from blue ($-1$) to red ($+1$), as in the lesson

Architecture

Layer sizes $k_0,k_1,\dots,k_L$ separated by commas, each 1–6, 2–6 layers in total. Initial weights follow PyTorch's default nn.Linear rule $U(-1/\sqrt{k_{i-1}},\,1/\sqrt{k_{i-1}})$.

Click a neuron to see its computation and edit its bias. Click a connection to edit its weight. The output layer has no activation, matching the definition: the last map is $Af_L$.

05 · Worked example

A 2 → 2 → 1 network by hand

Tutor addition

Take $L=2$, $\sigma_1=\text{ReLu}$ and $$W^1=\begin{bmatrix}1&-1\\2&1\end{bmatrix},\quad b^1=\begin{bmatrix}0\\-1\end{bmatrix},\quad W^2=\begin{bmatrix}1&1\end{bmatrix},\quad b^2=0.5,\quad x=X^0=\begin{bmatrix}1\\2\end{bmatrix}.$$

Compute $NN(x)=Af_2\circ\text{ReLu}\circ Af_1(x)$

1

$$W^1x=\begin{bmatrix}1\cdot1+(-1)\cdot2\\ 2\cdot1+1\cdot2\end{bmatrix}=\begin{bmatrix}-1\\4\end{bmatrix}$$

Each row of $W^1$ is one neuron's weight vector; dot it with $x$.

2

$$Af_1(x)=W^1x+b^1=\begin{bmatrix}-1+0\\4-1\end{bmatrix}=\begin{bmatrix}-1\\3\end{bmatrix}$$

Add the bias vector. These are the pre-activations.

3

$$X^1=\text{ReLu}\begin{bmatrix}-1\\3\end{bmatrix}=\begin{bmatrix}0\\3\end{bmatrix}$$

The gate closes for neuron 1 (negative) and passes neuron 2.

4

$$NN(x)=W^2X^1+b^2=1\cdot0+1\cdot3+0.5=3.5$$

The last layer is affine only; no activation after it.

5

$$\text{Without the gate: } W^2(W^1x+b^1)+b^2=-1+3+0.5=2.5$$

The gate changed the answer. Check the collapse: $W^2W^1=[3,\;0]$ and $W^2b^1+b^2=-0.5$, so $3\cdot1+0\cdot2-0.5=2.5$. Same number, one affine map.

Try it liveSet the simulator above to architecture 2,2,1 and ReLu, then click the connections and enter these weights. You will see $3.5$ at the output.

06 · Coding example

The same mathematics in PyTorch

These snippets are tutor additions that mirror this chapter. Nothing runs in this page: there is no Python runtime here. Outputs below are the values you should see, worked out by hand.

worked_example.py

import torch

W1 = torch.tensor([[1., -1.], [2., 1.]])   # W^1, shape (2, 2)
b1 = torch.tensor([0., -1.])               # b^1
W2 = torch.tensor([[1., 1.]])              # W^2, shape (1, 2)
b2 = torch.tensor([0.5])                   # b^2
x0 = torch.tensor([1., 2.])                # X^0

z1 = W1 @ x0 + b1        # Af_1(x): pre-activation
x1 = torch.relu(z1)      # sigma_1, applied pointwise
out = W2 @ x1 + b2       # Af_2: last layer, no activation
print(z1, x1, out)

no_gate = W2 @ z1 + b2   # same network with the gate removed
print(no_gate)

Expected outputtensor([-1., 3.]) tensor([0., 3.]) tensor([3.5000]) tensor([2.5000])

@Matrix product. W1 @ x0 is $W^1X^0$.
torch.reluApplies $\text{ReLu}$ to every entry: the "pointwise" of the definition.
1., -1.The dots make the entries floats. @ refuses to mix integer and float tensors (it raises a dtype error), so keep every tensor float.

activations.py

import torch
import torch.nn.functional as F

x = torch.tensor([-2., 0., 2.])
print(torch.relu(x))
print(torch.sigmoid(x))
print(F.silu(x))          # x * sigmoid(x)
print(torch.tanh(x))
print(F.softplus(x))      # ln(1 + e^x)

Expected output (4 decimals)tensor([0., 0., 2.]) tensor([0.1192, 0.5000, 0.8808]) tensor([-0.2384, 0.0000, 1.7616]) tensor([-0.9640, 0.0000, 0.9640]) tensor([0.1269, 0.6931, 2.1269])

collapse_check.py

import torch
import torch.nn as nn

A = nn.Linear(2, 3)     # Af_1 : R^2 -> R^3
B = nn.Linear(3, 1)     # Af_2 : R^3 -> R^1
x = torch.randn(5, 2)   # a batch of 5 inputs

with torch.no_grad():
    W = B.weight @ A.weight            # W^2 W^1
    b = B.weight @ A.bias + B.bias     # W^2 b^1 + b^2
    print(torch.allclose(B(A(x)), x @ W.T + b))

Expected outputTrue

The last snippet is the collapse argument from §3.3 as a test: two affine layers with nothing in between are one affine layer. Insert torch.relu between A and B and the check will generally print False.

07 · Practice

Check your understanding

08 · Summary

Key ideas to keep

Neuron: weighted sum $(W_j^i)^TX^{i-1}$. Layer: matrix product $W^iX^{i-1}$, with $W^i\in\mathbb{R}^{k_i\times k_{i-1}}$.
Linear layers alone collapse: $W^L\cdots W^1$ is one matrix (affine layers collapse too).
A gate (activation) with threshold breaks the collapse: $X^{i+1}=\sigma(W^{i+1}X^i+b^{i+1})$.
Definition: $NN=Af_L\circ\sigma_{L-1}\circ\cdots\circ\sigma_1\circ Af_1$, with $\sigma$ applied pointwise and no activation after $Af_L$.
Architecture = number of layers + neurons per layer + choice of activation.
Hard gate ReLu vs soft gate SiLu; also sigmoid, tanh, softplus.

Common mistakes

✗Writing $W^i$ as $k_{i-1}\times k_i$. Rows belong to the receiving layer: $k_i\times k_{i-1}$.

✗Putting an activation after the last layer when stating the definition. The definition ends with $Af_L$.

✗Thinking biases alone make a deep network more expressive. Without $\sigma$, affine maps still compose to one affine map.

✗Mixing up threshold and bias signs: $\text{ReLu}(x-\theta)$ corresponds to $b=-\theta$.

✗Reading $W^i$'s superscript as a power. It is a layer index.

Exam tips

When asked for a parameter count, sum $k_ik_{i-1}+k_i$ over layers.
For "why do we need activations?", give the two-line collapse proof, including biases.
Always state domain and codomain of each $Af_i$; it shows you understand the shapes.

Section 1.1 · Code

Coding Neural Networks with PyTorch

Tensors, slicing, nn.Linear as an affine map, and a first network written as a class.

≈ 45 minDifficulty ●●○○○Prereq: Chapter 1, basic Python

01 · Introduction

What you will learn

Tutor addition

Import torch and matplotlib.pyplot, create tensors and do pointwise arithmetic.
Slice a tensor with x[1,:] and x[:,2], remembering Python counts from 0.
Read nn.Linear(n,m) as the affine map $L(x)=Wx+b$, with $W\in\mathbb{R}^{m\times n}$, acting on batches of shape [batch_size, n].
Write the two-layer network $N=Af_2\circ\tanh\circ Af_1:\mathbb{R}\to\mathbb{R}^{20}\to\mathbb{R}$ as an nn.Module and plot the function it computes.

02 · Intuition

Tensors are the vectors and matrices of Chapter 1

From the lesson

The basic objects in PyTorch are tensors: multi-dimensional arrays. Coding is introduced throughout the notes layer by layer, alongside the theory.

Picture itA tensor of shape (3,) is a list of 3 numbers (a vector). Shape (2,3) is a table with 2 rows and 3 columns (a matrix). Every $X^i$, $W^i$ and $b^i$ from Chapter 1 becomes one tensor.

From the lesson · objects

In object-oriented programming an object has attributes (its internal state or data) and methods (actions on that state). nn.Linear(n,m) is an object whose attributes include the weight $W$ and bias $b$. Each of them carries more than numbers: it is a pair (value, gradient), which training will use later.

03 · Theory, line by line

From tensors to nn.Module

3.1 Imports and tensors

From the lesson

To use a package you import it. We need matplotlib.pyplot for plotting and torch for tensors.

lesson code · imports and tensors

import torch
import matplotlib.pyplot as plt

a=torch.tensor([-1,2]) # create a tensor from a list of numbers from package torch
b=torch.tensor([2,-7])

From the lesson

Operations are pointwise: a+b adds entrywise, 3*a multiplies each entry, b**2 squares each entry.

3.2 Plotting a function from a tensor

lesson code · plot a² − 3a

import torch
import matplotlib.pyplot as plt # shorten matplotlib.pyplot to plt
a = torch.linspace(-4, 4, 30) # create a tensor of 30 elements of numbers evenly spaced between -4 and 4
plt.plot(a,a**2-3*a) # plot a^2-3a on a axis
plt.title('graph of a^2-3a')
plt.xlabel('a')
plt.ylabel('a^2-3a')

Advanced · what linspace really doestorch.linspace(s, e, n) returns $s+\frac{k(e-s)}{n-1}$ for $k=0,\dots,n-1$: $n$ points including both ends. With $(-4,4,30)$ the spacing is $8/29\approx0.276$. The curve you see is the 30 points joined by straight segments.

3.3 Slicing

lesson code · slicing

x=torch.rand(2,3) # this create a tensor of 2 by 3 whose elements are random number between 0 and 1
x1=x[1,:] # we define a tensor x1 to be the second row of x
y1=x[:,2]
#Now we can print the above tensors
print(f'x= {x}')
print(f'x1={x1}')
print(f'y1={y1}')

Output shown in the lesson (random, so yours will differ)x= tensor([[0.9607, 0.1908, 0.6339], [0.9940, 0.0890, 0.3960]]) x1=tensor([0.9940, 0.0890, 0.3960]) y1=tensor([0.6339, 0.3960])

From the lesson

Python counts from 0, so x[1,:] is the second row and x[:,2] the third column: $\texttt{x1}=(x_{21},x_{22},x_{23})$ of shape (3,), $\texttt{y1}=(x_{13},x_{23})$ of shape (2,).

3.4 nn.Linear is an affine map

lesson code · nn.Linear

import torch.nn as nn
L=nn.Linear(2,3) # this create an affine map taking 2 dimensional input and output 3 dimension

W=L.weight
b=L.bias
print(f'Weight W of L: {W}')
print(f'Bias b of L: {b}')

Output shown in the lesson (random initialisation)Weight W of L: Parameter containing: tensor([[-0.2916, -0.7016], [ 0.3956, 0.5720], [ 0.3072, 0.3156]], requires_grad=True) Bias b of L: Parameter containing: tensor([-0.6838, -0.4431, -0.2809], requires_grad=True)

From the lesson

Mathematically, nn.Linear(2,3) creates $L(x)=Wx+b$ with $$W=\begin{bmatrix} w_{11} & w_{12}\\ w_{21} & w_{22}\\ w_{31} & w_{32}\end{bmatrix}\in\mathbb{R}^{3\times 2},\qquad b\in\mathbb{R}^{3},\qquad x\in\mathbb{R}^{2}.$$ Its entries are random. nn.Linear(2,3,bias=False) sets $b=0$. $W$ has 3 rows and 2 columns, the shape PyTorch prints, matching $W^i\in\mathbb{R}^{k_i\times k_{i-1}}$ from Chapter 1. PyTorch itself computes $xW^T+b$ with $x$ written as a row, which gives the same numbers.

During training it is efficient to process several data points at once, a batch. So the natural input of nn.Linear(n,m) has shape [batch_size, input_dim]: a stack $[v_1,\dots,v_k]$, $v_i\in\mathbb{R}^n$, is processed in parallel to give $[L(v_1),\dots,L(v_k)]$.

Advanced · why $xW^T+b$ is the sameIf $x$ is a row vector ($1\times n$), then $(Wx^T)^T=xW^T$, so $xW^T+b$ is the row form of $Wx+b$. For a batch $V\in\mathbb{R}^{k\times n}$ whose rows are $v_i$, $VW^T+\mathbf 1b^T$ has rows $(Wv_i+b)^T$: one matrix product handles the whole batch.

Advanced · default initialisation (tutor addition)PyTorch's default nn.Linear(n,m) draws every weight and bias from $U\!\left(-\tfrac{1}{\sqrt n},\tfrac{1}{\sqrt n}\right)$. For $n=2$ that is $(-0.707,0.707)$, which is why all printed entries above have absolute value below $0.71$.

3.5 A two-layer network as a class

From the lesson

Build $N:\mathbb{R}\to\mathbb{R}$ by composing an affine map $A_1:\mathbb{R}\to\mathbb{R}^{20}$, then $\tanh$, then $A_2:\mathbb{R}^{20}\to\mathbb{R}$.

lesson code · NeuralNet

class NeuralNet(nn.Module):
    def __init__(self): # this is the part where we define attribute inside object NeuralNet
        super().__init__() # this initializes the base class nn.Module
        self.A1=nn.Linear(1,20) # create A1 as an affine map from 1d to 20 dim
        self.A2=nn.Linear(20,1)
    def forward(self,x): # this is where we assemble all part/attribute of NeuralNet
        y=self.A1(x)
        y=torch.tanh(y)
        y=self.A2(y)
        return y

N = NeuralNet() # this create a neural network

From the lesson

$$\begin{aligned} N&=Af_2\circ\tanh\circ Af_1:\ \mathbb{R}\to\mathbb{R}^{20}\to\mathbb{R},\\ Af_1(x)&=W^1x+b^1,\qquad W^1\in\mathbb{R}^{20\times 1},\ b^1\in\mathbb{R}^{20},\\ Af_2(h)&=W^2h+b^2,\qquad W^2\in\mathbb{R}^{1\times 20},\ b^2\in\mathbb{R}\end{aligned}$$ self.A1 and self.A2 are $Af_1$ and $Af_2$; torch.tanh is $\sigma$, applied pointwise. Writing N(x) calls the method forward, where the composition is written.

nn.ModuleThe base class for all networks. Inheriting from it makes PyTorch register A1, A2 and their parameters automatically.
super().__init__()Runs nn.Module's own setup. Forgetting it gives an error as soon as you assign self.A1.
forwardThe composition $Af_2\circ\tanh\circ Af_1$, read top to bottom. You never call forward directly; call N(x).

3.6 Plotting what the network computes

lesson code · plot N

x = torch.linspace(-20, 20, 300) # create domain
with torch.no_grad(): # this code is there to stop pytorch from keeping track of derivative
    y = N(x.view(-1,1)) # the 300 points as a stack of shape [300,1], processed at once
#plot the result
plt.figure(figsize=(10,5))
plt.plot(x, y)
plt.show()

From the lesson

Training the network comes later, after stochastic gradient descent (Section 4.1–4.2).

04 · Interactive visualization

Play with tensors and shapes

Everything below is computed by JavaScript that re-implements the same formulas. It is not running PyTorch; the output formatting imitates PyTorch's printing.

What does an untrained NeuralNet() compute?

$Af_2\circ\sigma\circ Af_1$ on $[-20,20]$, 300 points

width

$\sigma$ show hidden units

Weights use PyTorch's default initialisation, so each "new network" is one possible output of the lesson's code. With tanh the graph flattens for large $|x|$: each $\tanh(W^1_jx+b^1_j)$ saturates at $\operatorname{sign}(W^1_jx)$, so $N(x)\to b^2+\sum_j W^2_{1j}\operatorname{sign}(\pm W^1_j)$ as $x\to\pm\infty$.

05 · Worked example

Tracing shapes through N(x.view(-1,1))

Follow one batch of 300 points

1

x = torch.linspace(-20,20,300) → shape [300]

A 1-D tensor: 300 numbers, not yet a batch of inputs.

2

x.view(-1,1) → shape [300, 1]

-1 means "work this out": 300 rows of one feature each. This is the [batch_size, input_dim] layout nn.Linear(1,20) expects.

3

self.A1(x) → shape [300, 20]

Each row computes $W^1x_i+b^1\in\mathbb{R}^{20}$; in batch form $XW^{1T}+b^1$ with $X\in\mathbb{R}^{300\times1}$, $W^{1T}\in\mathbb{R}^{1\times20}$.

4

torch.tanh(y) → shape [300, 20]

Pointwise, so the shape is unchanged.

5

self.A2(y) → shape [300, 1]

$[300,20]\cdot[20,1]$ gives one output per input point.

6

Parameters: $(20\cdot1+20)+(1\cdot20+1)=40+21=61$

$W^1$ (20), $b^1$ (20), $W^2$ (20), $b^2$ (1). Check with sum(p.numel() for p in N.parameters()).

06 · Coding example

Extend the lesson code

Tutor additions. Not executed on this page; expected outputs worked out by hand.

inspect_network.py

import torch
import torch.nn as nn

class NeuralNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.A1 = nn.Linear(1, 20)
        self.A2 = nn.Linear(20, 1)
    def forward(self, x):
        return self.A2(torch.tanh(self.A1(x)))

N = NeuralNet()
for name, p in N.named_parameters():
    print(name, tuple(p.shape))
print(sum(p.numel() for p in N.parameters()))

x = torch.linspace(-20, 20, 300).view(-1, 1)
with torch.no_grad():
    y = N(x)
print(x.shape, y.shape)

Expected outputA1.weight (20, 1) A1.bias (20,) A2.weight (1, 20) A2.bias (1,) 61 torch.Size([300, 1]) torch.Size([300, 1])

batch_equals_loop.py

import torch
import torch.nn as nn

L = nn.Linear(2, 3)
V = torch.randn(4, 2)                 # batch of k=4 vectors in R^2
batch = L(V)                          # shape [4, 3]
loop = torch.stack([L.weight @ v + L.bias for v in V])
print(batch.shape, torch.allclose(batch, loop))

Expected outputtorch.Size([4, 3]) True

07 · Practice

Check your understanding

08 · Summary

Key ideas to keep

Tensors are multi-dimensional arrays; arithmetic like a+b, 3*a, b**2 is pointwise.
Indexing starts at 0: x[1,:] is row 2, x[:,2] is column 3.
nn.Linear(n,m) is $L(x)=Wx+b$ with weight of shape (m,n) and bias of shape (m,); it maps a batch [k,n] to [k,m] via $xW^T+b$.
A network is an nn.Module: layers are attributes set in __init__, the composition is written in forward, and N(x) calls it.
torch.no_grad() turns off gradient tracking when you only evaluate.

Common mistakes

✗Passing shape [300] into nn.Linear(1,20). It needs the last dimension to be 1: use x.view(-1,1).

✗Expecting L.weight to be (n,m). It is (m,n), output by input.

✗Calling N.forward(x) instead of N(x); it works but skips PyTorch's hooks.

✗Forgetting super().__init__() in a custom module.

Lecture · 22 slides · 8 sections

Language Modeling: A First Look

Vanny Khon's short lecture for the Department of Applied Mathematics and Statistics at the Institute of Technology of Cambodia, rebuilt as one connected lesson. It starts from “a model turns a sentence into a sentence” and ends with a recurrent network that learns to tell approval from disapproval.

≈ 80 minDifficulty ●●○○○Prereq: vectors, functions, a little probabilityEvery slide mapped below

01 · Introduction

What you will learn, and where every slide goes

Tutor addition · goals

State a language model as a map $M:\mathcal{L}\to\mathcal{L}$ and see why it is hard to build.
Tell apart words $\Sigma$, all word sequences $\Sigma^*$ and legitimate sentences $\mathcal{L}\subset\Sigma^*$.
Build $M$ from a next-word map $M_0:\mathcal{L}\to\Sigma$.
Give a computer numbers to work with: an embedding $E:\Sigma\to\mathbb{R}^d$ for the input and a probability vector in $\mathbb{R}^{|\Sigma|}$ for the output.
Read a sentence word by word with a recurrent neural network $f:\text{words}\times\text{memory}\to\text{memory}$, and train one on approval vs disapproval.

In plain wordsThe lecture answers one question in eight small steps: how can a computer, which only knows numbers, take in a sentence and say something sensible back?

Tutor addition · how it connects

The lecture is a first pass over ideas that the full notes develop later. This table says where each section meets a chapter of those notes. Chapters 2–10 are still outline pages here, so the links lead to their outlines.

Slides	Lesson section	Chapter of the full notes	Why they meet

Original deck: open the slides ↗. Blocks tagged “From the lesson · slide n” restate that slide; blocks tagged “Tutor addition” are explanations, toy models and code written for this site.

Section 1

A model that turns a sentence into a sentence

From the lesson · slide 2

A language model is an object that can turn a sentence into a sentence.

From the lesson · slide 3

Let $\mathcal{L}$ be the set of sentences. A language model is a map $$M:\mathcal{L}\to\mathcal{L}.$$ But how do we build it?

Tutor addition

The definition only fixes the type of $M$: a sentence goes in and a sentence comes out. Many everyday tasks have that shape. A question goes in and an answer comes out, as in slide 2's “Where do you study?”. A sentence in one language goes in and its translation comes out. The start of a text goes in and a continuation comes out. Notice what the definition does not say: it gives no recipe. The rest of the lecture is the recipe.

Why a list of answers is not a language model

Toy lookup table · entries invented for this page

Ask

The “model” here is a table with two entries. It answers those two questions and nothing else, even when you change one word. A real $M:\mathcal{L}\to\mathcal{L}$ must handle every sentence in $\mathcal{L}$, and $\mathcal{L}$ is not a short list. So $M$ has to be a rule that computes its answer, not a list that stores it.

Try “Where do you study” without the question mark, or “Where do you work?”.

In plain wordsThink of a very patient person who must answer any sentence you give them. They cannot memorise every possible sentence, so they must follow some method. A language model is that method, written in mathematics.

Advanced · determinismAs written, $M$ is a function, so one input sentence gives one output sentence. Real models are usually probabilistic: they compute a distribution over answers and a sentence is drawn from it. Sections 6 and 8 make this precise by letting the model output probabilities.

Section 2

Words, sequences of words, and real sentences

From the lesson · slide 4

$\Sigma$ is the set of words. For Khmer, $\Sigma=\{$ឆ្មា, ភ្លៀង, ទន្លេ, ញ៉ាំ, ខៀវ$,\dots\}$. For English, $\Sigma=\{\text{cat},\text{rain},\text{river},\text{eat},\text{blue},\dots\}$.

From the lesson · slide 5

$\Sigma^*$ is the big region of all word sequences. Inside it sits the dashed region $\mathcal{L}$ of legitimate sentences, so $$\mathcal{L}\subset\Sigma^*.$$ The slide's examples inside $\mathcal{L}$: “the cat eats fish”, “it rains”, “I study NLP” and ឆ្មាស៊ីត្រី. Outside $\mathcal{L}$ but still in $\Sigma^*$: “rain cat blue”, “eat eat the”, “river of study at” and ខៀវភ្លៀងឆ្មា.

Tutor addition · notation

$\Sigma^*$ is every finite sequence of words: $\Sigma^*=\bigcup_{n\ge0}\Sigma^n$, where $\Sigma^n$ holds the sequences of exactly $n$ words. If $\Sigma$ has $|\Sigma|$ words then $|\Sigma^n|=|\Sigma|^n$. Being legitimate is a judgement by speakers of the language. There is no formula for “is in $\mathcal{L}$”, which is one reason a language model has to be learned from examples later on.

In plain wordsΣ is a bag of words. Σ* is every way of lining words up, sense or nonsense. 𝓛 is the small part of those lineups that a speaker would accept.

Advanced · countable but huge$\Sigma^*$ is countably infinite whenever $\Sigma\neq\emptyset$, since it is a countable union of finite sets. $\mathcal{L}$ may be finite or infinite depending on how sentences are defined, but in practice a model only ever sees finitely many of them.

Section 3

Building $M$ from a next-word map $M_0$

From the lesson · slide 6

A language model is a function $$M:\Sigma^*\to\Sigma^*.$$ $M$ can be built from $M_0:\mathcal{L}\to\Sigma$. The diagram starts with $s_0=s$, feeds it into the box $M_0$, and $M_0$ produces an output.

Tutor addition · the loop behind the diagram

$M_0$ reads a sentence and returns one word. The slide shows only the first step. The usual way to get a whole sentence out is to append the new word and run $M_0$ again: $$s_{k+1}=(s_k,\;M_0(s_k)),\qquad k=0,1,2,\dots$$ and stop when an “end” word appears. This repetition is what Chapter 8 calls an auto-regressive language model. A much smaller problem than “build $M$” remains: predict one next word.

Run the loop by hand

Toy bigram table, written by hand · not a trained model

Start $s_0$ pick

The bars show what this toy $M_0$ believes comes next after the last word. It looks only at that last word. A real $M_0$ reads the whole sentence $s_k$, which is why slide 6 gives it the domain $\mathcal{L}$ (the sentence so far), not a single word.

In plain wordsFinishing someone's sentence is easier than answering a whole question. If you can always guess the next word, you can write a whole reply by guessing again and again.

Advanced · a typing detailSlide 6 gives $M:\Sigma^*\to\Sigma^*$ and $M_0:\mathcal{L}\to\Sigma$. A half-written sentence $s_k$ lies in $\Sigma^*$ but need not lie in $\mathcal{L}$, so in practice $M_0$ is applied to every prefix, that is, on all of $\Sigma^*$. The lecture leaves this refinement unstated.

Section 4

How can a computer hold $M$?

From the lesson · slides 7–8

The question: how to represent $M$ in a computer? The answer starts with a better way to represent words: embed words into vectors. Slide 8 sends the words cat, dog, apple and cabbage of $\Sigma$ to points of the plane $\mathbb{R}^2$, with cat and dog near each other and apple and cabbage near each other. The slide's side notes say there are “more words”, and that real embeddings use more than two dimensions (“3 dimensions” and beyond).

Tutor addition · why vectors?

A computer can only do arithmetic. Words are not numbers. The cheapest fix is to number them: cat = 1, dog = 2, apple = 3. But then “dog is one more than cat” means nothing, and cat and apple being “two apart” is an accident of the numbering. A vector gives each word a position, and positions have distances. Place similar words close together and arithmetic on positions starts to carry meaning.

In plain wordsGive every word an address on a map. Words that are used alike get neighbouring addresses. Then “how similar are these two words?” becomes “how far apart are their addresses?”.

Section 5

The embedding $E:\Sigma\to\mathbb{R}^d$

From the lesson · slide 9

An embedding $E:\Sigma\to\mathbb{R}^d$ represents a word as a vector. The slide lists real models:

As the slide notes, for BERT, GPT, Llama, Qwen and DeepSeek the “words” $\Sigma$ are pieces of words (tokens). The right-hand column is a tutor addition: the count $|\Sigma|\cdot d$ of numbers an embedding table needs.

Tutor addition · what $E$ really is

For a finite $\Sigma$, $E$ is a table with one row per word: a matrix $\mathbf{E}\in\mathbb{R}^{|\Sigma|\times d}$. Looking a word up means picking its row. Writing $e_w\in\mathbb{R}^{|\Sigma|}$ for the one-hot vector that has a single 1 at position $w$, the lookup is the matrix product $E(w)=e_w^{\,T}\mathbf{E}$. The numbers in the table are parameters: they are learned along with the rest of the model.

In plain wordsThe embedding is a big spreadsheet. Each word owns one row of numbers. “Embedding a word” just means reading out its row.

Section 6

Numericalising the output

From the lesson · slide 10

With embedded input, a language model is a map $M:(\mathbb{R}^d)^*\to\Sigma$. How to numericalise the output?

From the lesson · slides 11–12

Draw the output $M(s)$ as a bar over every word. In the slide's example the bars for a, at, cat, I, rain, study, we, you are $0.03,\,0.07,\,0.01,\,0.48,\,0.01,\,0.10,\,0.17,\,0.05$, with the tallest at “I”, and the dots (⋮) stand for the remaining words. Stacking the bars gives a vector, so $$M:(\mathbb{R}^d)^*\to\mathbb{R}^{|\Sigma|}.$$

Tutor addition · reading the picture

Entry $i$ of $M(s)$ is the model's confidence that the next word is word $i$ of $\Sigma$. Such a vector has non-negative entries that add up to 1, a probability vector. The eight visible entries add up to $0.92$, so the unseen words share the remaining $0.08$. A network's last layer usually produces unconstrained scores $z_i$, and the standard way to turn scores into probabilities is the softmax: $$p_i=\frac{e^{z_i}}{\sum_j e^{z_j}}.$$ The slides do not show how their bars are produced; softmax is the usual choice and returns in Chapter 6.

In plain wordsInstead of the model blurting out one word, it hands back a score for every word, as percentages that add to 100%. Then a word can be chosen: the highest one, or a random one weighted by those percentages.

Advanced · temperatureDividing scores by $T$ before softmax sharpens ($T\lt1$) or flattens ($T\gt1$) the distribution. As $T\to0$ it concentrates on the arg-max word; as $T\to\infty$ it tends to the uniform distribution. Temperature is a tutor addition, not part of the slides.

Section 7

Reading with a memory: the recurrent neural network

From the lesson · slides 13–14

The section opens with a person asked “What is your favorite food?”, and then shows a long geometry problem (an acute triangle $ABC$ inscribed in a circle $\Gamma$, with points $A_1,B_1,C_1$ and $P$) next to an empty thought bubble.

Tutor addition · how to read those two slides

The lecture's own commentary is not on the slides, so this is an interpretation. A short question like the first is easy to take in at a glance. The geometry problem is too long to hold in your head at once: you read it a word at a time and keep a running understanding of what you have read so far. That running understanding is exactly what a recurrent network calls its memory.

From the lesson · slides 15–17

A memory vector $h_0$ with blank entries sits above the sentence “What is your favorite food ?”, and the memory changes as the words are read. An RNN is a function $$f:\text{words}\times\text{memory}\to\text{memory}.$$

Tutor addition · unrolling $f$

Read the words $w_1,\dots,w_n$ one at a time, always with the same $f$: $$h_t=f(w_t,\,h_{t-1}),\qquad t=1,\dots,n.$$ The slide's blank $h_0$ is the starting memory (zeros are the usual choice) and $h_n$ is a vector that summarises the whole sentence. The parameter names on slide 21, $\theta=(A_w,A_h,b,v,c)$, contain the three that make up $f$, so the standard form is $$h_t=\tanh\!\big(A_w\,e(w_t)+A_h\,h_{t-1}+b\big),$$ where $e(w_t)=E(w_t)\in\mathbb{R}^d$ is the embedding from Section 5. Which activation the lecture uses is not visible on the slides; $\tanh$ is the common one.

In plain wordsAn RNN reads like you do: one word at a time, keeping notes in its head. After each word it rewrites its notes using the new word and the old notes. When the sentence ends, the notes are a summary of everything it read.

Advanced · one function, many stepsThe same $f$, with the same parameters, is applied at every position. That is why an RNN can read sentences of any length with a fixed number of parameters. Training pushes gradients back through all $n$ applications of $f$ (back-propagation through time); a long chain can make those gradients shrink, which is the usual reason long-range memory is hard for plain RNNs.

Section 8

Example: approval or disapproval

8.1 The pipeline (slides 18–19)

From the lesson · slides 18–19

Feed “the food was not bad” word by word through a chain of RNN boxes. The last memory $h_n$ goes into $v\cdot h_n+c$ to give a number $z$, and $\sigma$ turns $z$ into $p$: $$z=v_1h_{n,1}+v_2h_{n,2}+\cdots+v_hh_{n,h}+c=v\cdot h_n+c,\qquad p=\sigma(z).$$ Here $v_i$ says how much part $i$ of the memory counts toward approval.

Tutor addition

The function $\sigma(z)=1/(1+e^{-z})$ squeezes any real number into $(0,1)$, so $p$ can be read as a probability of approval. A common decision rule is: approval if $p\ge\tfrac12$, i.e. if $z\ge0$. Since $z=v\cdot h_n+c$ is linear in the memory, the whole classifier is one RNN followed by one weighted vote.

8.2 Do the word vectors know about approval? (slide 20)

From the lesson · slide 20

The slide plots word vectors in a plane (PC1 against PC2, with a 2D/3D switch). Blue words appear mostly in approvals, orange words mostly in disapprovals, black words in both. Words such as “delicious”, “tasty” and “enjoy” sit on one side, “terrible”, “bad” and “worse” on the other.

The same kind of picture, for this page's toy data

Tutor addition · toy vectors built by hand, so the clusters are built in

ignore sentences that contain “not”, “never” or “no”

mostly in approvalsmostly in disapprovalsboth

PCA finds the two directions along which the 8-dimensional word vectors vary most. These vectors were made by hand so that similar words sit together, as pretrained embeddings do, so the separation here is a design choice and not a discovery. The checkbox shows something real, though: untick it and negations come back, so the same adjective turns up in both classes (“not clean” is a disapproval), which is why bag-of-words counts are not enough and word order has to be read.

8.3 Parameters and loss (slide 21)

From the lesson · slide 21

The model has parameters $\theta=(A_w,A_h,b,v,c)$, and $M_\theta(s)=p$, the probability that the sentence $s$ is an approval.

Tutor addition · counting and scoring

With embedding size $d$ and memory size $h$: $A_w\in\mathbb{R}^{h\times d}$ has $hd$ entries, $A_h\in\mathbb{R}^{h\times h}$ has $h^2$, and $b$, $v$ have $h$ each, plus the single number $c$: $$\#\theta=hd+h^2+2h+1.$$ The slide stops at defining $\theta$ and $M_\theta$. For a probability output the natural loss on labelled sentences $(s_k,y_k)$, with $y_k=1$ for approval, is the binary cross-entropy of Chapter 6: $$\mathcal{J}(\theta)=-\frac1N\sum_{k=1}^{N}\Big[y_k\ln M_\theta(s_k)+(1-y_k)\ln\big(1-M_\theta(s_k)\big)\Big].$$ Training lowers $\mathcal{J}$ by gradient descent (Chapter 4).

8.4 Train it (slide 22)

From the lesson · slide 22

The demo uses 865 sentences, 692 to train and 173 to test. Each has a label $y_k$ (0 for disapproval, 1 for approval); the slide's rows include “the city is not clean” (0), “the library is quiet and comfortable” (1), “not a good meal at all” (0) and “the slides look nice” (1). Before training, test accuracy is 51%. You choose the memory size $h\in\{1,2,4,12\}$ and train. For “the food was not bad” the untrained model gives $p=\sigma(-0.10)=0.48$, so it says “disapproval”.

Train a small RNN on approval vs disapproval

Tutor addition · own 100-sentence toy set (80 train, 20 test), runs in your browser

memory size $h$

training losstest accuracy

The 80 training sentences

sentence

The bars show the entries of $h_n$ for the sentence above, as on slide 22. Words the toy vocabulary does not know are treated as neutral.

Gradients are computed exactly by back-propagation through time and applied with the Adam optimiser in mini-batches of 8. Results depend on the random start: press “Start again” a few times with $h=1$ and with $h=12$ and compare. With $h=1$ the memory is a single number, too little to remember that “not” flips the meaning of the next word.

In plain wordsThe network starts out guessing. After each pass over the training sentences it nudges its numbers so that its guesses move toward the right labels. Testing on sentences it never trained on tells you whether it learned a rule or just memorised.

Coding example

The lecture's pieces in PyTorch

These snippets are tutor additions that mirror Sections 5, 6 and 8. Nothing runs in this page: there is no Python runtime here. The outputs shown are the values and shapes the code is expected to give, worked out by hand.

Section 5 · an embedding is a table

import torch
import torch.nn as nn

vocab = ["cat", "dog", "apple", "rain"]            # a toy Sigma
idx = {w: i for i, w in enumerate(vocab)}

E = nn.Embedding(num_embeddings=len(vocab), embedding_dim=3)   # E : Sigma -> R^3
ids = torch.tensor([idx["cat"], idx["rain"]])
print(E(ids).shape)      # one row per word looked up
print(E.weight.shape)    # the table itself, |Sigma| x d

Expected outputtorch.Size([2, 3]) torch.Size([4, 3])

Section 6 · scores to probabilities

z = torch.tensor([2.0, 1.0, 0.0])      # scores for three words
p = torch.softmax(z, dim=0)
print(p)
print(torch.allclose(p.sum(), torch.tensor(1.0)))

Expected outputtensor([0.6652, 0.2447, 0.0900]) True

Section 8 · the approval classifier, theta = (A_w, A_h, b, v, c)

class ApprovalRNN(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.Aw = nn.Linear(d, h, bias=False)   # A_w
        self.Ah = nn.Linear(h, h)               # A_h (and the bias b)
        self.v  = nn.Linear(h, 1)               # v and c

    def forward(self, X):                       # X: [batch, n, d] embedded words
        m = torch.zeros(X.shape[0], self.h)     # h_0 = 0
        for t in range(X.shape[1]):             # read one word at a time
            m = torch.tanh(self.Aw(X[:, t]) + self.Ah(m))   # h_t = f(w_t, h_{t-1})
        return torch.sigmoid(self.v(m)).squeeze(-1)         # p = sigma(v . h_n + c)

model = ApprovalRNN(d=3, h=4)
X = torch.randn(5, 6, 3)                        # 5 sentences, 6 words, d = 3
print(model(X).shape)
print(sum(q.numel() for q in model.parameters()))

Expected outputtorch.Size([5]) 37

Section 8 · training with binary cross-entropy

y = torch.tensor([1., 0., 1., 0., 1.])           # labels: 1 = approval
loss_fn = nn.BCELoss()
opt = torch.optim.Adam(model.parameters(), lr=0.05)

for epoch in range(30):
    opt.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()                               # back-propagation through time
    opt.step()

Expected behaviourthe loss starts near ln 2 ≈ 0.69 and falls as training proceeds (exact numbers depend on the random start)

Parameter count check: $A_w$ has $4\cdot3=12$ numbers, $A_h$ has $4\cdot4+4=20$ (matrix and bias), and $v,c$ add $4+1=5$, total $37=hd+h^2+2h+1$. PyTorch's own nn.RNN keeps two bias vectors, whose sum plays the role of the single $b$ here.

Practice

Check your understanding

Summary

Key ideas to keep

Slides 2–3. A language model turns a sentence into a sentence: $M:\mathcal{L}\to\mathcal{L}$.
Slides 4–5. $\Sigma$ = words, $\Sigma^*$ = all word sequences, $\mathcal{L}\subset\Sigma^*$ = legitimate sentences.
Slide 6. $M:\Sigma^*\to\Sigma^*$ can be built from a next-word map $M_0:\mathcal{L}\to\Sigma$ by repeating it.
Slides 7–9. To compute, embed words as vectors: $E:\Sigma\to\mathbb{R}^d$, a table of $|\Sigma|\cdot d$ learned numbers.
Slides 10–12. The output is a vector of next-word probabilities: $M:(\mathbb{R}^d)^*\to\mathbb{R}^{|\Sigma|}$.
Slides 13–17. An RNN reads word by word with a memory: $f:\text{words}\times\text{memory}\to\text{memory}$.
Slides 18–22. For approval vs disapproval, $z=v\cdot h_n+c$, $p=\sigma(z)$, with parameters $\theta=(A_w,A_h,b,v,c)$ trained on labelled sentences and judged on held-out ones.

Common mistakes

✗Treating $\Sigma^*$ and $\mathcal{L}$ as the same set. Most sequences in $\Sigma^*$, like “eat eat the”, are not sentences.

✗Reading $M_0$ as producing a sentence. It produces one word; the loop does the rest.

✗Thinking an embedding is a formula. For a finite vocabulary it is a table of learned numbers.

✗Expecting the visible bars on slide 11 to sum to 1. They sum to 0.92; the hidden words hold the other 0.08.

✗Forgetting that order matters to an RNN: the same words in a different order generally end in a different memory $h_n$.

✗Judging a classifier by its training sentences. Slide 22 holds out 173 sentences to test.

Where to go next

Each step of this lesson is developed properly in the full notes:

Bridge lesson · tutor-written · added 2 Nov 2026

Learning by gradients

Homework 2 trained a network by searching (one coordinate at a time, or random directions). Lecture 2 trains its RNN with Adam, back-propagation and cross-entropy. This lesson explains that missing step, in the order you need it.

≈ 60 minDifficulty ●●●○○Prereq: Homework 2, derivativesEvery number re-run in Python

Why this lesson

From searching to sliding downhill

Tutor addition

The notes cover this material in Chapter 4 (Gradient Descent), 4.1 (Stochastic Gradient Descent) and 6 (Cross-Entropy), which are only outlines on this site. No new course file was needed: everything here is standard mathematics, written to connect what you have already done (Homework 2) with what Lecture 2 uses.

You know (Homework 2)	Here you add	Used in
loss $L(P)\ge0$ measures error	its gradient $\nabla L(P)$ says which way is uphill	every training step
step $P+\eta v$ in a random direction $v$	step $P-\eta\nabla L(P)$: the best local direction	Lecture 2 (Adam)
learning rate: small = slow, large = jumpy	the same, now exactly predictable	Lecture 2, lr 0.01
squared ($L^2$) loss	cross-entropy for probabilities, softmax for many classes	Lecture 2 §6, §8

Step 1

The gradient points uphill

Tutor addition

For one parameter, the derivative $L'(w)$ is the slope: if $L'(w)>0$, increasing $w$ increases the loss. For $K$ parameters the gradient collects all the slopes:

$$\nabla L(P)=\Big(\frac{\partial L}{\partial p_1},\dots,\frac{\partial L}{\partial p_K}\Big)\in\mathbb{R}^K$$

It points in the direction where $L$ grows fastest, so $-\nabla L(P)$ is the steepest way down. Coordinate descent (Homework 2 Q7) moved along one axis at a time; the gradient uses all $K$ slopes at once.

Numerical check: $\dfrac{\partial L}{\partial p_k}\approx\dfrac{L(P+he_k)-L(P-he_k)}{2h}$ with a tiny $h$. Slow, but a perfect test for a hand-derived gradient (used in the scripts below).

Step 2

Gradient descent and the learning rate

Tutor addition $$P\ \leftarrow\ P-\eta\,\nabla L(P)$$

Example: $L(w)=(w-3)^2$, $L'(w)=2(w-3)$, start at $w=0$.

$\eta$	$w_1,\dots,w_6$	behaviour
0.1	0.6, 1.08, 1.464, 1.771, 2.017, 2.214	slow but steady (each step closes 20% of the gap)
0.5	3, 3, 3, …	exact in one step (lucky for this parabola)
0.9	5.4, 1.08, 4.536, 1.771, 3.983, 2.214	jumps across the valley, slowly settles
1.1	6.6, −1.32, 8.184, −3.221, 10.465, −5.958	too big: diverges

Here $w_{t+1}-3=(1-2\eta)(w_t-3)$: the gap is multiplied by $1-2\eta$ each step, so it shrinks only if $|1-2\eta|<1$, i.e. $0<\eta<1$. That is the exact version of Homework 2 Q9: small $\eta$ is slow, large $\eta$ jumps and may never settle.

Step 3

Back-propagation = the chain rule, layer by layer

Tutor addition

The Homework 2 network on one data point $(x,y)=(1.5,\,1)$ with $P=(w_1,b_1,w_2,b_2,v_1,v_2,c)=(1,0,-1,0.5,2,1,-0.5)$:

Forward: $z_1=w_1x+b_1=1.5$, $z_2=w_2x+b_2=-1$; $a_1=\mathrm{ReLU}(z_1)=1.5$, $a_2=0$; $NN=v_1a_1+v_2a_2+c=2.5$; loss $(y-NN)^2=2.25$.
Backward, last layer first: $\frac{\partial L}{\partial NN}=-2(y-NN)=3$. Then $\frac{\partial L}{\partial c}=3$, $\frac{\partial L}{\partial v_1}=3a_1=4.5$, $\frac{\partial L}{\partial v_2}=3a_2=0$.
Through the gate: $\frac{\partial L}{\partial z_1}=3\cdot v_1\cdot\mathrm{ReLU}'(z_1)=3\cdot2\cdot1=6$, so $\frac{\partial L}{\partial w_1}=6x=9$, $\frac{\partial L}{\partial b_1}=6$. Neuron 2 is switched off ($z_2<0$, $\mathrm{ReLU}'=0$): $\frac{\partial L}{\partial w_2}=\frac{\partial L}{\partial b_2}=0$.
Check: the numerical gradient gives exactly the same 7 numbers. One step with $\eta=0.05$ lowers the loss from 2.25 to 0.516.

Back-propagation reuses each intermediate derivative once, so the whole gradient costs about as much as one extra forward pass, even with millions of parameters. A closed gate passes no gradient: a neuron that is off for every data point never learns (a “dead ReLU”).

Script: GD_ex1.py · output

Step 4

Probabilities need cross-entropy

Tutor addition

For a yes/no answer the model outputs $p=\sigma(s)=1/(1+e^{-s})$ and is scored with $L=-[y\log p+(1-y)\log(1-p)]$ (Lecture 2 §6). Using $\sigma'(s)=p(1-p)$, the gradient becomes remarkably simple:

$$\frac{\partial L}{\partial s}=p-y$$

Compare squared error on $p$, whose gradient is $2(p-y)\,p(1-p)$:

$s$	$p$	$y$	cross-entropy $\partial L/\partial s$	squared error $\partial L/\partial s$
−4.6	0.010	1	−0.990	−0.0195
−1.9	0.130	1	−0.870	−0.197
0	0.500	1	−0.500	−0.250
1.9	0.870	1	−0.130	−0.0295

When the model is confidently wrong ($p=0.01$ for an approval), cross-entropy pushes hard (−0.99) while squared error barely moves (−0.02): that is why classifiers are trained with cross-entropy. The second row is Lecture 2's coffee sentence if its label were approval.

Step 5

Many classes: softmax

Tutor addition

A language model chooses among all words (First Look §6) or all 73 Khmer characters (Lecture 2 §8). Scores $z_c$ become probabilities and the loss is the minus log of the correct one:

$$p_c=\frac{e^{z_c}}{\sum_{c'}e^{z_{c'}}},\qquad L=-\log p_y,\qquad \frac{\partial L}{\partial z_c}=p_c-[c=y]$$

Example: $z=(2,\,1,\,0.1)$ and the correct class is the first one → $p=(0.659,\,0.242,\,0.099)$, $L=0.417$, gradient $(-0.341,\,0.242,\,0.099)$: raise the correct score, lower the others in proportion to how likely they looked. With two classes this is exactly the sigmoid case.

Script: GD_ex2.py · output

Step 6

Mini-batches and Adam

Tutor addition

Stochastic gradient descent: the loss is an average over $N$ examples; instead of the full gradient, use the average gradient of a small random batch (Lecture 2: 16 sentences). Noisier but far cheaper; one pass over all data is an epoch.
Adam (what Lecture 2 uses, lr 0.01): keeps a running mean $m$ of recent gradients (momentum, $\beta_1=0.9$) and of their squares $v$ ($\beta_2=0.999$), and steps $-\eta\,\hat m/(\sqrt{\hat v}+\varepsilon)$. Each parameter gets its own step size, similar in spirit to Homework 2's adaptive learning rate.

Step 7

Back-propagation through time (for RNNs)

Tutor addition

An RNN uses the same $A_w, A_h, b$ at every word. Unroll it over the sentence ($h_1\to h_2\to\dots\to h_n$) and it is just a deep network with shared weights; back-propagate from the loss at the end back to the first word, and add up the gradient contributions of every time step for each shared weight.

Going back $k$ words multiplies by $A_h$ (and gate derivatives) $k$ times. In one dimension: $0.5^{20}\approx10^{-6}$ (the signal vanishes: early words cannot be learned), $1.5^{20}\approx3300$ (it explodes). Lecture 2's code therefore limits (clips) the gradient. LSTM and attention (Chapter 9, Transformer) were invented to fix this.

Practice

Check your understanding

Summary

Key ideas to keep

$\nabla L(P)$ points uphill; gradient descent steps $P\leftarrow P-\eta\nabla L(P)$. The step size $\eta$ must be small enough to converge.
Back-propagation computes the whole gradient with the chain rule from the output back; a closed ReLU gate passes no gradient.
Sigmoid + cross-entropy gives $\partial L/\partial s=p-y$; softmax + cross-entropy gives $p_c-[c=y]$. Both push hardest when the model is confidently wrong.
In practice: mini-batches (SGD) and Adam. For RNNs, back-propagation through time, with vanishing / exploding gradients and clipping.
Next: Lecture 2 uses every one of these ideas.

Lecture 2 · 26 slides · 8 sections · added 2 Nov 2026

Recurrent Neural Network

Vanny Khon's second lecture (Department of Applied Mathematics and Statistics, ITC). It rebuilds the RNN from one question, “what does the memory $h_n$ store?”, trains it on 865 sentences labelled approval / disapproval, and ends with a character-level language model trained on Khmer tales.

≈ 90 minDifficulty ●●●○○Prereq: First Look lesson, Chapter 1, dot productsToy example re-run in Python

Map

Where every slide goes

Slides	Deck section	Here
1–2	Title · 1. Approval or disapproval	1 · The task
3–4	2. Recurrent Neural Network (reading word by word)	2 · Reading with a memory
5	3. Word embedding (EmbeddingGemma)	3 · Words as vectors
6–13	4. What is a concept? (features, gate, keys, the RNN equation)	4 · What the memory stores + toy RNN lab
14	5. Classifying a sentence	5 · From memory to a decision
15–19	6. Training (parameters θ, data, cross-entropy)	6 · Training
20–21	7. Example (training live, what each feature looks for)	7 · The trained example
22–26	8. Language model (Khmer characters)	8 · A character language model

Original deck: open the slides ↗ (needs your Google sign-in). Blocks tagged “From the lesson” restate the slides; “Tutor addition” blocks are explanations and checks written for this site. The interactive slides (training live, 2-D/3-D embedding views, the PyTorch model) only run in the original deck.

Section 1 · slides 1–2

Approval or disapproval?

From the lesson

Given a sentence, is it an approval or a disapproval? A model reads the sentence and answers:

sentence	model answer
I enjoy the coffee in the morning	approval
the food was bad and cold	disapproval
I do not dislike my job	approval

Tutor addition

The third sentence is the hard one: it contains the “bad” word dislike, yet it is an approval because of not. Counting good and bad words is not enough; the model has to remember what came before each word. That is exactly what the recurrent memory $h_n$ is for.

Section 2 · slides 3–4

Reading with a memory

From the lesson

Slide 3 reads “What is your favorite food?” one word at a time. Slide 4 shows a long olympiad geometry statement (“Let $ABC$ be an acute triangle inscribed in a circle $\Gamma$ … Point $P$ makes $AB_1PC_1$ convex with the same area as $ABC$”): by the last sentence you must still remember $A_1$, $B_1$, $C_1$ from the beginning.

Tutor addition

We read the same way: one word at a time, keeping a running summary. An RNN does the same with a vector: after word $n$ it holds $h_n\in\mathbb{R}^h$, a summary of everything read so far, and it updates the summary with each new word: $h_n=f(x_n,h_{n-1})$, $h_0=0$. The same function $f$ (the same weights) is used at every word, so a sentence of any length can be read.

Section 3 · slide 5

Words as vectors

From the lesson

We use $E=$ EmbeddingGemma (Google, 2025), $E:\Sigma\to\mathbb{R}^{768}$. The slide shows the words of the data set projected to 2-D and 3-D.

Tutor addition

In the deck's trained example the 768 numbers of each word are reduced to 48 by principal components (PCA), and this embedding is kept fixed: only $A_w, A_h, b, v, c$ are trained. Words with similar meaning land close together, so a feature vector $a_i$ that points toward “good” also responds to “enjoy” and “love”.

Section 4 · slides 6–13

What does the memory store?

From the lesson

$h_n$ = the information in the sentence up to word $n$. It looks for features: feature 1 = a good word (good, enjoy, love, clean, …), feature 2 = a bad word (bad, boring, dirty, cold, …).

Feature $i$ has a sample vector $a_i$, and $a_i\cdot x$ is big $\iff$ word $x$ looks like $a_i$.
Combine the new word with what was read before, then pass it through a gate: $h_n=\mathrm{ReLU}(z_n+b)$ with $\mathrm{ReLU}(t)=\max(0,t)$. ReLU together with $b$ = gate.
Toy model (slide 9): $z_{n,i}=a_i\cdot x_n+u_i\cdot h_{n-1}$, $h_n=\mathrm{ReLU}(z_n-1)$, $h_0=(0,0)$, $u_1=(1,0)$, $u_2=(0,1)$.
In general there are $h$ features $a_1,\dots,a_h\in\mathbb{R}^d$ (a good word, a bad word, a negation, …). Stack them as rows of $A_w\in\mathbb{R}^{h\times d}$; then $A_wx_n=(a_1\cdot x_n,\dots,a_h\cdot x_n)$.

$$h_n=\mathrm{ReLU}\big(\underbrace{A_wx_n}_{\text{new word}}+\underbrace{A_hh_{n-1}}_{\text{information up to word }n-1}+\,b\big)$$

$A_h$ = how the previous information interacts with the new information. We call this equation a recurrent neural network. Each $a_i$ is a key; the same neurons read every word.

Tutor addition · how to read the gate

With $b_i=-1$ a feature only “switches on” when $z_{n,i}>1$: weak evidence is ignored. Because $u_i\cdot h_{n-1}=h_{n-1,i}$ in the toy model, each feature also carries its own past forward, minus 1 per word. So a memory slowly leaks: a strong “bad” signal survives a couple of neutral words, then fades. That is both a feature (old, weak evidence is forgotten) and a weakness (long sentences lose early information), which later lectures fix with LSTM / attention.

Interactive · tutor addition

Run the toy RNN word by word

The deck's own toy numbers: words in $\mathbb{R}^2$, $a_1$ at 25°, $a_2$ at 115° (unit vectors), $A_h=I$, $b=(-1,-1)$, $v=(1,-1)$, $c=0$. Pick a sentence and step through it.

Sentence

Tutor addition · checked in Python

For the coffee was good but cold the exact values are $h_6=(0.72,\,2.57)$, $s=v\cdot h_6+c=-1.85$, $p=\sigma(-1.85)=0.136$ → disapproval. The slide shows the same numbers rounded: $h_6=(0.7,\,2.6)$, $s=-1.9$, $p=0.13$. Swap the order to …cold but good and the answer flips to approval ($p=0.98$): the “bad” memory leaks away before the end (see the gate note above). Script: RNN_ex1.py · output.

Section 5 · slide 14

From memory to a decision

From the lesson

Add the concepts together to get a score, then turn the score into a probability:

$$s=v_1h_{n,1}+\dots+v_hh_{n,h}+c=v\cdot h_n+c,\qquad p=\sigma(s)=\frac{1}{1+e^{-s}}$$

$v_i$ = how much concept $i$ counts toward approval. the coffee was good but cold: $h_6=(0.7,\,2.6)$, $v=(1,-1)$, $c=0$ → $s=-1.9$, $p=0.13$ → disapproval.

Tutor addition

$\sigma$ squeezes any score into $(0,1)$: $s=0$ gives $p=0.5$ (undecided), large positive $s$ gives $p\to1$. The sign of $v_i$ says whether a feature votes for approval ($v_1=1$ for “good”) or against it ($v_2=-1$ for “bad”). Compare the First Look example: the food was not bad, $h_n=(0.9,-0.2)$, $v=(2,1)$ → $z=1.6$, $p=0.83$.

Section 6 · slides 15–19

Training: what θ should be

From the lesson

To solve the problem we need the parameters $\theta=\{A_w,\,A_h,\,b,\,v,\,c\}$ with $A_w=(a_1;\dots;a_h)$, $A_h=(u_{ij})$, $b=(b_1,\dots,b_h)$, $v=(v_1,\dots,v_h)$. We want $\theta$ to predict as a human would; human judgement comes from experience, and this experience is the data $(s_1,y_1),\dots,(s_N,y_N)$ with $y_k=1$ approval, $0$ disapproval. With $M_\theta(s)$ = probability that $s$ is an approval:

$$\mathrm{loss}(\theta)=-\frac1N\sum_{k=1}^{N}\Big[y_k\log M_\theta(s_k)+(1-y_k)\log\big(1-M_\theta(s_k)\big)\Big]$$

For one sentence with $p=M_\theta(s)$: if $y=1$ the term is $-\log p$, very large when the model disagrees ($p\to0$) and close to 0 when it agrees ($p\to1$); if $y=0$ the term is $-\log(1-p)$, the mirror image. We want $\theta$ that makes the loss small.

Tutor addition · numbers

$p$	loss if $y=1$	loss if $y=0$
0.99	0.010	4.605
0.90	0.105	2.303
0.70	0.357	1.204
0.50	0.693	0.693
0.10	2.303	0.105
0.01	4.605	0.010

Always answering $p=0.5$ gives loss $\log 2\approx0.693$: a trained model must beat that. A confident mistake ($p=0.01$ for an approval) costs 4.6, almost 500 times a confident correct answer: cross-entropy punishes overconfidence hard. This is the same loss as Chapter 6 of the notes (cross-entropy for classification). Script: RNN_ex2.py · output.

Section 7 · slides 20–21

The trained example

From the lesson

Slide 20 trains the RNN live (choose $h=1,2,4$ or $12$ features) and shows, for test sentences, $M_\theta(s)$ for approvals and $1-M_\theta(s)$ for disapprovals, plus the features $h_n$. Slide 21 asks “what does $a_i$ look for?”: a table where each cell is $a_i\cdot x_n$, with the words of largest and smallest $a_i\cdot x$.

Tutor addition · the set-up, read from the deck's code

Data	865 short English sentences labelled 1 / 0 (e.g. “the app never crashes” 1, “I do not dislike my job” 1, “not worth the price” 0); the first 20% (173) are the test set, 692 train.
Words	EmbeddingGemma 768 → 48 numbers by PCA, fixed.
Model	$h=12$ features: $A_w$ 12×48, $A_h$ 12×12, $b,v\in\mathbb{R}^{12}$, $c$ → 745 trained numbers (52 for $h=1$, 105 for $h=2$, 217 for $h=4$).
Training	Adam, learning rate 0.01, batches of 16, 30 passes over the training set, loss = the cross-entropy above.

Accuracy depends on the run and is shown live in the deck; it was not re-run here.

Section 8 · slides 22–26

A character language model for Khmer

From the lesson

A language model is a function $M:\Sigma^*\to\mathbb{R}^{|\Sigma|}$ with $\sum_{c\in\Sigma}M(c_1\dots c_n)_c=1$. We use an RNN $M_\theta$ to approximate $M$. Text: $c_{n+1}\sim M_\theta(c_1\dots c_n)$, then $c_{n+2}$, … Training: make $-\frac1N\sum_t\log M_\theta(c_1\dots c_t)_{c_{t+1}}$ small.

$\Sigma$ = the Khmer characters, $x_n=E(c_n)\in\mathbb{R}^d$ with $E$ learned, and

$$h_n=\sigma(A_wx_n+A_hh_{n-1}+b)\in\mathbb{R}^h,\qquad y_n=Vh_n\in\mathbb{R}^k,\qquad p(c)=\frac{e^{u_c\cdot y_n+b_c}}{\sum_{c'\in\Sigma}e^{u_{c'}\cdot y_n+b_{c'}}}$$

Data: six Khmer tales from Wikipedia. A chunk of $n+1$ characters gives $n$ data points (each prefix predicts the next character); slides show $n=16,32,64$ and train in the browser or load a PyTorch model.

Tutor addition · numbers from the deck's trained models

$|\Sigma|=73$ characters (space, consonants, independent vowels, dependent vowels, signs, ។ ៖ ៗ). Guessing uniformly costs $\ln 73\approx4.29$ per character.
After 400 passes (PyTorch): loss 1.948 with $d=2$ and 1.417 with $d=16$. As perplexity $e^{\text{loss}}$: about 7.0 and 4.1, i.e. the model is as unsure as choosing among ~7 or ~4 equally likely characters instead of 73.
The deck's code uses $\sigma=\tanh$ by default (ReLU, SiLU, sigmoid selectable), Adam, back-propagation through time, and clips the gradient because ReLU can blow up through the recurrence.
Same equation as the classifier; only the head changes: instead of one score $v\cdot h_n+c$, a softmax over all 73 characters, and a prediction after every character instead of only at the end.

Sources of the data in the deck: Khmer Wikipedia tales (CC BY-SA). The six titles include រឿងភ្នំនាងកង្រី and រឿងខ្លា ស្វា និងទន្សាយ.

Practice

Check your understanding

Summary

Key ideas to keep

An RNN reads word by word and keeps a memory $h_n=\mathrm{ReLU}(A_wx_n+A_hh_{n-1}+b)$, $h_0=0$, with the same weights at every word.
Row $a_i$ of $A_w$ is a feature (a key): $a_i\cdot x_n$ is large when word $x_n$ looks like it. $A_h$ mixes in the past; ReLU with $b$ is a gate.
Decision: $s=v\cdot h_n+c$, $p=\sigma(s)$. Training: find $\theta=(A_w,A_h,b,v,c)$ minimising the cross-entropy over labelled data.
The same machinery with a softmax head over characters is a language model; on Khmer tales the loss falls from $\ln73\approx4.29$ (guessing) to 1.42.
Weakness to remember: the toy gate makes old information leak away, so word order and distance matter.

Homework 1 · 12 questions · graded 12 / 12

Neural Networks and Embedding

Read a network, build one from its architecture and count its parameters, then see the whole network as a point $P\in\mathbb{R}^K$ that becomes a continuous function $NN_P$.

Goes with Chapter 1Answers re-checked in PythonAdded 2 Nov 2026

From the homework

1 · Parts of a network (Q1–Q3)

The signal flows left to right: input layer → hidden layers → output layer. The example network maps $\mathbb{R}^2\to\mathbb{R}$ ($n=2$, $m=1$) with 3 hidden layers, counted as layers 0 to 4 (layer 0 is the input), so it has 4 layers in the course's counting.

Weight notation: $w^{i}_{k,j}$ is the connection from neuron $N^{i-1}_k$ (layer $i-1$) to neuron $N^{i}_j$ (layer $i$). So $w^{3}_{2,3}$ is the line from the 2nd neuron of layer 2 to the 3rd neuron of layer 3.

Worked solution

Q1–Q3 · reading a network diagram

What you need. Layers are counted from the input (layer 0). In $w^{i}_{k,j}$, the upper index is the layer the weight enters; the lower pair is (from-neuron, to-neuron). See Chapter 1.

1

Follow the signal from left to right: input → hidden layers → output.

Why: every layer only receives from the layer before it (feed-forward).

2

Read the sizes: 2 inputs, 3 hidden layers, 1 output, so $NN:\mathbb{R}^2\to\mathbb{R}$, layers 0–4.

Why: the input layer does no computation, so it has no weights.

3

Locate $w^{3}_{2,3}$: it enters layer 3, from neuron 2 of layer 2 to neuron 3 of layer 3.

Why: knowing the index order avoids the classic row/column swap.

Answer. $\mathbb{R}^2\to\mathbb{R}$, 4 layers after the input; $w^{3}_{2,3}$ joins $N^2_2$ to $N^3_3$.

✔ What it provesYou can translate between a drawing and the symbols $W^i, b^i, w^i_{k,j}$.

✖ What it does not sayCourses count layers differently (some count only hidden layers); always state your convention.

→ Where it leadsQ4–Q9: turning the drawing into matrices and parameter counts.

From the homework

2 · Building a network and counting parameters (Q4–Q9)

A network with layer sizes $s_0,s_1,\dots,s_L$ is a composition of affine maps and activations, e.g. $NN = Af_3\circ\sigma_2\circ Af_2\circ\sigma_1\circ Af_1$ with $Af_i(x)=W^i x+b^i$.

$$W^i\in\mathbb{R}^{s_i\times s_{i-1}},\qquad b^i\in\mathbb{R}^{s_i},\qquad \#Af_i = s_i\,s_{i-1}+s_i$$

Q	Question	Answer	Why
4	Build $2\to(3,5,3)\to1$	sizes 2-3-5-3-1	input 2, three hidden layers, output 1
5	$Af_1:\mathbb{R}^4\to\mathbb{R}^2$, $Af_2:\mathbb{R}^2\to\mathbb{R}^2$, $Af_3:\mathbb{R}^2\to\mathbb{R}^3$	sizes 4-2-2-3	read the dimensions in order
6	Shape of $W^2$	5 × 3	maps 3 neurons to 5: rows = output size
7	Parameters of $Af_2$	20	5·3 + 5
8	Whole network	51	9 + 20 + 18 + 4
9	One extra neuron in hidden layer 2 (3, 6, 3)	+7	$Af_2$: 20→24, $Af_3$: 18→21

Worked solution

Q4–Q9 · build the network and count parameters

What you need. $Af_i(x)=W^ix+b^i$ with $W^i\in\mathbb{R}^{s_i\times s_{i-1}}$, $b^i\in\mathbb{R}^{s_i}$: $s_is_{i-1}+s_i$ parameters per map.

1

Q4: $2\to(3,5,3)\to1$ means sizes 2-3-5-3-1.

2

Q6: $W^2$ maps 3 neurons to 5, so it is $5\times3$.

Why: rows = size of the new layer, columns = size of the previous one, so that $W^2x$ is defined.

3

Q7: $\#Af_2=5\cdot3+5=20$.

4

Q8: $9+20+18+4=51$ (figure).

Why: add every map; activations have no parameters.

5

Q9: one more neuron in hidden layer 2 (3, 6, 3): $Af_2$ 20 → 24, $Af_3$ 18 → 21: +7.

Why: a new neuron brings its incoming weights + bias (3 + 1) and its outgoing weights (3).

Figure. The Q4 network. Each affine map from s_i−1 to s_i neurons has s_i·s_i−1 weights (lines) and s_i biases (one per circle it feeds). Counts computed in this page.

Answer. $W^2$ is $5\times3$; $Af_2$ has 20 parameters; the network has 51; one extra neuron adds 7.

✔ What it provesParameter count is $\sum_i (s_is_{i-1}+s_i)$: linear in each layer’s width, quadratic in width overall.

✖ What it does not sayMore parameters do not guarantee a better model (overfitting, harder training); and the count ignores that some networks share weights (RNNs, CNNs).

→ Where it leadsLecture 2: an RNN reuses the same weights at every word, so its count does not grow with sentence length.

From the homework

3 · Embedding the parameters into functions (Q10–Q12)

For a fixed architecture every parameter vector $P=(W^1,\dots,W^L,b^1,\dots,b^L)$ is a point of $\mathbb{R}^K$, and it gives a continuous function $NN_P$ on $X$. This is a map $\mathbb{R}^K\to C(X),\ P\mapsto NN_P$. Training means moving $P$ until $NN_P$ is close to a target $f$ in the sup-distance $d_\infty(NN_P,f)=\max_{x\in X}|NN_P(x)-f(x)|$.

Q10. $NN_P(x)=\mathrm{ReLU}(x)+\mathrm{ReLU}(-2x+1)$ passes through $(-1,3)$, $(0,1)$, $(0.5,0.5)$, $(1,1)$: the bottom-left graph.
Q11. One neuron $\mathrm{ReLU}(a_1x+a_2)$: $P=(-1.5,\,0.5)$ matches the target exactly (distance 0.00, pass mark 0.2).
Q12. Fit $f(x)=\tfrac16\big((2x-1)^2+3|2x-1|\big)$ on $[-1,1]$ with two ReLU neurons. Submitted $(w_1,b_1,w_2,b_2,v_1,v_2,c)=(-3,1.5,1,0.5,1,1.5,-1.5)$; grade $=(1+\tfrac16)/(1+d_\infty)$.

Worked solution

Q10–Q12 · a parameter vector is a function

What you need. Fixing the architecture, each $P\in\mathbb{R}^K$ gives a function $NN_P$; distance between functions $d_\infty(g,f)=\max_{x\in X}|g(x)-f(x)|$.

1

Q10: evaluate $\mathrm{ReLU}(x)+\mathrm{ReLU}(-2x+1)$ at $x=-1,0,0.5,1$: 3, 1, 0.5, 1 → the bottom-left graph.

Why: four points pin down a 3-piece graph with kinks at 0 and ½.

2

Q11: one neuron $\mathrm{ReLU}(a_1x+a_2)$; $P=(-1.5,0.5)$ reproduces the target exactly (distance 0).

Why: read the kink position $-a_2/a_1$ and the slope from the target graph.

3

Q12: two neurons for $f(x)=\tfrac16\big((2x-1)^2+3|2x-1|\big)$ on $[-1,1]$; submitted $P=(-3,1.5,1,0.5,1,1.5,-1.5)$.

4

Measure the fit: $d_\infty=1/6$ exactly (figure), so grade $(1+\tfrac16)/(1+\tfrac16)=1$.

Why: the formula gives full marks when the distance reaches 1/6, which this network cannot beat.

target f(x)NN_P(x), two ReLU neurons

Figure. Q12: target f (curved, with a kink at x = ½) and the submitted two-neuron network. The worst error is exactly 1/6, so the grade (1 + 1/6)/(1 + d_∞) is 1. Computed on 2001 points in this page.

Answer. Q10: bottom-left graph · Q11: $P=(-1.5,0.5)$ · Q12: $d_\infty=1/6$, grade 1.

✔ What it provesTraining is a search in $\mathbb{R}^K$ for the point whose function is closest to the target.

✖ What it does not sayTwo ReLU neurons give at most 3 straight pieces, so a curved target always leaves some error; $d_\infty$ measures the worst point only, not the average.

→ Where it leadsHomework 2 searches $\mathbb{R}^7$ automatically with a loss; universal approximation with more neurons.

Tutor addition · checked by running code

The script below recomputes every number above. Q12's distance is exactly 1/6 = 0.1667 (the PDF says ≈ 0.17), which is the best this two-neuron network can reach, so the grade is exactly 1.

HW1_ex1.py

"""Homework 1 checks: parameter counts and the two ReLU networks (plain Python)."""
relu = lambda z: max(z, 0.0)

def params(sizes):
    # an affine map from s_{i-1} to s_i neurons has s_i*s_{i-1} weights + s_i biases
    return [s * p + s for p, s in zip(sizes, sizes[1:])]

for sizes in ([2, 3, 5, 3, 1], [2, 3, 6, 3, 1], [4, 2, 2, 3]):
    per = params(sizes)
    print(f"{'-'.join(map(str, sizes)):12s} per map {per}  total {sum(per)}")

# Q10: NN_P(x) = ReLU(x) + ReLU(-2x + 1)
q10 = lambda x: relu(x) + relu(-2 * x + 1)
print("Q10 points:", [(x, q10(x)) for x in (-1, 0, 0.5, 1)])

# Q12: best two-neuron fit of f(x) = ((2x-1)^2 + 3|2x-1|)/6 on [-1, 1]
f = lambda x: ((2 * x - 1) ** 2 + 3 * abs(2 * x - 1)) / 6
w1, b1, w2, b2, v1, v2, c = -3, 1.5, 1, 0.5, 1, 1.5, -1.5
nn = lambda x: v1 * relu(w1 * x + b1) + v2 * relu(w2 * x + b2) + c
xs = [-1 + 2 * i / 20000 for i in range(20001)]
d = max(abs(nn(x) - f(x)) for x in xs)
print(f"Q12 sup-distance on [-1, 1]: {d:.4f}  (1/6 = {1/6:.4f})  grade = {(1 + 1/6) / (1 + d):.4f}")

output (run 2 Nov 2026)

2-3-5-3-1    per map [9, 20, 18, 4]  total 51
2-3-6-3-1    per map [9, 24, 21, 4]  total 58
4-2-2-3      per map [10, 6, 9]  total 25
Q10 points: [(-1, 3.0), (0, 1), (0.5, 0.5), (1, 1.0)]
Q12 sup-distance on [-1, 1]: 0.1667  (1/6 = 0.1667)  grade = 1.0000

Files: HW1_ex1.py · output

Tutor addition

Why Q12 cannot reach 0

Two ReLU neurons give a piecewise-linear graph with at most 2 bends (3 straight pieces). The target is curved (a parabola plus a kink at $x=\tfrac12$), so some error must remain. More neurons → more pieces → smaller possible $d_\infty$. This is the first step toward the universal approximation idea.

The self-check questions of this homework are now in Part 3 · Practice → Revision mode (flashcards).

Source: Homework1_Neural_Networks_Summary.pdf (3 pages, all read).

Homework 2 · 11 questions · graded 11 / 11

Training Neural Networks

A network is a function with parameters; a loss scores it on data; training searches parameter space, first one coordinate at a time, then with all coordinates moving together.

Goes with Chapters 1–4Numbers re-checked in PythonAdded 2 Nov 2026

From the homework

1 · The network (Q1–Q3)

$$NN_P(x)=v_1\,\mathrm{ReLU}(w_1x+b_1)+v_2\,\mathrm{ReLU}(w_2x+b_2)+c$$ $$P=(w_1,b_1,w_2,b_2,v_1,v_2,c)\in\mathbb{R}^7$$

$w$ changes the steepness of a neuron, $b$ moves its kink left/right, $v$ scales it vertically ($v=0$ switches it off), $c$ shifts the whole graph.
Piecewise linear with at most 2 kinks. $|x|$ is exact; $x^2$ can only be approximated.

Worked solution

Q1–Q3 · what each parameter does

What you need. $NN_P(x)=v_1\mathrm{ReLU}(w_1x+b_1)+v_2\mathrm{ReLU}(w_2x+b_2)+c$, $P\in\mathbb{R}^7$.

1

Change one parameter at a time and watch the graph.

Why: isolating one parameter shows its role.

2

$w$ changes the slope of a hinge, $b$ moves its kink ($x=-b/w$), $v$ scales it ($v=0$ switches it off), $c$ shifts the whole graph.

3

Shape: piecewise linear with at most 2 kinks; $|x|$ is exact, $x^2$ only approximate.

Why: see Problem 2 of the Checking sheet for the proof.

Answer. w = steepness, b = kink position, v = height/on-off, c = vertical shift; at most 3 straight pieces.

✔ What it provesEach of the 7 numbers has a visible geometric effect.

✖ What it does not sayParameters interact: moving $b$ also changes the value at every $x$ past the kink, so “one parameter = one effect” is only approximate.

→ Where it leadsQ4–Q6: score a choice of $P$ with a loss.

From the homework

2 · Loss (Q4–Q6)

$$\mathrm{loss}_D(P)=\frac1N\sum_{i=1}^{N}\big(y_i-NN_P(x_i)\big)^2\ \ge 0$$

Squaring stops positive and negative errors cancelling. Worked example, errors $-0.70,-0.80,-0.05,0.45,0.55$:

Loss	Formula	Value
$L^2$ (mean squared)	$\frac1N\sum e_i^2$	1.6375 / 5 = 0.3275
$L^1$ (mean absolute)	$\frac1N\sum \|e_i\|$	0.51
$L^\infty$ (max)	$\max_i \|e_i\|$	0.8
signed mean (don't use)	$\frac1N\sum e_i$	−0.11: errors cancel

Worked solution

Q4–Q6 · scoring a network with a loss

What you need. Errors $e_i=y_i-NN_P(x_i)$; losses $L^2=\frac1N\sum e_i^2$, $L^1=\frac1N\sum|e_i|$, $L^\infty=\max|e_i|$.

1

Compute the errors on the five data points: −0.70, −0.80, −0.05, 0.45, 0.55.

2

Square and average: $L^2=(0.49+0.64+0.0025+0.2025+0.3025)/5=0.3275$.

Why: squaring makes every error count positively and punishes big errors more.

3

Compare: $L^1=0.51$, $L^\infty=0.8$; the signed mean is $-0.11$ (figure).

Why: the signed mean lets positive and negative errors cancel, so a bad model can score near 0.

4

Check: loss $\ge0$, and $=0$ only if every error is 0.

Figure. The same five errors (−0.70, −0.80, −0.05, 0.45, 0.55) scored four ways, computed in this page. The signed mean (red) lets errors cancel, which is why it is never used as a loss.

Answer. $L^2=0.3275$, $L^1=0.51$, $L^\infty=0.8$; never use the signed mean.

✔ What it provesA loss turns “how good is $P$” into one number we can minimise.

✖ What it does not sayDifferent losses rank models differently (outliers weigh more in $L^2$ and $L^\infty$); the choice is part of the model.

→ Where it leadsCross-entropy for classification (bridge lesson, Lecture 2).

From the homework

3 · Coordinate descent (Q7–Q8)

Fix all parameters but one, move that one to the minimum of the 1-D loss curve, go to the next, repeat. Each move never increases the loss; a second move on the same coordinate changes nothing; the order of coordinates leads to a different $P$ with about the same loss; it can stall short of the global minimum. Result: loss 0.328 → about 0.03 in one round, from either order.

Worked solution

Q7–Q8 · coordinate descent

What you need. Minimise along one coordinate at a time, keeping the others fixed; cycle through all coordinates.

1

Fix 6 parameters, plot the loss as a function of the 7th, move it to that curve’s minimum.

Why: a 1-D minimum is easy to find (scan or formula).

2

Go to the next coordinate and repeat; one round = all 7.

3

Observe: each move never increases the loss; redoing the same coordinate changes nothing; another order gives another $P$ with similar loss. Loss 0.328 → ≈ 0.03 in one round.

Why: the loss can only go down, but the path depends on the order.

Answer. One round of coordinate descent cuts the loss from 0.328 to about 0.03; the result depends on the order.

✔ What it provesCoordinate descent is a simple, monotone optimiser.

✖ What it does not sayIt can stall at a point that is not a minimum when the valley is diagonal (no single-axis move helps), and it scales badly to millions of parameters.

→ Where it leadsMoving all coordinates together: random directions (Q9–Q11) and then the gradient (bridge lesson).

From the homework

4 · Evolutionary selection (Q9–Q11)

Try many random directions $v$ and move to the best $P+\eta v$; each round is a generation; $\eta$ is the learning rate.

Method	What the runs showed
Fixed $\eta$ (Q9)	0.01: slow but settles (0.741 after 50 gen.); 0.1: fast then jitters (0.464); 0.5 and 2: bounce and stay high (0.527, 1.088). Start large, end small.
Adaptive $\eta$ (Q10)	Success → $\eta\times1.5$, failure → $\eta\times0.7$. From 0.01 it grows, from 2 it shrinks; both meet after ~10 generations; near the bottom $\eta\to10^{-6}$ and the loss freezes. 0.415 vs 0.741 after 50 generations.
Average of $k$ best (Q11)	Weights $a_j=\dfrac{2(k+1-j)}{k(k+1)}$; $k=3$: ½, ⅓, ⅙. $k=5,20$ jump less (noise cancels); $k=50$ averages in poor points and stays higher.

Worked solution

Q9–Q11 · random directions and the learning rate

What you need. Try many random directions $v$, move to the best $P+\eta v$; $\eta$ = step size (learning rate).

1

Fixed $\eta$ (Q9): after 50 generations 0.741 (η = 0.01), 0.464 (0.1), 0.527 (0.5), 1.088 (2) (figure).

Why: small steps are safe but slow; large steps jump across the valley.

2

Adaptive $\eta$ (Q10): success → $\eta\times1.5$, failure → $\eta\times0.7$; both starts meet after ≈ 10 generations; 0.415 after 50.

Why: the step size tunes itself to the local shape of the loss.

3

Average of the $k$ best (Q11): weights $a_j=\frac{2(k+1-j)}{k(k+1)}$; for $k=3$: ½, ⅓, ⅙.

Why: scores $k,\dots,1$ sum to $k(k+1)/2$, so dividing makes the weights sum to 1.

4

Compare k: $k=5,20$ jump less (noise cancels); $k=50$ averages in poor directions and stays higher.

Figure. Evolutionary search with a fixed step η (the homework’s reported runs). Too small: slow; too large: bounces and stays high. The adaptive rule reached 0.415.

Answer. A good step size matters most: adaptive η reaches 0.415 vs 0.741 with η = 0.01; weights for k = 3 are ½, ⅓, ⅙.

✔ What it provesMoving all parameters at once with a tuned step beats one-at-a-time search.

✖ What it does not sayThese are single runs with random directions: the exact numbers change with the seed, and near the bottom the adaptive η shrinks to ~10⁻⁶ and freezes.

→ Where it leadsGradient descent: the best direction computed instead of sampled (bridge lesson, Checking Problem 4).

Tutor addition · checked by running code

HW2_ex1.py

"""Homework 2 checks: the three losses on the worked example, and the k-best weights."""
from fractions import Fraction as F

errors = [-0.70, -0.80, -0.05, 0.45, 0.55]
n = len(errors)
print("squares:", [round(e * e, 4) for e in errors])
print("L2 (mean squared error):", round(sum(e * e for e in errors) / n, 4))
print("L1 (mean absolute error):", round(sum(abs(e) for e in errors) / n, 4))
print("Linf (max absolute error):", max(abs(e) for e in errors))
print("signed mean (why we square):", round(sum(errors) / n, 4))

for k in (1, 3, 5):
    scores = [k + 1 - j for j in range(1, k + 1)]
    total = sum(scores)
    print(f"k={k}: scores {scores}, sum {total} = k(k+1)/2, weights {[str(F(s, total)) for s in scores]}")

output (run 2 Nov 2026)

squares: [0.49, 0.64, 0.0025, 0.2025, 0.3025]
L2 (mean squared error): 0.3275
L1 (mean absolute error): 0.51
Linf (max absolute error): 0.8
signed mean (why we square): -0.11
k=1: scores [1], sum 1 = k(k+1)/2, weights ['1']
k=3: scores [3, 2, 1], sum 6 = k(k+1)/2, weights ['1/2', '1/3', '1/6']
k=5: scores [5, 4, 3, 2, 1], sum 15 = k(k+1)/2, weights ['1/3', '4/15', '1/5', '2/15', '1/15']

Files: HW2_ex1.py · output. The generation-by-generation losses in the table come from your homework runs (random directions), so they are not re-run here.

Tutor addition · how this connects

Coordinate descent and evolutionary selection need no derivatives. Gradient descent (Chapter 4 of the notes) replaces “try 100 random directions” by the single best local direction $-\nabla\,\mathrm{loss}_D(P)$, and the learning-rate lessons from Q9–Q10 carry over unchanged.

The self-check questions of this homework are now in Part 3 · Practice → Revision mode (flashcards).

Source: homework2_training_neural_networks_lesson.md (all sections read).

Part 2 · Problems with solutions · 10 problems

Checking: Neural Networks — worked solutions

Full step-by-step solutions to the ten problems of the sheet. Every step says what we do and why, every problem has a figure that shows the idea, and every problem ends with the edge of the result: what it proves, what it does not say, and where it leads.

Uses Lessons: Ch 1, Bridge, Lecture 1–210 figures drawn from computed numbersEvery number re-run in Python

Each problem: the problem → what you need → solution in steps (with “Why”) → figure → answer → edge of the knowledge (✔ what it proves · ✖ what it does not say · → where it leads). To test yourself afterwards, go to Part 3 · Practice and choose “Problems · Checking NNs”.

Problem 1Sizes, parameters, and why we need an activation

The problem

A network has input in $\mathbb{R}^3$, one hidden layer with $4$ neurons and output in $\mathbb{R}^2$. Find the sizes of $W^1$ and $W^2$, and count the parameters.

Without an activation function, $NN(x)=W^2(W^1x+b^1)+b^2$. Prove that this is an affine map $Wx+b$. What does this say about adding more layers?

What you need. A layer is $x\mapsto Wx+b$; its matrix has one row per output neuron and one column per input number (Chapter 1). An affine map is “matrix times $x$ plus a constant vector”.

1

Size of $W^1$. Layer 1 sends $\mathbb{R}^3\to\mathbb{R}^4$, so $W^1\in\mathbb{R}^{4\times3}$ and $b^1\in\mathbb{R}^4$.

Why: $W^1x$ must be a vector of 4 numbers (one per hidden neuron), and $x$ has 3 numbers, so the matrix is out × in $=4\times3$. Row $j$ holds the 3 weights of neuron $j$.

2

Size of $W^2$. Layer 2 sends $\mathbb{R}^4\to\mathbb{R}^2$, so $W^2\in\mathbb{R}^{2\times4}$ and $b^2\in\mathbb{R}^2$.

Why: same rule: 2 outputs, 4 inputs.

3

Count. $\underbrace{4\cdot3}_{W^1}+\underbrace{4}_{b^1}+\underbrace{2\cdot4}_{W^2}+\underbrace{2}_{b^2}=12+4+8+2=26.$

Why: each entry of a weight matrix and each bias is one trainable number. General rule: a layer $\mathbb{R}^{n}\to\mathbb{R}^{m}$ has $mn+m$ parameters.

4

Expand the composition. $NN(x)=W^2(W^1x+b^1)+b^2=W^2W^1x+W^2b^1+b^2.$

Why: matrix multiplication distributes over addition: $W^2(u+v)=W^2u+W^2v$.

5

Name the pieces. Put $W=W^2W^1\in\mathbb{R}^{2\times3}$ and $b=W^2b^1+b^2\in\mathbb{R}^2$. Then $NN(x)=Wx+b$: an affine map. ∎

Why: $W$ does not depend on $x$ (a product of fixed matrices) and $b$ does not depend on $x$ (a fixed vector), which is exactly the definition of affine.

6

More layers. By induction, if $k$ linear layers give $W_kx+b_k$, one more gives $W^{k+1}(W_kx+b_k)+b^{k+1}$, affine again. So any depth without activation is still one affine map.

Why it matters: stacking layers adds parameters but no new functions. The power of a neural network comes from the non-linear activation between layers.

Figure 1. Left: 26 parameters. Right: the same function written as one affine layer. Without an activation the hidden layer disappears.

Answer. $W^1$: $4\times3$, $W^2$: $2\times4$, 26 parameters. Without activation $NN(x)=(W^2W^1)x+(W^2b^1+b^2)$; adding linear layers never makes the network more expressive.

✔ What it provesDepth without activations is useless: a 100-layer linear network can only compute one affine map (here $\mathbb{R}^3\to\mathbb{R}^2$).

✖ What it does not sayIt does not say linear stacks are as good as one layer: a narrow hidden layer limits the rank of $W=W^2W^1$ to the hidden width, so a linear stack can be weaker, never stronger. (Here rank ≤ 2 is no limit.) It also says nothing about how training behaves.

→ Where it leadsPut a non-linear $\sigma$ between layers (Problem 2). The rank idea returns in low-rank factorisation (small embedding size $d$, LoRA).

Problem 2Two ReLU neurons: what they can and cannot draw

The problem

Let $NN_P(x)=v_1\,\text{ReLU}(w_1x+b_1)+v_2\,\text{ReLU}(w_2x+b_2)+c$. Find $P$ such that $NN_P(x)=|x|$. At most how many straight pieces can the graph of $NN_P$ have? Can $NN_P$ equal $x^2$ on an interval? Explain.

What you need. $\mathrm{ReLU}(t)=\max(t,0)$. The graph of $\mathrm{ReLU}(wx+b)$ is a “hinge”: flat on one side of $x=-b/w$, a straight line on the other (Homework 2).

1

Split $|x|$ by sign. $|x|=x$ for $x\ge0$ and $|x|=-x$ for $x<0$.

Why: we want each neuron to handle one side.

2

One hinge per side. $\mathrm{ReLU}(x)$ equals $x$ for $x\ge0$ and $0$ otherwise; $\mathrm{ReLU}(-x)$ equals $-x$ for $x<0$ and $0$ otherwise. Their sum is $|x|$ everywhere.

Why: for each $x$ exactly one of the two is “on”, and it gives the right value; the other adds 0.

3

Read off $P$. $(w_1,b_1,w_2,b_2,v_1,v_2,c)=(1,0,-1,0,1,1,0)$. (Checked on 13 points from −3 to 3: exact.)

4

Count kinks. Neuron $j$ changes slope only at $x=-b_j/w_j$ (if $w_j\ne0$). Two neurons → at most 2 kinks → at most 3 straight pieces.

Why: between kinks every neuron is either off (0) or a straight line, and a sum of straight lines is a straight line. $k$ cut points split a line into at most $k+1$ parts.

5

Can it be $x^2$ on an interval $I$? Suppose yes. $I$ has positive length but only ≤ 2 kinks, so it contains a sub-interval $J$ with no kink, where $NN_P(x)=\alpha x+\beta$. Then $x^2=\alpha x+\beta$ for all $x\in J$: the polynomial $x^2-\alpha x-\beta$ has infinitely many roots, impossible for a non-zero polynomial. So no.

Why (second view): on $J$ the second derivative of $NN_P$ is $0$, but $(x^2)''=2$. A curved function is curved on every interval; a ReLU network is straight on every small enough interval.

ReLU(x)ReLU(−x)sum = |x| (wide pale band)

Figure 2a. Two hinges add up to $|x|$ (kink of both neurons at $x=0$).

x²NN with 2 kinks (3 pieces)

Figure 2b. Two kinks → at most 3 straight pieces. They can follow $x^2$ (curved everywhere) but never equal it on an interval.

Answer. $P=(1,0,-1,0,1,1,0)$; at most 3 pieces; $NN_P$ can never equal $x^2$ on an interval — it can only approximate it.

✔ What it provesWith $n$ ReLU neurons in one hidden layer (1-D input), the graph has at most $n+1$ pieces and is never exactly a curved function on any interval.

✖ What it does not sayIt does not mean ReLU nets are bad at curves: with enough pieces they approximate $x^2$ as closely as you like on $[a,b]$ (the error of a good $n$-piece fit shrinks like $1/n^2$). “Cannot equal” ≠ “cannot approximate”.

→ Where it leadsThe universal approximation theorem (one wide hidden layer approximates any continuous function on a compact set). Depth multiplies the number of pieces much faster than width.

Problem 3The best constant: mean for $L^2$, median for $L^1$

The problem

The model is a constant $c$ and the data is $y_1,\ldots,y_N$. Prove that $L^2(c)=\frac1N\sum_i(y_i-c)^2$ is smallest when $c$ is the mean of the $y_i$.

For $y=0,1,2,5$, find the $c$ that minimizes $L^2$ and all the $c$ that minimize $L^1(c)=\frac1N\sum_i|y_i-c|$. Which loss is moved more by the point $5$?

What you need. A smooth convex function is smallest where its derivative is 0. The derivative of $|y-c|$ in $c$ is $+1$ if $c>y$ and $-1$ if $c<y$ (Homework 2, losses).

1

Differentiate $L^2$. $\dfrac{dL^2}{dc}=\dfrac1N\sum_i2(y_i-c)(-1)=-\dfrac2N\sum_i(y_i-c).$

Why: chain rule on each square; the inner derivative of $y_i-c$ in $c$ is $-1$.

2

Set it to 0. $\sum_i(y_i-c)=0\iff\sum_iy_i=Nc\iff c=\bar y=\frac1N\sum_iy_i.$

3

Check it is a minimum. $\dfrac{d^2L^2}{dc^2}=2>0$, so $L^2$ is a parabola opening up and $\bar y$ is its unique minimum. ∎

Why (without calculus): $\sum_i(y_i-c)^2=\sum_i(y_i-\bar y)^2+N(\bar y-c)^2$ (expand and use $\sum_i(y_i-\bar y)=0$). The first term does not depend on $c$; the second is $\ge0$ and is $0$ only at $c=\bar y$.

4

Numbers for $L^2$. $\bar y=\frac{0+1+2+5}{4}=2$ and $L^2(2)=\frac{(0-2)^2+(1-2)^2+(2-2)^2+(5-2)^2}{4}=\frac{4+1+0+9}{4}=3.5.$

5

Slope of $L^1$. $\dfrac{dL^1}{dc}=\dfrac1N\big(\#\{y_i<c\}-\#\{y_i>c\}\big)$ away from the data points.

Why: each point below $c$ contributes $+1$ (moving $c$ right increases its distance), each point above contributes $-1$.

6

Where the slope is 0. For $c<1$: at most 1 point below, at least 3 above → slope < 0 (going down). For $1<c<2$: 2 below, 2 above → slope $=0$ (flat). For $c>2$: slope > 0. So every $c\in[1,2]$ minimizes $L^1$, with $L^1=\frac{1+0+1+4}{4}=1.5$.

Why: these $c$ are the medians: half of the data on each side. With an even number of points the median is a whole interval.

7

Effect of the point 5. In $L^2$ its error is squared: $3^2=9$ of the total $14$, and it pulls the mean up. In $L^1$ only its side matters. Test: replace 5 by 100 → the mean jumps to $25.75$, the $L^1$ minimizers stay $[1,2]$. So $L^2$ is moved more.

Why: the gradient of $L^2$ is proportional to each error, so a far point pulls hard; the gradient of $L^1$ gives every point a vote of size 1.

L²(c) mean squaredL¹(c) mean absolute

Figure 3. $L^2(c)$ is a parabola with one bottom at the mean. $L^1(c)$ is made of straight pieces and is flat between the two middle points, so every median wins. Black dots: the data $0,1,2,5$.

Answer. $L^2$: $c=2$ (the mean), $L^2=3.5$. $L^1$: every $c\in[1,2]$ (the medians), $L^1=1.5$. The point 5 moves $L^2$ much more.

✔ What it provesChoosing the loss chooses the statistic: $L^2$ → mean (sensitive to outliers), $L^1$ → median (robust). The $L^1$ minimizer need not be unique.

✖ What it does not say“$L^1$ is always better” is wrong: it has a corner (not differentiable at 0) and a non-unique minimum; $L^2$ is smooth and is the right choice when errors are Gaussian. Each loss matches a noise model.

→ Where it leadsMaximum likelihood: $L^2$ ↔ Gaussian noise, $L^1$ ↔ Laplace noise, cross-entropy ↔ Bernoulli labels (Problem 10). Huber loss mixes $L^2$ and $L^1$.

Problem 4Steepest descent and the learning rate

The problem

Let $\|V\|=1$. From $D_Vf(P_0)=\nabla f(P_0)\cdot V$, prove that $f$ decreases fastest in the direction $-\nabla f(P_0)/\|\nabla f(P_0)\|$.

Let $f(x,y)=x^2+3y^2$ and $P_0=(1,1)$. Compute $f(P_1)$ for $P_1=P_0-\eta\nabla f(P_0)$, with $\eta=0.1$ and with $\eta=0.5$. What does this say about the learning rate?

What you need. The directional derivative $D_Vf$ is the rate of change of $f$ when moving in direction $V$. Cauchy–Schwarz: $|u\cdot V|\le\|u\|\,\|V\|$, with equality only when $u$ and $V$ are parallel (Bridge · gradients).

1

Bound the rate of change. By Cauchy–Schwarz, $D_Vf(P_0)=\nabla f\cdot V\ge-\|\nabla f\|\,\|V\|=-\|\nabla f\|.$

Why: no unit direction can make $f$ go down faster than $\|\nabla f\|$ per unit step.

2

Why: $\nabla f\cdot V=\|\nabla f\|\cos\theta$, smallest when $\cos\theta=-1$, i.e. angle $180°$.

3

Gradient at $P_0$. $\nabla f=(2x,\,6y)=(2,\,6)$ and $f(P_0)=1+3=4$.

4

$\eta=0.1$. $P_1=(1,1)-0.1(2,6)=(0.8,\,0.4)$, $f(P_1)=0.64+3(0.16)=1.12$. The value drops from 4 to 1.12.

5

$\eta=0.5$. $P_1=(1,1)-0.5(2,6)=(0,\,-2)$, $f(P_1)=0+3(4)=12$. The value rises from 4 to 12.

Why: the gradient is only a local direction. A big step goes past the bottom of the valley and lands higher on the other side.

6

See it per coordinate. One step maps $x\mapsto(1-2\eta)x$ and $y\mapsto(1-6\eta)y$. For $\eta=0.5$ the $y$-factor is $-2$: $y$ flips sign and doubles every step (12, 48, 192, … — it diverges). For $\eta=0.1$ both factors are in $(0,1)$ (1.12, 0.49, 0.27, … — it converges).

Why: convergence needs $|1-2\eta|<1$ and $|1-6\eta|<1$, i.e. $0<\eta<1/3$. The steepest direction (curvature 6) sets the limit.

η = 0.1 (converges)η = 0.5 (overshoots)

Figure 4. Level curves of $f=x^2+3y^2$ (dashed). Green: six steps with $\eta=0.1$, sliding down. Coral: one step with $\eta=0.5$ jumps across the valley to a higher level ($f=12$).

Answer. $-\nabla f/\|\nabla f\|$ is the steepest-descent direction. $f(P_1)=1.12$ for $\eta=0.1$, but $12$ for $\eta=0.5$. The learning rate must be small enough (here $\eta<1/3$), or gradient descent overshoots and can diverge.

✔ What it proves$-\nabla f$ is the best direction for an infinitely small step. For this quadratic, gradient descent converges exactly when $0<\eta<1/3$ (in general $\eta<2/L$, $L$ = largest curvature).

✖ What it does not say“Steepest” is local: for a finite step the best direction is not $-\nabla f$ (from $(1,1)$ the minimum is along $-(1,1)$, not $-(2,6)$). It says nothing where $\nabla f=0$ (minimum, maximum or saddle).

→ Where it leadsIll-conditioning (curvatures 2 vs 6 make the path zig-zag), momentum and Adam that adapt the step per coordinate (Bridge), Newton’s method, learning-rate schedules.

Problem 5One gradient step by hand, and a dead neuron

The problem

Let $NN(x)=\text{ReLU}(wx+b)$, with one data point $(x,y)=(2,3)$ and $\text{loss}=(y-NN(x))^2$. Start at $w=1$, $b=0$. Compute $\partial\,\text{loss}/\partial w$ and $\partial\,\text{loss}/\partial b$, take one step with $\eta=0.1$, and find the new loss.

Now start at $w=-1$, $b=0$. Compute the gradient. Why can training not move from here?

What you need. Chain rule (back-propagation): multiply the local slopes along the path from loss to parameter. $\mathrm{ReLU}'(z)=1$ for $z>0$, $0$ for $z<0$. Update: $\theta\leftarrow\theta-\eta\,\partial\,\mathrm{loss}/\partial\theta$ (Bridge).

1

Forward pass. $z=wx+b=1\cdot2+0=2$, $NN=\mathrm{ReLU}(2)=2$, $\mathrm{loss}=(3-2)^2=1$.

Why first: the backward pass needs the values $z$ and $NN$ to evaluate each local slope.

2

Local slopes. $\dfrac{\partial\,\mathrm{loss}}{\partial NN}=-2(y-NN)=-2$, $\dfrac{\partial NN}{\partial z}=\mathrm{ReLU}'(2)=1$, $\dfrac{\partial z}{\partial w}=x=2$, $\dfrac{\partial z}{\partial b}=1$.

3

Chain rule. $\dfrac{\partial\,\mathrm{loss}}{\partial w}=(-2)(1)(2)=-4$, $\dfrac{\partial\,\mathrm{loss}}{\partial b}=(-2)(1)(1)=-2$.

Why negative: the output is too small (2 < 3), so increasing $w$ or $b$ lowers the loss; the gradient points to “decrease”, so its sign is negative.

4

Update. $w\leftarrow1-0.1(-4)=1.4$, $b\leftarrow0-0.1(-2)=0.2$.

5

New loss. $NN=\mathrm{ReLU}(1.4\cdot2+0.2)=3.0$, so $\mathrm{loss}=(3-3)^2=0$.

Why exactly 0: a lucky hit for these numbers. In general one step only reduces the loss (for small enough $\eta$).

6

Start at $w=-1$, $b=0$. $z=-2<0$, $NN=0$, $\mathrm{loss}=9$. Now $\mathrm{ReLU}'(-2)=0$, so $\dfrac{\partial\,\mathrm{loss}}{\partial w}=(-6)(0)(2)=0$ and $\dfrac{\partial\,\mathrm{loss}}{\partial b}=(-6)(0)(1)=0$.

7

Why training is stuck. The update is $\theta\leftarrow\theta-\eta\cdot0=\theta$: nothing moves, forever. The loss is large (9), but the closed gate multiplies the error signal by 0. This is a dead ReLU.

Why: the gradient only sees the local slope. In the flat region small changes of $w,b$ do not change the output, so the gradient has no information about the fix (making $z$ positive).

Figure 5a. Forward pass (green numbers) and backward pass (coral): the chain rule multiplies one local slope per box.

Figure 5b. On the coral side the ReLU is flat, so every gradient that passes through it is multiplied by 0.

Answer. $\partial\,\mathrm{loss}/\partial w=-4$, $\partial\,\mathrm{loss}/\partial b=-2$; after one step $w=1.4$, $b=0.2$, new loss $0$. From $w=-1$ the gradient is $(0,0)$: the neuron is dead, so gradient descent cannot move.

✔ What it provesZero gradient ≠ good model. A ReLU neuron whose input is negative for every data point never learns again by gradient descent.

✖ What it does not sayAt exactly $z=0$ ReLU has no derivative; libraries choose $0$ by convention. With many data points a neuron is dead only if it is closed for all of them; here one point is enough.

→ Where it leadsLeaky ReLU / GELU / SiLU (a small slope for $z<0$), careful initialisation; and the same “product of slopes” problem over time: vanishing gradients in RNNs (Lecture 2).

Problem 6Words as vectors: inner product vs distance

The problem

Why do we represent words by vectors and not by the numbers $1,2,3,\ldots$?

Let $|u|=|v|=1$. Prove that $|u-v|^2=2-2\,u\cdot v$. What does this say about $u\cdot v$ when two words are close?

Let $a=(0.8,\,0.6)$, $x=(1.6,\,1.2)$ and $y=(0.9,\,3.7)$. Compute $a\cdot x$, $a\cdot y$, $|a-x|$ and $|a-y|$. Which vector has the larger inner product with $a$, and which is closer to $a$? Explain why they differ.

What you need. $|w|^2=w\cdot w$; $u\cdot v=|u||v|\cos\theta$. An embedding gives every word a vector in $\mathbb{R}^d$ (Lecture 1).

1

Why not numbers. Numbers $1,2,3$ put all words on one line with an order and fixed distances: word 2 would be “between” words 1 and 3, and $3=1+2$ would mean something. These relations are accidents of the numbering, not of meaning. Also one number cannot hold many independent properties (noun? positive? about food?).

Why vectors work: $\mathbb{R}^d$ has $d$ directions, so each direction can carry one kind of property, and similarity can be measured by geometry and learned from data.

2

Expand the distance. $|u-v|^2=(u-v)\cdot(u-v)=u\cdot u-2u\cdot v+v\cdot v=|u|^2-2u\cdot v+|v|^2=2-2u\cdot v.$ ∎

Why: the dot product is bilinear and symmetric; then use $|u|=|v|=1$.

3

Interpret. Distance small ⇔ $u\cdot v$ close to its maximum 1. Examples: $u\cdot v=1$ → distance 0; $u\cdot v=0$ → distance $\sqrt2$; $u\cdot v=-1$ → distance 2.

Why: for unit vectors $u\cdot v=\cos\theta$, so “close” = “small angle”.

4

Inner products. $a\cdot x=0.8(1.6)+0.6(1.2)=1.28+0.72=2.00$; $a\cdot y=0.8(0.9)+0.6(3.7)=0.72+2.22=2.94$.

5

Distances. $a-x=(-0.8,-0.6)$, $|a-x|=\sqrt{0.64+0.36}=1$; $a-y=(-0.1,-3.1)$, $|a-y|=\sqrt{0.01+9.61}=\sqrt{9.62}\approx3.10$.

6

Compare. $y$ has the larger inner product, but $x$ is closer.

Why they differ: $x$ and $y$ are not unit vectors ($|x|=2$, $|y|\approx3.81$), so step 2 does not apply. $a\cdot y=|a||y|\cos\theta$ is big because $y$ is long: $\cos(a,y)=2.94/3.81\approx0.77$, while $\cos(a,x)=1$ because $x=2a$ points exactly the same way.

x: a·x = 2.00, distance 1y: a·y = 2.94, distance 3.10

Figure 6. $x$ lies on the same ray as $a$ (cos = 1) and is close to it. $y$ points elsewhere (cos ≈ 0.77) but is long ($|y|\approx3.81$), so $a\cdot y=2.94$ beats $a\cdot x=2$. Dashed segments: the distances.

Answer. $a\cdot x=2$, $a\cdot y=2.94$, $|a-x|=1$, $|a-y|\approx3.10$. $y$ has the larger inner product, $x$ is closer. They disagree because the inner product mixes direction and length; only for unit vectors is “large $u\cdot v$” the same as “close”.

✔ What it provesFor unit vectors, ranking by inner product = ranking by distance = ranking by cosine. For general vectors these rankings can disagree.

✖ What it does not sayIt does not say that close vectors are words with the same meaning — that only holds after training, and only for the similarity the training rewarded (“hot” and “cold” are often close: they appear in the same contexts).

→ Where it leadsCosine similarity for word search; in attention (transformers, Ch 10) the raw score $q\cdot k$ is used and divided by $\sqrt d$, so length does matter there.

Problem 7The ReLU gate is the answer to a sparsity problem

The problem

Let $z=a\cdot x$ and $h=\text{ReLU}(z+b)$. For which $z$ is the gate open?

For $\lambda>0$, prove that the $h\ge 0$ minimizing $\tfrac12(h-z)^2+\lambda|h|$ is $h=\text{ReLU}(z-\lambda)$. Which $b$ does this gate have?

Find the $h\ge 0$ minimizing $\tfrac12(h-z)^2+\lambda h^2$ when $z>0$. Can this $h$ be $0$? So which penalty should we use to have only a few concepts present at each word?

What you need. $z=a\cdot x$ measures how much word $x$ matches feature $a$ (Lecture 2). To minimise a convex function on $[0,\infty)$: if its derivative is $\ge0$ at $0$ the minimum is at $0$, otherwise at the zero of the derivative.

1

Gate open. $h=\mathrm{ReLU}(z+b)>0\iff z+b>0\iff z>-b$.

Why: $-b$ is a threshold: the feature is “present” only if the word matches it by more than $-b$.

2

Simplify the $L^1$ problem. For $h\ge0$, $|h|=h$, so minimise $g(h)=\tfrac12(h-z)^2+\lambda h$ on $h\ge0$. Its derivative is $g'(h)=h-z+\lambda$, and $g''=1>0$ (convex).

3

Case $z>\lambda$. $g'(h)=0$ at $h=z-\lambda>0$, which is allowed, so $h=z-\lambda$.

4

Case $z\le\lambda$. $g'(h)=h-(z-\lambda)\ge h\ge0$ for all $h\ge0$, so $g$ never decreases on $[0,\infty)$: the minimum is at $h=0$.

Why: the pull towards $z$ is weaker than the price $\lambda$ per unit of $h$, so it is cheapest to switch the feature off.

5

Combine. $h=\max(0,z-\lambda)=\mathrm{ReLU}(z-\lambda)$. ∎ Comparing with $\mathrm{ReLU}(z+b)$: $\boldsymbol{b=-\lambda}$.

Why it matters: a ReLU with a negative bias is exactly “keep the feature only if the evidence is worth its price $\lambda$”.

6

The $L^2$ penalty. $g(h)=\tfrac12(h-z)^2+\lambda h^2$, $g'(h)=h-z+2\lambda h=0\Rightarrow h=\dfrac{z}{1+2\lambda}$. For $z>0$ this is positive, so it is allowed and it is the minimum.

7

Can it be 0? No: $z>0$ and $1+2\lambda>0$ give $h>0$. $L^2$ only shrinks features; it never turns them off.

Why: near $h=0$ the slope of $\lambda h^2$ is $2\lambda h\to0$, so any positive pull $z$ wins. The slope of $\lambda|h|$ stays $\lambda$, so a pull $z\le\lambda$ loses.

8

Conclusion. Use the $L^1$ penalty $\lambda|h|$ to have only a few concepts present at each word. Numbers ($\lambda=1$): $z=0.5$ → $L^1$ gives $0$, $L^2$ gives $0.167$; $z=3$ → $2$ vs $1$.

L¹ penalty: ReLU(z − λ)L² penalty: z / (1+2λ)no penalty

Figure 7. With $\lambda=1$. The $L^1$ answer $\mathrm{ReLU}(z-\lambda)$ is exactly 0 on the whole shaded zone: a gate with threshold $\lambda$ ($b=-\lambda$). The $L^2$ answer $z/(1+2\lambda)$ only shrinks and is never 0 for $z>0$. Grey dashed: no penalty, $h=\mathrm{ReLU}(z)$.

Answer. Open when $z>-b$. The $L^1$ minimiser is $\mathrm{ReLU}(z-\lambda)$, a gate with $b=-\lambda$. The $L^2$ minimiser $z/(1+2\lambda)$ is never 0 for $z>0$. For few active concepts, use $L^1$.

✔ What it provesA ReLU with negative bias is exactly the solution of “match $z$, but pay $\lambda$ per unit of feature”: the bias is a threshold that buys sparsity.

✖ What it does not sayIt does not prove that a trained network is solving this problem — it is an interpretation of one gate. Training may give $b>0$ (no $\lambda>0$ then). For signed $h$ the answer is soft-thresholding $\mathrm{sign}(z)\max(|z|-\lambda,0)$, not ReLU.

→ Where it leadsLasso regression, sparse coding and ISTA (one step = one ReLU layer), sparse autoencoders used to find “concepts” inside language models.

Problem 8Running an RNN by hand

The problem

Let $h_n=\text{ReLU}(A_wx_n+A_hh_{n-1}+b)$, with $h_0=0$ and $$A_w=\begin{pmatrix}1&1\\-1&1\end{pmatrix},\quad A_h=\begin{pmatrix}0.5&0\\-1&0.5\end{pmatrix},\quad b=\begin{pmatrix}-1\\-1\end{pmatrix},\quad x_1=\begin{pmatrix}1\\1\end{pmatrix},\quad x_2=\begin{pmatrix}2\\0\end{pmatrix}.$$ Compute $h_1$ and $h_2$. What does row $i$ of $A_w$ represent? What does the entry $-1$ of $A_h$ do?

With $h=12$ and $d=768$, how many numbers do $A_w$, $A_h$ and $b$ have together? How does this number change for a longer sentence? Explain.

What you need. The RNN update of Lecture 2: new word through $A_w$, old memory through $A_h$, add $b$, apply ReLU to each coordinate.

1

Word 1, input part. $A_wx_1=\begin{pmatrix}1+1\\-1+1\end{pmatrix}=\begin{pmatrix}2\\0\end{pmatrix}$, memory part $A_hh_0=\begin{pmatrix}0\\0\end{pmatrix}$.

2

Add bias, apply ReLU. $\begin{pmatrix}2\\0\end{pmatrix}+\begin{pmatrix}-1\\-1\end{pmatrix}=\begin{pmatrix}1\\-1\end{pmatrix}$, so $h_1=\mathrm{ReLU}\begin{pmatrix}1\\-1\end{pmatrix}=\begin{pmatrix}1\\0\end{pmatrix}$.

Why: concept 1 matched strongly enough (2 > threshold 1) and is on; concept 2 did not (0 < 1) and is off.

3

Word 2. $A_wx_2=\begin{pmatrix}2\\-2\end{pmatrix}$, $A_hh_1=\begin{pmatrix}0.5\cdot1+0\cdot0\\-1\cdot1+0.5\cdot0\end{pmatrix}=\begin{pmatrix}0.5\\-1\end{pmatrix}$.

4

Add and gate. $\begin{pmatrix}2+0.5-1\\-2-1-1\end{pmatrix}=\begin{pmatrix}1.5\\-4\end{pmatrix}$, so $h_2=\begin{pmatrix}1.5\\0\end{pmatrix}$.

Why 1.5: the new word gives 2, half of the old concept 1 is kept (0.5), the threshold removes 1.

5

Rows of $A_w$. $(A_wx_n)_i=a_i\cdot x_n$ where $a_i$ is row $i$. So row $i$ is a feature (key) vector: it scores how much the new word looks like feature $i$. Here $a_1=(1,1)$, $a_2=(-1,1)$.

6

The entry $-1$ of $A_h$ (row 2, column 1). It adds $-1\times(\text{concept 1 in the memory})$ to concept 2: if concept 1 was present, concept 2 is pushed down now — inhibition, like a “not” cancelling what follows. At $n=2$ it lowers concept 2 from $-3$ to $-4$.

Honest note: concept 2 is closed either way here, so the effect does not show in $h_2$; with another word it could decide whether the gate opens. The diagonal $0.5$ keeps half of each concept (memory that fades).

7

Count. $A_w$: $12\times768=9216$; $A_h$: $12\times12=144$; $b$: $12$. Total $9216+144+12=9372$.

Why these shapes: $A_w$ maps a word ($d=768$) to $h=12$ concepts; $A_h$ maps 12 old concepts to 12 new ones; $b$ has one threshold per concept.

8

Longer sentence. The number does not change: the same $A_w,A_h,b$ are used at every word (weight sharing). A longer sentence only means more steps of the same update.

Figure 8. The RNN unrolled over two words. Teal: the new word enters through $A_w$. Red: the memory enters through $A_h$. The same weights are reused at every step.

Answer. $h_1=(1,0)$, $h_2=(1.5,0)$. Row $i$ of $A_w$ is the feature vector of concept $i$. The $-1$ in $A_h$ lets concept 1 in memory suppress concept 2. 9 372 numbers, the same for any sentence length.

✔ What it provesAn RNN handles sentences of any length with a fixed number of parameters and a fixed memory size $h$.

✖ What it does not say“Fixed parameters” ≠ “handles any length well”: a long sentence must squeeze into 12 numbers, and gradients through many steps vanish or explode. The meaning of a row is an interpretation given after training.

→ Where it leadsLSTM/GRU gates that protect the memory, and attention/transformers (Ch 10), which drop the fixed-size memory and look back at every word.

Problem 9Does the RNN see word order?

The problem

Let $A_h=I$ and $b=0$, without ReLU, so $h_n=A_wx_n+h_{n-1}$. Prove that $h_T=A_w\sum_{n=1}^Tx_n$. What does this say about the order of the words?

Now take one neuron with $h_n=\text{ReLU}(z_n+h_{n-1})$ and $h_0=0$. Compute $h_3$ for $z=(3,-5,2)$ and for $z=(-5,3,2)$. Explain why the results differ.

What you need. Proof by induction; a sum does not depend on the order of its terms; ReLU clips negative values to 0.

1

Base case $T=1$. $h_1=A_wx_1+h_0=A_wx_1=A_w\sum_{n=1}^1x_n$.

2

Induction step. Assume $h_{T-1}=A_w\sum_{n=1}^{T-1}x_n$. Then $h_T=A_wx_T+A_w\sum_{n=1}^{T-1}x_n=A_w\sum_{n=1}^{T}x_n$. ∎

Why: $A_w$ is linear, so $A_wu+A_wv=A_w(u+v)$.

3

Order. $\sum x_n$ is the same for every ordering of the words, so this RNN cannot see word order: “dog bites man” and “man bites dog” give the same $h_T$. It is a bag of words. (Python check: a sequence and its reverse give the same $h_T$.)

4

Order $z=(3,-5,2)$. $h_1=\mathrm{ReLU}(3)=3$; $h_2=\mathrm{ReLU}(-5+3)=\mathrm{ReLU}(-2)=0$; $h_3=\mathrm{ReLU}(2+0)=2$.

5

Order $z=(-5,3,2)$. $h_1=\mathrm{ReLU}(-5)=0$; $h_2=\mathrm{ReLU}(3+0)=3$; $h_3=\mathrm{ReLU}(2+3)=5$.

6

Why different. Without ReLU both orders give $3-5+2=0$. With ReLU: in the first order the $-5$ arrives after the $3$, wipes it out, and the extra $-2$ is clipped and forgotten; in the second order the $-5$ comes first, is clipped while the memory is still 0, and costs nothing.

Why in one line: “add then clip” does not commute, while “add” does. The non-linearity is what makes an RNN sensitive to order.

z = (3, −5, 2) → h₃ = 2z = (−5, 3, 2) → h₃ = 5

Figure 9. Same three numbers, two orders. Dots below 0 joined by a dotted line: the value before ReLU when it is negative. In coral, the $-5$ arrives after the $3$ and its excess $-2$ is cut off. In teal it arrives first and costs nothing. Without ReLU both would end at $0$.

Answer. $h_T=A_w\sum_nx_n$, so the linear RNN ignores word order. With ReLU, $h_3=2$ for $(3,-5,2)$ and $h_3=5$ for $(-5,3,2)$: clipping at 0 forgets differently depending on order.

✔ What it provesA purely additive memory cannot tell word orders apart; the ReLU (or $A_h\ne I$) is what lets an RNN read order.

✖ What it does not sayIt does not say an RNN with ReLU reads order usefully — only that it can. And a linear RNN with $A_h\ne I$ (e.g. $0.5I$) already weights words by position; the bag-of-words result needs $A_h=I$.

→ Where it leadsTransformers also sum over positions, so they need explicit position encodings; the RNN gets order from its step-by-step update.

Problem 10Sigmoid + cross-entropy: the gradient $-(1-p)$

The problem

A sentence has the score $s=v\cdot h_T+c$ and $p=\sigma(s)=\dfrac{1}{1+e^{-s}}$, with $\text{loss}=-\big[y\log p+(1-y)\log(1-p)\big]$. For $y=1$, prove that $\text{loss}=\log(1+e^{-s})$ and $\dfrac{d\,\text{loss}}{ds}=-(1-p)$. What happens to this derivative when the model is confident and right?

Let $h_T=(3,\,1)$, $v=(1,\,-1)$ and $c=0$. Compute $p$, and compute the loss for $y=0$ and for $y=1$.

What you need. $\log(1/u)=-\log u$; $\dfrac{d}{ds}\log(1+e^{-s})=\dfrac{-e^{-s}}{1+e^{-s}}$. Cross-entropy as the training loss of the RNN classifier (Bridge, Lecture 2). $\log$ is the natural log.

1

Loss for $y=1$. The second term vanishes: $\mathrm{loss}=-\log p=-\log\dfrac{1}{1+e^{-s}}=\log(1+e^{-s})$. ∎

2

Differentiate. $\dfrac{d\,\mathrm{loss}}{ds}=\dfrac{1}{1+e^{-s}}\cdot(-e^{-s})=-\dfrac{e^{-s}}{1+e^{-s}}$.

3

Recognise $1-p$. $1-p=1-\dfrac{1}{1+e^{-s}}=\dfrac{e^{-s}}{1+e^{-s}}$, so $\dfrac{d\,\mathrm{loss}}{ds}=-(1-p)$. ∎ (Numerical derivative agrees at $s=-4,0,2,6$.)

Why it is so clean: the $\log$ in the loss undoes the $\exp$ in the sigmoid. In general $\dfrac{d\,\mathrm{loss}}{ds}=p-y$: “predicted minus true”.

4

Confident and right. $s\to+\infty$ ⇒ $p\to1$ ⇒ $\dfrac{d\,\mathrm{loss}}{ds}\to0$ (e.g. $s=6$: $-0.0025$). The example is already learned and barely moves the weights.

Contrast: confident and wrong ($s\to-\infty$) gives slope $\to-1$, a full push: the model learns most from its confident mistakes.

5

Score and probability. $s=v\cdot h_T+c=1\cdot3+(-1)\cdot1+0=2$, $p=\sigma(2)=\dfrac{1}{1+e^{-2}}\approx0.881$.

6

Losses. $y=1$: $-\log0.881\approx0.127$. $y=0$: $-\log(1-0.881)=-\log0.119\approx2.127$.

Why they differ by exactly 2: $\log(1+e^{s})-\log(1+e^{-s})=\log e^{s}=s=2$.

loss if y = 1loss if y = 0d loss/ds for y = 1

Figure 10. For $y=1$ the loss is $\log(1+e^{-s})$ and its slope is $-(1-p)$ (dashed). The slope tends to $-1$ when the model is confidently wrong and to $0$ when it is confidently right. At $s=2$ the two losses differ by exactly $2$.

Answer. $\mathrm{loss}=\log(1+e^{-s})$, $\frac{d\,\mathrm{loss}}{ds}=-(1-p)\to0$ when confident and right. $s=2$, $p\approx0.881$; loss $\approx0.127$ for $y=1$ and $\approx2.127$ for $y=0$.

✔ What it provesWith sigmoid + cross-entropy, $\partial\,\mathrm{loss}/\partial s=p-y$ for both labels, so the gradient never vanishes on a confident mistake.

✖ What it does not sayWith sigmoid + squared error the gradient is $2(p-y)\,p(1-p)$, which does vanish when $p\approx0$ on a $y=1$ example — why we do not pair sigmoid with $L^2$. A small training loss also does not mean good calibration on new data.

→ Where it leadsSoftmax + cross-entropy for many classes (gradient $p-\text{one-hot}(y)$, used in the character language model), perplexity $e^{\text{loss}}$, label smoothing.

Tutor addition · checked by running code

Every number and every figure on this page comes from running Python (3 Nov 2026): LessonWeb/NLP/code/CHK_ex.py (numbers, output in CHK_ex.out.txt) and LessonWeb/NLP/code/CHK_figs.py (figures). Main lines:

CHK_ex.py (main lines)

import numpy as np
relu = lambda z: np.maximum(z, 0); sig = lambda s: 1/(1+np.exp(-s))
# Q4  f = x^2 + 3y^2 from (1,1)
P0, g = np.array([1., 1.]), np.array([2., 6.])
for eta in (0.1, 0.5): P1 = P0 - eta*g; print(eta, P1, P1[0]**2 + 3*P1[1]**2)
# Q8  RNN by hand
Aw = np.array([[1, 1], [-1, 1.]]); Ah = np.array([[.5, 0], [-1, .5]]); b = np.array([-1, -1.])
h1 = relu(Aw@[1, 1] + b); h2 = relu(Aw@[2, 0] + Ah@h1 + b); print(h1, h2)
# Q9  order with ReLU
def one(zs):
    h = 0.
    for z in zs: h = max(z + h, 0.)
    return h
print(one([3, -5, 2]), one([-5, 3, 2]))
# Q10 sigmoid + cross-entropy
s = np.dot([1, -1], [3, 1]); p = sig(s); print(p, -np.log(p), -np.log(1 - p))

0.1 [0.8 0.4] 1.1199999999999999
0.5 [ 0. -2.] 12.0
[1. 0.] [1.5 0. ]
2.0 5.0
0.8807970779778823 0.12692801104297263 2.1269280110429714

Source: online problem sheet “Checking: Neural Networks” (ITC, Dept. of Applied Mathematics and Statistics; 10 problems, read 3 Nov 2026; opens with your Google sign-in). Solutions, explanations, figures and the “edge” boxes are tutor-written.

Practice mode

Question bank

Multiple-choice and numerical questions with instant feedback and step-by-step solutions. Your answers are remembered in this browser.

Section Difficulty

0 / 0answered correctly

Revision mode

Formulas, definitions, flashcards

Everything from the First Look lesson, Chapter 1 and Section 1.1 in compact form, plus exam-style questions.

Formula reference

Open "What the symbols mean" under any formula for a symbol-by-symbol reading.

Key definition

Neural network · from the lesson

$NN(x)=Af_L\circ\sigma_{L-1}\circ\cdots\circ Af_2\circ\sigma_1\circ Af_1(x)$, with affine $Af_i(x)=W^ix+b^i:\mathbb{R}^{k_{i-1}}\to\mathbb{R}^{k_i}$, $k_0=n$, $k_L=m$, and non-linear $\sigma_i$ applied pointwise.

Language model · from the First Look lesson

$M:\mathcal{L}\to\mathcal{L}$, built from a next-word map $M_0:\mathcal{L}\to\Sigma$; with embedded inputs and probability outputs, $M:(\mathbb{R}^d)^*\to\mathbb{R}^{|\Sigma|}$. An RNN is $f:\text{words}\times\text{memory}\to\text{memory}$, and the approval classifier computes $p=\sigma(v\cdot h_n+c)$.

Key fact (proved in Chapter 1)

Without activations, $W^L\cdots W^1$ is a single matrix, so depth adds no expressive power. With biases: $W^2(W^1x+b^1)+b^2=(W^2W^1)x+(W^2b^1+b^2)$.

Flashcards

Exam-style questions

Build status

Source coverage

What was read from the original notes and what this site has built so far.

Section	Original lesson	This site

First Look lecture: slide coverage

The lecture “Language Modeling (A First Look)” has 22 slides. Each one was read, restated in the lesson section shown, and extended with tutor additions.

Slide	Title on the slide	Taught in	Original lesson

Original deck: open the slides ↗

Original source

Vanny Khon, Language Modeling: A Mathematical Perspective (lecture notes, published as a Google Apps Script web page). This site restates its mathematics for study purposes; read the original for the author's full wording and figures.

From a single neuron to the transformer

Your progress

How to read this site

Chapters

Preface

Artificial Neural Network

What you will learn

A signal travelling through neurons

The ceiling drip: a gate with a threshold

From weighted sums to the definition

3.1 One neuron is a dot product

3.2 A layer is a matrix

3.3 Why purely linear layers collapse

Step-by-step derivation

3.4 The gate: ReLU with a threshold

Clickable formula: what each symbol means

3.5 Linear and affine maps

3.6 Architecture, hard and soft gates

See the mathematics move

A single neuron $a=\sigma(w\cdot x+b)$

Activation explorer: hard vs soft gates

Linear collapse, checked numerically

Forward propagation simulator

A 2 → 2 → 1 network by hand

Compute $NN(x)=Af_2\circ\text{ReLu}\circ Af_1(x)$

The same mathematics in PyTorch

Check your understanding

Key ideas to keep

Common mistakes

Exam tips

Coding Neural Networks with PyTorch

What you will learn

Tensors are the vectors and matrices of Chapter 1

From tensors to nn.Module

3.1 Imports and tensors

3.2 Plotting a function from a tensor

3.3 Slicing

3.4 nn.Linear is an affine map

3.5 A two-layer network as a class

3.6 Plotting what the network computes

Play with tensors and shapes

Pointwise tensor operations

torch.linspace and the graph of $a^2-3a$

Slicing explorer for x = torch.rand(2,3)

nn.Linear(n, m) on a batch

Evaluate $L(x)=Wx+b$ with the weights printed in the lesson

What does an untrained NeuralNet() compute?

Tracing shapes through N(x.view(-1,1))

Follow one batch of 300 points

Extend the lesson code

Check your understanding

Key ideas to keep

Common mistakes

Language Modeling: A First Look

What you will learn, and where every slide goes

All 22 slides, each placed in its section

A model that turns a sentence into a sentence

Why a list of answers is not a language model

Words, sequences of words, and real sentences

Sort the slide's examples

How big is $\Sigma^*$?

Building $M$ from a next-word map $M_0$

Run the loop by hand

How can a computer hold $M$?

Slide 8's picture, with distances

The embedding $E:\Sigma\to\mathbb{R}^d$

A lookup is a row pick

How much memory does a table need?

Numericalising the output

From scores to a probability vector

Reading with a memory: the recurrent neural network

Read a sentence one word at a time

Example: approval or disapproval

8.1 The pipeline (slides 18–19)

8.2 Do the word vectors know about approval? (slide 20)

The same kind of picture, for this page's toy data

8.3 Parameters and loss (slide 21)

8.4 Train it (slide 22)

Train a small RNN on approval vs disapproval

The lecture's pieces in PyTorch