Your study site for the course: three lessons rebuilt with real-data worked examples, both TPs solved (including the TP2 parts left as “To do”), a gallery of 20+ chart types and a one-page revision sheet.
Lecturer
Dr. HAS Sothea
TP instructor
Dr. UANN Sreyvi
Tools
Python · Jupyter · Matplotlib · Seaborn · Plotly
YOUR PROGRESS0/5
Mark lessons and TPs as done below or from any lesson. Progress is saved in this browser only.
No page matches “”. Try a method name such as “quartile”, “chi-square” or “imputation”.
Exams
No midterm, final or quiz paper is in the course folder yet. When one is added, it gets its own solved page here with each question mapped to the chapter it tests. Until then, the revision sheet lists likely question types per chapter.
Other pages
Graph gallery: which chart for which variable types, 20+ samples
The course folder holds 3 lecture PDFs and 2 notebooks and no existing study pages, so this site is new; the original files were only read, never changed. Every chart is drawn from the real dataset named under it, except the shape sketches marked “illustration”. Lecture I quotes “891 rows” for the Titanic preprocessing example, but its missing counts (Age 86, Fare 1, Cabin 327) are those of Kaggle’s 418-row test file; Lesson III redoes the example on the 891-row training file.
Look at one column at a time: decide its type, summarise it with the right numbers, draw it with the right graph, and spot problems before any model sees the data.
Garbage in, garbage out. A model can only be as good as the data it is fed. Real data arrive with missing values (a blank salary in a survey), outliers (an impossible clinical reading) and inconsistent entries (“Male”, “male”, “M”). Exploratory Data Analysis (EDA) is the first step of any analysis: you get curious about the data to find its problems, its patterns and the relationships inside it.
Done well, EDA lets you clean the data without destroying the patterns, raise new questions, notice things you would otherwise miss and pick informative inputs for prediction. Its goals are to inspect, detect issues, uncover patterns and prepare for modelling (check assumptions, select features, choose methods).
Descriptive statistics
Summarise the data you have with numbers and graphs: see the distribution, detect abnormal values, discover patterns. This is the toolkit of EDA.
Inferential statistics
Use a sample to say something about the whole population, through estimation and hypothesis tests.
Univariate analysis looks at each column separately. The first question is always its type, because the type decides which numbers and graphs make sense. You can compare and subtract ages (Age is quantitative), but “S − C” for the port of embarkation means nothing (Embarked is qualitative).
Slide 16 challenge, completed for every Titanic column
Column
Type
Why
Age, Fare
Quantitative · continuous
any value in an interval; differences are meaningful
SibSp, Parch
Quantitative · discrete
counts of relatives: 0, 1, 2, …
Sex, Embarked
Qualitative · nominal
labels with no order
Pclass
Qualitative · ordinal
1st > 2nd > 3rd is an order, but “2 − 1” is not a quantity, although it is stored as an integer
Survived
Qualitative · nominal (binary)
0/1 codes a category; its mean is the survival rate, a proportion
03Definitions & formulas
Data x1, …, xn; sorted data x(1) ≤ … ≤ x(n). Units: a statistic of location (mean, quartiles) has the unit of the data; the variance has the unit squared.
QUALITATIVE
Frequencies and mode
absolute nk = #{i : xi = k} relative fk = nkn, Σfk = 1 mode = the category with the largest nk
LOCATION
Mean and median
x̄ = 1nΣi=1nxi Q2 = x((n+1)/2) if n odd; x(n/2) + x(n/2+1)2 if n even
QUARTILES (COURSE METHOD)
Split at the median
Q1 = median of the lower half, Q3 = median of the upper half. For odd n the median itself is left out of both halves (slide 22 only shows an even example). Percentile p: value below which p% of the data fall.
DISPERSION
Variance and standard deviation
s2 = 1n − 1Σi=1n (xi − x̄)2, s = √s2 pandas .var()/.std() use n − 1; numpy uses n unless ddof=1.
BOXPLOT RULE
IQR and Tukey fences
IQR = Q3 − Q1 (the middle 50%) [Q1 − 1.5·IQR, Q3 + 1.5·IQR] For normal data 99.3% fall inside; points outside are flagged as outliers.
SHAPE (USED IN TP1)
Skewness and kurtosis
γ1 = E[((X − μ)/σ)3]: > 0 long right tail (mean > median) γ2 = E[((X − μ)/σ)4] = 3 for a normal; pandas .kurt() returns γ2 − 3 (excess)
HISTOGRAM
Counts per bin
Bins Bk = [bk, bk+1). For x ∈ Bk: h(x) = Σi 𝟙{xi ∈ Bk} Density version divides by n·(bin width) so the area is 1.
KERNEL DENSITY
A smooth histogram
f̂h(x) = 1nhΣiK(x − xih), K(u) = 1√(2π)e−u²/2 h = bandwidth: small → wiggly, large → over-smoothed.
04Worked examples
Four examples, from hand calculation to the full dataset. Every number was recomputed in Python.
EXAMPLE 1 · BY HAND · SLIDE 22 DATA
Quartiles, mean and spread of 1, 1, 2, 3, 5, 6, 7.5, 8, 8, 9
Trap:pd.Series(x).quantile([.25, .75]) gives 2.25 and 7.875, because pandas interpolates linearly between order statistics. Both are valid quartile definitions; in an exam, use the course method unless told otherwise and say which one you used.
IQR = 45.175 → upper fence 53.1 + 67.76 = 120.86: no outlier in this tiny sample.
x̄ = 207.93 / 7 = 29.70 ≫ Q2 = 8.46 → three expensive tickets pull the mean up: right skew. s = 27.89.
median 8.46 < mean 29.70 → right-skewed
EXAMPLE 3 · FULL DATASET · n = 891
How many fares are outliers, and which rule do you trust?
Q1 · Q2 · Q3
7.91 · 14.45 · 31.00
IQR · upper fence
23.09 · 65.63
mean · std
32.20 · 49.69
skewness
4.79 → 0.90 after log₁₀
The Tukey rule flags 116 fares (13.0%) above 65.63. A |z| > 3 rule flags only 20, because the huge fares inflate the standard deviation that defines them. Thirteen percent is far more than the 0.7% expected under normality: the fences are telling you the distribution is skewed, not that 116 tickets are errors. Fare is a price, so a log scale is the natural fix (see the graphs below).
EXAMPLE 4 · QUALITATIVE
Frequency table of Pclass
Pclass
1st
2nd
3rd
Total
count nk
216
184
491
891
proportion fk
0.24
0.21
0.55
1.00
mode = 3rd class (55%)
A mean of Pclass (2.31) is not meaningful: it is ordinal. You may still report the median class (3rd), because medians only need an order.
05Graphs on real data
Titanic training set (seaborn). Hover a chart to fade its points.
Bar chart · Pclass
Ordinal categories keep their natural order on the axis.
Bar chart · Sex
64.8% male, 35.2% female.
Donut chart · Embarked
Pie/donut only for few categories that sum to a whole; 2 values missing.
Histogram + KDE · Age (n = 714 known)
Two bumps: young children (0–5) and a main mode at 20–30. Mean slightly above median: mild right skew (skewness 0.39).
Boxplot · Fare, with the 116 flagged points
75% of passengers paid under £31; the box is squeezed against zero by a long right tail up to £512.33.
Histogram · log₁₀(Fare): the same data, made readable
293 tickets cost between £6.3 and £8.9 (mostly third class). The log scale exposes price tiers that the raw histogram hides. log(0) is undefined, so the 15 free tickets must be handled first (drop, or use log(1 + x)).
ECDF · Fare (new)
F̂(x) = share of values ≤ x. Read any quantile straight off the curve, no bins to choose.
Normal Q-Q plot · Age (new)
Points on the line = normal. Here the left tail sits above the line (no ages below 0) and the middle bends: Age is close to, not exactly, normal.
Illustration · how skew moves the mean (dashed) away from the median (solid)
Shapes drawn from beta curves, not data.
left skew · mean < median · γ₁ < 0 (lifeExp 2002: −0.63)
symmetric · mean = median · γ₁ ≈ 0 (Age: 0.39, close)
Univariate analysis describes each variable, understands its distribution and detects problems (invalid min/max, extreme values, skew, missing values) before you look at relationships in Lesson II.
09Self-check
1. A column stores postal codes as integers. Quantitative or qualitative?
Qualitative nominal. Codes are labels; their average or difference has no meaning.
2. Compute Q1, Q2, Q3 of 4, 8, 15, 16, 23, 42 with the course method.
n = 6: Q2 = (15 + 16)/2 = 15.5; lower half 4, 8, 15 → Q1 = 8; upper half 16, 23, 42 → Q3 = 23. IQR = 15, upper fence 45.5, so 42 is not an outlier.
3. Mean 32.2, median 14.45. What shape, and which centre do you report?
Right-skewed (a long tail of large values pulls the mean up). Report the median with the IQR.
4. Why does the |z| > 3 rule find fewer fare outliers than the boxplot rule?
The extreme fares inflate the standard deviation (49.7), which widens the z-score band; quartiles are robust to those values, so the fences stay tight.
5. When would you prefer an ECDF to a histogram?
When you need exact quantiles or want to compare several distributions without choosing bins; an ECDF uses every point and has no tuning parameter.
10Glossary · FR ↔ EN
variable quantitative ↔ quantitative variable
variable qualitative ↔ categorical variable
effectif / fréquence ↔ count / relative frequency
moyenne ↔ mean
médiane / quartile ↔ median / quartile
écart-type ↔ standard deviation
écart interquartile ↔ interquartile range
asymétrie / aplatissement ↔ skewness / kurtosis
boîte à moustaches ↔ boxplot
diagramme en barres / circulaire ↔ bar / pie chart
valeur aberrante ↔ outlier
fonction de répartition empirique ↔ ECDF
11Sources
Course: I. Introduction to EDA & Descriptive Statistics (PDF, 35 slides), HAS Sothea.
Data: Titanic training set via seaborn.load_dataset("titanic"), 891 rows.
Every question restated, solved and boxed. Your notebook already answered all of them; each answer below was re-run and checked against the data, and the notes say where it agrees or could be tighter.
Q2 · VARIABLE TYPES
Which Gapminder columns are quantitative, which qualitative?
Worked solution
Classify the six Gapminder columns
What you need. Quantitative = numbers you can average (continuous or discrete); qualitative = labels (nominal or ordinal). See C01.
1
Look at the dtypes.gapminder.dtypes: country, continent are object; year, lifeExp, pop, gdpPercap are numeric.
Why: the dtype is only a first hint — it says how the value is stored, not what it means.
2
Check the meaning of each numeric column. lifeExp, pop, gdpPercap are measurements: averaging them makes sense → quantitative (continuous).
3
Check year separately. int64, but only 12 equally spaced values (1952, 1957, …, 2007) that index time.
Why: it is ordered and used to group or sort, not to average: treat it as an ordered time index.
4
Labels. country and continent have no order → qualitative nominal.
✔ What it provesYou can decide the type from meaning, which decides the summaries and charts allowed.
✖ What it does not sayStorage type and statistical type are different: a postal code stored as int is still nominal; “year as a number” can be treated as quantitative in a trend model.
→ Where it leadsQ2 uses only the quantitative columns for mean, skewness and kurtosis.
gapminder.dtypes gives object for country, continent and numeric for the rest. Check the meaning, not just the dtype: year is stored as int64 but takes 12 equally spaced values and indexes time.
Quantitative: lifeExp, pop, gdpPercap (continuous) · Qualitative: country, continent (nominal) · year: ordered time index
Q2.1.1 · SUMMARY, SKEWNESS, KURTOSIS (2002, n = 142)
Summarise each quantitative variable and describe its shape
Worked solution
Summaries and shape (2002, n = 142)
What you need. Mean vs median shows the direction of skew; skewness measures asymmetry; excess kurtosis compares tails with a normal distribution (0).
1
Filter the year (year == 2002, 142 countries) and run describe(), skew(), kurt().
Why: one row per country: mixing years would count each country 12 times.
2
lifeExp: mean 65.69 < median 70.83, skew −0.63, excess kurtosis −0.95.
Why: mean below median → a long left tail (a group of low values); negative kurtosis → flatter than normal, light tails.
3
pop: mean 41.5 M vs median 10.4 M, skew 7.47, excess kurtosis 59.6.
Why: two giants (China, India) pull the mean far above the median: extreme right skew and very heavy tail.
4
gdpPercap: mean 9,918 ≈ 1.9 × median 5,320, skew 1.26, excess kurtosis 0.34.
Why: right-skewed, but its tails are close to normal.
Answer. lifeExp left-skewed and flat; pop extremely right-skewed with the heaviest tail; gdpPercap right-skewed. Report the median for pop and gdpPercap.
✔ What it provesThe sign of (mean − median) and the skewness agree here, and they decide which centre to report.
✖ What it does not saySkewness and kurtosis are themselves sensitive to a few extreme values (pop’s 59.6 is mostly China and India); they describe this sample, not a law.
→ Where it leadsLog-transforming pop and gdpPercap (C01 · Graphs) — and TP02, where the same skew distorts Pearson’s r.
2002
mean
median
std
min
max
skew
excess kurt
lifeExp
65.69
70.83
12.28
39.19
82.00
−0.63
−0.95
pop
41.5M
10.4M
140.8M
0.17M
1280M
7.47
59.60
gdpPercap
9918
5320
11154
241
44684
1.26
0.34
lifeExp 2002: 68 of 142 countries sit at 70–80 years, over a flat plateau from 40 to 70.
pop: extremely right-skewed and heavy-tailed; China and India alone stretch the scale. Roughly log-normal.
gdpPercap: right-skewed (mean ≈ 1.9 × median), tails close to normal.
lifeExp left-skewed · pop and gdpPercap right-skewed · pop by far the heaviest tail
Q2.1.3 – Q2.1.4 · EXTREMES
Richest, poorest, healthiest, unhealthiest in 2002 and 1977
Worked solution
Extremes in 2002 and 1977
What you need.idxmax() / idxmin() return the row label of the largest / smallest value.
1
Filter one year, then for gdpPercap and lifeExp take .loc[d[col].idxmax()] and idxmin().
Why: one helper function answers all four questions for any year — less code, fewer copy-paste errors.
2
2002: richest Norway $44,684 · poorest Congo, Dem. Rep. $241 · healthiest Japan 82.0 y · unhealthiest Zambia 39.2 y.
3
1977: richest Kuwait $59,265 · poorest Myanmar $371 · healthiest Iceland 76.1 y · unhealthiest Cambodia 31.2 y.
4
Explain the surprise. Cambodia: 40.3 (1972) → 31.2 (1977) → 51.0 (1982), the Khmer Rouge period.
Why: an extreme value is a question to explain, not just a number to report.
Answer. 2002: Norway / Congo DR / Japan / Zambia. 1977: Kuwait / Myanmar / Iceland / Cambodia.
✔ What it provesExtremes are found per year with idxmax/idxmin, and each one can be checked against history.
✖ What it does not sayOne extreme says nothing about the distribution; ties return only the first row; and Gapminder values are estimates, especially for crisis years.
→ Where it leadsOutlier detection with IQR fences and z-scores (C03).
2002
Richest Norway $44,684 Poorest Congo, Dem. Rep. $241 Healthiest Japan 82.0 y Unhealthiest Zambia 39.2 y
1977
Richest Kuwait $59,265 Poorest Myanmar $371 Healthiest Iceland 76.1 y Unhealthiest Cambodia 31.2 y
The guess for 1977 was Cambodia: under the Khmer Rouge (1975–79) life expectancy fell from 40.3 in 1972 to 31.2, then recovered to 51.0 by 1982. Use idxmax() / idxmin() rather than a query per question, as your extremes() helper does.
Q2.2 · QUALITATIVE VERSIONS (2002)
Cut each variable into three groups and describe them
Worked solution
Turn each quantitative variable into three groups
What you need.pd.cut = equal-width intervals; pd.qcut = equal-count intervals (quantiles).
1
Compute tertile cut-points with pd.qcut(x, 3): gdpPercap $2,141 and $9,022; pop 5.41 M and 19.58 M; lifeExp 61.3 and 73.8 y.
Why: tertiles put one third of the countries in each group.
2
Compare with equal widths (pd.cut(x, 3)): pop gives 98.6 / 0 / 1.4 %, gdp 76 / 13 / 11 %, lifeExp 23 / 18 / 59 %.
Why: on a skewed variable, equal widths are set by the few extreme values, so almost everything falls in one bin.
3
Choose qcut for skewed variables and label the groups (small / medium / large).
4
Describe the groups by their cut-points and widths, not by the bar heights.
Why: with tertiles every bar is ≈ 33 % by construction, so the information is in where the boundaries fall.
qcut (equal counts)cut (equal widths)
Figure. Share of countries in the low / middle / high group of each variable (2002). Green: pd.qcut tertiles (≈ 33 % each by construction). Orange: equal-width pd.cut: for population 98.6 % land in “small”. Numbers from the TP01 page.
Answer. Use qcut tertiles (≈ 34/33/33 %): e.g. small < 5.4 M ≤ medium < 19.6 M ≤ large.
✔ What it provesFor skewed data, quantile bins give usable groups; equal-width bins can leave groups empty.
✖ What it does not sayqcut groups are relative to this year’s sample: “large” in 2002 is not the same population as “large” in 1952 unless the cut-points are fixed (Q7).
→ Where it leadsBinning loses information; keep the numeric column for correlations (TP02).
New column
Tertile cut-points
Shares with qcut
Shares with equal-width cut
gdpQual
$2,141 · $9,022
34 / 33 / 33%
76 / 13 / 11%
popQual
5.41M · 19.58M
34 / 33 / 33%
98.6 / 0 / 1.4%
lifeExpQual
61.3 · 73.8 y
34 / 33 / 33%
23 / 18 / 59%
The hint suggests pd.cut, your notebook used pd.qcut. Both are defensible; the last column shows why qcut is the better choice for skewed variables: equal widths would put 98.6% of countries in “small”. With tertiles the bar charts are flat by construction, so the interesting information moves to the cut-points and the widths of the groups.
small < 5.4M ≤ medium < 19.6M ≤ large
Q3.1 · EVOLUTION OF QUANTITATIVE COLUMNS
Life expectancy by continent, 1952–2007
Worked solution
Life expectancy by continent, 1952–2007
What you need.groupby(["year", "continent"]).lifeExp.mean() and a line chart, one line per continent.
1
Aggregate lifeExp by year and continent (mean over countries).
Why: one value per continent and year makes 5 comparable time series.
2
Read the endpoints: Oceania 69.3 → 80.7, Europe 64.4 → 77.6, Americas 53.3 → 73.6, Asia 46.3 → 70.7, Africa 39.1 → 54.8.
3
Compare the gains: Asia +24.4 years is the fastest; Africa flattens after 1987.
Why: the change, not the level, answers “who improved most”.
4
Interpret carefully: Africa’s stall coincides with the HIV/AIDS epidemic.
Why: a coincidence in time is consistent with a cause but does not prove it.
Answer. All continents improve; Asia catches up fastest; Africa stalls from the late 1980s, so the gap widens again.
✔ What it provesContinental means rose everywhere between 1952 and 2007.
✖ What it does not sayAn unweighted mean treats Iceland and China equally; a population-weighted mean would tell a different story (e.g. Asia’s average person).
→ Where it leadsTP02 measures how much continent explains with η².
Every continent improves. Means 1952 → 2007: Oceania 69.3 → 80.7, Europe 64.4 → 77.6, Americas 53.3 → 73.6, Asia 46.3 → 70.7 (+24.4, the fastest), Africa 39.1 → 54.8. Africa flattens after 1987, consistent with the HIV/AIDS epidemic, so the gap to the other continents widens again.
Asia catches up; Africa stalls after the late 1980s
Q3.1.3 · CAMBODIA vs THAILAND vs VIETNAM
Three neighbours, three trajectories
Worked solution
Cambodia, Thailand, Vietnam
What you need. Filter three countries and plot each indicator over time; compare levels and timing.
1
Filtercountry.isin(["Cambodia", "Thailand", "Vietnam"]) and pivot by year.
Find the crossing: Vietnam overtakes Thailand on life expectancy around 1992, while its GDP per capita is under a quarter of Thailand’s ($989 vs $4,617).
Why: health and income do not move in lockstep: policy (e.g. primary health care) matters.
4
Spot the break: Cambodia 1977 is the only collapse in the three series.
Answer. Vietnam is healthiest in 2007 at a third of Thailand’s income; Cambodia lags on both after its 1970s collapse.
✔ What it provesComparing neighbours shows that similar regions can follow very different paths.
✖ What it does not sayThree countries cannot establish a general relation between income and health — that needs all countries (TP02).
→ Where it leadsTP02 · Q7 compares Cambodia and Thailand on three indicators together.
2007
KH
TH
VN
lifeExp
59.7
70.6
74.2
pop (M)
14.1
65.1
85.3
gdp/cap ($)
1,714
7,458
2,442
Vietnam overtakes Thailand on life expectancy around 1992 while its GDP per capita is still under a quarter of Thailand’s ($989 vs $4,617): health and income do not move in lockstep. Cambodia’s 1977 collapse is the only break in any of the three series.
Q3.2 · EVOLUTION OF QUALITATIVE COLUMNS
Asia’s lifeExpQual, using the fixed 2002 cut-points
Worked solution
Asia’s life-expectancy groups over time, fixed cut-points
What you need. Re-use the 2002 tertile boundaries (61.3 and 73.8 years) for every year with pd.cut(x, [-inf, 61.3, 73.8, inf]).
1
Fix the boundaries from 2002 instead of recomputing tertiles each year.
Why: with yearly tertiles every year would be 33/33/33 % and no change could be seen.
2
Apply them to Asia for each year and compute the share in each group.
3
Read the result: “unhealthy” falls from 94 % (1952) to 9 % (2007); “healthy” rises from 0 % to 36 %.
4
Compare: in Africa “unhealthy” only falls from 100 % to 79 %.
Why: the same fixed scale makes the two continents directly comparable.
✔ What it provesFixed thresholds turn a category into a stable measuring stick over time.
✖ What it does not sayThe thresholds come from 2002 data, so the labels are relative to 2002; another reference year would shift every share.
→ Where it leadsStacked bar charts of category shares over time (C02 · Graphs).
Fixing the thresholds at the 2002 tertiles keeps “healthy” meaning the same thing every year (above 73.8 years). In 1952, 94% of Asian countries were “unhealthy”; by 2007 only 9% were, and 36% were “healthy”. In Africa (your notebook, Q3.2.3) “unhealthy” only falls from 100% to 79%, and stays there after 2002.
hypothesis test vocabulary: H₀, test statistic, p-value, significance level α
02Explanation
Gapminder follows 142 countries every five years from 1952 to 2007. Summaries of each column in 2007 tell you that population is extremely skewed (mean 44M, median 10.5M) and life expectancy ranges from 39.6 to 82.6 years. They cannot tell you whether richer countries live longer. That is a bivariate question, and the right tool depends on the pair of types.
The intuition. Two columns are related when knowing one changes what you expect of the other. For two numbers, that shows up as a trend in a scatterplot. For a number and a category, it shows up as groups whose values are well separated. For two categories, it shows up as row profiles that differ from one row to the next.
Course rule: begin the analysis with indicators, then visualise the pairs with potential impact. With 10 columns there are 45 pairs; a correlation matrix finds the five worth drawing.
Clear purpose distribution, relationship or comparison?
Right chart scatter, box, heatmap, bar, line
Fit the audience experts, public, decision-makers
Clean, honest, accessible labels, log scale when needed, colour-blind-safe palettes
03Definitions & formulas
QUANT × QUANT
Covariance
Cov(X, Y) = 1n − 1Σi (xi − x̄)(yi − ȳ) Sign = direction of the linear trend. Unit = unit(X)·unit(Y), so its size is not comparable across pairs.
QUANT × QUANT
Pearson correlation
rXY = Cov(X, Y)sXsY ∈ [−1, 1] ±1: perfect linear trend; 0: no linear trend or trend hidden by noise. Unit-free.
QUANT/ORDINAL × QUANT/ORDINAL
Spearman rank correlation
ρXY = rR[X], R[Y] = 1 − 6 Σidi2n(n2 − 1) di = R[xi] − R[yi]. Measures a monotonic trend; robust to outliers. The short formula assumes no ties.
QUANT × QUAL
η² coefficient
BSS = Σgng(x̄g − x̄)2, TSS = Σi (xi − x̄)2 η2 = BSS / TSS ∈ [0, 1]: share of the variance of X explained by the groups G. Sensitive to outliers.
QUANT × QUAL · TEST
One-way ANOVA
H₀: all group means equal. F = BSS / (G − 1)WSS / (n − G), WSS = TSS − BSS Assumes normal groups with equal variances; otherwise use Welch’s ANOVA or Kruskal–Wallis.
In every cell |O − E| = 112.47: in a 2 × 2 table the four deviations always share one absolute value. scipy’s default adds Yates’ continuity correction for 2 × 2 tables; pass correction=False to match the hand result.
Caveat: Oceania has 2 countries, so its expected counts are about 0.7 per cell, far below 5. The χ² approximation is shaky for that row; merging Oceania with Asia or using an exact or simulated test would be safer. The conclusion (strong dependence) does not change.
05Graphs on real data
Hover a scatter to fade a group and compare.
Scatter · tip vs total bill, coloured by sex (slide 9–10), with least-squares line
Upward trend with a fan shape: tips vary more on large bills. Pearson summarises the trend, not the fan.
What different correlations look like (slide 12, redrawn)
Simulated: 70 points each, fixed random seed.
The last panel has a perfect pattern and a correlation near zero: r ≈ 0 means “no linear trend”, not “no relationship”.
Anscombe’s quartet (new): four datasets, one correlation
Each set: mean x = 9, mean y = 7.5, var x = 11, var y ≈ 4.12, r = 0.816, same fitted line y = 3 + 0.5x.
Always plot before you trust a coefficient. Set IV previews Lesson III’s high-leverage points.
Pearson matrix · 2007Spearman matrix · 2007
Spearman raises gdp–lifeExp from 0.68 to 0.86: the link is strong but not linear. Population is unrelated to both.
Bubble chart · four variables at once (slides 33–35)
x = GDP (log), y = lifeExp, colour = continent, area ∝ population. On the log axis the trend is close to a straight line. Swaziland (39.6 years) is far below countries with a similar income: health is not only income.
Conditional boxplot · lifeExp by continent · η² = 0.635Conditional boxplot · gdpPercap by continent · η² = 0.424
Life-expectancy boxes barely overlap (Africa vs Europe): high η². GDP boxes overlap a lot within Asia and Europe: lower η².
100% stacked bar · GDP category by continent (slide 28)
The more the bars differ, the stronger the association. Under independence every bar would show the overall split (33 / 33 / 34%).
06Python, with real output
bivariate.pypandas · scipy
import numpy as np, pandas as pd
from scipy import stats
from gapminder import gapminder as g
d7 = g.query("year == 2007")
q = ["gdpPercap", "lifeExp", "pop"]
d7[q].corr().round(2) # gdp-life 0.68, pop ≈ 0
d7[q].corr("spearman").round(2) # gdp-life 0.86def eta2(df, y, g="continent"):
m = df[y].mean()
bss = sum(len(s) * (s.mean() - m)**2 for _, s in df.groupby(g)[y])
return bss / ((df[y] - m)**2).sum()
eta2(d7, "lifeExp"), eta2(d7, "gdpPercap") # (0.635, 0.424)
d7 = d7.assign(gcat=pd.qcut(d7.gdpPercap, 3, labels=["Developing", "Emerging", "Developed"]))
ct = pd.crosstab(d7.continent, d7.gcat)
chi2, p, dof, exp = stats.chi2_contingency(ct) # 82.29, 1.7e-14, 8
V = np.sqrt(chi2 / (ct.values.sum() * (min(ct.shape) - 1))) # 0.538
stats.contingency.association(ct, method="cramer") # same, built in
07Common mistakes & exam traps
Correlation is not causation. A strong r shows a tendency, not a mechanism; a third variable can drive both.
r ≈ 0 ≠ independent. A U-shape or a hidden subgroup gives r near zero.
One point can make or break r. Kuwait alone drags 1952 gdp–life Pearson from 0.75 to 0.28 (see TP02).
Comparing covariances. Covariance depends on units; compare correlations instead.
Your notebook now answers every question and adds further explanations. This page gives a worked solution for each question, adds your explanations after checking them on the real data, and marks the four statements that need changing.
REVIEW · NOTEBOOK EDITED 4 NOV 2026
What changed, what is right, what to fix
Your update adds a Further Explanation to most answers, and a fallback in cell 22 for when pingouin is not installed. I re-ran the analysis on the real Gapminder data and tested each statement that the data can test.
✔ All printed numbers reproduce
Correlations, η², ANOVA, Levene, Shapiro–Wilk, Kruskal–Wallis and Welch all match your outputs.
✔ The fallback code is right
Without pingouin it still gives η² and Welch values equal to your outputs: F = 81.51 on (4, 17.58) df.
✖ 4 statements need changing
One changes a conclusion: the “economic divide came later” story is mostly the Kuwait effect. See the table below.
Statements to change
Cell
You wrote
What the data say
Suggested wording
25, 48, 53
Continents “became economically different only later”; the divide is “a later phenomenon”; 42% vs 5% shows it widened.
Without Kuwait, η²(continent, GDP) is 0.443 in 1952 and 0.448 in 2007. It is flat. Your own cell 44 already blames Kuwait for the 5%, so the cells disagree.
“Continent already explains about 44% of GDP variance in 1952 once Kuwait is set aside. The rise from 5% to 42% is mainly Kuwait falling back. The gap did grow in ratio: Europe’s mean was 4.5× Africa’s in 1952 and 8.1× in 2007.”
8
r rises because by 1987 “many countries had grown richer and spread out”, making the linear fit stronger.
Without Kuwait, r is already 0.747 in 1952 and 0.751 in 1987. The linear link did not get stronger. Kuwait fell from $108,382 to $28,118.
“r rises because Kuwait’s GDP per capita falls back among the other countries, not because the underlying relationship strengthens.”
34
log(GDP) makes each continent “much closer to normal” and stabilises the variances.
Partly. Skew falls in Africa, Americas and Asia, and Americas now passes. But Africa and Asia still fail, Europe gets worse (p 0.088 → 0.0055), and Levene still rejects (p = 0.0016).
“The log reduces skew and narrows the variance ratio, but does not fully meet the assumptions, so Welch or Kruskal–Wallis is still needed.”
53
“Global Convergence in Health”.
SD of lifeExp across countries: 12.23 (1952), 10.56 (1987), 12.07 (2007): no net narrowing. Asia and the Americas did close in on Europe; Africa barely (25.3 → 22.8 years behind).
“Asia and the Americas converged towards Europe in health, but Africa did not, so the spread across all countries is unchanged.”
Smaller fixes
Before you submit, use Restart & Run All. Cell 22 has no execution number, so it was edited after the last run, and its output is older than its code.
Cell 33 prints p_unc = 0.0 because of .round(4). The real values are 4.2e-11 and 7.1e-07. Never report p = 0; write p < 0.0001, or format the p column separately.
Cell 51: “After 1982, Cambodia recovered rapidly” should read “by 1982”. The fast part is 31.2 → 51.0 (1977–1982); after that it is 51.0 → 59.7 over 25 years.
Cell 16: question d sits inside the explanation of question c. Put it in its own cell so that it is not missed.
Historical reasons such as Kuwait’s oil boom, the HIV/AIDS epidemic, the 1997 crisis and family planning are reasonable context, but this dataset cannot test them. Present them as interpretation, not as findings.
1a – 1b · PEARSON AND SPEARMAN
Correlation matrices in 1952, 1987 and 2007
Worked solution
Correlation matrices in 1952, 1987, 2007
What you need. Pearson r measures linear association; Spearman ρ is Pearson on ranks, so it measures monotonic association. See C02.
1
Compute both matrices per year with d[cols].corr() and corr(method="spearman").
Why: Spearman is high every year: richer countries are nearly always healthier.
3
pop: |r| ≤ 0.06 and |ρ| ≤ 0.18 with both variables in all years.
Why: population is unrelated to health and income.
4
Notice the gap Spearman − Pearson, largest in 1952 (0.769 vs 0.278).
Why: a big gap means “monotonic but not linear” or “an outlier”: that is the next question.
Answer. Strong monotonic gdp–lifeExp link every year (ρ 0.77–0.89); Pearson weak only in 1952; population unrelated.
✔ What it provesComparing Pearson and Spearman tells you whether a relation is linear or only monotonic.
✖ What it does not sayCorrelation is not causation, and a low r does not mean “no relation” — only no linear one.
→ Where it leadsQ2 finds the single country responsible for 1952’s low r.
pair
Pearson 1952
1987
2007
Spearman 1952
1987
2007
lifeExp–gdp
0.278
0.750
0.679
0.769
0.892
0.857
lifeExp–pop
−0.003
0.033
0.048
0.175
0.053
0.003
gdp–pop
−0.025
−0.052
−0.056
0.087
0.008
−0.065
Description (1a): GDP per capita and life expectancy are positively related, strongly in 1987 and 2007 but only weakly in 1952 by Pearson. Population is linearly unrelated to both in every year. Observation (1b): Spearman is high every year (0.77–0.89) and always above Pearson. The relationship is monotonic but not linear, and in 1952 something specific is suppressing Pearson.
gdp–lifeExp: strong monotonic link every year · pop: no link
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
Why Spearman beats Pearson here
Ranks measure monotonicity
Spearman turns values into ranks first. It asks “does y go up when x goes up?”, whatever the rate.
Concave curve (Preston curve)
Income buys many years at low GDP and few at high GDP. A straight line underfits it: in 2007, r = 0.679 on raw GDP and 0.809 on log GDP.
Robust to outliers
Kuwait at $108,382 or at $15,000 has the same rank, 142. So it cannot pull Spearman around.
1c – 1d · WHY IS PEARSON SO LOW IN 1952?
One country explains it
Worked solution
Why is Pearson so low in 1952?
What you need. Pearson uses the actual distances from the mean, so one extreme point can dominate it; ranks (Spearman) and logs reduce that influence.
1
Look at the scatter plot. Kuwait: $108,382 per person, seven times the next (Switzerland $14,734), with an ordinary 55.6 years.
Why: always plot before trusting r.
2
Remove Kuwait: Pearson 0.278 → 0.747.
Why: a single point was flattening the fitted line.
3
Keep Kuwait, log-transform GDP: Pearson = 0.748.
Why: the log pulls the extreme value back towards the others.
4
Check Spearman: 0.769 — Kuwait is just rank 142, so it was never fooled.
Figure. One country changes Pearson’s r from 0.28 to 0.75. Removing Kuwait or taking log₁₀(GDP) gives the same answer as Spearman, which only uses ranks. Numbers from the TP02 page.
Answer. The low 1952 r is caused by one outlier on a skewed axis, not by a weak relationship.
✔ What it provesA single high-leverage point can change Pearson from weak to strong; three remedies agree.
✖ What it does not sayIt does not license deleting inconvenient points: Kuwait is real data. Report the choice (log scale or rank method) instead of silently dropping it.
→ Where it leadsOutliers vs leverage points (C03); robust statistics.
Kuwait’s 1952 GDP per capita is $108,382, seven times the next country (Switzerland, $14,734), with an ordinary life expectancy of 55.6 years. On a linear axis it sits alone and flattens the fitted line.
Drop Kuwait: Pearson jumps from 0.278 to 0.747. Keep Kuwait but use log₁₀(gdpPercap): Pearson = 0.748. Spearman (0.769) was never fooled, because Kuwait is simply rank 142.
1952’s low r is an outlier + skew effect, not a weak relationship
Remark (cell 19): a Pearson matrix is a fast screen, but outliers, non-linearity, small samples and confounders can all distort it. Confirm with Spearman, a log transform and a scatterplot before interpreting.
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
Why one point has so much power
Pearson is r = Σ(x−x̄)(y−ȳ) / √[Σ(x−x̄)² · Σ(y−ȳ)²]. Your cell 18 says Kuwait inflates the denominator much more than the numerator. The numbers confirm it.
Kuwait is 11.2 standard deviations above the mean GDP. It supplies 89.4% of the GDP spread in the denominator but only 15.2% of the co-movement in the numerator, and this is why r collapses.
2a · 3a · η² BY YEAR
Do health and economy differ across continents?
Worked solution
How much does continent explain? (η²)
What you need. η² = BSS / TSS: the share of total variance that lies between group means. 0 = no difference between continents, 1 = all variation is between continents.
1
Compute TSS (total sum of squares around the global mean) and BSS (sum over continents of nk·(meank − global mean)²).
Why: continent explains about 60 % of life-expectancy differences every year.
3
gdpPercap: 0.048 / 0.471 / 0.424 — the 1952 value is suspicious.
Why: Kuwait inflates the within-Asia variance, so BSS/TSS collapses.
4
log₁₀ gdpPercap: 0.431 / 0.516 / 0.513 — stable.
Why: the same outlier fix as for Pearson.
Answer. Continent explains ≈ 0.6 of lifeExp variance and ≈ 0.4–0.5 of log-GDP variance; 0.048 in 1952 is an outlier artefact.
✔ What it provesη² gives an effect size, not just yes/no: “how much” continent matters.
✖ What it does not sayη² is descriptive and outlier-sensitive; it does not test significance and does not say which continents differ.
→ Where it leadsQ4 tests the difference with ANOVA and checks its assumptions.
η² with continent
1952
1987
2007
lifeExp
0.650
0.592
0.635
gdpPercap
0.048
0.471
0.424
log₁₀ gdpPercap
0.431
0.516
0.513
Your notebook’s pingouin values (0.650 and 0.048 for 1952) match. Are they reasonable? lifeExp: yes, continent explains about 60% of its variance every year. gdpPercap in 1952: no. Kuwait inflates the within-Asia sum of squares so much that BSS/TSS collapses to 0.05, the same outlier problem as Pearson (η² is also sensitive to outliers, slide 23). On log GDP the 1952 value is 0.43 and the series is stable.
continent explains lifeExp (≈ 0.6) more than GDP (≈ 0.4–0.5 on log scale)
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
The key correction: is the economic divide a later phenomenon?
The raw η² climbs from 0.048 to 0.424, which looks like continents drifting apart. But one country sits inside that number. Recompute it for every year without Kuwait.
all countrieswithout Kuwaitlog GDP, all countries
η² looks like it climbs from 0.05 to 0.42. Without Kuwait it is flat: 0.44 to 0.45.
Pearson, all countriesPearson, without KuwaitSpearman, all countries
Same story for r: 0.28 to 0.75 with Kuwait, 0.75 to 0.75 without it.
year
Kuwait GDP/cap
η² all
η² no Kuwait
η² log GDP
r all
r no Kuwait
1952
108,382
0.048
0.443
0.431
0.278
0.747
1962
95,458
0.101
0.469
0.456
0.383
0.732
1972
109,348
0.142
0.418
0.438
0.460
0.693
1982
31,354
0.419
0.451
0.488
0.723
0.730
1987
28,118
0.471
0.494
0.516
0.750
0.751
1997
40,301
0.398
0.424
0.497
0.704
0.709
2007
47,307
0.424
0.448
0.513
0.679
0.682
Conclusion. Continent explains about the same share of income differences throughout (0.40–0.49 without Kuwait, with a mild rise on the log scale, 0.43 → 0.51). Income levels did spread in ratio: Europe’s mean was 4.5× Africa’s in 1952, 7.5× in 1987 and 8.1× in 2007. Both are true; they measure different things.
2c · ONE-WAY ANOVA (2007)
Confirm the continent effect, then question the test
Worked solution
ANOVA 2007, then check its assumptions
What you need. One-way ANOVA tests H₀: all group means are equal, assuming normal groups with equal variances. Levene tests equal variances; Shapiro–Wilk tests normality; Kruskal–Wallis is the rank-based alternative.
1
Run ANOVA: lifeExp F(4, 137) = 59.71, p = 4.2e-29; gdpPercap F = 25.24, p = 1.1e-15 → reject H₀.
2
Check equal variances: Levene W = 6.82 (p = 4.9e-05) and 6.90 (p = 4.3e-05) → rejected. Europe’s lifeExp sd is 3.0, Africa’s 9.6.
Why: ANOVA’s F assumes one common within-group variance.
3
Check normality: Shapiro–Wilk rejects for several groups (e.g. Africa lifeExp p = 0.011); Oceania has only 2 countries.
Why: with 2 observations a group mean and variance are barely estimable.
4
Use robust alternatives: Welch’s ANOVA without Oceania: lifeExp F = 84.1 on (3, 68.6) df, gdp F = 38.9 on (3, 51.1) df, both p < 10⁻¹⁵; Kruskal–Wallis H = 88.10 and 71.09, p ≈ 10⁻¹⁸ and 10⁻¹⁴.
Why: if every method agrees, the conclusion does not depend on the violated assumption.
Answer. Continents differ significantly in both lifeExp and gdpPercap; plain ANOVA’s assumptions fail, but Welch and Kruskal–Wallis confirm the result.
✔ What it provesThe continent effect is real and robust to the choice of test.
✖ What it does not sayA tiny p-value says the means are not all equal, not how large the differences are or which pairs differ (use η² and post-hoc tests such as Games–Howell).
→ Where it leadsPost-hoc pairwise comparisons; transforming gdpPercap before testing.
2007
ANOVA F (4, 137)
p
Levene W
p
Kruskal H
p
lifeExp
59.71
4.2e-29
6.82
4.9e-05
88.10
3.3e-18
gdpPercap
25.24
1.1e-15
6.90
4.3e-05
71.09
1.3e-14
Result: both p-values are tiny → reject “all continents have the same mean”.
Reliable? Not fully. ANOVA assumes normal groups with equal variances and reasonable group sizes. Here Levene rejects equal variances for both variables (Europe’s lifeExp sd is 3.0, Africa’s 9.6); Shapiro–Wilk rejects normality for Africa, Asia and Europe lifeExp (p = 0.011, 0.021, 0.033) and for gdpPercap in Africa, Americas and Asia (p < 0.001); and Oceania has only 2 countries.
Fixes: (1) Welch’s ANOVA, which drops the equal-variance assumption: without Oceania, lifeExp F = 84.1 on (3, 68.6) df and gdpPercap F = 38.9 on (3, 51.1) df, both p < 10⁻¹⁵. (2) Kruskal–Wallis on ranks (table above). (3) Log-transform gdpPercap before testing. (4) Merge Oceania with another region or drop it. Every route reaches the same conclusion.
continents differ significantly · confirmed by Welch and Kruskal–Wallis
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
Does log(GDP) fix the assumptions? Only partly
2007
Africa
Americas
Asia
Europe
Levene p
Shapiro p, GDP
< 0.001
< 0.001
< 0.001
0.088
0.00004
Shapiro p, log GDP
0.008
0.777
0.029
0.005
0.00157
Skew, GDP
1.66
2.16
1.20
-0.10
Skew, log GDP
0.53
-0.10
0.22
-0.72
The log helps three continents but over-corrects Europe, which is now left-skewed. That is why the robust tests still matter.
Welch ANOVA, 2007
F
df
p
lifeExp, all 5 continents (your cell 33)
81.51
4, 17.58
4.2e-11
log GDP, all 5 continents (your cell 33)
53.26
4, 10.39
7.1e-07
lifeExp, without Oceania
84.10
3, 68.64
6.1e-23
log GDP, without Oceania
65.29
3, 68.48
4.7e-20
Why is df₂ so small with Oceania? Welch weights each group by n/variance and pays for small groups in the degrees of freedom. With n = 2 in Oceania, df₂ is 17.6 instead of 68.6. Every version still rejects equal means, so your conclusion holds. η² on log GDP is 0.513, against 0.424 on raw GDP.
2d – 2e · POPULATION AS SIZE, THEN DIRECTLY
Does population matter?
Worked solution
Does population matter?
What you need. Encode a third variable as bubble size; then plot it directly on a log axis; confirm with Spearman.
1
Bubble chart (x = gdpPercap on log scale, y = lifeExp, size = pop): China and India sit in the middle of the cloud.
Why: large and small countries are spread everywhere, so size shows no pattern.
2
Plot pop directly on a log x-axis against lifeExp and gdpPercap: no trend.
Why: a log axis is needed because pop spans 0.06 M (1952) to 1,319 M (2007).
3
Confirm with a number: 2007 Spearman pop–lifeExp = 0.003, pop–gdp = −0.065.
Answer. Population size is unrelated to life expectancy and income (|ρ| < 0.07 in 2007).
✔ What it provesNo monotonic association between country size and health or wealth in this data.
✖ What it does not sayCountry-level data cannot answer person-level questions (ecological fallacy); population density or growth might still matter.
→ Where it leadsMultivariate analysis: controlling for continent before judging a variable.
The bubble chart in C02 · Graphs encodes pop as bubble area: China and India are huge bubbles sitting in the middle of the cloud, while small and large countries are spread over the whole range. Plotted directly (pop on a log axis), neither lifeExp nor gdpPercap shows a trend: 2007 Spearman pop–lifeExp = 0.003 and pop–gdp = −0.065. Population size does not help predict health or income.
pop is unrelated to lifeExp and gdpPercap (|ρ| < 0.07 in 2007)
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
What the bubble chart adds
Five variables in one plot
x = GDP (log), y = lifeExp, area = pop, colour = continent, panel = year.
One country, one vote
Your correlations give China (1,319 M) and São Tomé (0.2 M) the same weight, 1/142 in 2007. Bubble size shows what that hides.
China + India ≈ 39% of the population
Share of the 142 countries’ population: 38.6% in 1952, 38.9% in 2007. Your “over one-third of humanity” holds for this dataset.
4a · EVOLUTION 1952 → 2007
How the four indicators change over time
Worked solution
Four indicators, 1952 → 2007
What you need. Recompute Pearson, Spearman and η² for each of the 12 years and plot them together.
1
Loop over years (groupby("year")) and store the four statistics.
2
Pearson 0.28 (1952) → 0.75 (1987) → 0.68.
Why: Kuwait’s GDP falls from $109k (1972) to $31k (1982); once it rejoins the other rich countries the linear link appears.
3
Spearman stays high: 0.77 → 0.90 (1992) → 0.86.
Why: the ranking of rich vs healthy barely changes.
4
η²: lifeExp ≈ 0.6 (dip 0.57 in 1992); gdp 0.05 → ≈ 0.42.
Why: most of that rise is Kuwait’s fall, not continents separating: without Kuwait η² is 0.44 in 1952 and 0.45 in 2007.
Answer. The health–wealth ranking is stable; the rise in both Pearson and η²(GDP) is mostly Kuwait falling back among the other countries.
✔ What it provesTracking a statistic over time separates a real trend from one-off distortions.
✖ What it does not sayTwelve correlations on overlapping country sets are not independent tests; a trend in r is descriptive.
→ Where it leadsPanel-data models (country fixed effects) for change within countries.
Pearson climbs from 0.28 (1952) to a peak of 0.75 (1987), then eases to 0.68: Kuwait’s GDP per capita falls from $109k (1972) to $31k (1982), back among other rich countries, and the linear link appears.
Spearman is high throughout (0.77 → 0.90 in 1992 → 0.86): the ranking of rich vs healthy barely changes.
η² continent → lifeExp stays near 0.6, with a dip in 1992 (0.57) when African life expectancy stalls while others keep rising.
η² continent → gdp rises from 0.05 to about 0.42, but this is mostly Kuwait falling back: without Kuwait it stays at 0.44–0.45 (see the η² section).
from scipy.stats import spearmanr
rows = []
for yr, d in gapminder.groupby("year"):
rows.append({"year": yr,
"Pearson": d.lifeExp.corr(d.gdpPercap),
"Spearman": spearmanr(d.lifeExp, d.gdpPercap)[0],
"eta2 lifeExp": eta2(d, "lifeExp"),
"eta2 gdp": eta2(d, "gdpPercap")})
ev = pd.DataFrame(rows).melt("year", var_name="indicator")
px.line(ev, x="year", y="value", color="indicator", markers=True)
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
The dip in η²(lifeExp) around 1992
BotswanaSouth AfricaSwazilandZimbabwe
From their peak these countries lost 13 to 22 years by 2002–2007. The data show the fall; the link to HIV/AIDS is historical context from outside the dataset.
Africa’s average stalls: 53.3 (1987), 53.6 (1992), 53.6 (1997), 53.3 (2002). Meanwhile other continents keep rising. Your cell 48 is right that this widens the gap with Africa, and η²(continent, lifeExp) goes from 0.573 (1992) to 0.636 (2002).
4b · CAMBODIA vs THAILAND
Population, life expectancy and GDP side by side
Worked solution
Cambodia vs Thailand on three indicators
What you need. Put the three series on comparable axes (log scale for GDP and population), then compare start, end and path.
1
Endpoints: lifeExp KH 39.4 → 59.7, TH 50.8 → 70.6; pop KH 4.7 → 14.1 M, TH 21.3 → 65.1 M; gdp/cap KH $369 → $1,714, TH $758 → $7,458.
2
Population: both triple.
3
GDP: Thailand ≈ ×10 on a steady log-linear path (pause after 1997); Cambodia flat for 40 years, dips in 1972, ×1.9 between 2002 and 2007.
Why: a straight line on a log axis = constant growth rate.
4
Life expectancy: the gap is 11 years in both 1952 and 2007, but Cambodia fell from 45.4 to 31.2 years between 1967 and 1977 and only passed its 1967 level again in 1982.
Why: equal endpoints can hide very different paths.
Answer. Same population growth, very different income paths; the health gap returned to its 1952 size after Cambodia’s collapse and recovery.
✔ What it provesA side-by-side, log-scaled comparison shows growth rates and breaks clearly.
✖ What it does not sayTwo countries are a case study, not evidence for a general rule; GDP per capita ignores inequality within a country.
→ Where it leadsThe one-paragraph synthesis (next section).
KH 1952
KH 2007
TH 1952
TH 2007
lifeExp
39.4
59.7
50.8
70.6
pop (M)
4.7
14.1
21.3
65.1
gdp/cap
369
1,714
758
7,458
Both populations triple. Thailand’s GDP grows almost tenfold on a steady log-linear path (a pause after the 1997 crisis); Cambodia’s is flat for 40 years, falls in 1972 and only takes off after 2002 (×1.9 between 2002 and 2007). The life-expectancy gap of 11 years is the same in 2007 as in 1952, but Cambodia fell from 45.4 to 31.2 years between 1967 and 1977 and only passed its 1967 level again in 1982.
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
The 1970s collapse, year by year
ThailandCambodia
Life expectancy by year. The recovery to 51.0 happens by 1982; after that the rise is slow, 51.0 to 59.7 over 25 years.
Your claim checks out: Cambodia in 1977 is the lowest in the world that year (31.2; next is Sierra Leone at 36.8). The gap with Thailand is 11.4 years in 1952 and 10.9 in 2007, but 31.3 years in 1977.
4c · THE WORLD 1952 → 2007
A one-paragraph answer
Between 1952 and 2007 the median country gained 27 years of life expectancy (45.1 → 71.9) and tripled its GDP per capita ($1,969 → $6,124). Health and wealth stayed strongly linked (Spearman 0.77–0.90 throughout; Pearson’s rise from 0.28 to 0.68 is mainly Kuwait falling back), and the link is logarithmic: each tenfold rise in income buys roughly the same gain in years. Continents explain about 60% of the differences in life expectancy and, once Kuwait is set aside, about 40–50% of income differences all along. Asia and the Americas closed most of their health gap with Europe, while Africa stalled from the late 1980s. Population size played no role in either.
healthier and richer · Asia and Americas catch up, Africa lags · pop irrelevant
FROM YOUR NOTEBOOK · CHECKED ON THE DATA
Is the world converging in health?
If health had converged globally this line would keep falling. It narrows until 1987, then returns to its 1952 level.
Years behind Europe (continent means)
1952
2007
Americas
11.1
4.0
Asia
18.1
6.9
Africa
25.3
22.8
Everyone rose, but not together. 128 of 142 countries gained in both life expectancy and GDP per capita from 1952 to 2007, so “almost all moved up and to the right” is right. Asia and the Americas closed most of their gap with Europe; Africa did not, and that keeps the overall spread at its 1952 level. Twelve countries ended poorer than they began, and two (Swaziland and Zimbabwe) ended with lower life expectancy.
Self-check
1. η²(continent, GDP) goes from 0.05 in 1952 to 0.42 in 2007. Can you conclude continents grew apart economically?
Not from that alone. Kuwait inflates the within-Asia variance in 1952. Without it, η² is 0.44 in 1952 and 0.45 in 2007. Always check for a single influential point.
2. In 1952 Kuwait supplies 89% of the denominator’s GDP spread but only 15% of the numerator. What happens to Pearson’s r?
It falls, because the denominator grows far more than the numerator: r is 0.278 with Kuwait and 0.747 without it.
3. Why does Kuwait not change Spearman’s ρ much?
Spearman uses ranks. However extreme its GDP, Kuwait is just the top rank.
4. A log transform reduces skew in three continents but Shapiro–Wilk still rejects in three. What should you do?
Keep the log for interpretation if useful, but use a robust test (Welch on the log scale, or Kruskal–Wallis), and report that the assumptions are only partly met.
5. A notebook prints p_unc = 0.0. How should you report it?
A p-value is never exactly 0: the zero comes from rounding. Report p < 0.0001, or the actual value in scientific notation.
surveys, interviews, observations, experiments. You control the design, and pay for it.
SECONDARY
Collected by others
government reports, books, Kaggle, NGO and industry data. Cheap, but incompleteness is the most common problem.
FORMAT
Structured vs unstructured
tables with a schema (Excel, CSV) vs free text, images, audio, video, web pages.
The six dimensions of data quality
1
Timeliness
Up to date for the use? Last year’s stock prices won’t price today’s trade.
2
Uniqueness
No accidental duplicates. A patient counted twice biases results; a row in both train and test inflates accuracy.
3
Validity
Values inside the valid range: no zero heights, no negative cholesterol. Needs domain knowledge.
4
Consistency
Same format and units everywhere: 15/03/2004 vs 03/15/2004, “Male” vs “M”, height in m vs cm (slide 12).
5
Accuracy
Measures what it should. The hardest to detect: a non-anonymous survey invites polite answers.
6
Completeness
No missing values: NaN, NA, null, or disguised as 0 or −1.
03Definitions & rules
MCAR
Missing completely at random
Missingness unrelated to the column itself and to every other column. Dropping rows leaves other distributions unchanged (it only loses power).
MAR
Missing at random
Missingness depends on other observed columns, not on the missing value itself. Dropping shifts those columns; impute conditionally on them.
MNAR
Missing not at random
Missingness depends on the unobserved value (very high or low incomes left blank). Needs domain knowledge or extra data; if few, dropping is common.
Rules of thumb (slide 24)
Share of NA in a column
Rule
< 5%
drop the rows
5 – 10%
can drop with a large sample, but check the missingness type
10 – 20%
impute according to the type
20 – 30%
remove the column if it is not important
> 30%
remove the column
Z-SCORE
z = x − x̄sx
mean 0, sd 1; unit-free
MIN–MAX
x′ = x − minmax − min ∈ [0, 1]
one outlier squeezes all others
POSITIVE DATA
x → log(x) or √x
tames a right tail; log(1 + x) if zeros
CAPPING vs TRIMMING
Capping limits extremes to a threshold (e.g. the fence); trimming removes them. No absolute solution: it depends on the analysis.
04Worked examples
Run on the 891-row Titanic training file. The lecture’s own example uses a file whose counts (Age 86 NA, Fare 1, Cabin 327) match Kaggle’s 418-row test set, so numbers differ but the method is the same.
EXAMPLE 1 · DISGUISED MISSING VALUES (SLIDE 19)
What’s wrong with this data?
Gender
Age
Height
Weight
F
68
0
20
F
68
0
18
F
65
105
0
F
63
0
15
F
68
112
0
A child cannot have height 0 cm or weight 0 kg: these are validity failures, i.e. missing values disguised as 0.
Convert them first: df[["Height","Weight"]] = df[["Height","Weight"]].replace(0, np.nan).
Only then compute statistics; leaving the zeros would drag the mean height far down.
Next question: are they MCAR, MAR or MNAR? That decides drop vs impute.
zeros in Height/Weight are NaN in disguise → recode before anything else
EXAMPLE 2 · APPLY THE RULES OF THUMB
A decision for each incomplete column
deck 77.2% → remove the column (the lecture drops Cabin for the same reason).
age 19.9%, at the edge of the “impute” band, and important for survival → impute, after checking its type (Example 3).
embarked 2 rows (0.2%) → drop them. Both are first-class passengers with fare £80 who survived; drop or fill with the mode “S”.
Survived, Pclass are int64 → convert to category before analysis.
EXAMPLE 3 · IS AGE MCAR? (NEW)
Compare the missingness across groups
Under MCAR the share of missing ages would be about the same (19.9%) in every class. It is 13.9%, 6.0% and 27.7%.
Survival rate: 38.4% overall but 29.4% among passengers with no recorded age. Dropping those rows would make the sample look more likely to survive.
So missingness depends on observed columns (class, survival) → behaves like MAR. (We cannot rule out MNAR from the data alone.)
Impute conditionally: the median age is 37 / 29 / 24 in 1st / 2nd / 3rd class, so a single global median (28) is too old for 3rd class and too young for 1st.
Age missingness depends on Pclass → impute by class (or model-based), don’t just drop
The lecture compares distributions before and after dropping and concludes “barely impacts, may be MCAR” on its file. Both checks are valid; they answer slightly different questions. The class-wise rate above is the sharper test because it compares the missing group directly with the rest.
EXAMPLE 4 · WHAT MEDIAN IMPUTATION DOES
Filling 177 ages with 28
The 28–32 bin jumps from 91 to 268 passengers and the standard deviation shrinks from 14.53 to 13.02: every imputed value sits exactly at the centre, so spread and correlations are understated. Class-wise medians soften this (sd 13.24); KNNImputer or IterativeImputer (slides 37–38) spread the filled values by using Fare and family size.
simple imputation keeps n but shrinks variance (14.53 → 13.02)
EXAMPLE 5 · SCALING BY HAND
Ages 22, 38, 26, 35, 54
x
22
38
26
35
54
z = (x − 35) / 12.45
−1.04
0.24
−0.72
0.00
1.53
(x − 22) / 32
0.000
0.500
0.125
0.406
1.000
x̄ = 175 / 5 = 35; Σ(x − 35)² = 169 + 9 + 81 + 0 + 361 = 620; s = √(620/4) = 12.45. min = 22, max = 54, range 32.
z: mean 0, sd 1 · min–max: [0, 1]
EXAMPLE 6 · CAP OR TRANSFORM FARE?
Three treatments of the 116 high fares
RAW
mean 32.20 · skew 4.79
CAPPED AT 65.63
mean 24.05 · skew 1.08
log(1 + fare)
skew 0.39 · order kept
Capping changes 116 real prices to the same value and erases the difference between a £70 and a £512 ticket. The log keeps every value and its order and gives the most symmetric shape. Trimming would delete 13% of the passengers, mostly first class. For fare, transform; reserve capping for values you believe are errors.
log(1 + fare): skew 4.79 → 0.39 without losing information
EXAMPLE 7 · ONE-HOT ENCODING (SLIDE 29)
continent = [Asia, Europe, Africa]
row
Africa
Americas
Asia
Europe
Oceania
0 Asia
0
0
1
0
0
1 Europe
0
0
0
1
0
2 Africa
1
0
0
0
0
Each row has exactly one 1, so the five columns always sum to 1. With an intercept in a linear regression that is perfect collinearity (the “dummy trap”): keep K − 1 columns with pd.get_dummies(..., drop_first=True). Never encode a nominal variable as 1, 2, 3, 4, 5: that invents an order and distances.
05Outliers, leverage, influence
Detect univariate outliers early, because they cause trouble later in multivariate work.
Why the 1.5 · IQR fence: 99.3% of a normal sample falls inside
Standard normal density, drawn exactly. Q1, Q3 = ∓0.674σ; fences = ∓2.698σ.
Outlier vs high leverage vs influential (slides 26–27)
Illustration: 25 simulated points (fixed seed) plus one special point in orange. Dashed = fit without it, solid = fit with it.
Outlier: extreme y for its x. Moves the intercept a little, the slope barely.
High leverage: extreme x. Harmless if it follows the trend.
Influential: removing it changes the analysis (slope, predictions, test results). Extreme x off the trend is the classic case, like Kuwait in 1952 (TP02) or Anscombe’s set IV (C02).
Missingness matrix · first 120 Titanic rows (new)
Orange = missing. One row per passenger: deck is mostly empty, age has scattered gaps, embarked has one gap (row 61). Patterns lined up across columns would hint that values go missing together.
06Python, with real output
preprocess.pypandas · scikit-learn
import numpy as np, pandas as pd, seaborn as sns
from sklearn.experimental import enable_iterative_imputer # noqafrom sklearn.impute import IterativeImputer, KNNImputer
t = sns.load_dataset("titanic")
(t.isna().mean() * 100).round(1).sort_values(ascending=False).head(4)
# deck 77.2 · age 19.9 · embarked 0.2 · embark_town 0.2
t = t.drop(columns=["deck"]).dropna(subset=["embarked"])
t[["survived", "pclass"]] = t[["survived", "pclass"]].astype("category")
t.groupby("pclass", observed=True).age.apply(lambda s: s.isna().mean()) # 0.139 0.060 0.277
t["age_cls"] = t.age.fillna(t.groupby("pclass", observed=True).age.transform("median"))
X = t[["age", "fare", "sibsp", "parch"]]
t["age_knn"] = KNNImputer(n_neighbors=5).fit_transform(X)[:, 0]
t["age_iter"] = IterativeImputer(random_state=0).fit_transform(X)[:, 0]
t["log_fare"] = np.log1p(t.fare) # skew 4.79 -> 0.39
t["fare_z"] = (t.fare - t.fare.mean()) / t.fare.std()
X_cat = pd.get_dummies(t["embarked"], drop_first=True) # Q, S (C is the baseline)
07Common mistakes & exam traps
Counting only NaN. Zeros, −1, “?”, “unknown” can all be missing values in disguise.
MCAR means the variable is unrelated? No: only the missingness is random (slide 20 warning).
Imputing before splitting. Fit imputers and scalers on the training set only, then apply to test.
Mean-imputing skewed data. Use the median (or a group median) when outliers exist.
Deleting every outlier. Only influential points that are errors deserve removal; genuine extremes may carry the signal.
log(0). Fare has 15 zeros: use log(1 + x) or handle zeros first.
Label-encoding nominal data. 1, 2, 3 for continents invents an order; one-hot instead.
Min–max with outliers. A single £512 fare puts half of all fares below 0.03 on the [0, 1] scale.
08Summary
1 · AUDIT
source, six quality dimensions, dtypes
2 · RECODE
disguised NA → NaN, fix types and formats
3 · MISSING
% per column → rule of thumb; MCAR/MAR/MNAR → drop or impute
4 · OUTLIERS
fences, leverage, influence → transform, cap or trim
5 · ENCODE
scale numbers, one-hot categories
09Self-check
1. A survey has 8% missing income, mostly among the highest earners. Which type, and what do you do?
MNAR (missingness depends on the income itself). Dropping biases income downwards; flag the rows, use external data or a model with domain assumptions, and report the limitation.
2. Height column: 1.65, 1.70, 176, 1.58. Which quality dimension fails and how do you fix it?
Consistency (mixed units, m vs cm, as on slide 12). Convert values above 3 from cm to m (divide by 100), then check validity ranges.
3. 25% of a column is missing and the column is weakly related to the target. Rule of thumb?
20–30% and not important → remove the column.
4. A point has an extreme x but lies exactly on the regression line. Outlier, leverage or influential?
High leverage, not influential: removing it would not change the fitted line much.
5. Why does pd.get_dummies(..., drop_first=True) matter for linear regression?
The full set of dummies sums to 1, which duplicates the intercept (perfect collinearity). Dropping one category makes it the baseline and the model identifiable.
10Glossary · FR ↔ EN
données primaires / secondaires ↔ primary / secondary data
qualité des données ↔ data quality
valeur manquante ↔ missing value
imputation ↔ imputation
doublon ↔ duplicate
point levier / influent ↔ leverage / influential point
écrêtage / troncature ↔ capping / trimming
centrage-réduction ↔ standardisation (z-score)
normalisation min-max ↔ min–max scaling
codage disjonctif complet ↔ one-hot encoding
11Sources
Course: III. Data Quality & Preprocessing (PDF, 39 slides), HAS Sothea.
Enders, Applied Missing Data Analysis (Guilford) and McKinney, Python for Data Analysis, 3rd ed. (the two references on slide 3).
Stacked density curves compare many distributions at once. The world moved from one peak near 40 years (1952) to a two-humped shape with most countries above 70 (2007).
Lesson II. Relationships between pairs of columns.
2-D histogramQUANT × QUANT · NEW
When points overplot (1704 country-years), count them per cell instead. Darker = more countries. The densest cell is rich and long-lived (around $10k, 70–75 years).
Two time points, one line per unit: steeper = faster gain. Cambodia gained 20 years but still ends last of the five; the 1977 collapse is invisible here, which is the limit of a slope chart.
Counts, not shares: you see that Africa has the most countries and Oceania only 2, which a 100% stacked bar hides. Pick by question: sizes (grouped) or profiles (stacked %).
sns.countplot(d7, x="continent", hue="gcat")
Mosaic plotQUAL × QUAL · NEW
Width = group size, height = share within the group, so area = count. The visual partner of the χ² example in C02 (χ² = 263, V = 0.54).
from statsmodels.graphics.mosaicplot import mosaic; mosaic(t, ["sex", "survived"])
<5 drop rows · 5–10 drop if n large · 10–20 impute · 20–30 drop column if unimportant · >30 drop column
missingness types
MCAR: unrelated to anything · MAR: depends on other observed columns · MNAR: depends on the missing value
outlier / leverage / influential
extreme y · extreme x · changes the fit when removed
Which indicator?
quant × quant
Pearson (linear) · Spearman (monotonic, robust)
ordinal × ordinal
Spearman
quant × qual
η² · ANOVA / Welch / Kruskal–Wallis
qual × qual
χ² test · Cramér’s V (bias-corrected)
χ² critical values, α = 5%
df 1 3.84
df 2 5.99
df 3 7.81
df 4 9.49
df 5 11.07
df 6 12.59
df 7 14.07
df 8 15.51
df 9 16.92
df 10 18.31
Numbers worth remembering
Titanic: 891 rows · 38.4% survived · age mean 29.7, median 28
Fare: median 14.45, mean 32.20, skew 4.79, 116 Tukey outliers
Gapminder 2007: r = 0.68, ρ = 0.86 (gdp–life)
η² continent → life 0.635, → gdp 0.424
1952 Kuwait: Pearson 0.28 → 0.75 without it
Sex × survival: χ² = 263, V = 0.54
Chapter → question map
No exam paper is in the folder yet, so this maps the question types the lessons and TPs prepare you for to the worked example that answers each. Replace with real exam questions when a paper is added.