Statistics and probability

Lesson 1 of 81890s–today7 min

Descriptive statistics

A start-up advertises an average salary of 127,000 euros. It has ten people: nine earn 30,000 and the founder pays herself a million. The average is arithmetically correct and tells a job applicant almost nothing; the median salary, 30,000, tells them what they will probably earn.

Descriptive statistics is the art of compressing a column of numbers into a few that do not lie. Karl Pearson, who introduced the term standard deviation in the 1890s, spent his career on exactly that.

Centre: mean, median, mode

The mean x̄ = ∑xᵢ / n is the balance point of the data. It uses every value, which makes it efficient for symmetric data and fragile when there are outliers. The median is the middle value of the sorted data (the average of the two middle values when n is even); it ignores how extreme the extremes are, so it is robust. The mode is the most frequent value, the only one of the three that makes sense for categories. In a right-skewed distribution, such as salaries, house prices or response times, the long tail drags the mean above the median, which is why official statistics report median income.

Spread: variance and standard deviation

Two datasets can share a mean and differ completely in how scattered they are. The variance averages the squared deviations from the mean; the standard deviation is its square root, back in the original units. For a whole population, divide by n: σ² = ∑(xᵢ − μ)² / n. For a sample used to estimate the population, divide by n − 1: s² = ∑(xᵢ − x̄)² / (n − 1). The reason is that deviations measured from the sample's own mean are on average slightly too small, because x̄ is fitted to the same data; dividing by n − 1 (Bessel's correction) makes s² an unbiased estimator of σ². Robust alternatives exist too: the interquartile range (IQR = Q3 − Q1) is the spread of the middle half of the data, and the basis of the box plot's outlier rule, 1.5 × IQR beyond the quartiles.

Worked exercise

Orders per day

A shop receives 4, 8, 6, 5, 3 and 10 orders on six days. Compute the mean, the median, the sample variance and the sample standard deviation. Then replace the 10 with 46 and recompute the mean and the median.

Try it first, then reveal the solution one step at a time.

python
from statistics import mean, median, variance, pvariance, stdev

orders = [4, 8, 6, 5, 3, 10]    # orders per day, six days
print(mean(orders))             # 6
print(median(orders))           # 5.5
print(variance(orders))         # 6.8   sample: divides by n - 1
print(pvariance(orders))        # 5.666...   population: divides by n
print(round(stdev(orders), 3))  # 2.608

orders[-1] = 46                 # one extreme day
print(mean(orders), median(orders))  # 12 5.5

Key terms

Mean
The sum of the values divided by their number; sensitive to outliers.
Median
The middle value of the sorted data; robust to outliers.
Sample variance
s² = ∑(xᵢ − x̄)² / (n − 1), an unbiased estimate of the population variance.
Standard deviation
The square root of the variance, in the same units as the data.
Interquartile range
Q3 − Q1: the spread of the middle 50% of the data.

Check what you learned

  1. 1.House prices in a town are strongly right-skewed. Which summary best describes a typical house?

  2. 2.Why does the sample variance divide by n − 1 instead of n?

  3. 3.Q1 = 20 and Q3 = 30. Under the 1.5 × IQR rule, which value is an outlier?

Practise it for the exam

Sources