The learning problem and loss functions

The formal setup

Supervised learning starts from examples (x, y), where x ∈ X is a feature vector and y the target: a real number in regression, a label in classification. The examples are assumed drawn independently from an unknown distribution D over X × Y; this i.i.d. assumption is what lets the past say anything about the future. The learner picks a hypothesis h from a hypothesis class H (all straight lines, all trees of depth 3, all networks of a given shape), and a loss function ℓ(h(x), y) ≥ 0 measures how wrong a single prediction is.

What we want is a small true risk R(h) = E[ℓ(h(x), y)], the expected loss on fresh data from D. We cannot compute it, because D is unknown. What we can compute is the empirical risk on the n training examples, R̂(h) = (1/n) ∑ ℓ(h(xᵢ), yᵢ). Empirical risk minimisation chooses the h in H with the smallest R̂. The whole subject lives in the gap between R̂ and R: a rich H can drive R̂ to zero by memorising, and that says nothing about R. Learning theory bounds the gap in terms of n and the capacity of H.

Common loss functions

• Squared loss (y − ŷ)²: the default for regression; its mean is the MSE. Smooth and convex in linear models, but one large error dominates because it is squared.
• Absolute loss |y − ŷ|: its mean is the MAE. More robust to outliers; its minimiser is the median rather than the mean.
• 0–1 loss: 1 if the predicted class is wrong, 0 otherwise. It is what accuracy counts, but it is flat almost everywhere, so it cannot be optimised by gradients.
• Cross-entropy (log loss) −[y log p + (1 − y) log(1 − p)] for a predicted probability p: the standard surrogate for classification, punishing confident wrong answers without limit.
• Hinge loss max(0, 1 − y·f(x)) with y ∈ {−1, +1}: the loss of support vector machines, zero once a point is on the correct side with a margin.

import numpy as np

y = np.array([3, 5, 2, 10.0])
y_hat = np.array([2.5, 5, 3, 4.0])

mse = np.mean((y - y_hat) ** 2)   # 9.3125
mae = np.mean(np.abs(y - y_hat))  # 1.875

# log loss for binary labels and predicted probabilities
labels = np.array([1, 0, 1, 1])
p = np.array([0.9, 0.2, 0.6, 0.01])
eps = 1e-15                        # avoid log(0)
p = np.clip(p, eps, 1 - eps)
log_loss = -np.mean(labels * np.log(p) + (1 - labels) * np.log(1 - p))
print(mse, mae, round(log_loss, 3))  # 9.3125 1.875 1.361