Performance Metrics

Source: Unit 1 §6

Once features are engineered and a model is trained, you test it and get an output: a class or a probability. Six metrics measure how good that output is - accuracy, precision, recall, specificity, ROC, AUC - and every one of them is read off the same table, the confusion matrix.

The confusion matrix

A confusion matrix is a layout of how many predictions were correct versus incorrect, per class. Take class A = apple and class B = all other fruits; there are exactly four outcomes.

The four outcomes, with A as the positive class. Read the diagonal for agreement and the off-diagonal for the two kinds of mistake.

Term	Meaning, taking A (positive) as the target
TP true positive	A correctly predicted as A
TN true negative	B correctly predicted as B
FP false positive	B incorrectly predicted as A
FN false negative	A incorrectly predicted as B

◎Exam cueThe error-type names

FP = Type-1 error. FN = Type-2 error. Same convention as the IDS course.

The goal is as many correct predictions as possible, which is to say more True than False: weight in the two diagonal cells rather than the two off-diagonal ones.

⚠GotchaPick the positive class before you compute anything

Every metric on this page is defined relative to whichever class you call positive. For a binary or one-vs-all problem, name the target class first; TP and FP swap meaning the moment you flip it, and so do precision and specificity.

A worked matrix

Take $N = 200$, made of 100 actual A and 100 actual B.

N = 200. Row totals 100 and 100; column totals 130 predicted-A and 70 predicted-B.

→StepsReading it off

1. Correct predictions = TP + TN = 60 + 30 = 90.
2. Wrong predictions = FN + FP = 40 + 70 = 110.
3. More wrong than right, so whatever accuracy comes out will be below 0.5.

Accuracy

The fraction of all predictions that were correct.

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

⚠GotchaAccuracy is only honest on balanced classes

On a dataset that is 99% negative, a model that predicts "negative" for everything scores 99% accuracy and has learned nothing. Accuracy is the right headline metric only when the target classes are close to balanced.

Precision and recall

$$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN}$$

⇄ComparePrecision vs recall, in one question each

PrecisionOf everything I flagged positive, how many were actually positive? It is a column of the matrix: predicted-positive.

Recall (sensitivity)Of all the real positives, how many did I catch? It is a row of the matrix: actual-positive.

Specificity and F1

$$\text{Specificity} = \frac{TN}{TN + FP} \qquad F_1 = \frac{2 \times \text{Recall} \times \text{Precision}}{\text{Recall} + \text{Precision}}$$

Specificity is recall's mirror image: of all the real negatives, how many did I catch? $F_1$ is the harmonic mean of precision and recall, which is why one terrible value drags it down instead of being averaged away.

≡FactsWhich slice of the matrix each rate uses

• Precision = TP / (TP + FP) - the predicted-positive column.
• Recall (TPR, sensitivity) = TP / (TP + FN) - the actual-positive row.
• Specificity = TN / (TN + FP) - the actual-negative row.
• Accuracy = (TP + TN) / total - the whole table.
• F1 = harmonic mean of precision and recall.

The worked numbers

Using TP = 60, FN = 40, FP = 70, TN = 30:

Metric	Substitution	Value
Precision	60 / (60 + 70)	0.46
Recall	60 / (60 + 40)	0.60
Specificity	30 / (30 + 70)	0.30
Accuracy	(60 + 30) / 200	0.45

ROC: receiver operating characteristics

A model like logistic regression outputs a probability, not a hard class. To turn a probability into a class you pick a threshold (0.5 by default): above it the prediction is positive (say, "obese"), below it negative.

Every threshold produces a different confusion matrix, and therefore a different (FPR, TPR) point. The ROC curve is what you get by sweeping the threshold from 0 to 1 and plotting all of them.

≡FactsThe two axes

• y-axis = TPR, the true positive rate, which is exactly recall / sensitivity.
• x-axis = FPR, the false positive rate, which is exactly $1 - \text{specificity}$.

Threshold	TPR	FPR
0 (everything classified positive)	1.0	1.0
0.3	1.0	0.75
0.4	1.0	0.5
0.6	0.75	0.25
0.7	0.75	0.0
0.9	0.5	0.0

Each threshold is one confusion matrix, and therefore one (FPR, TPR) point. The curve is the whole sweep; the diagonal is a coin flip.

A threshold is better when it buys a higher proportion of correctly classified positives for a lower proportion of incorrectly classified negatives, which is the same as saying it sits closer to the top-left corner. The value of the curve is that it summarises every confusion matrix at once, so thresholds can be compared at a glance.

AUC: area under the curve

AUC is the area under the ROC curve, collapsed to a single number.

AUC compresses the whole curve into one number, which is what makes two models comparable.

AUC value	Interpretation
Close to 1	Excellent - the model separates the classes very well
Around 0.5	Poor - the model cannot distinguish the classes, no better than random

◎Exam cueComparing two models

The greater the AUC, the better the model. Given two ROC curves, the one that encloses more area is the model to ship.