Lesson 8.2: Evaluating and Improving Performance
Evaluating & Improving Model Performance
You can't improve what you don't measure. This lesson gives you the metrics, diagnostic tools, and iterative improvement process that turn a deployed model from a static artefact into a continuously improving system.
Measurement is the engine of improvement
A deployed AI model is not a finished product — it is the beginning of a measurement and improvement cycle. Without defined metrics and regular evaluation, you cannot know whether your model is improving or degrading, whether it is serving all user groups equally well, or whether a retraining effort actually made things better or worse.
This lesson teaches you to select the right metrics for the right context, read diagnostic outputs like confusion matrices, diagnose the patterns behind poor performance, and run a structured improvement process that turns measurement into action.
"You can't improve what you don't measure." This is not just a management cliché — in AI, it is literally true. Every improvement hypothesis (more data, different features, adjusted threshold, new architecture) must be validated against defined metrics using a consistent evaluation method. Otherwise you are guessing, not improving.
The language of model evaluation and iteration
These terms define both the measurement tools and the improvement techniques you will use throughout this lesson. Precision and recall in particular are frequently confused — know the distinction precisely.
| Term | Definition |
|---|---|
| Performance Metric | A quantitative measure of an AI system's effectiveness. Different tasks require different metrics — classification uses accuracy, precision, recall, F1; regression uses MAE, MSE, R²; clustering uses silhouette score. Choosing the wrong metric for your use case is one of the most common AI evaluation errors. |
| Precision | Of all the cases the model predicted as positive, what fraction were actually positive? Formula: TP ÷ (TP + FP). High precision means few false alarms — important when false positives are costly (e.g., blocking a legitimate bank transaction). |
| Recall (Sensitivity) | Of all the actual positive cases, what fraction did the model correctly identify? Formula: TP ÷ (TP + FN). High recall means few missed cases — critical when false negatives are dangerous (e.g., missing a cancer diagnosis or a fraudulent transaction). |
| F1-Score | The harmonic mean of precision and recall — a single number that balances both. Formula: 2 × (Precision × Recall) ÷ (Precision + Recall). Most useful when you need to balance both false positives and false negatives and can't prioritise one over the other. |
| Confusion Matrix | A table showing a classification model's results broken into four categories: True Positives (correctly predicted positive), True Negatives (correctly predicted negative), False Positives (incorrectly predicted positive), and False Negatives (incorrectly predicted negative). The confusion matrix is the starting point for all classification model diagnosis. |
| AUC-ROC | Area Under the Receiver Operating Characteristic curve — a metric that measures a model's ability to distinguish between classes across all decision thresholds. Values range from 0 to 1; above 0.9 is generally considered excellent. Particularly useful when class imbalance makes accuracy misleading. |
| MAE / MSE / R² | Regression metrics. Mean Absolute Error (MAE) is the average absolute difference between predictions and actuals — interpretable in the same units as your target. Mean Squared Error (MSE) penalises large errors more heavily. R² measures the proportion of variance in the target that the model explains — closer to 1 is better. |
| Cross-Validation | A technique for validating models by testing on multiple held-out subsets of data. In k-fold cross-validation, data is split into k parts; the model trains k times, each time using a different part as the test set. Provides more reliable performance estimates than a single train/test split. |
| Model Drift | The gradual degradation of model performance over time as real-world data patterns diverge from the training distribution. Can manifest as data drift (input distribution changes), concept drift (the relationship between inputs and outputs changes), or both. Regular monitoring and periodic retraining are the standard responses. |
| Hyperparameter Tuning | The process of systematically adjusting model settings (learning rate, tree depth, regularisation strength, number of hidden layers) to improve performance. Methods include Grid Search (exhaustive), Random Search (probabilistic), and Bayesian optimisation. Always use cross-validation during tuning to avoid overfitting to the validation set. |
| Overfitting / Underfitting | Overfitting: the model learns the training data too specifically and performs poorly on new data. Underfitting: the model is too simple to capture the true patterns. Both are diagnosed through learning curves — overfitting shows a large gap between training and validation performance; underfitting shows poor performance on both. |
Choosing the right metric for the right model
The single most common evaluation mistake in AI is applying the wrong metric to the wrong problem. Accuracy looks great on an imbalanced dataset. MAE is meaningless for a classification problem. Select each model type to see the right metrics, when to use each, and a real-world case study of iterative improvement.
Metric Selector — by Model Type
Select a model type · Explore the right evaluation metrics · See a real-world iteration example
Classification metrics — spam filter, fraud detector, medical diagnosis
Accuracy
Proportion of all predictions that were correct. Intuitive but misleading on imbalanced datasets — a model predicting "not fraud" 99.9% of the time looks 99.9% accurate but catches no fraud.
Precision
Of everything the model flagged as positive, how many were real? Prioritise when false positives are costly — blocking legitimate transactions, unnecessary biopsies, spam filtering of real emails.
Recall (Sensitivity)
Of all actual positives, how many did the model catch? Prioritise when false negatives are dangerous — missed cancer diagnoses, missed fraud, missed security threats.
F1-Score
The harmonic mean of precision and recall. Use when you need to balance both and can't prioritise one. Common default metric for imbalanced binary classification.
AUC-ROC
Measures discrimination ability across all thresholds. Especially valuable on imbalanced datasets where accuracy is misleading. Above 0.9 is generally excellent; 0.7–0.9 is acceptable depending on context.
Case Study — Bank Fraud Detection Iteration
A retail bank's fraud detection model starts with 85% recall and 70% precision. Goal: improve recall to 95% without precision dropping below 60%. Error analysis reveals a pattern — international transactions under $200 are being missed. The data science team adds international transaction pattern features and retrains using 5-fold cross-validation. Result: recall jumps to 96%, precision drops slightly to 68% — an acceptable trade-off the bank explicitly approved in advance. The improvement was validated on an identical held-out test set to ensure the comparison was apples-to-apples.
Regression metrics — house price prediction, sales forecasting, demand planning
MAE (Mean Absolute Error)
Average absolute difference between predictions and true values. Expressed in the same units as the target — easy to interpret. "Our forecast is off by $4,200 on average." Most stakeholder-friendly regression metric.
MSE / RMSE
Squared errors penalise large mistakes more heavily than small ones. RMSE (root MSE) brings units back to the target scale. Use when large errors are especially damaging — e.g., severely over-ordering inventory has a disproportionate cost.
R² (R-squared)
Proportion of variance in the target that the model explains. R² = 1.0 means perfect predictions. R² = 0.0 means the model does no better than predicting the mean. Useful for understanding how much of the problem the model actually solves.
MAPE (Mean Absolute Percentage Error)
Expresses error as a percentage of the actual value — makes comparisons across different scales possible. Problematic when actual values are near zero. Common in supply chain and demand forecasting contexts.
Case Study — Retail Demand Forecasting Iteration
An AI demand forecasting model has an initial MAE of $5,200 against a target of $3,500. R² is 0.61 — meaning the model explains 61% of demand variance. Error analysis shows the model performs well for standard products but poorly for seasonal items. The team adds seasonality features (week of year, holiday indicators) and incorporates 3-year promotional history. After retraining with cross-validation: MAE drops to $3,100 (below target), R² rises to 0.81. Both improvements validated on the same held-out test set used previously.
Clustering metrics — customer segmentation, anomaly grouping, document categorisation
Silhouette Score
Measures how similar each point is to its own cluster versus other clusters. Closer to +1 means well-separated, dense clusters. Near 0 means overlapping clusters. Negative values suggest points may be in the wrong cluster.
Inertia (Within-Cluster Sum of Squares)
Measures how tightly packed each cluster is — the sum of squared distances from each point to its cluster centre. Lower is better, but always decreases as you add more clusters (so use the "elbow method" to find the optimal number of clusters).
Business validation
Clustering metrics only tell you if segments are statistically distinct — not if they are useful. Always validate clusters against business criteria: do marketing teams recognise these customer types? Does each segment respond differently to interventions? This is the most important test.
Case Study — Customer Segmentation Iteration
An e-commerce company's customer segmentation model produces 4 clusters with a silhouette score of 0.32 — reasonable but not strong. Marketing reviews the segments and notes that two clusters behave almost identically in campaign response rates (within 2% of each other). The data scientist experiments with 3 clusters: silhouette score drops slightly to 0.29, but marketing confirms the 3-segment model is much more actionable. Lesson: statistical metrics and business utility are both necessary — neither alone is sufficient.
Time-series metrics — stock forecasting, energy consumption prediction, traffic prediction
MAE / RMSE over time
Standard regression metrics applied to time-series predictions. Track not just the overall error but how error changes over the forecast horizon — most models are more accurate at short horizons and degrade at longer ones.
MAPE (Mean Absolute Percentage Error)
Percentage-based error makes it comparable across different magnitude time periods. Common in supply chain, retail, and energy forecasting. Watch for instability near zero-value periods.
Drift monitoring metrics
Track error trend over time, not just error level. If RMSE has been increasing month-over-month for three consecutive months, the model is likely drifting. Setting automated alerts when error crosses a threshold is standard practice for production time-series models.
Baseline comparison
Always compare your model against naive baselines: "predict last week's value" (naïve), "predict the same week last year" (seasonal naïve), or simple moving averages. If your model barely beats a naïve baseline, the added complexity may not be justified.
Case Study — Energy Demand Forecasting Drift Detection
An energy utility's AI load forecasting model performs well for the first 18 months post-deployment. Monthly RMSE monitoring then shows a gradual upward trend over 4 months — from 2.1% to 3.8%. Investigation reveals the grid now serves three new industrial facilities with dramatically different consumption patterns that weren't present in training data. The model is retrained with 6 months of data including the new facilities and a seasonality re-calibration. RMSE returns to 2.0%. The drift monitoring alert that triggered investigation is estimated to have prevented $1.2M in balancing costs from over/under-procurement.
Read and diagnose a confusion matrix
The confusion matrix below comes from a fraud detection model evaluated on 1,000 transactions. The metrics are pre-calculated. Select a diagnostic question to read the interpretation — then you'll know exactly how to read any confusion matrix in practice.
Confusion Matrix Lab — Fraud Detection Model
Read the matrix · Calculate the metrics · Select a diagnostic to interpret the results
| Predicted: Fraud | Predicted: Not Fraud | |
|---|---|---|
| Actual: Fraud | 45 True Positive |
5 False Negative |
| Actual: Not Fraud | 40 False Positive |
910 True Negative |
Select a diagnostic question to interpret what these numbers mean:
Six steps from measurement to improvement
Improvement is not random — it follows a structured process. Expand each step to understand what happens there, what common mistakes to avoid, and what good practice looks like.
Before any tuning begins, document exactly what "better" means: which metric, what threshold, and what constitutes an acceptable trade-off. This prevents the common failure mode of iterating indefinitely with no clear stopping condition.
Example: "We will improve this fraud detection model when recall exceeds 95% AND precision stays above 60%, as validated on the held-out test set. Any result that doesn't meet both criteria will be discarded regardless of how good individual metrics look."
Use cross-validation or a rigorously held-out test set to evaluate every change. The cardinal rule: never tune against the test set. Once a model has been evaluated on a test set, that set's results are consumed — using the same data for further tuning leads to overfitting and inflated performance estimates.
For time-series data, always use temporal splits (train on past, test on more recent data) rather than random splits, which would leak future information into training.
Aggregate metrics (precision = 0.72) tell you there is a problem. Error analysis tells you where and why. Before changing anything, examine the actual errors the model is making. Look for patterns: are false negatives concentrated in a particular demographic, time period, geographic region, or data quality category?
Example: A medical imaging model has 88% overall recall but only 71% recall for patients over 70. This pattern, invisible in aggregate metrics, tells you exactly where to focus: gather more training examples for older patients and potentially apply different preprocessing for their imaging data.
Each improvement attempt should begin with a specific hypothesis: "Adding international transaction pattern features will improve recall for cross-border fraud by addressing the pattern identified in error analysis." This framing prevents random experimentation and makes it possible to learn from each attempt — even failed ones.
Test one change at a time wherever possible. If you change the model architecture, the feature set, and the training data simultaneously, you cannot attribute any improvement or regression to its cause.
Metric improvements don't always translate to business improvements — and business stakeholders often need to make explicit decisions about trade-offs that pure metric optimisation can't make for them. A fraud detection model that achieves 99% recall but generates 500 false alarms per day may overwhelm the investigation team and produce worse overall outcomes than a 90% recall model with 50 false alarms.
Similarly, a recommendation model that achieves higher click-through rate may drive short-term engagement at the expense of long-term customer satisfaction. Always ask: what does this metric improvement translate to in the real world, and is that actually what we want?
Deploying an improved model is not the end of the improvement cycle — it is the beginning of the next measurement phase. Set up monitoring dashboards with alerts for key metrics. Define drift thresholds that trigger investigation (e.g., if monthly recall drops below 85%, initiate error analysis). Schedule periodic retraining reviews (quarterly for most models, monthly for fast-changing domains).
Track both model metrics (precision, recall, error rates) and business metrics (fraud losses prevented, customer complaints, conversion rates). A gap between the two signals a misalignment between what the model is optimising and what the business actually needs.
Ask the Model Evaluation Coach
Get help interpreting metrics, diagnosing model problems, choosing the right evaluation approach, or designing a targeted improvement experiment. Type your question or use a quick prompt.
Test your understanding of Lesson 8.2
Select the best answer for each question, then submit for your score and detailed feedback.