YouTip

Regression Model Evaluation

In the world of machine learning, building a model is just the first step, just like baking a cake, we must taste it to know if it is successful. For regression models (a type of model used to predict continuous numerical values, such as predicting house prices, temperatures, or sales), **model evaluation** is our **tasting** step. It tells us how accurate the model's predictions are, where it does well, and where there is room for improvement. This article will guide you through a systematic study of regression model evaluation methods, from core concepts to specific metrics, and then to code practice, enabling you to not only understand evaluation reports but also score your models with your own hands. * * *

Core Concepts of Regression Models and Evaluation

Before diving into the metrics, we need to understand a few basic concepts that are the cornerstone of all evaluation methods.

What is a Regression Problem?

A regression problem is a type of supervised learning whose goal is to predict a **continuous numerical value** based on input features. * **Example**: Predicting the selling price (a continuous numerical value) based on the house's area, age, and geographical location (features).

Why is Evaluation Needed?

1. **Model Selection**: Comparing which of different algorithms (such as linear regression, decision tree regression) performs better on your data. 2. **Parameter Tuning**: When adjusting model parameters (such as regularization strength), an objective standard is needed to judge whether the adjustment is effective. 3. **Performance Confirmation**: Ensuring that the model also has reliable predictive ability on unseen data, avoiding "armchair strategy" (theoretical planning without practical testing).

Key Term: Error

All evaluation metrics revolve around "error". Error is the difference between the model's predicted value (ŷ) and the actual true value (y). * **Error for a single point**: `Error = y - ŷ` * The essence of evaluation is to **aggregate and analyze these errors in different ways**. * * *

Detailed Explanation of Core Evaluation Metrics

We will introduce the most commonly used and important evaluation metrics, using a simple example throughout. **Suppose we predicted the prices of 5 houses:** | True Price (y) | Predicted Price (ŷ) | Error (y - ŷ) | | --- | --- | --- | | 200 | 210 | -10 | | 150 | 145 | 5 | | 300 | 310 | -10 | | 400 | 380 | 20 | | 250 | 255 | -5 |

1. Mean Absolute Error (MAE)

**Plain understanding**: Add up the absolute values of all prediction "errors", and then calculate an average. It directly reflects "on average, how many units the predicted value deviates from the true value". **Calculation Formula**: `MAE = (1/n) * Σ|y_i - ŷ_i|` where `n` is the number of samples, and `Σ` is the summation symbol. **Calculation Example**: `MAE = (|-10| + |5| + |-10| + |20| + |-5|) / 5 = (10+5+10+20+5)/5 = 50/5 = 10` **Interpretation**: On average, our house price prediction deviates from the true price by **10 (ten thousand yuan)**. **Characteristics**: * **Pros**: Intuitive and easy to understand; not overly affected by extreme error values (outliers). * **Cons**: The absolute value function is not differentiable everywhere mathematically, which is inconvenient in some optimization scenarios.

2. Mean Squared Error (MSE)

**Plain understanding**: First "square" each error (making negative signs disappear and amplifying the error), and then calculate the average. It is **very sensitive to large errors**. **Calculation Formula**: `MSE = (1/n) * Σ(y_i - ŷ_i)^2` **Calculation Example**: `MSE = [(-10)^2 + (5)^2 + (-10)^2 + (20)^2 + (-5)^2] / 5 = (100+25+100+400+25)/5 = 650/5 = 130` **Interpretation**: The average of the squared errors is 130. This value itself has no direct unit meaning (it is ten thousand yuan^2), but it is very effective for comparing models. **Characteristics**: * **Pros**: Excellent mathematical properties (differentiable everywhere); it is the objective function minimized during training for many models (like linear regression). * **Cons**: The dimension is different from the original data, making the numerical magnitude hard to interpret; sensitive to outliers.

3. Root Mean Squared Error (RMSE)

**Plain understanding**: It is simply the square root of MSE. This is equivalent to "reverting" the MSE back to its original form, bringing it back to the same dimension as the true and predicted values. **Calculation Formula**: `RMSE = sqrt(MSE)` **Calculation Example**: `RMSE = sqrt(130) ≈ 11.4` **Interpretation**: On average, our prediction deviates from the true value by approximately **11.4 (ten thousand yuan)**. This interpretation is similar to MAE (10 ten thousand yuan), but because RMSE squares first and then takes the root, it gives higher weight to large errors, so it is usually larger than MAE. **Characteristics**: * **Pros**: Has the same dimension as the original data, making it more interpretable than MSE; penalizes large errors more heavily, and is commonly used in scenarios where large errors are critical (such as financial risk prediction). * **Cons**: Also sensitive to outliers.

4. R² Score (Coefficient of Determination)

**Plain understanding**: How much better is my model than "blind guessing" (guessing using the average)? It is a **ratio value** that measures the model's ability to explain the variance in the data. **Calculation Formula**: `R² = 1 - (Σ(y_i - ŷ_i)^2 / Σ(y_i - y_mean)^2)` where `y_mean` is the average of the true values. **"Blind guessing" model**: For any house, I predict the average house price (`y_mean`). * Calculated `y_mean = (200+150+300+400+250)/5 = 260` * Sum of squared errors for the "blind guessing" model: `Σ(y_i - 260)^2 = 1600+12100+1600+19600+100 = 35000` **Calculation Example**: `R² = 1 - (650 / 35000) = 1 - 0.01857 ≈ 0.9814` **Interpretation**: Our model can explain **98.14%** of the variance in the target variable (house price). This indicates that the model fits very well. **Characteristics**: * **Range**: Theoretically, the range of R² is `(-∞, 1]`. * `R² = 1`: Perfect prediction. * `R² = 0`: The model performs exactly the same as simply predicting the average. * `R² < 0`: The model is worse than simply predicting the average (indicating the model is completely unsuitable for the data). * **Pros**: Dimensionless, making it easy to compare model performance across different datasets. * **Cons**: As model features increase, R² will naturally increase even if the added features are useless, which can lead to overfitting. * * *

Metrics Comparison and How to Choose

To understand more clearly, we use a table to summarize: | Metric | Calculation Formula (Brief) | Dimension | To Outliers | Characteristics & Applicable Scenarios | | --- | --- | --- | --- | --- | | **MAE** | Mean Absolute Error | Same as y | Insensitive | **Most intuitive interpretation**. Focuses on "how large is the average deviation", applicable to all regression scenarios, especially data with many outliers. | | **MSE** | Mean Squared Error | Squared y | **Very Sensitive** | **Good mathematical properties**. Commonly used as a loss function for model training. The value itself is hard to interpret. | | **RMSE** | Square root of MSE | Same as y | **Very Sensitive** | **Most commonly used**. Combines the mathematical advantages of MSE with interpretability. Penalizes large errors heavily; popular in finance, forecasting, etc. | | **R²** | 1 - (Model Error / Baseline Error) | Dimensionless | Sensitive | **Relative performance metric**. Used to compare the model's improvement over a simple baseline. Easy to compare across datasets. | **Selection Advice**: 1. **First choice: Report RMSE and R²**: RMSE gives the actual magnitude of the error, and R² gives the relative performance of the model. This is the most universal combination. 2. **Focus on MAE when there are many outliers in the data**. 3. **Use MSE as the loss function during model training and optimization phases**. 4. **Never look at only one metric**! Combining multiple metrics is necessary for a comprehensive model evaluation. * * *