When we build machine learning models, whether to forecast oil production, predict equipment failure, or detect operational anomalies, it’s tempting to jump straight to accuracy. After all, accuracy feels like a neat, tidy scorecard.
Admittedly, error and accuracy are often thought of two sides of the same coin, but understanding error often offers much deeper inside.
1️⃣ The Role of Error in Model Assessment
Error tells us how far off our predictions are from reality. Think of it like a gap between what we thought would happenand what actually happened.
If we don’t measure and analyse this gap, we risk deploying models that are consistently off-target, sometimes in subtle but costly ways.
2️⃣ Regression: Measuring “How Wrong” We Are
In regression problems (predicting continuous values like daily production rates, pressure, or costs), error is often measured in numeric terms:
- Mean Absolute Error (MAE): The average “absolute distance” between prediction and reality. Great for interpretability, “on average, we’re off by 100 psi”
- Mean Squared Error (MSE) / Root Mean Squared Error (RMSE): Squares the error before averaging, which punishes large mistakes more heavily. Useful when big errors are costly.
- Mean Absolute Percentage Error (MAPE): Puts errors in percentage terms, helpful when your audience thinks in relative performance, but beware of divide-by-zero issues.
Your choice of metric depends on your application and baseline. If your engineers have been “guessing left, right, and centre,” beating their guesses might be good enough.
Sometimes, the median is used instead of the mean, especially when error distributions are skewed. In most real-world cases, absolute and squared error distributions are right-skewed, i.e. the mean will be larger than the median.
3️⃣ Classification: Understanding “What Kind of Wrong”
In classification problems (e.g. predicting pump failure categories or classifying well states), error isn’t just about “how far”, it’s about misclassifications.
Here, accuracy can be misleading. Imagine predicting pump failure in a dataset where 95% of pumps are fine. A lazy model that always says “fine” will score 95% accuracy… but will miss every real failure.
This is where the Confusion Matrix comes in
- False Positives (FP): The model cries “Anomaly!” when there’s none. Often easier to measure because users can reject the model’s alert. Every time an engineer dismisses a pump failure warning that turned out to be fine, that’s a false positive in your log.
- False Negatives (FN): The model stays silent when an anomaly is actually happening. Much harder to measure because the model never raises a flag, it relies on users spotting the anomaly themselves, often manually and after the fact. In industrial operations, this can mean the problem is only discovered when a well underperforms for days, or a pump has already failed.
4️⃣ Biases in Error
When examining model error, look for biases, systematic differences in performance between certain conditions, locations, or equipment types.
Sometimes, these biases aren’t caused by the model’s features, but by how data is collected. For example, if some wells are monitored more frequently than others, your anomaly detection model might simply be better at catching issues in those wells, not because they’re more problematic, but because they’re more visible.
5️⃣ Why This Matters in Operations
In some applications, false negatives are often more costly than false positives.
- A false positive might waste an hour of an engineer’s time
- A false negative could cause thousands of barrels in lost production.
That’s why many teams in oil & gas, manufacturing, and utilities aim for high recall, catch as many true anomalies as possible, even if it means tolerating more false positives. The extra alerts are a small price to pay for avoiding a costly miss.
6️⃣ Bottom Line
Whether you’re predicting production rates or detecting anomalies, accuracy is just the headline. Error is the full article.
The next time you train a model, don’t stop at “How good is it?” Ask “How exactly is it wrong, and why?” That’s where the real performance story lies.