Growing up in the 90s, Excel was the gold standard for scientific and academic works. At the turn of the 21st century, petroleum engineers used Excel for everything: pressure transient analysis, production decline curves, and countless other applications. That green grid became as familiar as a wellsite.

I still remember the satisfaction of building my first NPV macro. It felt like magic, watching the spreadsheet automate tasks that once took hours. For a generation of engineers, Excel wasn’t just a tool. It was the tool.

But something has shifted. The questions we’re asking today have outgrown what spreadsheets can answer.

When the Familiar Became the Bottleneck

Consider a scenario most petroleum engineers would recognise. You need to perform linear regression over pressure and rate across all wells’ production history, perhaps running a seven day moving window on daily data from thousands of wells over several decades. In Excel, you would build a VBA macro, hit run, and wait. And wait. Thirty minutes or an hour later, you’d have results for a single scenario.

The architecture of VBA created unavoidable bottlenecks. When the macro looped through Excel cells, each operation involved expensive overhead as Excel located each cell, performed type conversions, updated the worksheet, and potentially triggered recalculation events. For our moving window regression example, a typical VBA macro that looped over all cells would take more than an hour and remain frustratingly unstable. Some engineers with exceptional VBA skills might have optimised this significantly, but even then, the fundamental constraints remained.

Now what if you wanted to run the algorithms on hourly data or fine tune the window size? Too scary to try. And in that hesitation lies the problem. When your tools constrain your curiosity, you stop asking the questions that create value.

Python: Vectorisation Changes Everything

When engineers wrote operations in NumPy, Python handed off the heavy lifting to highly optimised code written in faster languages like C and Fortran. Rather than processing data one cell at a time, these libraries operated on entire columns simultaneously. The same analysis that took an hour in VBA completed in seconds. The code was also clearer and less prone to errors than equivalent VBA macros spanning hundreds of lines.

Pandas added another layer perfectly suited to petroleum engineering workflows: the DataFrame. This tabular structure mirrored how engineers naturally thought about well data, reservoir properties, and production histories. Operations like grouping by well, filtering by formation, and joining pressure data with production became single line expressions rather than complex nested loops.

For most engineering teams, Python with Pandas solved the immediate pain points. Analyses that were previously impractical became routine. But as sensor networks expanded and data volumes grew, new challenges emerged.

Spark: Analytics at Scale

Given the plethora of sensors deployed across modern oilfields and how fast data transmission developed, what if you wanted to analyse intraday data, hourly or by the minute? Data volumes quickly exceeded what any single machine could store and process. The issue couldn’t be solved by optimisation alone, and hardware upgrades weren’t scalable or future proof.

Apache Spark, built upon the Hadoop framework and emerging around 2009, addressed this by distributing data across multiple machines and processing partitions in parallel. For truly massive datasets like intraday readings from thousands of wells over decades, Spark provided horizontal scalability. Terabytes of data that would have been impossible for Python on a single machine instead took minutes across a cluster.

For enterprise scale operations running exception based surveillance across entire portfolios, Spark became the tool of choice.

Polars: Bridging the Gap

The jump from Pandas on a single machine to Spark represented a significant leap in both data scale and operational complexity. A grey area existed between them: datasets too large for Pandas to handle efficiently, but not massive enough to justify the overhead of distributed computing.

Pandas, for all its strengths, carried design decisions from 2008. It was memory hungry and single threaded by default, which left performance on the table with modern multi core processors.

Polars emerged more recently to fill precisely this gap. It achieved 5 to 10 times better performance than Pandas through several innovations. It was fully parallelised by default, automatically using all available CPU cores. Its query optimiser restructured operations to eliminate unnecessary work and minimise memory usage. The result was that analyses which pushed Pandas to its limits ran comfortably in Polars, often without requiring the leap to distributed infrastructure.

The practical impact extended beyond raw speed. Polars made a new category of analysis feasible on standard hardware, filling the space between desktop computing and enterprise clusters.

Choosing the Right Tool

The progression from Excel VBA through Python to Spark isn’t about one tool being universally superior. It’s about matching computational architecture to problem scale.

Excel remains adequate for small, ad-hoc analyses and one-off calculations. The familiarity and ubiquity have value. But the moment you’re writing VBA loops over thousands of rows, you’re using the wrong tool.

Python with Pandas or Polars should be the default for serious data analysis in petroleum engineering. The ecosystem for scientific computing, machine learning, and visualisation is unmatched.

Spark becomes essential when you’re dealing with streaming data from hundreds of thousands of sensors, enterprise level surveillance systems, or automated machine learning workflows at portfolio scale.

More importantly, faster tools enable different thinking. When analysis took minutes, you ran it once and moved on. When analysis takes seconds, you iterate, explore alternatives, and develop intuition through rapid experimentation. The tool shapes the thought process.

Looking Forward

Five years ago, Python fluency was a differentiator. Today, it’s rapidly becoming a baseline expectation. The engineers who delay this transition aren’t standing still. They’re falling behind as the industry’s computational centre of gravity shifts beneath them.

This isn’t about abandoning hard won expertise. Domain knowledge remains irreplaceable. Understanding reservoir behaviour, recognising operational constraints, applying engineering judgment: these capabilities matter more than ever. But they’re no longer sufficient on their own. The future belongs to engineers who can pair deep domain expertise with computational tools that amplify their insight.

Excel isn’t disappearing tomorrow, but its role has fundamentally changed. The serious computational work has moved to frameworks designed for modern data scales and hardware. The analytical techniques we want to apply, from machine learning to real time optimisation, require computational capabilities Excel never had.

The engineers who master these tools won’t just work faster. They’ll ask questions the rest of us haven’t thought to formulate. They’ll find value hidden in data we’ve been collecting for years but never truly examined. They’ll build the surveillance systems and optimisation engines that define how this industry operates in the decades ahead.

The choice isn’t whether to make this transition. It’s whether to lead it or be left explaining why you’re still waiting for that macro to finish running.