Last Friday, I had the opportunity to present our updated MLOps strategy to an expert team in the energy sector, which was one of the first few times I presented to a team who is familiar with data science but does not work with a modern software development stack. 👀

The software engineering perspective on deployments

Why is that relevant? Well, if you tell a software engineer that you’re pushing changes straight to production without a test environment, chances are you’ll get some raised eyebrows (and rightly so 😒). In software development, skipping CI/CD pipelines and proper testing is considered unprofessional - a risk you simply don’t take.

But in the world of machine learning and data pipelines, the lines are blurrier. We do test, of course (🧐). But we don’t always deploy frequently or at all. Our test coverage isn’t always complete. CI/CD? Still a work in progress in many teams. But we understand the principles, especially since most of us come from a computer science background. We know how it should work and the MLOps movement is a sign that we are implementing this more and more.

The production-industry perspective on deployments

Some teams in the industry don’t write their own code. They rely on complex, production-level vendor software that was purely developed for their field of engineering - and those systems don’t always offer separate testing or staging environments. No Git. No branching. No “let’s test this before we go live.” You change a setting, and it’s live. Immediately. Every day. No rollback. No diff-check.

And that’s a huge communication challenge for us as data scientists and ML engineers.

When we explain our setup — “We test before pushing to production”, it can unintentionally sound condescending. Like we’re assuming the audience doesn’t test, when in fact, they simply work within a different paradigm. Their tools, constraints, and standards are not less professional - just different.

​

What I learned from this exchange

We’re in a unique spot. We talk to domain experts and IT teams. We live in data and code. And we borrow best practices from both sides.

So what can we do?

🧠 Tip: Before your next presentation or collaboration, ask how the team currently handles software or configuration changes. What tooling do they use? How do they manage versions or rollouts? You’ll learn a lot - and it’ll help you frame your approach with empathy and clarity.

💡 On our side, we’re currently working on a testing framework that supports different data versions within the same Databricks workspace. That means separate schemas for testing and production, and proper controls to ensure changes in the pipeline don’t accidentally affect live data used by ML models. Managing this interplay between data, code, and infrastructure is one of the biggest challenges companies face today - and no one has it completely figured out, though there are several promising approaches.

But we’re getting there. One schema, one CI job, one conversation at a time 😅

​

Until the next time I have some harrowing tale about ML in real life...

Sarah 👋🏻