Operational data is the lifeblood of modern digital systems. It powers real-time decisions, user personalization, fraud detection, predictive maintenance, and more. But while this data drives AI and analytics, and provides the necessary context, using it effectively isn’t as straightforward as it seems.

I saw this firsthand while working at a UK-based digital lender focused on small business loans. Unlike large banks, speed and agility were our edge. We automated most of the loan underwriting process using machine learning models powered by real-time Kafka data streams. Borrowers could be approved, signed, and funded very rapidly. The success of this flow depended on the quality and performance of our data pipelines and models.

Where it Broke Down

Training the models followed standard ML workflows: engineers and data scientists collect large historical datasets, clean and enrich them, run simulations, flag anomalies, tune parameters. All typical. The problem came later, when those models were deployed into production.

In production, the models ran on real-time operational data: raw, inconsistent, and fast-moving. But the training datasets were drawn from our analytical environment, where the data had been ETL’d: transformed, normalized, and cleaned. In short, the models were trained on a polished version of reality that didn’t exist in the wild.

That disconnect had a cost. To make the models work in production, engineers had to study the ETL pipelines that produced the training data and build an “adapting” wrapper for the model in the real-time data world to effectively repeat these transformations. It was hard to maintain, and constantly required careful testing and monitoring for correctness.

As a result, this system designed for speed became slowed by development complexity. Deploying new models took longer. Debugging got harder. Data mismatches could potentially be costly so conservative testing and change control set in. And the keeping to the core principle that models should learn from the same data they’d eventually operate on became ever more expensive.

Two Data Worlds, One Problem

The root of the issue was the split between operational and analytical data. There were effectively two different data worlds:

• One used for model training (transformed, structured, and enriched).

• One used in production (raw, unfiltered, and constantly changing).

Unifying them would have solved many problems. But building a single operational-analytical model that spanned streaming pipelines and data lakes wasn’t trivial. Data quality issues, fragmented ownership, and misaligned priorities across teams made it hard to execute. We managed it, but could there have been an easier way?

Shifting Left, but With Guardrails

What we really needed was to "shift left": to define and implement data standards, rules, and validations earlier in the data lifecycle, ideally within the applications themselves. But in a distributed microservice environment, that’s expensive to implement, tough to govern, and nearly impossible to scale without resistance. Developers resist added complexity, and centralized governance often clashes with team autonomy.

This is exactly the kind of problem that Conduktor Trust is designed to solve.

Trust allows teams to define and enforce data rules at the source. It tracks anomalies, enforces structure, and guarantees field-level data quality for streaming pipelines, preventing them from entering data streaming pipelines and data lakes in the first place. This makes it possible to maintain a consistent data model that works across both analytical and operational environments.

If we had used Trust, we could have caught inconsistencies before they reached production. The models would have been trained and deployed on the same data structure. There would have been no need for wrapper logic or patchwork fixes. We could have moved faster and still delivered the reliable results we took pride in.

Why I Joined Conduktor

That experience taught me a painful lesson about how fragile data systems can be when teams operate in silos. It also made me realize how valuable a solution like Trust could be.

That’s why I joined Conduktor. I wanted to help build the tools that I wish we had used. Trust gives data engineers, developers, and analysts a common foundation for clean, reliable data.

If you're facing the same disconnect between training and production, take a look at Conduktor Trust or sign up for a free demo. It can prevent the exact kind of failure we experienced.