Are You Collecting the Right Data: Why Data Captured by Accident Can't Run a Business on Purpose

Every post in this series so far has taken the data organisations already hold as its starting point. The mess exists; the question has been what to do about it. This final post asks the question that sits upstream of all of it: why does the mess keep forming in the first place?

The answer, more often than not, is that the data was never really collected at all. It just happened. It's a byproduct: the exhaust of systems and processes that were designed to do something else. The CRM was bought to manage a sales pipeline, and the customer data is whatever fell out of that. The timesheet system was set up to run payroll, and the effort data is whatever people remembered to type in on a Friday afternoon. Nobody sat down and asked what data the business actually needs to run well, and then worked backwards to how it should be captured. So what accumulates is whatever the tools happened to produce, in whatever shape they happened to produce it.

You can clean byproduct data, define it, curate it, and govern it, and the earlier posts in this series argue you should. But there's a limit to how good it can ever get, because the flaw is baked in at the point of creation.

Manual entry is where good data goes to die

Take one small, concrete example: knowing when something actually happened. There are two broad ways to get that. A person types it in, or the system stamps it automatically the moment the event occurs.

Manual entry feels flexible and cheap. It's also where accuracy quietly dies. People enter time in batches, days later, rounded to the nearest half hour, reconstructed from memory and calendar archaeology. They fill in the fields that are mandatory and skip the ones that aren't. Two people describe the same activity in different words, in different levels of detail, in different fields. None of this is a failing of the people involved. It's the predictable output of asking humans to do, repetitively and after the fact, what a system could have recorded automatically and precisely at the moment it happened.

A datestamp captured by the system isn't just more convenient. It's a different class of data: complete, consistent, and trustworthy without anyone having to remember anything. Multiply that one design decision across every field, every form, and every process in a business, and you get the real answer to why some organisations' data can be trusted and most can't. It was decided, mostly invisibly, at the point each system was chosen or built.

The build-versus-buy question is really a data question

Which brings us to a conversation that's usually had entirely in terms of cost and features, and almost never in terms of data: how you source your software.

Licensed off-the-shelf software and SaaS sit at one end. They're fast to adopt, cheap to start with, and someone else keeps them running. The trade-off is that you inherit the vendor's model of the world. Their fields, their definitions, their workflow. If their process matches yours, that discipline is a gift. If it doesn't, people bend the tool: they repurpose fields, keep the real record in spreadsheets on the side, and the system of record slowly stops recording reality. The data you accumulate is shaped by someone else's assumptions about how your business works.

Platform-built applications, the ServiceNow and Power Platform end of the spectrum among others, sit in the middle. You get the plumbing for free: authentication, hosting, audit trails, integration, and increasingly the data capture discipline of a mature platform. And you get to shape the application around your actual process, your definitions, and the events you actually need recorded. For a lot of internal business processes, this is the sweet spot: enough customisation that the data reflects your business, enough standardisation that it doesn't become an unmaintainable one-off.

Fully custom builds sit at the far end. Total control over what gets captured, how, and when: every datestamp, every field, every definition yours by design. Historically that came at a price that reserved it for the processes that genuinely differentiate a business, and it carried the long tail of maintenance that off-the-shelf buyers never think about until they don't have to pay it.

There's no universally right answer. But notice what the spectrum actually trades: at one end you buy speed and accept someone else's data model; at the other you pay for the data model your business actually needs. Framed that way, the sourcing decision is one of the most consequential data decisions an organisation makes, and it's routinely made without a single data person in the room.

What AI acceleration changes

Here's what's genuinely different now. The cost curve on that spectrum has bent.

AI-assisted development has dramatically compressed the effort of building software: scaffolding an application, wiring up integrations, generating the unglamorous code that used to consume most of a build's budget. Work that took a team months increasingly takes a small team weeks. That doesn't just make custom builds cheaper. It moves the break-even point along the whole spectrum. Processes that could never have justified purpose-built capture, the ones that have limped along for years on spreadsheets and repurposed SaaS fields precisely because they were too small to justify anything better, can now justify it.

The same acceleration applies within platforms. Building a well-designed application on ServiceNow or Power Platform with AI-assisted development is faster still, which makes "capture this process properly, with the right fields and automatic timestamps, designed around how the work actually flows" an affordable ambition for ordinary internal processes, not just flagship ones.

The organisations that recognise this get to attack the data problem at its source. Instead of endlessly cleaning byproduct data downstream, they can afford, for the first time, to fix the point of creation: to have systems that capture the right data, the right way, as a designed outcome rather than a lucky accident.

Start with the process, not the software

Which points at the real starting line, and it isn't a tooling decision at all.

Before any of the sourcing options can be evaluated, you have to know what the business actually needs to see about itself: which processes matter, which events within them need recording, which definitions have to hold across departments, and where the current data is a byproduct standing in for the real thing. That's business process and organisational analysis, and it's the least glamorous, most skippable-looking step in any of this. Skipping it is how organisations end up automating a broken process faster, or buying a beautiful system that captures precisely the wrong things.

So if this series ends with one recommendation, it's this. Don't start by asking which software to buy or build. Commission a proper analysis of your processes and your organisation first: map how the work actually flows, decide what you genuinely need to know about it, and only then choose the tools, on a spectrum that AI has just made considerably wider and cheaper than it was two years ago. The businesses that get this right won't just have cleaner data. They'll be the ones whose data was never rubbish to begin with.

This is the final post in a series on why more data isn't always better data. The earlier posts covered why deletion is a strategy, what AI does to untrusted data, the definitions problem, analyst overwhelm, hoarding as a risk management failure, and a keep-or-kill framework. This one closes the loop: the best data problem is the one you design out at the source.