Data migration
How to Clean Master Data Before You Migrate Anything
Every ERP migration has a moment where the interface lights up, the finance team runs the first trial balance, and someone finds that customer X exists three times in three different ledgers. The system works. The data behind it does not. That is the story of most delayed go-lives, and it is almost always a master data problem rather than a software problem.
Master data is the slow, unglamorous, expensive part of the work, and it is the part most project plans under-resource. Software vendors sell you the new system. Nobody sells you clean data, because cleaning data is yours to do. Get it right and the migration is a heavy lift done well. Get it wrong and no amount of clever integration saves you. If you are still weighing the whole decision, start with whether you are ready to migrate at all.
Why master data sinks migrations
Legacy systems accumulate data like sediment. Over a decade, the same customer appears under different names, suppliers split and merge, and part codes proliferate because no one dared delete anything. A new ERP does not clean that for you. It imports it faithfully, defects included, and then it holds you to a standard of consistency the old system never did.
New ERP platforms enforce uniqueness and validation rules that old ones did not. A customer record that was tolerated for years becomes an error the day the finance month-end reconciles against a single master. The consequences surface as open purchase orders in the wrong legal entity, duplicate invoices, or a stock ledger that does not tie out. Once dirty master data enters cutover, it is enormously costly to unpick.
On one public sector programme we found that almost a third of the vendor master records had not been touched in over a decade, including several marked active that had supplied nothing for years. Cleaning them before go-live saved the implementation from a procurement freeze in the first month.
What counts as master data
Master data is the reference data that everything else hangs off: customers, suppliers, products and materials, locations, chart of accounts, cost centres, and the pricing and tax structures tied to them. It is distinct from transactional data, the orders and invoices and movements that attach to these masters, which you migrate separately and usually in less depth. The DAMA data management body of knowledge is the standard reference for defining and governing exactly this reference data.
The trap is assuming master data is just "the lists". In reality much of it lives inside transaction history, in the form of a customer ID that no longer matches any parent record. Your clean-up has to decide what is truly still reference data, what needs a legacy archive, and what is safe to leave behind.
Audit before you touch anything
Do not open the cleaning tools on day one. Start with an audit that answers four questions. What data exists and where does it live? How much of it is still in active use? What is duplicated or incomplete? And what is standard across systems versus siloed? The audit gives you a baseline you can measure clean-up against, which matters because nobody believes a data programme without numbers.
Sample heavily rather than sampling lightly. The second most common mistake after skipping the audit is basing decisions on a five hundred row extract. Pull the full record counts, profile the fields with real duplication and completeness analysis, and let the numbers set the scope. For a clear step-by-step view of where this work sits in the wider effort, see how to plan an ERP migration step by step.
Standardise, deduplicate, enrich
Cleaning is a three-move sequence. First standardise, so that addresses, phone formats, units and naming conventions follow one defined rule. Second deduplicate, matching records that are the same thing under different spellings and merging them with a clear survivor. Third enrich, filling the gaps that matter, such as VAT numbers, payment terms or price lists, so the new system starts with decision-ready data rather than blanks to fix later.
Rule of thumb: only migrate what the new business process genuinely needs for the first few years. Decades of dormant records add risk without value and slow every subsequent data load. Archive what you legally must keep, and let the rest go. This is the same judgement call you will make about the wider system when you weigh the arguments in legacy versus cloud ERP.
Ownership that outlasts go-live
Cleaning master data once is meaningless if nobody owns it afterwards. Even the best migration builds a small master data management function, one person, sometimes two, responsible for the definitions, the standards and the exceptions. Put that ownership in the business, not in IT, because it is a business discipline. Finance owns the chart of accounts. Procurement owns the supplier base. Operations owns the material masters.
Without this, the pristine data you shipped at go-live decays within a year and you are back where you started, only on a more expensive platform. The cost of that decay is a theme that applies across your whole estate, which is why it is worth reading the hidden ERP costs that blow up budgets before you commit. Clean data is not a project closure item. It is an operating discipline.
Frequently asked questions
How long does master data cleaning take?
For a mid-size enterprise with a decade of history, count on weeks of profiling and validation, not days, running alongside the build. It is rarely the longest workstream, but it is the one that is hardest to compress, because it depends on business experts making judgement calls.
Should we migrate all historical master data or only active records?
Migrate what the new processes need for the first few years. Archive the rest for legal and reporting reasons. Migrating a full decade of mostly dormant masters adds risk, slows loading and pollutes reporting for no operational benefit.
Can software clean master data automatically?
Tools can profile, suggest matches and handle standardisation, but a human owner must confirm each merge and each survivor. Leaving duplicates or matches to software alone is how you inherit a new set of wrong records on day one.
Who should own master data cleaning in a migration?
The business, not IT. Assign named owners by domain, finance for the chart of accounts, procurement for suppliers, operations for materials, and give them the authority to make exceptions. This resets the discipline that must survive go-live.
Sources and further reading
The argument in this article draws on public research. Where you want to go deeper, these are the sources cited in the text and further reading.
