Rubbish in, rubbish out: why data quality decides a migration
Key points
- Data quality problems aren’t always obvious errors. They can be caused by hidden relationships and conflicting migration criteria.
- Rules that appear sensible in isolation can exclude current, valuable records when applied across connected data.
- A Data Quality Review identifies these risks before the final migration.
- Leaving data quality until the end can result in missing records, broken processes and totals that don’t reconcile.
- Data quality needs to be treated as part of the migration plan from the beginning.
George Tye
CEO & Founder
Data: You get out what you put in
A new system can only work with the data it receives.
If important records are missing, duplicated, incorrectly linked or excluded by the migration rules, those problems don’t disappear when the new platform goes live.
They become harder to find, more expensive to correct and far more disruptive to the people trying to use the system.
Poor data doesn’t become good data when you move it
It’s tempting to think of data migration as a technical transfer: extract the information from the old system, map it to the new one and load it across.
The reality is more complicated. Most business data has accumulated over years, sometimes decades. Different teams have entered information in different ways. Systems have been customised, replaced or connected to other platforms. Rules that made sense when the data was first created may no longer reflect how the organisation works today.
There may be obvious quality issues such as duplicate contacts, incomplete addresses and inconsistent formatting. However, some of the most serious problems only appear when you look at how records relate to one another and how the migration criteria will treat them.
A record can be accurate and still be left out of the migration. A contact can meet the agreed criteria and still fail to load because the account it belongs to has been excluded. A historical record can look too old to retain while still supporting an active payment today.
This is why data quality can’t be reduced to tidying a few spreadsheets before go-live.
What is a Data Quality Review?
A Data Quality Review, usually shortened to DQR, is an assessment of the source data before it’s moved into a new system.
You can think of it as the inspection before the move. It identifies what should be taken, what needs attention and what could cause problems if it reaches the new platform unchanged.
In practice, the review usually covers three areas:
Audit
The source data is assessed for missing information, duplicates, inconsistencies, obsolete records and structural issues. The review also looks at the relationships between records and how the proposed migration criteria will affect them.
Clean and prepare
Problems are corrected where possible, unnecessary data is removed and important records are reformatted to meet the requirements of the destination system.
Verify
Test migrations and validation checks are used to confirm that the expected information has been transferred, linked correctly and made available in the new system.
A good DQR also gives the client an opportunity to challenge assumptions about the migration. Which records are genuinely no longer needed? What counts as “active”? How much history should be retained? Are there older records that still support current services, payments or customer relationships?
These decisions are much safer when they’re made deliberately rather than discovered accidentally at go-live.
At least 80% of data-related issues we see come down to missing or inadequate client data, not technical problems with the migration itself. This is almost always resolved through a proper Data Quality Review (DQR).
What bad data quality looks like in a real migration
Some data problems are easy to spot. The more dangerous ones can sit behind perfectly reasonable migration rules.
Here are three examples our team has encountered during real projects. The organisations have been anonymised, but the underlying problems are common.
Important files attached to an excluded record
During one project, the agreed migration criteria excluded a particular record. The same criteria had previously been used for a migration carried out by another supplier, so there was no immediate reason to believe it would cause a problem.
However, a large number of important files had been allocated to that record during the earlier migration.
Because the parent record was excluded, the associated files were excluded with it. The client only realised what had happened during go-live, when the documents they expected to find in the new system weren’t there.
The individual files weren’t damaged or incorrectly formatted. The problem was the relationship between those files and a record the migration rules had been told to leave behind.
Recent contacts linked to older accounts
In another migration, the client asked for accounts and contacts from the previous three years to be moved.
On the surface, that’s a clear and sensible rule. However, an account created more than three years ago could have a contact added to it within the last year.
The contact met the migration criteria. The account didn’t.
Because the contact needed to sit beneath an account that hadn’t been migrated, it couldn’t be created correctly in the new system. From the client’s perspective, it looked as though a valid recent contact had simply been missed.
The cause was a hidden conflict between two sets of migration rules. Each rule made sense on its own, but they didn’t work when applied to related records.
Important files attached to an excluded record
During one project, the agreed migration criteria excluded a particular record. The same criteria had previously been used for a migration carried out by another supplier, so there was no immediate reason to believe it would cause a problem.
However, a large number of important files had been allocated to that record during the earlier migration.
Because the parent record was excluded, the associated files were excluded with it. The client only realised what had happened during go-live, when the documents they expected to find in the new system weren’t there.
The individual files weren’t damaged or incorrectly formatted. The problem was the relationship between those files and a record the migration rules had been told to leave behind.
Historical contacts with active recurring payments
A third client asked for accounts, contacts and transactions from the previous 15 years to be migrated.
The organisation then discovered that it still had active recurring donations dating back to 1998. Some of the contacts connected to those donations had been created well outside the 15-year window.
Excluding those older records would have removed people who were still actively giving. It also meant that the total donation value in the new system didn’t match the total held in the old one.
Again, the migration rule itself wasn’t unreasonable. The problem was using the age of the contact as a proxy for whether the relationship was still active or valuable.
Why migration criteria need to be tested against the data
Every migration needs a defined scope. Moving everything without question can carry years of clutter into a new platform, increase the cost of the project and make the new system harder to use.
The difficulty is that apparently simple criteria can produce unintended consequences.
| Migration criterion | The question that also needs to be asked |
|---|---|
| Only migrate records created within a set period | Are older records connected to current contacts, payments or services? |
| Exclude inactive accounts | Are files, activities or active contacts still attached to them? |
| Migrate recent contacts | Can those contacts be created without their older parent records? |
| Leave historical transactions behind | Are they needed for reconciliation, reporting or audit purposes? |
| Remove duplicate records | Are they true duplicates, or do they contain different history and relationships? |
| Migrate complete records only | Would excluding incomplete records remove information the organisation still relies on? |
A Data Quality Review tests the proposed scope against the data that actually exists. It gives the project team a chance to find exceptions, conflicting rules and dependencies before they affect the live migration.
What happens when data quality issues reach go-live?
When data quality hasn’t been addressed, the consequences usually appear as soon as people begin working in the new system.
Important information is missing
Users may be unable to find the customer records, documents, transactions or history they need. In some cases, the information is present but attached to the wrong place or no longer connected to the related records.
Business processes stop working
Incomplete or incorrectly structured data can prevent workflows from running as intended. Billing, payroll, reporting and other automated processes may fail because the system hasn’t received the information or relationships it expects.
Figures don’t reconcile
Financial totals, transaction histories and record counts may differ between the old and new systems. The team then has to determine whether information has been deliberately excluded, transformed incorrectly or missed altogether.
Users lose confidence in the new system
Employees don’t see the migration logic operating behind the screen. They see missing contacts, duplicated accounts and figures that don’t look right.
Once users begin to assume that the new platform can’t be trusted, adoption becomes much harder. Even correctly migrated information may be questioned.
The project moves into firefighting
Instead of supporting users through a planned launch, the internal team and migration specialists have to investigate live problems under pressure.
Cleaning and correcting data in a production environment is slower and riskier than resolving it during preparation and testing. It can also extend the project and create additional costs at the point when everyone expected the implementation to be complete.
Why is data quality so often addressed too late?
Data migration is still regularly treated as a secondary part of a system implementation.
The new platform naturally receives most of the attention. Teams focus on functionality, configuration, integrations and how the system should work. The migration is sometimes reduced to a task that can be completed once the rest of the implementation is nearly ready.
The problem is that functionality depends on the information sitting behind it.
A well-configured CRM can’t provide a reliable view of a customer if their records are incomplete or divided across several duplicate accounts. A finance platform can’t produce trustworthy reports if historical balances and active transactions haven’t been migrated correctly.
When data quality is left until testing or go-live, there may no longer be enough time to understand the problem properly. The project is forced to choose between delaying the launch, accepting known issues or applying hurried fixes.
When should a Data Quality Review happen?
The DQR should begin early enough to influence the scope and design of the migration.
It shouldn’t be a final check performed once the mapping has been completed. Findings from the review may change which records need to move, how different sources should be consolidated and what the destination system needs to accommodate.
At Migrate Data, we involve the client throughout this process. Technical specialists can identify patterns, exceptions and dependencies, but the organisation still needs to explain what the data means and how it’s used.
Someone needs to recognise that an old contact is still an active donor. Someone needs to know why files were attached to an otherwise obsolete record. Someone needs to decide whether historical transactions are operationally necessary or can be archived.
The client knows the context. The migration team knows where that context can be lost. A successful DQR brings those two sides together before the final move.
Give your new system data it can actually use
Data quality isn’t separate from a successful migration. It determines whether the new system contains the right information, whether that information works as intended and whether users trust what they see when they log in.
Finding these issues early gives you time to investigate them properly. It also means that exclusions, corrections and compromises are documented decisions rather than unpleasant surprises.
Learn more about Migrate Data’s data quality services and how a Data Quality Review can help you prepare for your next migration.
Frequently Asked Questions
What is data quality in a migration?
Data quality describes whether the information being migrated is accurate, complete, consistent and suitable for use in the destination system.
It also includes the relationships between records and whether the migration criteria will preserve the information the organisation needs.
What does DQR stand for?
DQR stands for Data Quality Review. It’s the process of assessing source data, identifying potential issues and agreeing how those issues should be handled before the final migration.
Does a Data Quality Review only look for incorrect data?
No. A DQR may identify incorrect or incomplete information, but it also examines duplicates, formatting, obsolete records, record relationships and conflicts within the proposed migration criteria.
Some records may be accurate in isolation but cause problems because they depend on other data that hasn’t been included.
Can poor data quality cause a migration to fail?
Yes. Poor data quality can result in records being rejected, information being missed, relationships breaking and totals failing to reconcile.
Even when the technical transfer completes successfully, the migration can still fail in practical terms if users can’t trust or work with the data in the new system.
Should all historical data be migrated?
Not necessarily. Some historical data may no longer provide enough value to justify moving it into the new platform.
However, older records shouldn’t be excluded based on age alone. They may still be connected to active payments, current contacts, important documents or reporting requirements.
Who should be involved in a Data Quality Review?
The review should involve migration specialists and people within the client organisation who understand how the data is used.
The migration team can identify technical issues and unexpected patterns, while the client provides the operational context needed to make informed decisions about what should be migrated.
Can data quality issues be corrected after go-live?
They can, but correcting data in a live system is usually slower, riskier and more disruptive. Users may already be working with the affected records, while automated processes may depend on them.
It’s safer to identify and resolve as many issues as possible during preparation and test migrations.
Don't Fall in to the Migration Trap
Speak with a Migration Expert
Weather you’re migrating from Salesforce, Spotify or Microsoft – getting help from a migration expert today can save you a lot of time and headaches tomorrow.
Get Unity through Data with Migrate Data.