Usually, companies store data in multiple database engines. In situations where data needs to be migrated from legacy systems to a new database engine or system, the challenge is to ensure the data is correct in the new system.
The comparison of data in both systems is an essential action to ensure reliable data. Row counts and column sums are not enough. Furthermore, the comparison performed on a sampling basis does not guarantee 100% accurate data migration but could be one strategy used when moving Big data.
In-house Test Platform
We at Onepoint have developed a Testing Platform that compares heterogeneous data sources. This platform can compare entire data from different sources thus providing 100% data coverage.
The comparison may have criteria like handling null values, applying different field operations, checking for values being greater, less or equal as an example. For tech-savvy people, they may even use java expressions in the comparisons.
Besides database data sources, the platform also supports other data source types, such as excel spreadsheets, CSV files and JSON files.
The platform generates excel reports that specify the results of the test Passed or Failed. In case of failures, the mismatches or extra records of source/target are also listed in separate excel files.