|
TIP: We would always advise the Data Mapping and Assessment to be done on offline database, as it gives a lot of flexibility to the teams to work at their own pace. Extensive DMA activities can place a significant system load on the production systems, especially in critical period-end processing. This however may not be a simple, as a medium scope DMA also will be involving multiple systems, and all the systems test environments may not be available. Therefore, there can be millions of different ways like all offline DMA, offline+online DMA or totally online DMA approach.
TIP- Don't use DMA on a staging platform of a Data Warehouse. Sometimes, it is suggested that if the staging layer of data warehouse has extracted data from the source systems, and if that staging Data is in DMA scope, why not to use the staging layer for DMA (or a part of it)? Our reply would be not to muddy the waters. DMA as an exercise can have many objectives, and by mixing DW and DMA, one may create confusion in the whole work management. Secondly, staging layer in DW is a complex environment having different layers of data:
- extracted data
- Partially transformed data,
- Fully transformed data
- Data Sets ready for loading
Given this complexity, one has a risk of wrong data set to be selected for DMA.
TIP: As you do DMA, one may not like to mix the single column and multi-column analysis. We have seen that sometimes, we get into the trap of jumping the steps and then interpolating the results. For example, if there is a mis-match between the master tables (multi-column analysis), one may conclude that the customer code field (single column) in one of the systems has faulty data. This is a risky approach. We recommend going in the sequential path of single column analysis followed by Data Model analysis followed by multi-column analysis.
TIP- When you are testing DMA scripts, try to do it in individual pieces, and start with single column, followed by Data-Model, followed by Multiple Columns. The reason is-
- Single Column testing sets up the context of the basic accuracy of data. It helps you to fine-tune the multi-column and data-model DMA. For example- If your location_ID in customer table are not following a standard format, it might mean that there is a lack of referential integrity between the customer_master and Location_master.
- Data-Model Testing contributes as much to finding the root-cause of data-issues, as to find the data issues itself. For example, if you find that referential integrity between the state_master and City_master is not being followed, it can be the root cause of many inconsistencies across customer, agent, vendor related data as all of them will be using city and country information.
- Multi-column analysis: this should be ideally the last point of your DMA, as all what is wrong with single column and data-model contributes to multi-column issues.
TIP- For Test-planning of DMA, avoid, running a one single script to churn out all the results in one go. Do it sequentially and in small bites. The reason is simple; the initial results could make you go back to your design phase to fine-tune your scripts.
TIP- DMA actuals have to be compared with the standards, to define the DQ gaps. Sometimes, you are not able to get the standards for all instances of your DMA scope. Given that there are hundreds of possible DMA data-checks you are going to do, it will be difficult for business to assign varying level of standards. The best way out for this, is to work in the back-ground with a business analysts, and come out with the proposed standards. Build your reasons on 'what? And why?’ Share it with business, and other stakeholders (like internal control, CFO...). As you do it, be on more aggressive in terms of being ideal. One also has to ensure that while you are presenting the draft to the stakeholders, they take the accountability for standards.
TIP- Taking it further- Even if you are not able to get the standards before you start with DMA, once you have done your DMA execution and you have the actual results on the ground, business (business owners, Data Custodians, Data Steward..), will be in a better position to provide their business need.
TIP- You will find this TIP at various points in our knowledgebase. Chances are that most of the time, Business will be looking for 100% data quality. One need not challenge this, as it may be true in most of the cases. As you get into the Gap analysis, solution and planning related to the Gaps, business will be a in a better position to understand the cost-benefit attached to resolving a Gap. That will be another check-point to fine-tune the business expectations.
|
