There is not much to write on the data monitoring and validation in terms of concepts. All the subjects covered in the 'Data Quality Assurance' chapter that is 'Interface Control', 'Input Controls', 'Entity-Relationship Controls', 'Domain and mapping assurance controls', 'Data Standard Assurance' controls, ' Value Property Assurance controls', 'Business Rules Assurance Controls' are the reference points for Data quality monitoring. While Quality Assurance is preventive, data quality monitoring is validating and leading to prevention and remedial methods.
Data quality assurance is more extensive and can obviate most of the data quality issues. Data monitoring is less extensive as one cannot monitor each and every data quality rules on ongoing basis. This will place a high overhead on the system. Some organization do deploy 100% online data monitoring for mission-critical processes.
The extent of data monitoring can be pro-active given the importance of quality requirements, and for the rest, it is driven by experience and issues faced. Wrong customer statements of accounts OR adverse audit findings lead to monitoring for the purpose of avoidance and to validate the remedial fixes.
Database MonitoringOnce you have a well-documented metadata and data quality rules, a wide range of data quality health check routines can be created. These routines can be run on the database in intensity and frequency depending upon the level of confidence and criticality of data. All the items mentioned in the data quality assurance can be the candidates for such kind of routines. You can run routines, which can:
Transaction Monitoring
Monitoring of transactions is a new concept, which ensures that any transaction is not committed to database unless its integrity is established. There are transaction monitoring tools available in the market.
TIP- For the online businesses, one should weigh between the criticality of transaction and the system load for monitoring at a transaction level.
Physical Monitoring
The data may fulfill all the data quality rules , but still be wrong due to wrong data entry (willful OR unintentional). The last mile of ensuring this is to do a physical verification. Some of this is manifested through undelivered mails, untraced customers, unfulfilled orders etc. Given the business case, a set of physical verification procedures should be put into place. This is an expensive method, but apart from resolving discrepancies, it throws up process improvements for the future. For example placing a maker – checker for stock transfer.
Batch monitoring
Whenever a batch is run, which includes interfacing of files across the systems OR end of day processing, the data accuracy, consistency and completeness health check can be run (pre and post a job) to ensure defined quality benchmark before progressing. This monitoring of batch is online and real time. |