Data Quality Monitoring

There is not much to write on the data monitoring and validation in terms of concepts. All the subjects covered in the 'Data Quality Assurance' chapter that is 'Interface Control', 'Input Controls', 'Entity-Relationship Controls', 'Domain and mapping assurance controls', 'Data Standard Assurance' controls, ' Value Property Assurance controls', 'Business Rules Assurance Controls' are the reference points for Data quality monitoring. While Quality Assurance is preventive, data quality monitoring is validating and leading to prevention and remedial methods.

Data quality assurance is more extensive and can obviate most of the data quality issues. Data monitoring is less extensive as one cannot monitor each and every data quality rules on ongoing basis. This will place a high overhead on the system. Some organization do deploy 100% online data monitoring for mission-critical processes.

The extent of data monitoring can be pro-active given the importance of quality requirements, and for the rest, it is driven by experience and issues faced. Wrong customer statements of accounts OR adverse audit findings lead to monitoring for the purpose of avoidance and to validate the remedial fixes.

Database Monitoring

Once you have a well-documented metadata and data quality rules, a wide range of data quality health check routines can be created. These routines can be run on the database in intensity and frequency depending upon the level of confidence and criticality of data. All the items mentioned in the data quality assurance can be the candidates for such kind of routines. You can run routines, which can:

• Check the business flow (Input + WIP opening balance = Production + closing WIP balance)
• Business to accounting (The premium received and applied to the premium income recognition accounting)
• The commissions and disbursements as per the business done and commission slabs.

  There is no limit to all the health-checks you can run on the system. Here is the typical list:

• Production assurance: These health checks typically ensure that there is not pilferage in the cycle of business coming in, getting processed and produced to the end customer.
• The financial integrity – Business done to accounting generated to financial information in the database, disbursement done and bank balances.
  
• The database reconciliation across the systems: This includes-->
  - The comparison on transactions, statuses and financials.
  - The master tables: The customer, supplier, vendors and other masters.
  - The parameters: The product set-ups etc.

Transaction Monitoring

Monitoring of transactions is a new concept, which ensures that any transaction is not committed to database unless its integrity is established. There are transaction monitoring tools available in the market.

TIP- For the online businesses, one should weigh between the criticality of transaction and the system load for monitoring at a transaction level.

Physical Monitoring

The data may fulfill all the data quality rules , but still be wrong due to wrong data entry (willful OR unintentional). The last mile of ensuring this is to do a physical verification. Some of this is manifested through undelivered mails, untraced customers, unfulfilled orders etc. Given the business case, a set of physical verification procedures should be put into place. This is an expensive method, but apart from resolving discrepancies, it throws up process improvements for the future. For example placing a maker – checker for stock transfer.

Batch monitoring

Whenever a batch is run, which includes interfacing of files across the systems OR end of day processing, the data accuracy, consistency and completeness health check can be run (pre and post a job) to ensure defined quality benchmark before progressing. This monitoring of batch is online and real time.