Quick Reference:

What is GAVI?

GAVI Fact Sheets

Governance

Board Documents

Guidelines & Forms

Press Releases

Immunization Information

Immunization Focus
Latest Issue

GAVI Update

The Data Quality Audit (DQA)

The DQA was conceived as a means to verify reported performance as well as enhance immunization monitoring and reporting systems. It reviews both the numbers of children reported to have received a DTP3 injection and the accuracy of the EPI reporting system. Effective programmes with accurate data recording and reporting systems receive the most credit in a DQA.

Three reasons for accurate recording and reporting practices are:

1. managers of immunization services need correct and timely information to detect improvement or decline in performance
2. partners of the GAVI Alliance working at all levels need reliable information to judge the impact of new efforts and new resources on performance
3. GAVI award money according to a system of ”shares”, one share being earned by the country for each additional child reported to have been immunized relative to the previous or baseline year

There are many pitfalls that can be avoided if the critical elements of quality are in place. The information collected from a DQA enables auditors to offer advice to health workers, managers and national leaders of EPI programmes. DQA is a powerful capacity building tool for immunization information systems and is a good example of the way in which GAVI can assist country programmes to achieve their own objectives.

Annual report form (February 2004):

• Annual report form (English) (Zip file - 63kb)
  Guidelines (34kb)



• Annual report form (French) (Zip file - 57kb)
  Guidelines (36kb)

Guidelines for preparing a national immunization program financial sustainability plan.

Please note: These forms are subject to revision.

Available monitoring reports received from countries

Data Quality Audits

Annual Progress Reports

Financial Sustainability Plans

The Data Quality Audit

Other Information

Members of the Independent Review Committee Monitoring and Evaluation Team

GAVI Update 2003 (Deadlines for receipt of documents from countries for Proposals and Monitoring & Evaluation reviews)

If you have questions, please contact your local UNICEF or WHO office, or the GAVI Secretariat at gavi@unicef.org

Q&As (Revised 2003)

1. Is the DQA justified?
2. Is the DQA approach equitable between countries?
3. Is the DQA costly?
4. Are the 4 sampled districts representative of one country, even if the country has more than 100 districts?
5. Are the DQA measurements precise?
6. Is the pass/fail approach to determine access to rewards acceptable?
7. Is the threshold (80%) approach to determine access to rewards acceptable?
8. Is the Verification Factor a single valid indicator of a country reporting system?
9. What should be done in a country to prepare for a DQA?
10. What should be done if a country’ fails’ the DQA?
11. What should be done if a country ‘passes’ the DQA?

1. Is the DQA justified?

YES, GAVI partners feel that the new performance-based approach promoted by GAVI necessitates an investment in a data verification tool. This reassures partners about the validity of reported data and justifies investment from donors. Furthermore the DQA is a useful diagnosis tool for evaluating reporting systems, helping countries to identify their weaknesses, and producing recommendations for improvement.

back to top

2. Is the DQA approach equitable between countries?

YES, the tool is standardized; the same methodology is applied by a trained independent auditing company in all countries receiving ISS funds (with the exception of countries receiving a small ( $100,000) amount of money from the Vaccine Fund).

back to top

3. Is the DQA costly?

As an exercise conducted by external auditors, the DQA is relatively costly (around $45,000 per audit) when compared with other assessments that could be conducted by partners or national experts. However, the cost of other type of surveys (such as an immunization coverage survey) conducted externally would be higher. One of the reasons for the relatively small sample sizes in the DQA has been the cost factor.

back to top

4. Are the 4 sampled districts representative of one country, even if the country has more than 100 districts?

YES, both the sampling method and the sample sizes were validated by statistical experts (from Ohio State University) and can be used whatever the number of districts in one country. Using random selection, the survey provides a representative picture of the monitoring and reporting systems of a country. Additionally, the small sample (4 districts and 24 health units) implicitly requires a country to have good practices and undertake corrective measures in virtually all centres, which, if reached, will promote an equally high monitoring quality level.

back to top

5. Are the DQA measurements precise?

Not precise enough; the measures (including the verification factor, VF) have very large confidence intervals (CI), usually greater than +/-20% (at the 95% confidence level) with the width increasing with intra and inter district variability. This width was an argument not to take the CIs into account for the access to rewards. This limits considerably the comparability of VFs across time and place. The interpretation of the VF is very non-statistical because of the limitation on the ability to draw valid statistical comparisons and conclusions.

back to top

6. Is the pass/fail approach to determine access to rewards acceptable?

The original approach to adjust rewards mathematically based on a verified number has proven difficult. As the verification was to be conducted starting at the most peripheral level (i.e. health unit), a sample of health units had to be taken, making the measures not precise enough. Instead, the pass/fail decision follows the approach of the lot quality assurance survey, in which a lot is accepted or rejected if a threshold is reached. The approach has also the advantage of being simple (yes/no).

back to top

7. Is the threshold (80%) approach to determine access to rewards acceptable?

YES, as long as one threshold has to be taken. This approach allows a 20% verification gap. The approach has allowed half of 16 audited countries in 2002 to be eligible for rewards. If a country’s VF is below the threshold, the opportunity of conducting a coverage survey is still possible, allowing the country to be eligible for rewards for one year.

back to top

8. Is the Verification Factor a single valid indicator of a country reporting system?

The VF correlates well (significantly) with the Quality Indices at health unit and district level, meaning that lower VFs indicate weaker monitoring systems. However, each level of the system (national/district/health unit) needs to be looked at independently to tackle specific weaknesses: one single measure cannot reflect all problems. At this stage, it is felt that the VF is a valid measure of accuracy and can be used as a basis for deciding on rewards. Most countries with repeat DQAs (2001 and 2002) have been able to improve the VF value, associated with actions undertaken to improve their reporting system based on the findings of the DQA.

back to top

9. What should be done in a country to prepare for a DQA?

As a start, the DQA process should be well understood by all ICC partners so that the best collaboration can happen between the auditing company and the country officials during the audit, resulting in fruitful discussions. The DQA briefing paper provides the basics of the DQA process. Then a country should also look at various indicators of the quality its monitoring system (e.g. variance between different coverage estimates, reporting completeness, etc) to determine where the weaknesses are and where and how the situation should be improved. Data quality assessments and supervisory checklists are two tools that can be used for a precise diagnosis and guide appropriate actions.

back to top

10. What should be done if a country’ fails’ the DQA?

The country Interagency Coordinating Committee (ICC) should convene and review recommendations described in the report. An action plan should be prepared (to be submitted with the next GAVI annual report). Actions should be discussed with the Regional Working Group and needs reported to the implementing partners. As an important action to be taken rapidly in the country, the availability of tally sheets and registers at the health unit should be promoted. A systematic follow-up based on the findings and recommendations of the DQA is crucial and is one of the key actions for the country to eventually improve coverage.

back to top

11. What should be done if a country ‘passes’ the DQA?

Even if the accuracy measure (the VF) indicates a reliable reporting system, a number of improvement areas might have been identified during the DQA, needing an appropriate follow-up and corrective measures. As the DQA is also looking at other indicators such as the vaccine wastage and adverse event following immunization surveillance implementation (reflected in the quality indices), it is perfectly possible to have a good coverage accuracy but a significant weakness in the monitoring of another area. The ICC should review those weaknesses and the report recommendations in order to implement ad hoc follow-up measures.

back to top

Note: the target audience is assumed to know the basics of the DQA and of the reward system implemented by GAVI. The answers provided refer to a number of terms related to the DQA methodology. The DQA briefing paper will be a good reference for less experienced readers.