June 2001

Return to June 2001 contents page

BRIEFING

Good management means good measurement

All countries need to be able to measure the performance of their national immunization services accurately. The World Health Organization and the Bill and Melinda Gates Children’s Vaccine Program at PATH have developed a new tool called the Data Quality Audit (DQA) to help countries check whether their information systems are working well. Specifically, the DQA will assess whether nationally reported data accurately reflect the number of children being immunized and recorded at district level. As auditors begin training this month in the use of the DQA in Kenya, Uganda and Pakistan, John Lloyd explains why it matters and how it works

Why do we need the DQA?

There are three good reasons why the Data Quality Audit (DQA) is needed to audit the system that reports on the performance of immunization services in each country.

First, managers of immunization services need correct and timely information to detect improvement or decline in performance. Second, the partners of the GAVI Alliance working at all levels need reliable information to judge the impact of new efforts and new resources on performance. Third, GAVI and the Vaccine Fund award money to countries to improve their immunization services according to a system of "shares", one share being earned by the country for each additional child reported to have been immunized relative to the previous year. The Vaccine Fund can only reward governments on the basis of children who have been correctly recorded and reported as immunized. So, the DQA aims to:

• Assess the quality, accuracy and completeness of administrative immunization reporting systems; and
• Provide practical feedback to health staff on how to improve the quality of reported data.

Where did the idea come from?

The DQA was born last summer, at the start of the Vaccine Fund application process, when it became clear that progress in reaching more children with vaccines should be verified annually.

How will the audit ensure independence and transparency?

The DQA is external and independent both of the national management and the local staff of the GAVI partners. The procedure is carried out by a team of international experts in private- and public-sector auditing and in the field of public health. The procedure, which lasts two to three weeks, may be carried out in any country receiving assistance from GAVI and the Vaccine Fund. But in practice, it is probable that it will be applied to a selection of countries, depending on evidence of the quality of reporting systems and the size of the Vaccine Fund grant.

How does it work?

The DQA country visit focuses on reporting practice in a sample of four districts and six health centres in each district – 24 health centres in all. The auditors check the accuracy of recording of the number of immunizations, the transcription and aggregation of these numbers and the reporting from level to level of the system. The audit focuses mainly on one key indicator of performance: the number of children who receive a third dose of diphtheria, tetanus and pertussis (DTP) vaccine. The third DTP dose, known as DTP3, is chosen because, though not perfect, it is considered the most reliable measure of the number of fully protected children.

A country may state in its reports to WHO, UNICEF and GAVI that a certain number of children received DTP3 in a given year. This number may include children who:

• are thought to have received DTP3 but were never recorded; or
• received DTP3 and were recorded and reported; or
• received DTP3 and were recorded but never reported; or
• were reported to have received DTP, but were never recorded.

The DQA does not substantiate the first group of children as having been immunized. But it does assess the ability of the administrative reporting system to count and report correctly those children that were recorded at the site of immunization. Since awards will be made on the basis of the additional number of children relative to the previous year who are recorded as having received DTP3, there is a strong incentive to reduce the first group of children – those who were immunized but never recorded – to zero.

What else does the DQA check, and how does it help countries?

In addition to re-counting and checking the data, the DQA judges the overall reliability and timeliness of the reporting system, using a set of standard indicators. For example, it looks at the proportion of records that get lost and the proportion of reports that arrive late. If the child’s immunization cards are lost, the wrong vaccine dose may be given and recorded. Late reports result in incorrect aggregations and coverage calculations at higher levels. There are many pitfalls that can be avoided if the critical elements of quality are in place. This information enables the auditors to offer advice to the health workers, managers and national leaders of immunization. DQA is a powerful capacity building block for each nation’s health management information system and is a good example of the way in which GAVI can strengthen the health system. What next? Dr Linda Archer, a WHO consultant based in Nairobi, has developed successive drafts and refinements of the DQA that have been tested in Kenya and Sri Lanka. Now, the latest revision of the DQA manual is ready(1).
		Hold onto it: the immunization record card is vital for an effective system

The GAVI partners decided to search for a suitable organization to implement the audit, and after a tendering procedure they chose a consortium headed by Liverpool Associates in Tropical Health, UK, a body associated with the Liverpool School of Tropical Medicine with a strong research knowledge base and global experience in health assessments and evaluations. The training of auditors begins this month in Kenya, Uganda and Pakistan and then six other countries will be visited by September 2001. Each year thereafter, a proportion of countries that receive assistance from GAVI and the Vaccine Fund will be visited by DQA auditors.

(1) Copies may be requested by email from Lisa Jacobs at the GAVI Secretariat: ljacobs@unicef.org

John Lloyd is Resident Adviser at the European office of the Bill and Melinda Gates Children’s Vaccine Program, implemented by PATH. Dr Lloyd took part in the creation of the DQA concept, the development of the methodology and has participated in the testing and the training of auditors.

 

Download this issue (PDF document – 10 pages/approx. 173k)

Download Adobe Acrobat (Required to read PDF documents)

Return to June 2001 contents page