Return to June
2001 contents page
Good management means good measurement
All countries need to be able to
measure the performance of their national immunization services
accurately. The World Health Organization and the Bill and Melinda
Gates Childrens Vaccine Program at PATH have developed a new tool
called the Data Quality Audit (DQA) to help countries check whether
their information systems are working well. Specifically, the DQA
will assess whether nationally reported data accurately reflect
the number of children being immunized and recorded at district
level. As auditors begin training this month in the use of the DQA
in Kenya, Uganda and Pakistan, John Lloyd explains why it matters
and how it works
Why do we need the DQA?
There are three good reasons why
the Data Quality Audit (DQA) is needed to audit the system that
reports on the performance of immunization services in each country.
First, managers of immunization services
need correct and timely information to detect improvement or decline
in performance. Second, the partners of the GAVI Alliance working
at all levels need reliable information to judge the impact of new
efforts and new resources on performance. Third, GAVI and the
Vaccine Fund award money to countries to improve their immunization
services according to a system of "shares", one share being earned
by the country for each additional child reported to have been immunized
relative to the previous year. The Vaccine Fund
can only reward governments on the basis of children who have been
correctly recorded and reported as immunized. So, the DQA aims to:
- Assess the quality, accuracy and completeness
of administrative immunization reporting systems; and
- Provide practical feedback to health staff on
how to improve the quality of reported data.
Where did the idea come from?
The DQA was born last summer, at the
start of the Vaccine Fund application
process, when it became clear that progress in reaching more children
with vaccines should be verified annually.
How will the audit ensure independence
The DQA is external and independent both
of the national management and the local staff of the GAVI partners.
The procedure is carried out by a team of international experts
in private- and public-sector auditing and in the field of public
health. The procedure, which lasts two to three weeks, may be carried
out in any country receiving assistance from GAVI and the
Vaccine Fund. But in practice, it is probable that it will
be applied to a selection of countries, depending on evidence of
the quality of reporting systems and the size of the
Vaccine Fund grant.
How does it work?
The DQA country visit focuses
on reporting practice in a sample of four districts and six health
centres in each district 24 health centres in all. The auditors
check the accuracy of recording of the number of immunizations,
the transcription and aggregation of these numbers and the reporting
from level to level of the system. The audit focuses mainly
on one key indicator of performance: the number of children who
receive a third dose of diphtheria, tetanus and pertussis (DTP)
vaccine. The third DTP dose, known as DTP3, is chosen because, though
not perfect, it is considered the most reliable measure of the number
of fully protected children.
A country may state in its reports to
WHO, UNICEF and GAVI that a certain number of children received
DTP3 in a given year. This number may include children who:
- are thought to have received DTP3 but were
never recorded; or
- received DTP3 and were recorded and reported;
- received DTP3 and were recorded but never reported;
- were reported to have received DTP, but were never
The DQA does not substantiate the first
group of children as having been immunized. But it does assess the
ability of the administrative reporting system to count and report
correctly those children that were recorded at the site of immunization.
Since awards will be made on the basis of the additional number
of children relative to the previous year who are recorded as having
received DTP3, there is a strong incentive to reduce the first group
of children those who were immunized but never recorded to zero.
What else does the DQA check, and
how does it help countries?
In addition to re-counting and
checking the data, the DQA judges the overall reliability
and timeliness of the reporting system, using a set of standard
indicators. For example, it looks at the proportion of records
that get lost and the proportion of reports that arrive late.
If the childs immunization cards are lost, the wrong
vaccine dose may be given and recorded. Late reports result
in incorrect aggregations and coverage calculations at higher
levels. There are many pitfalls that can be avoided if the
critical elements of quality are in place. This information
enables the auditors to offer advice to the health workers,
managers and national leaders of immunization. DQA is a powerful
capacity building block for each nations health management
information system and is a good example of the way in which
GAVI can strengthen the health system.
Dr Linda Archer, a WHO consultant
based in Nairobi, has developed successive drafts and refinements
of the DQA that have been tested in Kenya and Sri Lanka. Now,
the latest revision of the DQA manual is ready(1).
it: the immunization record card is vital for an effective system
The GAVI partners
decided to search for a suitable organization to implement the audit,
and after a tendering procedure they chose a consortium headed by
Liverpool Associates in Tropical Health, UK, a body associated with
the Liverpool School of Tropical Medicine with a strong research
knowledge base and global experience in health assessments and evaluations.
The training of auditors begins this month in Kenya, Uganda and
Pakistan and then six other countries will be visited by September
2001. Each year thereafter, a proportion of countries that receive
assistance from GAVI and the Vaccine Fund will be visited by DQA
(1) Copies may be
requested by email from Lisa Jacobs at the GAVI Secretariat: email@example.com
John Lloyd is Resident Adviser
at the European office of the Bill and Melinda Gates Childrens
Vaccine Program, implemented by PATH. Dr Lloyd took part in the
creation of the DQA concept, the development of the methodology
and has participated in the testing and the training of auditors.
Return to June 2001 contents