|
October 2001
Return to October
2001 contents page
SPECIAL REPORT
Do your data measure up?
Kenya, working with a team of auditors,
has just put its immunization data through an accuracy check. Lisa
Jacobs went along and listened to the audited and the auditors
PATRICK Mbugua, district public health
nurse for Muranga district in central Kenya, explains to Kangema
Health Unit's Medical Officer Julia Njagi, and Stanley Kagwi, the
nurse, why he and the other visitors have come this morning. They
are here to audit the unit's information system as part of a pilot
test of a new tool to assess the accuracy of national immunization
data. "The mission is to see, from top to bottom, the quality of
the information," says Mr Mbugua. "Because if there is a problem
at the bottom, it will go to the national level."
 |
The hand re-count: Vicki Doyle and Patrick
Mbugua check tally sheets |
With Mr Mbugua are Vicki Doyle, from
Liverpool Associates in Tropical Health (LATH), a UK-based company,
owned by the Liverpool School of Tropical Medicine, which heads
the independent consortium appointed by GAVI to do the audit, and
Kenyas national information officer for the Expanded Programme
on Immunization (EPI), David Kiongo.
"Didnt you know we were coming?"
asks Dr Akpala Kalu, immunization advisor from the national office
of the World Health Organization in Nairobi, who has joined the
audit today as an observer. "You didn't get the message that we
were coming?" Dr Kalu smiles. "Im just joking." Surprise is an
element of the audit.
Accurate numbers make for better
management
Why put people on the line like
this? For health workers struggling to provide a basic level of
service, keeping good records may seem less important than most
other aspects of their job. As one Kenyan nurse put it: "You have
mothers waiting for you, children waiting for you, curative waiting
for you, antenatal waiting for you, family planning waiting for
you. It is very difficult to rush back and tally."
Yet all countries need accurate immunization
data, so that their health managers can promptly detect downward
or upward trends, measure their own performance and direct their
resources efficiently to ensure the maximum number of children are
safely protected against killer diseases. The idiom, "If you can't
measure it, you cant manage it", is as true for immunization as
it is for any programme. And, since the emergence of GAVI and the
Vaccine Fund, the incentives to improve immunization data have sharpened.
Kenya is among the first countries to
receive funding from GAVI and the Vaccine Fund under the "share"
system, which provides incentives and rewards to countries for increasing
their immunization coverage. Under the system, in one years
time, the GAVI Board will need to decide how much to award each
country, based on its reported figures for the percentage of infants
receiving three doses of diphtheria, tetanus and pertussis (DTP3
coverage). Shares are awarded for each additional child reported
as immunized, relative to the previous year. Only reported immunizations
can be taken into account. If vaccinations are happening in health
units, but not getting reported, a country could be awarded less
money than it technically "deserves".
But the GAVI Board will also need assurance
that the immunization coverage data are correct. The immunization
data quality audit, or DQA, developed by health information experts
at WHO and the Bill and Melinda Gates Childrens Vaccine Program,
has been designed to do just that (see Box 1).
1. The DQ-What?
The immunization Data Quality
Audit assesses the accuracy of the immunization reporting
system that flows from the health units to the districts to
the national level. To do this, four districts are randomly
selected to ensure representativeness; within each district,
six health units are selected (24 health units in all). Two
teams, each comprised of one national immunization official
and one external auditor, split the districts; each team then
links up with a district official in their visits to the health
units.
Health unit records are compared
to district level records, district records are compared with
the nationally reported figures. In addition to the accuracy
checks, all aspects of the reporting system are assessed,
and the auditors also observe staff to ensure that their practice
is correct. The auditors give immediate feedback to national,
district and health unit staff on practical ways to strengthen
their performance and their recording system.
The audit was put out to tender
and GAVI eventually awarded the contract to a consortium that
is headed by LATH, in association with the Euro Health Group
from Denmark and the Deloitte and Touche Emerging Markets
Group in the US. The audit was initially implemented on a
pilot basis from May to September 2001 in eight countries
that were awarded the greatest support for strengthening their
immunization services in 2000 and early 2001: Côte dIvoire,
Kenya, Liberia, Mali, Pakistan, Rwanda, Tanzania, and Uganda.
Early indications from the pilot
sites reveal a problem in most countries with stock management
issues many staff do not have adequate training to keep accurate
vaccine ledgers, nor is this closely monitored. Data consistency
on the different levels varied in the countries tested, with
incidents of disagreement of data outnumbering those in which
data agreed. The DQA also highlighted that countries with
an integrated approach to data collection, such as Uganda
and Tanzania, have a problem with parallel reporting systems:
EPI data are reported twice, and inconsistently.
The experiences from this years
pilot audits will guide subsequent audits and will help GAVI
to decide whether it is an appropriate tool for adjusting
funding amounts provided to countries from the Vaccine Fund,
and if so, how this will be done. GAVI partners will review
the DQA pilot experience at the end of October; recommendations
will follow.
|
While its major emphasis is to
assess the quality, accuracy and completeness of immunization reporting
systems, the DQA has also been designed to provide practical feedback
to health staff on how to improve data quality. However, questions
have arisen regarding the emphasis on data reporting in such resource-poor
settings.
"Do you invest in quality of data
or reducing disease?" says Dr Kalu, the immunization adviser from
the national WHO office in Nairobi. Others working in immunization
in African settings argue, however, that there is no conflict between
good data management and combating disease.
"When I first came in I was quite
cynical," says Dr Doyle. But after conducting DQAs in Uganda and
Kenya, her attitude has changed. "As a starting point it's really
good its like a wake-up call." If vaccinations
are not being reported properly, it could be an indication of more
fundamental problems in the programme whether it is lack
of knowledge of policy and procedures, inadequate supervision, or
staff shortages. The auditors report on this information as well
as the numbers.
And, says Dr Doyle, inaccurate information
can lead to waste of scarce resources. "If theyre under-reporting
their immunization they may be spending money on areas they shouldnt."
For example, a district might invest in unnecessary outreach or
social mobilization efforts in a community if the reports say coverage
is low, when actually, children are being reached but not counted.
The health unit re-count
The DQA starts at the most basic
level of reporting: the immunization tally sheet. Nurses use tally
sheets to tick off each vaccine they administer over the course
of an immunization session whether it's a whole day, just
a morning, or an outreach activity.
As part of the DQA, Dr Doyle and Mr
Mbugua will need to re-count by hand the health unit's tally sheets
from the previous year, verifying the number of DTP3, measles, and
maternal tetanus vaccinations that were given over the year. The
recount is then compared with the data that the unit had reported
to the district; district data are compared with reports found at
the national ministry. Consistency is what the auditors look for.
Where are Kangema's tally sheets for
the year 2000? No one has ever asked to see them before.
Two searches through a file cabinet
finally produce a pile of crumpled tally sheets from the year 2000
all except January. In many of the health units visited in
Kenya, none of the tally sheets from 2000 could be found. In others,
they were found to have new uses: as liners for the scales used
for weighing babies, folded into envelopes to hold drugs from the
phamacy, or used to write out lab requests. Not surprising, perhaps,
considering that Kenya does not currently have any official policy
on what units should do with tally sheets after the data are transferred
to the monthly reports. "Now I think we will develop a policy,"
says Mr Kiongo, the EPI information officer.
While the particular problem of missing
tally sheets was more serious in Kenya than in other countries that
participated in the DQA pilot this year, it is widespread. So why
does the DQA look at them, especially since they seem to be in such
shortage? Simple: "Tally sheets are difficult to fake," says Dr
Doyle. In other words, if all the tally sheets look crisp and new,
and are filled out with the same pen (which was observed in one
health unit in another pilot country), the auditors are going to
catch it.
While Dr Doyle and Mr Mbugua re-count
the tally sheets, David Kiongo observes immunizations, assessing
whether the vaccinators are giving infants the right vaccines and
correctly marking the tally sheets and the child health cards. In
health units that do not conduct immunizations every day, a simulation
exercise has been developed to assess performance.
The DQA also looks for other signs that
the health information system is working. For example, is there
a chart or table showing child vaccination rates on display? Has
there been a supervisory visit in the last two months and is there
a record of the topics covered? And, does the unit maintain an accurate
ledger book to track stock of the different vaccines?
Other system issues are also assessed.
For example, does the districts senior medical officer
not the EPI person sign the reports? If so, this indicates
that immunization is integrated into the wider health system. Are
the best demographic estimates used in the calculation of the denominator?
And so on.
|
Getting it out on the table: the auditors help to identify strengths and weaknesses and give staff immediate feedback |
The feedback session
Kangema fares better than others,
but can still only account for 57% of DTP3 reported at the district
level. However, the auditors have found coverage and drop-out charts
prominently displayed, a vaccine stock ledger book unfortunately,
one month out of date, but there nonetheless and a reasonably
good system for keeping records filed. "We know that youre
doing a good job here," Dr Doyle tells the staff.
But there are some areas for improvement.
"You need to know your catchment area," says Dr Doyle. "Otherwise,
how do you know you are achieving the level that you should be doing?"
Careful stock recording is also essential. "When the new vaccine
comes, it will be very expensive," says Dr Doyle. "Reducing wastage
and stock management will be very important." And a practical suggestion:
"When you receive stock, write it in red. When you take it out,
write it in blue."
David Kiongo reports that the vaccination
sessions he observed were all correct. But there is another problem:
"You are doing immunizations in the same room where you have sick
babies coming in," he says. Obstetrics, antenatal and family planning
patients also use the room.
"There seems to be an empty room
available, not all of the rooms are being used," Mr Kiongo says.
Perhaps they could convert a room down the hall into another room
for outpatient mother-and-child-health care?
Mr Kiongo continued. "And another thing,
slightly outside the audit I saw your health officer re-capping
needles." Re-capping needles before discarding them raises the risk
that a health worker might prick a finger; if the blood is infected,
the worker might infect themselves. He advises the team on good
safety rules.
"Most of these things are within
our reach. We want to be able to do them," says Nurse Kagwi, who
has been taking fervent notes throughout the feedback session. "Thank
you for coming. Maybe if you are in the province again you could
come back to see how we are doing."
Meanwhile, in Bondo
Staff in the health units in Bondo
face greater challenges than those in Muranga. Bondo is in
the western part of Kenya, on the shores of Lake Victoria. The region
is one of the poorest in the country, and immunization rates are
low: whereas DTP3 coverage rates in Muranga district hover
around 85%, Bondo reports well below 50%.
Five out of the six health units selected
in Bondo do not have tally sheets available to the auditors for
re-count. The sixth is the district hospital, and staff there search
for three hours to locate the sheets.
In their feedback to the district officials
in Bondo, LATH auditor Max Moyo and his national counterpart, vaccine
control officer Dr Amos Chweya, lay down some hard truths. "We came
here to investigate: 'Is it true that the data we are getting from
you is the same as the data from the units?'" Dr Chweya begins.
"As you will see the picture looks a bit funny numbers from
you are different than numbers we found at the units and than we
get in our recounts."
Furthermore, none of the health units
has been found to keep records of vaccine stock. "This is a weakness
that we have in the country it is not just a problem in Bondo,"
says Dr Chweya. "At the moment we do not know our wastage rate."
This is a big problem, considering the higher cost of the new vaccines
being supplied by GAVI and the Vaccine Fund. "The days when we had
people bringing in vaccines when we asked for them are long gone,"
warns Dr Chweya.
Mr Moyo points out that none of the
health units have immunization coverage targets. "If you are travelling
to Nairobi, you need to know where Nairobi is. Otherwise, how do
you know if you have made it there?"
Even accessing the data has proved difficult.
The district data manager has been out of the office because of
a broken-down car, and no one else knows how to find the correct
records. "The way an information office should work is that the
information is available when you are away," says Mr Moyo, an information
systems expert. "So if parliament calls and asks what your immunization
rate is, anyone could go to a file and get the data." In other words,
"Dont take the keys with you when you go," he says. "Sorry
to say that but thats the way I think."
"We did not come to condemn,"
says Dr Chweya, Kenya's national vaccine control officer
Dr Francis Odira, the district medical
officer of health, listens quietly to the feedback. "I agree with
most of the findings. They are not exaggerated," he says finally.
"The quality of our health information system has been declining.
It is something we have known but we were looking for ways to improve."
But, he says, the problems can't be
solved without addressing staffing issues. "Most health units have
only one qualified nurseb one to see patients, take records,
summarize records and take them to the district. She also has to
collect revenue from patients," he says. Most districts in Kenya
require patients to pay small user fees for health services
including immunization. "Most of the way we work is a matter of
improvising," adds Herbert Onyando, the district records and information
officer.
"We know the constraints," says
Dr Chweya. "We didnt come to condemn. We are telling you what
we found out there from the people hat they said." He promises
to bring the staffing issue to the national level.
"We still have time to improve our
records"
The final step for the auditors is to
present their findings to a meeting of the Interagency Coordinating
Committee. Dr Doyle and Dr Chweya split the presentation between
them. Dr Doyle presents the main findings: poor data storage leading
to reporting inaccuracies at all levels; weak systems to monitor
and analyse immunization performance; and a lack of integration
in the health information system.
All of these issues contribute to inaccuracies
in Kenya's reporting system while Kenyas "best estimate"
of its coverage for 2000 was 63%, the DQA finds that the reported
coverage from administrative data was 51%.
But Dr Doyle is optimistic. "Though
weve pointed out a lot of the weaknesses here, we were very
impressed with the commitment of people to improve the system of
data collection."
And, as Dr Chweya says to the district
management team in Bondo: "When GAVI comes back to audit us they
will look at 2001. We still have time to improve our records."
Lisa Jacobs
Return to October 2001 contents
page
|