SPECIAL REPORT - October 2001
The hand re-count: Vicki Doyle and Patrick Mbugua check tally sheets
Do your data measure up?
Kenya, working with a team of auditors, has just put its immunization data through an accuracy check. Lisa Jacobs went along and listened to the audited and the auditors
PATRICK Mbugua, district public health nurse for Muranga district in central Kenya, explains to Kangema Health Unit's Medical Officer Julia Njagi, and Stanley Kagwi, the nurse, why he and the other visitors have come this morning. They are here to audit the unit's information system as part of a pilot test of a new tool to assess the accuracy of national immunization data. "The mission is to see, from top to bottom, the quality of the information," says Mr Mbugua. "Because if there is a problem at the bottom, it will go to the national level."
With Mr Mbugua are Vicki Doyle, from Liverpool Associates in Tropical Health (LATH), a UK-based company, owned by the Liverpool School of Tropical Medicine, which heads the independent consortium appointed by GAVI to do the audit, and Kenyas national information officer for the Expanded Programme on Immunization (EPI), David Kiongo.
"Didnt you know we were coming?" asks Dr Akpala Kalu, immunization advisor from the national office of the World Health Organization in Nairobi, who has joined the audit today as an observer. "You didn't get the message that we were coming?" Dr Kalu smiles. "Im just joking." Surprise is an element of the audit.
Accurate numbers make for better management
Why put people on the line like this? For health workers struggling to provide a basic level of service, keeping good records may seem less important than most other aspects of their job. As one Kenyan nurse put it: "You have mothers waiting for you, children waiting for you, curative waiting for you, antenatal waiting for you, family planning waiting for you. It is very difficult to rush back and tally."
Yet all countries need accurate immunization data, so that their health managers can promptly detect downward or upward trends, measure their own performance and direct their resources efficiently to ensure the maximum number of children are safely protected against killer diseases. The idiom, "If you can't measure it, you cant manage it", is as true for immunization as it is for any programme. And, since the emergence of GAVI and the Vaccine Fund, the incentives to improve immunization data have sharpened.
Kenya is among the first countries to receive funding from GAVI and the Vaccine Fund under the "share" system, which provides incentives and rewards to countries for increasing their immunization coverage. Under the system, in one years time, the GAVI Board will need to decide how much to award each country, based on its reported figures for the percentage of infants receiving three doses of diphtheria, tetanus and pertussis (DTP3 coverage). Shares are awarded for each additional child reported as immunized, relative to the previous year. Only reported immunizations can be taken into account. If vaccinations are happening in health units, but not getting reported, a country could be awarded less money than it technically "deserves".
But the GAVI Board will also need assurance that the immunization coverage data are correct. The immunization data quality audit, or DQA, developed by health information experts at WHO and the Bill and Melinda Gates Childrens Vaccine Program, has been designed to do just that (see Box 1).
1. The DQ-What?
The immunization Data Quality Audit assesses the accuracy of the immunization reporting system that flows from the health units to the districts to the national level. To do this, four districts are randomly selected to ensure representativeness; within each district, six health units are selected (24 health units in all). Two teams, each comprised of one national immunization official and one external auditor, split the districts; each team then links up with a district official in their visits to the health units.
Health unit records are compared to district level records, district records are compared with the nationally reported figures. In addition to the accuracy checks, all aspects of the reporting system are assessed, and the auditors also observe staff to ensure that their practice is correct. The auditors give immediate feedback to national, district and health unit staff on practical ways to strengthen their performance and their recording system.
The audit was put out to tender and GAVI eventually awarded the contract to a consortium that is headed by LATH, in association with the Euro Health Group from Denmark and the Deloitte and Touche Emerging Markets Group in the US. The audit was initially implemented on a pilot basis from May to September 2001 in eight countries that were awarded the greatest support for strengthening their immunization services in 2000 and early 2001: Côte dIvoire, Kenya, Liberia, Mali, Pakistan, Rwanda, Tanzania, and Uganda.
Early indications from the pilot sites reveal a problem in most countries with stock management issues many staff do not have adequate training to keep accurate vaccine ledgers, nor is this closely monitored. Data consistency on the different levels varied in the countries tested, with incidents of disagreement of data outnumbering those in which data agreed. The DQA also highlighted that countries with an integrated approach to data collection, such as Uganda and Tanzania, have a problem with parallel reporting systems: EPI data are reported twice, and inconsistently.
The experiences from this years pilot audits will guide subsequent audits and will help GAVI to decide whether it is an appropriate tool for adjusting funding amounts provided to countries from the Vaccine Fund, and if so, how this will be done. GAVI partners will review the DQA pilot experience at the end of October; recommendations will follow.
While its major emphasis is to assess the quality, accuracy and completeness of immunization reporting systems, the DQA has also been designed to provide practical feedback to health staff on how to improve data quality. However, questions have arisen regarding the emphasis on data reporting in such resource-poor settings.
"Do you invest in quality of data or reducing disease?" says Dr Kalu, the immunization adviser from the national WHO office in Nairobi. Others working in immunization in African settings argue, however, that there is no conflict between good data management and combating disease.
"When I first came in I was quite cynical," says Dr Doyle. But after conducting DQAs in Uganda and Kenya, her attitude has changed. "As a starting point it's really good its like a wake-up call." If vaccinations are not being reported properly, it could be an indication of more fundamental problems in the programme whether it is lack of knowledge of policy and procedures, inadequate supervision, or staff shortages. The auditors report on this information as well as the numbers.
And, says Dr Doyle, inaccurate information can lead to waste of scarce resources. "If theyre under-reporting their immunization they may be spending money on areas they shouldnt." For example, a district might invest in unnecessary outreach or social mobilization efforts in a community if the reports say coverage is low, when actually, children are being reached but not counted.
The health unit re-count
The DQA starts at the most basic level of reporting: the immunization tally sheet. Nurses use tally sheets to tick off each vaccine they administer over the course of an immunization session whether it's a whole day, just a morning, or an outreach activity.
As part of the DQA, Dr Doyle and Mr Mbugua will need to re-count by hand the health unit's tally sheets from the previous year, verifying the number of DTP3, measles, and maternal tetanus vaccinations that were given over the year. The recount is then compared with the data that the unit had reported to the district; district data are compared with reports found at the national ministry. Consistency is what the auditors look for.
Where are Kangema's tally sheets for the year 2000? No one has ever asked to see them before.
Two searches through a file cabinet finally produce a pile of crumpled tally sheets from the year 2000 all except January. In many of the health units visited in Kenya, none of the tally sheets from 2000 could be found. In others, they were found to have new uses: as liners for the scales used for weighing babies, folded into envelopes to hold drugs from the phamacy, or used to write out lab requests. Not surprising, perhaps, considering that Kenya does not currently have any official policy on what units should do with tally sheets after the data are transferred to the monthly reports. "Now I think we will develop a policy," says Mr Kiongo, the EPI information officer.
While the particular problem of missing tally sheets was more serious in Kenya than in other countries that participated in the DQA pilot this year, it is widespread. So why does the DQA look at them, especially since they seem to be in such shortage? Simple: "Tally sheets are difficult to fake," says Dr Doyle. In other words, if all the tally sheets look crisp and new, and are filled out with the same pen (which was observed in one health unit in another pilot country), the auditors are going to catch it.
While Dr Doyle and Mr Mbugua re-count the tally sheets, David Kiongo observes immunizations, assessing whether the vaccinators are giving infants the right vaccines and correctly marking the tally sheets and the child health cards. In health units that do not conduct immunizations every day, a simulation exercise has been developed to assess performance.
The DQA also looks for other signs that the health information system is working. For example, is there a chart or table showing child vaccination rates on display? Has there been a supervisory visit in the last two months and is there a record of the topics covered? And, does the unit maintain an accurate ledger book to track stock of the different vaccines?
Other system issues are also assessed. For example, does the districts senior medical officer not the EPI person sign the reports? If so, this indicates that immunization is integrated into the wider health system. Are the best demographic estimates used in the calculation of the denominator? And so on.
Getting it out on the table: the auditors help to identify strengths and weaknesses and give staff immediate feedback
The feedback session
Kangema fares better than others, but can still only account for 57% of DTP3 reported at the district level. However, the auditors have found coverage and drop-out charts prominently displayed, a vaccine stock ledger book unfortunately, one month out of date, but there nonetheless and a reasonably good system for keeping records filed. "We know that youre doing a good job here," Dr Doyle tells the staff.
But there are some areas for improvement. "You need to know your catchment area," says Dr Doyle. "Otherwise, how do you know you are achieving the level that you should be doing?" Careful stock recording is also essential. "When the new vaccine comes, it will be very expensive," says Dr Doyle. "Reducing wastage and stock management will be very important." And a practical suggestion: "When you receive stock, write it in red. When you take it out, write it in blue."
David Kiongo reports that the vaccination sessions he observed were all correct. But there is another problem: "You are doing immunizations in the same room where you have sick babies coming in," he says. Obstetrics, antenatal and family planning patients also use the room.
"There seems to be an empty room available, not all of the rooms are being used," Mr Kiongo says. Perhaps they could convert a room down the hall into another room for outpatient mother-and-child-health care?
Mr Kiongo continued. "And another thing, slightly outside the audit I saw your health officer re-capping needles." Re-capping needles before discarding them raises the risk that a health worker might prick a finger; if the blood is infected, the worker might infect themselves. He advises the team on good safety rules.
"Most of these things are within our reach. We want to be able to do them," says Nurse Kagwi, who has been taking fervent notes throughout the feedback session. "Thank you for coming. Maybe if you are in the province again you could come back to see how we are doing."
Meanwhile, in Bondo
Staff in the health units in Bondo face greater challenges than those in Muranga. Bondo is in the western part of Kenya, on the shores of Lake Victoria. The region is one of the poorest in the country, and immunization rates are low: whereas DTP3 coverage rates in Muranga district hover around 85%, Bondo reports well below 50%.
Five out of the six health units selected in Bondo do not have tally sheets available to the auditors for re-count. The sixth is the district hospital, and staff there search for three hours to locate the sheets.
In their feedback to the district officials in Bondo, LATH auditor Max Moyo and his national counterpart, vaccine control officer Dr Amos Chweya, lay down some hard truths. "We came here to investigate: 'Is it true that the data we are getting from you is the same as the data from the units?'" Dr Chweya begins. "As you will see the picture looks a bit funny numbers from you are different than numbers we found at the units and than we get in our recounts."
Furthermore, none of the health units has been found to keep records of vaccine stock. "This is a weakness that we have in the country it is not just a problem in Bondo," says Dr Chweya. "At the moment we do not know our wastage rate." This is a big problem, considering the higher cost of the new vaccines being supplied by GAVI and the Vaccine Fund. "The days when we had people bringing in vaccines when we asked for them are long gone," warns Dr Chweya.
Mr Moyo points out that none of the health units have immunization coverage targets. "If you are travelling to Nairobi, you need to know where Nairobi is. Otherwise, how do you know if you have made it there?"
Even accessing the data has proved difficult. The district data manager has been out of the office because of a broken-down car, and no one else knows how to find the correct records. "The way an information office should work is that the information is available when you are away," says Mr Moyo, an information systems expert. "So if parliament calls and asks what your immunization rate is, anyone could go to a file and get the data." In other words, "Dont take the keys with you when you go," he says. "Sorry to say that but thats the way I think."
"We did not come to condemn," says Dr Chweya, Kenya's national vaccine control officer
Dr Francis Odira, the district medical officer of health, listens quietly to the feedback. "I agree with most of the findings. They are not exaggerated," he says finally. "The quality of our health information system has been declining. It is something we have known but we were looking for ways to improve."
But, he says, the problems can't be solved without addressing staffing issues. "Most health units have only one qualified nurseb one to see patients, take records, summarize records and take them to the district. She also has to collect revenue from patients," he says. Most districts in Kenya require patients to pay small user fees for health services including immunization. "Most of the way we work is a matter of improvising," adds Herbert Onyando, the district records and information officer.
"We know the constraints," says Dr Chweya. "We didnt come to condemn. We are telling you what we found out there from the people hat they said." He promises to bring the staffing issue to the national level.
"We still have time to improve our records"
The final step for the auditors is to present their findings to a meeting of the Interagency Coordinating Committee. Dr Doyle and Dr Chweya split the presentation between them. Dr Doyle presents the main findings: poor data storage leading to reporting inaccuracies at all levels; weak systems to monitor and analyse immunization performance; and a lack of integration in the health information system.
All of these issues contribute to inaccuracies in Kenya's reporting system while Kenyas "best estimate" of its coverage for 2000 was 63%, the DQA finds that the reported coverage from administrative data was 51%.
But Dr Doyle is optimistic. "Though weve pointed out a lot of the weaknesses here, we were very impressed with the commitment of people to improve the system of data collection."
And, as Dr Chweya says to the district management team in Bondo: "When GAVI comes back to audit us they will look at 2001. We still have time to improve our records."
Immunization Focus October 2001 - Contents