Real-Time Monitoring of Vaccination Campaign Performance Using Mobile Phones — Nepal, 2016

Intra-Campaign Monitoring Through RCM

As part of Nepal’s continuing progress toward measles elimination (5), a nationwide measles-rubella vaccination campaign directed at children aged 9–59 months was implemented in four phases^§ during September 2015–March 2016 in all 75 districts of Nepal, which are administratively divided into village development committees (VDCs) and further divided into VDC wards. Intra-campaign monitoring was implemented according to the WHO Comprehensive Field Guide for Planning and Implementing High Quality Supplementary Immunization Activities for Measles and Rubella and other Injectable Vaccines (6). In each VDC, the SIA was conducted over 10–15 days, and RCM was conducted on the third and seventh days in VDCs identified by the Ministry of Health (MoH) as high risk (i.e., <90% immunization service delivery coverage; location near the India border with population movement; large population; and hard-to-reach).

RCM Monitoring Using Mobile Phones

All 33 districts included in the second phase of the campaign during February 2016 used paper-based RCM, but the MoH and WHO-Nepal selected 10 districts among them that included a mix of high- and low-performance in immunization service delivery and different geographic topographies (five were in the plains and five were hilly) for pilot testing RCM-MP on a limited scale. Thus, in the 10 pilot districts, there was a mix of VDCs where RCM was conducted using paper forms or mobile phones. For the RCM-MP, data collection forms were programmed into an electronic data collection tool^¶ and loaded onto Android phones.** National-level staff members were trained on use of the phones and software 2 weeks before deployment, and the national staff team then provided training for field monitors. Electronic data visualization software^†† was used to create two dashboards connected directly to the server, where data were uploaded, enabling real-time data visualization of SIA implementation performance indicators on the dashboards for national and district supervisors (Table 1). The first dashboard was designed to be action-oriented, displaying overall SIA performance, reasons for nonvaccination and refusal, and monitoring results by VDC and date of visit. The second dashboard was created to track monitors’ activities using a map showing GPS coordinates collected at each household and to display the number of reports received and results found by district (Figure).

In each pilot district, 10 monitors using mobile phones were responsible for two VDCs each. In each VDC, one or two visits were made to complete one in-house and one out-of-house RCM form^§§ during each visit. In total, 100 monitors collected RCM-MP data on 11,093 children in 377 visits in 196 VDCs. Among monitored children, 10,583 (95%) were vaccinated; 159 (42%) of 377 visited areas required remedial action (Table 2). Among the 311 incompletely vaccinated households, the primary reasons for nonvaccination were child absence during an SIA (126 of 311 [41%]) or vaccine refusal (68 of 311 [22%]); the primary reason for vaccine refusal was child being sick (53 of 68 [78%]). Six months after completion of the SIA, no RCM reports had been received at the central level from districts using paper-based RCM. In contrast, 94% (377 of 400) of expected reports were received from 98% (196 of 200) of VDCs where RCM-MP was conducted, and 87% (328 of 377) of these reports were received on the same day the data were collected.

Assessment of Use of Mobile Phones for RCM

All 100 monitors who used mobile phones, 10 district supervisors, and six national supervisors were asked to respond to a questionnaire about their experience. Completed questionnaires were received from 93 monitors and three district and five national supervisors. Common challenges reported by the monitors were finding and connecting to a third generation (3G) or Wi-Fi network^¶¶ (56 of 93 [60%]) and mistakenly striking incorrect keys (21 of 93 [22%]). Recording GPS location also was a challenge; only 61% (5,730 of 9,425) of expected GPS coordinates were captured. Despite challenges, 64% (54 of 84) of monitors found the mobile technology easy or somewhat easy to use, and 51% (47 of 93) found it easier than paper data collection.*** Ninety-six percent (78 of 81) of monitors thought that the technology increased data accuracy, and 90% (71 of 79) recommended its future use. Among the three district and five national supervisors, all found the technology helpful or somewhat helpful, and seven of the eight looked at the dashboard “almost every day” or “every few days.” Of the three district supervisors, two reported using the reasons given for nonvaccination to design specific actions, and all three took action in all VDCs requiring remedial action. All five national supervisors reported that the most helpful feature was having data at the central level in a timely manner, and four supervisors reported that tracking monitors’ activities through GPS coordinates and automatic analytics in real-time on the dashboard also was useful (Figure). All eight district and national supervisors recommended its future use in campaign settings.

Discussion

Although the reported SIA administrative coverage was >100%^††† for the 10 districts included in the pilot study (Table 2), RCM-MP identified a total of 510 (5%) unvaccinated children and 159 (42%) visited areas that needed mop-up vaccination activities. Many unvaccinated children were identified in four districts that reported >100% administrative coverage. The majority of monitors found the mobile technology easy or somewhat easy to use, and about half found it easier to use than paper-based RCM. More than 90% of monitors and all district and national supervisors who completed surveys recommended its future use.

Use of RCM-MP in Nepal resulted in increased reporting timeliness and completeness. The automated calculations and analyses displayed on dashboards eliminated the potential for manual calculation errors, a previous problem with paper-based RCM data (4). Mobile data collection provided information more rapidly to higher administrative levels than did paper-based RCM; most RCM results were available on the same day as monitoring. National supervisors cited the rapid availability of data as the most helpful aspect of this technology for ensuring SIA quality.

Timely reporting resulted in better supervision from the national and subnational levels to the VDCs. Aggregated reasons for nonvaccination and refusal were used by supervisors to tailor vaccination strategies and to take immediate actions, which had not been possible with paper-based RCM. However, supervisors pointed out that there was no RCM mechanism for reporting actions taken in the VDCs with unvaccinated children. Therefore, even if supervisors did take action to improve SIA coverage, the status of VDCs initially marked as needing action did not get updated. Future RCM implementation can address this deficiency by including a reporting system for actions taken in poorly performing areas; this change will enable supervisors to monitor follow-up actions routinely and ensure accountability of vaccination teams.

The findings in this report are subject to at least two limitations. First, the selection of VDCs for RCM was purposeful, and convenience sampling was used within VDCs; therefore, RCM results were not representative of the population and should not be considered as coverage estimates. Second, the impact of RCM-MP on the goal of achieving ≥95% SIA coverage was not quantitatively assessed because of time and resource limitations.

In the RCM-MP pilot, the main challenge to submitting real-time data was difficulty connecting to 3G or Wi-Fi networks. To avoid reliance on 3G coverage, encoding RCM data into text messages is a potential option. However, with expansion of 3G networks of better quality and coverage, it is anticipated that this problem will be overcome. As countries continue to implement campaigns to eliminate measles, rubella, and other vaccine-preventable diseases, the use of mobile phone technology for campaign monitoring might be considered to improve information systems and, ultimately, the quality of campaigns.