Amy Tsui of PMA2020 and Ian Salas of TCI Discuss the Power of Cross-Project Collaboration on M&E

With two-thirds of the world’s population expected to live in urban environments by 2050, improving access to family planning among urban residents, especially the urban poor, is a high priority. The Gates Institute is addressing this need via a collaboration between The Challenge Initiative (TCI), our urban reproductive health platform, and PMA Agile, a project that expands on innovations that work in Performance Monitoring and Accountability 2020 (PMA2020).

Launched in 2016, TCI is built off evidence from the Urban Reproductive Health Initiative (URHI), and is focused on rapidly scaling-up and sustaining proven reproductive health solutions for underserved urban poor communities by imparting key knowledge and skills to local governments through learning-by-doing and cultivating a culture of continuous improvement.

PMA Agile, which launched in 2017, is a continuous data monitoring and evaluation system, collecting data at quarterly intervals on overall health service delivery with an emphasis on family planning and contraceptive service provision. Agile is active in urban areas in multiple countries and uses the PMA2020 platform to capture information on health facility performance and client utilization. Clients are followed up at four months to assess continued use.  

In certain geographies, PMA Agile data informs TCI’s programmatic activities, allowing TCI’s monitoring and evaluation (M&E) arm to better assess the efficacy of TCI’s programmatic work and to plan more effective intervention practices and approaches. J.M. Ian Salas, Senior Technical Advisor at TCI, and Amy Tsui, Senior Technical Advisor at PMA2020 and co-Principal Investigator for PMA Agile, recently shared details about this synergistic collaboration. Jessica Mirano, Program Officer, M&E, at TCI, and Titilope Akinlose, Program Manager, PMA Agile, at PMA2020, also contributed.


Q: Given that TCI supports more than 83 cities/states and 153 local governments (as of December 2018) across 4 regional hubs with various systems and corresponding monitoring challenges, what are TCI’s main approaches to monitoring and evaluation?

IAN: The main purpose of TCI’s M&E is to gather evidence that guides programmatic decision-making and facilitates quick feedback loops. With that in mind, TCI utilizes data from various sources to pull together a well-rounded picture of our activities and our contribution to improving contraceptive uptake. We compile data from project records and service statistics (HMIS) on a monthly basis, and we collect data from health facilities and clients through quarterly or semi-annual surveys fielded by PMA Agile and other partners. We also use the Most Significant Change (MSC) technique to systematically capture stories on qualitative changes that are taking place in various domains (provider knowledge and attitudes, institutional practices, local government commitments and health systems, and FP user access and quality of care).


Q: With HMIS as one of your main data sources, how does TCI address concerns regarding the differences in definitions, reporting rates, data quality, and other potential issues with HMIS?

IAN: To extract meaningful signals from “noisy” HMIS data, we focus our attention on monitoring trends in the number of family planning (FP) clients. To estimate these trends, TCI developed a methodology that takes into account the duration of protection each FP method provides and any pattern of seasonal use. We worked with our implementing partners to uniformly and consistently apply this methodology to data from all of our sites in different countries. With this methodology, we are able to easily detect whether trends are stable, improving or worsening in TCI sites, and we have also started comparing these trends to the general trend exhibited in their respective countries. We cross-validate this HMIS trend analysis with insights from the field and signals coming in from other data sources, like PMA Agile surveys. The early signals we are receiving point to positive trajectories in most TCI sites, especially in places where we’ve been implementing for a longer time. Longer exposure equals more impact.


Q: How does TCI synthesize data from other data sources with the products of PMA Agile?

IAN: TCI compares the signals we receive from PMA Agile’s facility surveys with the signals we compile from HMIS data on FP client volume. We check if both data sources tell a consistent story, in which case our confidence increases that we are observing a real change in outcomes. For example, PMA Agile documented that the urban primary health centers (UPHCs) that TCI has been supporting in the city of Firozabad in Uttar Pradesh, India, were fully stocked with IUDs in three survey rounds that spanned 2018. In contrast, PMA Agile recorded stock-outs of IUDs in private medical colleges and hospitals in Firozabad, and in UPHCs located in Shikohabad and Tundla, two non-TCI cities next to Firozabad. Over that same time period, we saw a positive trend in the number of FP clients in Firozabad from HMIS data. This tells us that TCI’s strategy in India of ensuring availability of FP commodities and providers during fixed days of the week is gaining traction and producing tangible results.


Q: How does PMA Agile support TCI in its monitoring and evaluation approach? 

AMY: PMA Agile provides another source of performance data for TCI to examine. Agile indicator data is now accessible on dashboards — see, for example, Kenya data. Because youth may ask their partners to procure contraceptives at non-clinical sites, exhibiting what we call “hidden behaviors,” PMA Agile conducted a respondent-driven sampling study of some 2,000 unmarried females and males age 15–24 in Abidjan to inform TCI’s intervention designs there.


Q: What are some of the learnings from the implementation of PMA Agile, specifically in terms of covering various sites in a frequent manner?

AMY: It helps to plan early! It also helps to initiate cross-project collaboration early! In hindsight, early meetings between Agile country and TCI hub teams on planning the data collection would have been particularly helpful.


Q: How about utilization of data for decision-making?

AMY: Thus far Agile has disseminated findings directly to TCI and DKT International but will be broadening to local governments in the future. As noted earlier, Agile dashboards of performance data are being made publicly accessible and will gain in value as trends become visible with more rounds of data collected.


Q: Has the collaboration with PMA Agile influenced any of the other approaches TCI has towards monitoring and evaluation? And, conversely, has the collaboration with TCI influenced some components of the new Performance Monitoring for Action?

IAN: TCI’s recent supplemental grant provides funding for an intensive focus on improving contraceptive uptake among girls and women 15–24 years old, and so on the M&E side we are working with our partners to identify data sources that disaggregate information for this important age group. We are seeing only a few sources that provide this information, so in many cases we need to collect the data ourselves. TCI is grateful that PMA Agile recently tested in Abidjan, Cote d’Ivoire, an innovative approach to learning about the knowledge, behaviors and practices of both female and male unmarried adolescents and youth through a network-based survey called respondent-driven sampling (RDS). PMA Agile and TCI are now partnering to conduct another RDS in Nairobi, Kenya. TCI is also exploring other approaches that can help us better capture the age, marital status and parity of FP clients in TCI sites.

AMY: Collaborating with TCI has influenced some components of PMA2020’s future plans. The follow-on project, Performance Monitoring for Action, will be initiating client exit surveys in conjunction with the health facility surveys and considering telephone follow-up interviews with clients. In one urban Kenyan county, 1439 female clients were interviewed; 974 (67.7%) reached by phone were using contraception. Of the 974, 740 (76%) were still using the method obtained four months earlier, mostly implant or injectable. Many who stopped did so to have a planned pregnancy. Such findings can be useful to promoting contraceptive service access.


Q: Moving forward, how do TCI and PMA envision continuing to work together?

AMY: We plan to continue data collection to support TCI Baltimore/TCI country teams’ project activities. We hope to enable them to make better use of Agile data and directly access Agile datasets, and assist the teams with country-specific analysis. Also, as the Agile grant completes itself, we will be finding ways to incorporate its learnings, tools and established partnerships into the M&E practices of TCI and other partners.

IAN: TCI is engaging with PMA Agile on approaches we can take to further strengthen the dissemination of findings to our implementing partners and local government stakeholders. The results dashboard that PMA Agile is developing will help TCI visualize changes in important facility and client indicators over time. In turn, this will facilitate TCI’s ability to synthesize these signals with other quantitative and qualitative data we are receiving, improving our capability to undertake course-corrective actions as needed.


This Q&A was originally published in Positive Disruptions, the bimonthly email newsletter of the Gates Institute. To sign up to receive Positive Disruptions, fill out this contact form

Thursday, March 7, 2019 - 08:00