To improve the standard of maternal healthcare, we need to be able to measure the changes we aim to create.
Jacaranda believes that measuring impact is critical to success. To integrate monitoring and evaluation (M&E) into their model, they have partnered with researchers from the Harvard School of Public Health to form an evaluation team. Dr. Jessica Cohen and Dr. Maggie McConnell are economists who specialize in impact evaluation and have both have done research on health innovations in East Africa. I’ve been working as part of this team in the last few months to implement the first step in the evaluation process: establishing a baseline.
Baseline data represent the “before” in a before-and-after evaluation. The baseline provides a valid comparison group when we do our analysis, and contributes to our ability to make reliable conclusions about whether Jacaranda is having its intended impact. Without a baseline, it’s almost impossible to measure effectiveness, document progress and set goals for the future.
We have developed and tested a questionnaire that we plan to administer to over 2 thousand expectant moms and women who have just delivered. We ask about their experiences with antenatal care, birth planning during pregnancy, delivery, and postnatal care. We will repeat the same survey after Jacaranda’s maternity clinics have been up and running in the neighborhoods.
In planning our survey, we faced two big challenges:
- How can we achieve adequate coverage of the neighborhoods?
We need a representative sample of Jacaranda’s prospective client base, which is in a large and densely populated area of the city.
- How do we collect high-quality data quickly?
We need complete and accurate response data to make conclusions, and we need to collect it quickly enough to keep findings relevant.
These objectives are not always easy to achieve. For example, we had no roster of expectant mothers in the neighborhoods we needed to survey. Furthermore, we are largely unacquainted with these communities.
To address these challenges, we have invited Community Health Workers, or CHWs, to join the survey team. CHWs know their communities intimately. They are smart, hardworking and compassionate and have a nuanced understanding of the community’s needs and dynamics.
We wanted to pair CHWs with surveyors to implement a chain referral system – each woman interviewed is asked to direct us to other women that fit the criteria for the survey. But we didn’t know if CHWs would be willing to participate, as it would require an additional time commitment and we didn’t want to interfere with their work.
We met initially with four CHWs in different neighborhoods to discuss our ideas. Based on their positive feedback, we arranged three more information sessions, each with 10 to 20 CHWs. We now have over 40 CHWs working with us on the survey.
Once we had the CHWs on-board, we pilot tested the sampling strategy in two neighborhoods. I first shadowed Joyce* through the labyrinthine high-rises in her neighborhood where we visited 12 women in only 90 minutes. During our second test, I followed Stephen* through a quieter peri-urban neighborhood, where visits were punctuated with stops for goat-crossing.
In one test visit, a husband insisted on answering before his wife, which encouraged us to think critically about how to handle the potential effect of a spouse’s presence on a woman’s ability to be forthcoming during a real interview. When a small crowd gathered curiously near another woman who had been approached by our team just outside of her home in a place that had seemed out of the way at first, we were reminded of the importance of enforcing strict privacy for our interviews to ensure confidentiality.
Discussing the results of these pilots with Joyce, Stephen, and the other CHWs on the team has already enhanced the quality of our evaluation through these lessons learned. In lending their knowledge, skills, and ties with the community to allow us to carry out our project and help us identify respondents, Community Health Workers have helped us take important steps toward solving our initial challenges with evaluation.
While we have taken additional steps in the meantime to address our challenges – such as setting up a system to use mobile phone technology for data collection, which could eliminate the need to print and handle over 60,000 pages of paper for our analysis – ultimately, we believe that the relationships we build with Community Health Workers and others who are invested in their community will lead to the success of our evaluation, appropriately designed interventions, and in the end, better maternal health outcomes.
The baseline survey is an essential part of being able to make a “before and after” comparison of Jacaranda’s impact. Engaging in this kind of evaluation reflects that Jacaranda is serious about being a learning organization, which means experimenting periodically and sharing lessons learned with the broader maternal health community.
*Names have been changed to protect privacy.