Innovating the “Outcome Harvesting” Evaluation Method for Rural and Indigenous Contexts

Case Study

Paula Kantor, Alejandra González e Ingrid Son
February 11, 2025

The importance of starting by recognizing the intervention context

Between July and October 2024, Catalystas conducted the Final Evaluation of the project “Innovating and Scaling Proven Model to Improve Maya Women’s Access to Justice and End Violence Against Women in Rural Guatemala.” This three-year project was implemented in the municipalities of San Martín Jilotepeque and San José Poaquil, located in the department of Chimaltenango, Guatemala.

The project was spearheaded by the Women’s Justice Initiative (WJI), a nonprofit organization with over 10 years of experience working with Indigenous and rural women and girls in Guatemala. It received funding from the United Nations Trust Fund to End Violence Against Women (UNTF), which contributed $498,754.

During its implementation, the project delivered legal literacy workshops to 1,747 Maya women across 24 rural communities, 96% of whom improved their legal knowledge, and 91% reported feeling entitled to a life free from violence. Additionally, the project trained 114 public service providers who handled 427 violence cases. WJI provided 1,549 legal services to Indigenous Maya women survivors of violence and trained 44 Maya women as Community Advocates, who in turn offered legal support to other women. WJI also trained 461 community leaders and established 20 community referral pathways to address violence cases. Furthermore, the project produced 10 radio episodes to raise awareness about Violence Against Women and Girls (VAWG) and improve family communication.

The Catalystas team, composed of Paula Kantor, Alejandra González, and Ingrid Son, conducted a comprehensive evaluation of the project based on the criteria of effectiveness, relevance, efficiency, sustainability, impact, gender equality, and human rights. The evaluation identified key lessons, promising best practices, and recommendations for future programs of this nature. The evaluation methodology was grounded in Popular Education and centered on two core approaches: a participatory approach, which aimed to ensure the active involvement of the individuals consulted, and a learning-focused approach, which sought to make the process formative, reflective, and capacity-building.

The team made significant efforts to deeply understand the context in which the project operated. This was achieved by assembling a diverse team that included personnel from the intervention area and culture, a local Maya Kaqchikel translator and interpreter for data collection, conducting field visits, and carrying out interviews and focus groups in a representative sample of communities and institutions involved. Additionally, a thorough review of relevant literature was conducted.

In the two municipalities where the project operated, San Martín Jilotepeque (SMJ) and San José Poaquil (SJP), the population is predominantly Indigenous Maya Kaqchikel, facing high levels of poverty, low educational attainment, geographic isolation, and limited access to basic services such as healthcare, education, and justice. Although both municipalities share a history of violence, oppression, and poverty rooted in their Indigenous identity, their community dynamics differ. SMJ has historically been an important regional trade center since pre-Hispanic times, with its population undergoing significant historical processes of struggle and defense of their rights, resulting in some degree of empowerment and leadership. In contrast, SJP retains more deeply rooted elements of Maya Kaqchikel culture and language, with a more dispersed population, which contributes to isolation and increased vulnerability of the social fabric.

The Indigenous Kaqchikel women from the rural communities involved in the project face not only poverty and low educational attainment but also traditional gender roles and stereotypes. These include high rates of domestic violence with low reporting rates, social isolation, early marriage and pregnancies, and the exclusive responsibility for domestic and caregiving tasks. Additionally, like the rest of their communities, their primary source of income is subsistence farming, with daily life revolving around agriculture and the Milpa system, as discussed below. Among the common cultural elements that persist across Maya communities, despite their differing realities, are the prevalence of oral communication and tradition over written forms, a worldview shaped by cyclical, systemic perspectives in harmony with nature, and the centrality of the traditional farming system to their daily lives.

Innovating the Outcome Harvesting Method Using La Milpa

To conduct the evaluation through a feminist, intersectional, and culturally appropriate lens while ensuring the meaningful participation of the Indigenous women targeted by this project, the Catalystas team adapted the Outcome Harvesting Method by incorporating the concept of La Milpa as an analogy to deepen the learning processes and experiences of the women involved in the project.

According to the OECD (2022), the Outcome Harvesting Method examines the changes, outcomes, and impacts within a community over a specific period and then evaluates whether and how the project in question contributed to those changes, reversing the traditional evaluation sequence. As described by the Government of Mexico, La Milpa is a traditional farming system originating in Mesoamerica, characterized by the simultaneous and complementary cultivation of maize, beans, squash, and occasionally other plants such as chili or amaranth. Beyond being an agricultural method, the milpa is a holistic and sustainable model that integrates farming practices, ancestral knowledge, and an Indigenous worldview where the relationship with the land is central. In addition to serving as a subsistence strategy, the milpa symbolizes a space of cultural, spiritual, and social connection for the communities that practice it.

During the evaluation, the consulting team facilitated six in-person focus groups: two with Community Advocates and four with women from the legal literacy program, reaching a total of 78 women from 18 different communities. Using dynamic, participatory group techniques, the analogy of the Milpa growth process was introduced, allowing the focus group participants to identify which phase of the Milpa’s growth they were in concerning the knowledge and skills they had gained during the project.

The meeting began with a presentation and the “basket” activity, a familiar element for the Maya women, in which each participant introduced themselves with their name, community, and something they brought to share with the group. By using this everyday item, the focus group started creating a trusting environment where participants could contribute what they had brought to the meeting and see how each person’s contribution added to those of the others.

Next, the purpose of the group gathering was explained: just as when harvesting, the fruit is gathered, here the goal was to assess how well the project achieved its objectives—specifically, what fruits were reaped from the planting of knowledge that WJI had sown through its training sessions. The activity involved presenting three distinct stages of the Milpa, with questions linked to each stage. Each question offered closed response options (Yes/No, Much/Little/None, Easy/Complicated/Very Complicated), and the women selected their answers by either moving around the space or raising their hands. Afterward, they were asked to delve deeper into why they felt they were in that particular category. As shown in the photos, flip charts, colorful and visual images, and graphic materials were consistently used throughout the process.

Stage	Description	Questions	Answers
Preparing the soil and planting the seed	To plant corn, what is the first thing we need? The seed. But it’s not just thrown in the field, what needs to be done first? Prepare the soil, remove the weeds, fertilize, but it’s not done at any time; everything follows a process.	What knowledge did I have about my rights and violence against women before the WJI workshops?	Much \| Little \| None
		Before the project, did you know where to go for help in cases of violence?	Yes \| No
		What obstacles did you encounter when seeking support in cases of violence?	Images of beautiful paths, with hills and full of stones
		Why do you describe your experience that way?	Open Response
Germination of the seed (it’s planted and germinates)	We’ve prepared the soil and planted the seed, but for it to germinate, the area around the crop must be cleaned and the weeds removed so that the plant doesn’t drown. This is called “chenoj” in Kaqchikel. Here we recognize obstacles to further growth in the knowledge of rights and the dignification of our lives.	Did the WJI facilitators explain what the project was about (its objectives)?	🙅🏽‍♀️🤨😐😊🥰
		Did the WJI facilitators explain how the workshops would be conducted?	🙅🏽‍♀️🤨😐😊🥰
		Did you like the way the workshops were delivered? Did you like the materials?	🙅🏽‍♀️🤨😐😊🥰
		Do you feel that your opinions were considered by WJI?	🙅🏽‍♀️🤨😐😊🥰
		How was your learning process?	Easy \| Regular \| Difficult
		Did your family listen to the WJI radio programs? Has there been any change in your family after listening to the programs?	Yes \| No \| Don’t know
		Other trigger questions: Did WJI do anything to help you participate in the workshops? What made your participation more difficult? What did you like most about the workshops? What didn’t you like? Why?	Open response
Harvesting the crops	Harvesting allows us to identify the achievements, what we did well, and what we did wrong to improve the next harvest. Here, another task is performed, called "Poloj" in Kaqchikel, which involves strengthening the roots of the Milpa by building small mounds of soil around the root so that it stays firm and doesn’t fall in storms or winds, as it has gained height and the fruit is now heavy. This was used to explain that it is necessary to continue working to strengthen the learning.	After participating in the workshops, how is my Milpa of knowledge about my rights and violence against women now?	Images of Milpas in 3 stages: germination, flowering, in production, and harvesting.
		In germination: What do you think is needed to continue progressing in growth?	Open Response
		In flowering: What is missing for the Milpa to begin bearing fruit? What could be improved?	Open Response
		In production and harvest: What is the fruit of your harvest? How do you share your harvest with your family and community? What changes have arisen in you and your family as a result of your participation in the project?	Open Response
		Other questions: Would you be willing to continue your training process with WJI? Would you recommend the workshops to other women?	Open Response

This methodology was possible to apply because the implementing organization, WJI, already follows the same principles of participatory work, learning, and popular education in its workshops and monitoring and evaluation mechanisms with the participants.

Additionally, it was essential:

Coordinating with the local organization’s field technical team to manage data collection activities.
Respecting the cultural norms and language of each community.
Studying the cultural and historical context of each location beforehand, as allows for a better approach, generating trust and empathy.
Ensuring a safe and suitable space (with privacy and free from external noise) where the women can participate calmly and be heard without criticism or interruptions.
Considering the women’s caregiving and fieldwork responsibilities when setting the date and time for meetings. Providing child care services by a trusted person assigned for this purpose.
Facilitating from the perspectives of learning, participation, confidentiality, and respecting ethical evaluation standards.
Preparing teaching and administrative materials in advance for use during the focus group.
Obtaining informed consent from the focus group participants to safeguard their safety and ensure they don’t feel intimidated by photography and video recordings, explaining how the collected information will be used.

Key Results Obtained

The implementation of the Results Harvesting method, adapted to the analogy of the milpa, generated meaningful learning and tangible outcomes:

Ease of Identification and Participation: The women easily identified each stage of the process due to their daily relationship with the Milpa, an element deeply rooted in their everyday lives. The analogy helped connect their personal experiences with their learning and knowledge. Additionally, it allowed the inclusion of almost all evaluation criteria and questions within a limited timeframe, respecting the women’s time constraints due to their household and farming responsibilities.
Creation of a Trusting Environment: By using concepts familiar to the women, they were able to view the evaluation as an opportunity for improvement rather than criticism, overcoming the fear of speaking publicly. Additionally, the dynamic of moving around the space and raising hands encouraged active and spontaneous participation: one woman would begin a reflection, and others would follow suit, without the need for external interventions.
Identification of Internal and External Factors: Women were able to recognize internal needs for growth, such as additional skills and knowledge, represented by the milpa strengthened with practices like “chenoj” (cleaning after planting) and “poloj” (reinforcing the roots). They also identified external factors, such as the weather and soil quality, highlighting the importance of elements beyond their control to support their development.
Collection of Tangible Results: By using the milpa analogy and its stages of production and harvest, the women were able to identify key outcomes (fruits) generated by the project. Additionally, they were able to assess what other aspects are needed to continue strengthening those results.
Obtaining Quantitative Data: Although the main approach did not initially aim to collect quantitative data, the methodology allowed for the unification of responses and the gathering of key numerical information, regardless of the women’s varying education levels. This information was then cross-checked with the project’s quantitative results and analyzed in the final evaluation report.
Development of Recommendations: Thanks to the trusting environment and participatory dynamic, the women shared clear recommendations on how to improve future processes based on their own knowledge and experiences, making the project evaluation much more effective and relevant.
Adaptability and Scalability: The methodology, designed with visual and dynamic materials, proved to be adaptable to similar rural contexts and transferable to other countries, taking into account local cultural and agricultural particularities. The tool has the potential to become a reference for evaluations in rural settings, offering a replicable methodology that is sensitive to local realities.

You can access the official publication of the Final Evaluation at this link.