Case Study: Caritas Austria
Catalystas: Aviva Stein, Judith Vollebregt, Haley Dillan, Daphne Mulder
Over the summer of 2021, Catalystas took the Caritas Austria team on a learning journey, helping them to solidify the foundation of their Middle East education work and build a strong, forward thinking strategy for the coming years. Within the framework of this learning journey, our team of M&E and education experts also conducted a final impact evaluation of the second iteration of Caritas’ Regional Holistic Education Programme (RHEP II), implemented across Jordanian, Lebanese, and Syrian primary schools with consortium partners Caritas Jordan, Caritas Lebanon, Caritas Syria, and Caritas Switzerland. We worked together with local counterparts in each program location to successfully determine the effectiveness and impacts of the RHEP II program, providing our experts with a strong understanding of Caritas Austria’s strengths, weaknesses, and needs – both as an individual organization, and as part of the Caritas Internationalis consortium. From there, we led Caritas Austria in a deep dive to examine their internal processes and created a roadmap to support the Austria team in designing and implementing new and improved approaches to streamlining decision making, working across diverse cultures, and ultimately making the most positive impacts possible for all of the communities they support.
Caritas Austria is part of the Caritas Internationalis family, with a core mission of identifying and acting to resolve hardship. With international programs in Africa, South Asia, the Balkans, Europe, and the Middle East in addition to domestic programs in Austria, Caritas is an active partner in helping vulnerable and marginalized communities access economic independence, education, food security, and inclusive and safe places to call home. One such program is the Regional Holistic Education Program, which supports young children and their families in Jordan, Lebanon, and Syria in enabling access to primary school, early childhood education (in Lebanon and Jordan), and extra learning support. The RHEP II program operated from 2018-2021 on the four main objectives of enabling children’s access to education, teaching through learner-centered principles, creating protective and inclusive environments, and engaging members of the education community in supporting and caring relationships. As part of an overall learning journey that aimed to develop the next long term organizational strategy for Caritas Austria, Catalystas conducted an impact evaluation of the RHEP II program in all three countries of implementation.
Conducting multi-country evaluations always requires carefully planned methodological approaches and strategies for cross-comparing data and analyzing evidence. However, the summer of 2021 presented a challenging and complex reality in each of the countries in question. Syria, still in the depths of ongoing conflict, presented key challenges when it came to access and clarity. It remains extremely difficult to travel from city to city, with multiple permissions required and checkpoints prevalent, and the country remains under international sanctions and pressure, with security risks still a clear and present danger. In neighboring Jordan, the situation was much calmer, but there were signs of struggle simmering below the surface; the Covid-19 pandemic has taken its toll, and the influx of Syrian refugees – over 600,000 – has put a strain on Jordanian resources. Accommodating these families alongside host communities has proved a difficult task for an overburdened government and civil society. While the economy remains largely stable, the small nation’s proximity to collapsed Syria and imploding Lebanon present a risk to the continued stability and prosperity of the kingdom. On the contrary, Lebanon presented a quickly evolving and challenging environment, with the country’s spiraling economic collapse exacerbated by the resonating impacts of the Beirut Blast, ongoing political catastrophe, the Covid-19 pandemic, and continued struggles surrounding supporting their own influxes of refugees – both old and new. In such a complex and quickly shifting context, Catalystas had to ensure that conflict sensitive and trauma informed approaches were at the core of our evaluation methods, remaining adaptable and aware of the swiftly changing needs and concerns of the communities served by the RHEP II program.
Catalysta Judith poses with school staff while conducting site visits in Jordan.
Against this backdrop of complex environments and multifaceted challenges, Catalystas dove in headfirst to capture an accurate picture of Caritas’ impact among communities on the ground in each location. Taking each specific context into account, we deployed our field missions, working closely with local counterparts to ensure a comprehensive understanding of cultural nuance paired with unbiased outside perspective. In each location, our teams met with the local Caritas sister branches first to gain a better understanding of local operations, program implementation on the ground, and dynamics between the Austria team, local partners, and beneficiary communities. Following our initial introductions, our experts carried out more in depth interviews with key partner staff, alongside interviews and focus groups with teachers and counselors working at the schools served by the program, as well as students and parents. Keeping both child protection guidelines and participatory approaches at the core of our evaluation methodology, we ensured that all participants understood the purpose of the discussions, that school staff or parents were always present where children were involved (albeit far enough away so that the children could speak freely), and that we left space for feedback and open sharing from each group and individual. Our approach looked to measure the impact of the program against the OECD-DAC criteria of effectiveness, impact, efficiency, relevance, coherence, and sustainability, along with the Core Humanitarian Standards of strengthening local capacities and avoiding negative effects, and basing response on participation and feedback & welcoming and addressing complaints. We also looked to examine challenges, lessons learned, and best practices, alongside developing recommendations with an emphasis on preparing future strategic programming for improved interconnectivity of education and community development aspects and inclusion of children with disabilities.
Most of the classrooms our team visited were empty due to Covid-19 restrictions, with students returning to their seats to take their final exams before summer holidays.
Across the locations in each country, we heard similar stories: while families were often facing many challenges, the schools supported in the Caritas program provided a safe and nurturing environment for children, and students were openly enthusiastic about attending classes. At the end of each day in the field, our experts relayed initial findings to our HQ team for real time triangulation and cross comparison, allowing us to revise and adapt our tools to ensure tailored materials for each location and context. Conducting real time validation sessions also enabled our team to keep that Caritas Austria team apprised of our progress, and we held regular check-ins throughout the process to share and receive feedback and make adjustments to our approaches and tools as needed. Regular communication between HQ and the field also allowed our HQ team to analyze and triangulate findings across locations and Caritas teams, incorporating our learnings into building a foundation for a new strategic framework for future education programming in the MENA region.
However, we also faced a number of challenges that required flexibility and adaptive practices in each location. In Lebanon and Jordan, for example, the Covid-19 pandemic meant that schools were being taught virtually, and so special arrangements were made to hold focus groups with students and parents, with health safety precautions taken highly seriously. In Lebanon, traveling between schools was difficult due to the numerous checkpoints, increasing fuel prices, and long distances from Beirut, where our field team and the Caritas Lebanon office were based.
Students joined our team in Lebanon for a focus group.
However, we successfully conducted site visits to five out of six of the Lebanese RHEP schools, with our local consultant conducting a final visit after the official end of our field mission in order to accommodate school and Caritas staff. In Lebanon our experts also witnessed firsthand how swiftly the needs and priorities of local staff and communities were shifting, with emergency humanitarian assistance quickly becoming a higher priority than long term development work, as crisis after crisis continues to sweep the struggling nation. This perspective enabled our team to adjust the strategic roadmap in order to incorporate and accommodate the changing situation and address the ways in which international organizations, with Caritas as a leading player, can support local communities facing new and changing needs and challenges.
Alongside the evaluation process, Catalystas also helped Caritas Austria to zoom out, guiding the team on a learning journey to support the development of a clear strategic path and roadmap to improved internal processes. We conducted deep dive discussions with the full Caritas team as well as individual interviews to gain a deep understanding of strengths, weaknesses, opportunities, and risks as an organization, as well as insights into team structures and dynamics, relations with other Caritas and partner organizations, and overall and individual goals and objectives. We also undertook a mapping of all major players in the education ecosystem in the three countries of implementation, designing a visualized overview of themes and geographies covered in order to identify gaps and saturated areas for Caritas to assess in future program planning and development exercises.
Utilizing co-creative approaches and consistent feedback loop mechanisms with the Caritas Austria team, we designed a custom learning journey to fit the needs of the organization as it transitioned and merged with another office, ensuring that the newly combined team set off on the right strategic path. We also provided the team with a set of tools and reflective questions that they can continue to use and return to as a regular strategy and team building exercise to maintain or adjust their approach and aims as needed.
Although we could not lead these sessions in person due to Covid-19 restrictions, by using interactive remote learning tools such as Miro whiteboards and Prezi video presentation software, Catalystas was able to design interactive, engaging workshop environments that successfully kept discussions of up to 15 people highly engaged and actively contributing to the ideation and strategic development processes of the learning journey. Building on the foundations of continuing communicative, horizontal structures, Catalystas supported Caritas in a guided journey culminating in the delivery of a roadmap and strategic plan basis focused on empowerment of internal staff and external partners, capacity building with a key emphasis on cross cultural competency and knowledge exchange, and information management system development.
Ultimately, despite the challenges we faced – including limited access to program sites in Syria and increasingly difficult and changing contexts in Lebanon – Catalystas delivered a comprehensive final evaluation of the multi-country RHEP II program to the Caritas Austria team, along with a strategy document, mapping of ecosystem actors, and visualized roadmap to strategic planning and implementation. We look forward to keeping up with Caritas Austria and her sister organizations as they continue to provide children and families across the Middle East with essential education and support services, through their highly impactful and dynamic programming.
Scope of Work
Catalystas provided the following the services for Caritas Austria:
- Organizational Capacity Assessment
- Strategic Roadmap Development
- Learning Journey Facilitation and Guidance
- Ecosystem Stakeholder Mapping
- Final Program Evaluation