External evaluation 2024 report brief

Photo by Pawel Czerwinski on Unsplash 

The Strategic Plan of the ITCILO for the period 2022-2025 emphasizes the significance of adopting a quality-centric, data-informed methodology for monitoring and evaluation. It affirms the commitment to advancing excellence in training and learning through ongoing quality enhancement initiatives and external assessments. To this end, the Centre commits to commissioning annual external evaluations focusing on a cluster of activities associated with one of its specialized thematic domains.

In 2021 and 2022, the evaluations concentrated on the Centre's training activities conducted exclusively through online modalities. In 2023, the evaluation expanded its scope to encompass a sample of face-to-face, blended, and online training activities alongside a Diploma programme.

This external evaluation focuses on ITCILO's Workers' Activities Programme (ACTRAV) training activities implemented in 2023. ITCILO ACTRAV offers training activities to Trade Unions, funded by contributions from ACTRAV and support from ITCILO, aiming to support unions in countries where resources for training are limited.

The team delivers a core programme and additional activities, structured around the needs and challenges of Workers and their organizations.

The evaluation aims to assess the relevance, validity, coherence, effectiveness, efficiency, impact, and sustainability of training services provided by the Workers' Activities Programme. It focuses on evaluating different training modalities used to enhance Workers' Organizations and gathers insights into best practices and lessons learned. These activities, which include online, face-to-face, and blended modalities at various locations, are chosen to represent a broad spectrum of methodologies and regional diversity.

The desk review for the ITCILO ACTRAV Training programme in 2023 involved a detailed examination of twenty selected courses from a total of forty. The analysed documents included course materials, survey results from participants, timetables, information notes, course descriptions, resource materials, and PowerPoint presentations.

Institutional documents from both ITCILO and ILO were analyzed, providing a comprehensive overview of the training programme's documentation and helping to shape the subsequent phases of data collection.

Data collection for the evaluation was multifaceted, incorporating both qualitative and quantitative methods. An online survey was sent out via the ITCILO's CRM tool to 1152 former ACTRAV participants and deployed across four languages with a 29 per cent response rate. Interviews and focus groups were conducted, involving ITCILO management, ACTRAV team members, training participants, and institutional beneficiaries, to gain in-depth insights into the trainings' impact and effectiveness.

The data analysis stage leveraged data triangulation, combining findings from surveys, interviews, focus groups, and document reviews to enhance the reliability and validity of the conclusions.

The findings highlight that the ITCILO is strategically aligned with the ILO's objectives, notably the 2019 Centenary Declaration and the 2021 Call to Action for post-COVID-19 recovery, focusing on lifelong learning, gender equality, and social protection.

• The ITCILO delivered 496 training activities in 2023,  including 40 by ACTRAV, making up 8% of its total training offerings.

• The majority of ITCILO training activities in 2023 was distance learning and blended training. These modalities were the most engaging, attracting 68.2% of the participants. The shift towards online platforms reflects a broader approach to making training accessible globally, particularly in regional or global initiatives. ACTRAV activities show a different distribution with distance training being the most prevalent at 50%, Blended-F and In the Field trainings each represent 20% of the total. Blended-C and Turin Centre trainings are the least common for ACTRAV, each constituting 5% of the total.

• For ACTRAV's Core programme, no less than 72 per cent of the training activities are distance learning. This proportion is significantly higher than the ratios for all ITCILO training activities combined and the ACTRAV global training portfolio. Blended learning activities with a face-to-face component in countries/regions are more prevalent than in the global ITCILO training portfolio but are comparable to the global ACTRAV training portfolio. This illustrates that ACTRAV has successfully transitioned to distance learning, thereby operating in alignment with its strategic and operational plan.

• Survey results show high relevance of the training to participants' needs, with 98% reporting applicability of learned skills and 93% foreseeing institutional benefits.

• Geographically, participation in ACTRAV training activities was highest from Africa and the Americas with notable contributions from specific countries like Argentina and Uzbekistan, often due to targeted training activities.

• The demographic data shows a slight female majority among participants, predominantly in the 25-54 age range, highlighting broad and inclusive engagement across different demographics.

• The validity of the training activities is highly valued by the participants. Over 95% of the respondents were satisfied with how the courses were structured.

• Participants felt well-supported, with approximately 89% reporting that they received adequate support when needed and were satisfied with the learning resources, which they found relevant and of high quality.

• Despite some participants experiencing issues with internet connectivity, the overall technical support was viewed positively.

• The e-Campus online learning platform was noted for its ease of access and navigation. The use of various educational technologies like discussion forums, video conferencing, and content was deemed appropriate by most learners, and the delivery modes of the courses were effective in meeting the participants' schedules and learning preferences.

• The courses were flexible, provided ample opportunities for participation, and were conducive to understanding and applying content effectively.

• In terms of the Community of Inquiry framework, there were improvements in teaching presence, social presence, and cognitive presence, with no significant differences noted between face-to-face and distance learning modalities, underscoring the effectiveness of both.

• The findings highlighted that the training activities exhibit consistent effectiveness across various modalities, including both face-to-face and online formats, despite facing logistical challenges such as accommodating different time zones and ensuring stable internet access.

• Survey data revealed a high approval rating for these activities, with a significant majority of respondents acknowledging improved competencies and performance.

• The evaluation revealed that the unit faces challenges in inter-unit collaboration within ITCILO due to differences in operational approaches. Despite these challenges, instances of successful individual cooperation suggest potential for improved institutional collaboration, highlighting the need for enhanced inter-unit cooperation to optimize training effectiveness and resource utilization.

• ACTRAV Unit's secure funding model allows it to plan and execute its core training activities well in advance, facilitating more structured and predictable training schedules.

• Despite the structural and financial readiness for distance learning activities, their utilization remains sometimes low, with participation rates not meeting potential. The effectiveness of these activities, in terms of knowledge and skills acquisition, matches that of face-to-face training, suggesting that learning outcomes are comparable across different delivery modes.

• There is a preference among trainees for face-to-face interactions, which are highly valued for the networking and cultural exchanges they facilitate. This underscores a significant aspect of learning that extends beyond mere knowledge acquisition, emphasizing the human need for interaction. The evaluation suggests a potential efficiency improvement by separating training from networking objectives, proposing that training could be predominantly conducted online with networking reserved for specific activities. This could help manage costs more effectively while still meeting the diverse needs of participants.

• The current model's scalability is limited by staff workload and the necessity for high-quality internet access among participants, indicating that strategic adjustments are needed to expand the reach and impact without compromising quality.

• The evaluation revealed that the current focus of the Workers' Activities Programme is on individual capacity development activities (i.e. training) while the ILO capacity development approach followed by the Centre is more holistic, with a triple emphasis on individual capacity development, institutional capacity development and system level capacity development.

• In cases where the Workers' Activities Programme has carried out institutional capacity development activities, it appears that it is yet to fully internalize the related Centre’s monitoring and evaluation processes and tools.

• At the level of individual capacity development, the emphasis of the evaluation activities of the Centre is on the verification of out-takes and outcomes, where attribution confidence is high. The validation of long-term impact of the Centre’s activities on the world of work is taking place under the umbrella of higher level ILO impact evaluations. It is important to  accentuate the link between ITCILO inputs and ILO impact along a multi-year results chain.

• 53 per cent of the respondents of the survey submitted concrete examples of the way in which the training activity has been of practical use for them which is another demonstration of the beneficial effects ITCILO training activities succeed in producing. 

• Based on five case studies, significant applications of acquired competencies by trainees from various training activities, emphasizing practical outcomes across different areas could be identified. The key outcomes include enhanced integration of SDGs in organizational activities and improved communication strategies, which have facilitated greater engagement and inclusion within workplace practices, notably increasing trade union membership.

• Early indications suggest that trainees are actively applying their newfound skills in various professional settings.

• The survey data suggest high loyalty of the participants to the organization. A substantial 92.43% of the survey respondents expressed a definite interest in enrolling in additional courses, signifying strong ongoing engagement with the training institution.

• The survey facilitated the calculation of the Net Promoter Score (NPS), a key metric for gauging customer loyalty and satisfaction. With the majority of respondents rating their likelihood to recommend the training highly, the NPS reached a commendable 54.89 which reflects a dominant percentage of promoters (64.04% of respondents), who are enthusiastic about the training and likely to advocate for it.

The evaluation of the training activities of ITCILO's Workers' Activities Programme shows a compelling alignment with the strategic objectives of the ILO, particularly highlighted by the digital transformation and the adoption of hybrid training methods. The focus on gender equality (inclusivity), lifelong learning, ILS, SDT, and social protection through well received training methods has increased accessibility and engagement significantly.

Notably, the feedback from participants emphasizes the applicability of skills learned to professional settings, with high rates of anticipated institutional benefits.

Furthermore, the adaptation to online platforms has expanded global reach and inclusivity, showcasing the programme’s effectiveness and alignment with modern educational needs. By continuing to enhance and expand distance learning opportunities and implementing strategic recommendations for greater inter-unit collaboration and a comprehensive evaluation framework, the ITCILO can further capitalize on its strengths and address the evolving demands of its global audience.