External Evaluation of Training 2023

External evaluation carried out by:

Download full report

This external evaluation aligns with the ITCILO's vision to be the global centre of excellence for ILO constituents to source capacity development services on social justice for decent work, and with its mission to provide people across the world of work directly and via ILO constituents with access to digitally enhanced capacity development services to successfully manage their Future Work transitions.

The ITCILO continues to expand its base of beneficiaries and attracts more participants for its training activities each year, with total participant numbers in 2022 nearing 100.000. Most of this increase is due to the strong growth in the number of distance learners, which comprised around 92.000 participants in 2022.

The evaluation criteria were based on the OECD DAC evaluation principles, encompassing relevance of the sampled activities to beneficiaries' needs, their coherence, efficiency, effectiveness, impact and sustainability. It also provided a comparative analysis of the different modalities of training offered by the ITCILO, namely face-to-face, blended and online.

The methodology for this evaluation was carefully designed to gather a wide range of data and insights to address specific evaluation questions and comprehensively assess the organization's performance.

The methodology included quantitative and qualitative evaluation methods to provide conclusions and recommendations from the findings substantiated with desk research, statistical data and 5 case studies documenting good practice. 626 responses were collected from a participant’s survey, and in-depth interviews were conducted with 56 ITCILO staff members, 6 institutional partners, and focus group discussions were held with 9 former participants.

the main findings of the external evaluation were presented in accordance with the OECD DAC evaluation criteria.

3.1 Relevance and Outreach

• Through the survey, participants were asked whether they agree that the course was relevant to their needs. 96.91% of responding participants strongly agreed or agreed, while none strongly disagreed and only 0.39 % disagreed. These results were equally based on input from people who work for different types of organizations.

• These results demonstrate a clear pattern in line with previous years’ evaluations. This clearly signals that the ITCILO is on the right track when it comes to meeting the needs of its training activity participants.

• All interviewees expressed general satisfaction with the relevance of the ITCILO’s training activities for their own operations, organizational activities, policy development or capacity building. Interviewees acknowledged ITCILO’s internal (expertise) and external capacities (partnerships) in the domain that is relevant to them.

• Through a comprehensive review of the training activities, it was evident that the vast majority of the training activities align with the gender diversity principles and incorporate gender-sensitive elements. Overall, the evaluation of gender, International Labour Standards, and Social Dialogue and Tripartism markers across the activities has revealed a nuanced landscape of implementation.

• There is a noticeable emphasis on alignment with the ITCILO's mandate. This aspect holds particular significance for many of the interviewed staff members, who indicated that they viewed ensuring that activities are closely aligned with the ILO's mission as a critical element underpinning all their endeavours.

• The ITCILO demonstrated particular success in terms of outreach by providing a number of free online courses which attract a high number of participants. This is a commendable initiative and, judging from the participant numbers in these courses as well as the feedback received during the focus groups, widely welcomed.

• The expansion of distance training activities puts the Centre well on track to greatly surpass the ITCILO Strategic Plan 2022-2025 targets. Simultaneously, face-to-face activities have seen a resurgence in enrolment.

• An encouraging finding based on the participants' survey is that, although female participants do have a slightly lower assessment of course applicability and course relevance, these differences are not significant and the findings clearly demonstrate an impressively high level of course applicability and relevance for both female and male participants. 

• The ITCILO has once again showed a substantial outreach to the African continent. 

• The survey responses showed that the ITCILO manages to reach different income groups relatively equally and that there is no significant barrier detected for those who self-identify as lower or much lower than the average when it comes to household income.

• 14.5% of respondents self-identified as a member of an under-represented or vulnerable group (including categories as diverse as ethnic minority, disability, refugee background, etc.)This could be seen as an indication that the Centre manages to successfully reach participants in different conditions, relative to their societal context.

3.2 Coherence

• Demand-based training holds immense importance for the ITCILO and involves close collaboration with the ILO to align with their specific goals in various countries and regions. 

• The effort made by the Centre to develop training activities aligned closely with the ever-changing needs of its constituents, partner institutions, and clients, is evident. The focus on staying updated with new trends and topics showcases a proactive approach to ensure the relevance and effectiveness of the training activities.

• the ITCILO maintains regular and direct communication with clients and constituents through various channels. This ongoing interaction not only fosters a deeper understanding of their needs but also allows for continuous feedback and validation of assumptions. 

• ITCILO's emphasis on seeking partnerships with specialists and relevant entities further bolsters the credibility of its training offerings. This collaborative approach ensures that the training activities are designed to align precisely with its mandate and the evolving landscape of the field. 

3.3 Validity of Training Design

• The Turin Learning Approach represents a best practice example in enhancing the learning experience and ensuring the effectiveness of capacity development. This approach, rooted in the three pillars of the ITCILO's learning strategy (relevance, quality, and impact) emphasises customer-oriented activities, learner-centred methods, diverse learning environments, and flexibility in design.

• Each technical programme within the organization employs distinct approaches to course design. These methodologies are tailored to the specific requirements and objectives of each programme, reflecting the unique nature of the subject matter, beneficiaries' needs and the learning outcomes sought. These variations in design strategies ensure that training activities are optimised for their respective domains, accommodating diverse learning needs and effectively addressing the specific challenges and opportunities presented by each technical area. 

• The course evaluation platform is acknowledged to have some shortcomings, but it remains a valuable tool for improving training activities and gaining insights into participants' experiences and needs. The ITCILO's openness to sharing end-of-activity participants' survey results with donors reflects a commitment to transparency and accountability. 

• Active utilization of participants feedback to revise and enhance courses for subsequent iterations allows for continuous improvement and ensures that courses remain relevant and responsive to participants' needs. The current effort invested in revising the database of survey questions is commendable. The working group's ongoing efforts to develop a comprehensive database that includes a variety of question types demonstrate a commitment to gathering diverse and valuable feedback from participants. 

• The practice of sharing quality assurance results with consultants and individuals involved in course delivery is commendable. This approach fosters a culture of learning and improvement, as it enables the course organisers to identify areas for enhancement and make informed decisions to enhance future course delivery. 

• The dedication of course organisers to collecting and using feedback reflects their commitment to delivering high-quality and relevant training programs. It highlights their receptiveness to input from participants and stakeholders and their continuous efforts to refine and optimise their offerings.

• To assess participants' learning experience in ITCILO's training activities, an instrument was utilised to measure three dimensions of the Community of Inquiry (Col) framework: teaching presence, social presence, and cognitive presence. 

• Using a 5-point scale, the survey measured the three dimensions with their respective sub-dimensions. The results indicate that ITCILO's course designers and facilitators successfully delivered highly engaging, interactive, and supportive training activities, providing opportunities for rich and profound learning experiences. The average ratings were 4.38 for teaching presence, 4.13 for social presence, and 4.29 for cognitive presence.

• The analysis based on participants' feedback was complemented by an analysis of course materials conducted directly by the reviewers to include more perspectives and to broaden the evidence base. The conclusion was that ITCILO training activities scored fairly high on almost all indicators - even the lowest rated indicators still have 70-75% training activities clearly meeting the criteria.

3.4 Effectiveness

• ITCILO's training activities generally manage to achieve the intended effects. The collected evidence clearly points to participants achieving the intended learning objectives, as per a (self)reported increase in their competence levels. Participants often found the competences acquired and developed through the training activities relevant, applicable and beneficial for their professional activities, career advancement and/or specialisation in a specific domain.

• This success can in large part be contributed to the close proximity of the ITCILO to its primary stakeholders (ILO constituents, other organizations active in the field of labour policies and their implementation), individual professionals involved in developing and implementing the training activities and the support of a robust quality assurance mechanism, through which the ITCILO ensures that feedback is continuously collected, analysed and reacted upon. 

• 88.5% of the participants agreed or strongly agreed with that they were provided with all the necessary learning resources for completing the course successfully.

• Participants' feedback on their general level of satisfaction further supports the judgement that the ITCILO is highly effective in delivering its training activities. This level of satisfaction was relatively equal when respondents are categorised according to their professional background. The interviews with former ITCILO training participants made it clear that the ITCILO enjoys a high reputation.

• More detailed written feedback provided from the survey illustrate that the close contact that the ITCILO has with its beneficiaries is successfully transferred to the course design and implementation level. Not only were the learning objectives determined in an effective and fit-for-purpose manner, but these learning objectives have been successfully translated into a relevant learning experience.

• Interviews with workers' and employers' representatives clearly demonstrated that the ITCILO plays a significant role in the international dimension of tripartite functioning, with both workers and employers considering it a highly reputable, reliable, and successful provider of training. 

3.5 Effectiveness of Management Arrangements

• Flexibility is observed as a high-level attribute, with course adjustments prompted by course evaluations, external evaluations, and established quality standards.

• The ITCILO has a strong culture of continuous enhancement. Participants feedback plays a crucial role in refining training activities, often leading to changes and adaptations, including mid-term evaluations and tailored storyboards.

• The Centre would benefit from a more structured approach to capacity building in instructional design. This is especially important when considering the current environment in which quick adaptations are essential for different modalities to function in a manner which plays to the strengths of each specific modality.

• It is clear that there is a strong principle of decentralisation and subsidiarity with programme teams functioning in a highly autonomous manner. The crucial goal is to strike a balance between autonomy and standardised quality enhancement processes across the institution. 

• Enhancing collaboration among various technical programmes, while also simplifying financial processes, is recommended to facilitate the creation of interdisciplinary training activities that integrate diverse expertise. This approach aligns with learners' requirements, ensuring course relevance and addressing multifaceted learning needs. 

3.6 Efficiency

• The analysis of the 50 assessed training activities discloses an encouraging financial trajectory: all proceeds generated not only offset their immediate expenditures but also yielded surplus earnings in 2022. This outcome concurs with prior evaluation outcomes, emphasising the notable efficacy of ITCILO's online training endeavours, characterized by a clear-cut inputs-and-outputs structure. 

• A shared perception prevails that online training activities are more financially efficient than their face-to-face counterparts due to savings in travel and lodging expenditures. While this argument holds merit about outreach and accessibility, it neglects the considerable human effort involved in delivering effective online training.

3.7 Impact

• The positive responses to the survey underscore the courses' positive influence on the participants' learning journey, and affirm the training activities' constructive role in facilitating the practical application of newly acquired knowledge. 

• The evaluation results highlight the substantial influence of the training activities on participants' capacity to effectively apply their knowledge, skills and competences within real-world scenarios in their professional contexts. 

• A measure of ITCILO's potential impact is self-assessed level of improvement in competencies and job performance of course participants. A strong majority of participants reported that they made large or substantial improvements in terms of their competencies (75%) and job performance (71.5%) as a result of the training activities.

• 472 out of 519 respondents who completed the relevant section of the survey, submitted a specific example of the way in which the training activity has been of practical use for them. The fact that such an overwhelming number of participants readily submitted a practical example when prompted to answer an open text question is another demonstration of the beneficial effects ITCILO training activities succeed in producing. 

A sample of the survey responses to the question"Can you give a concrete example on the way in which the course itself has been of practical use for achieving results in your work?"

• The interviews findings demonstrate a very high level of confidence in the value of ITCILO trainings from the participants perspective and is one of the most relevant proxy indicator for the broader impact. 

3.8 Sustainability

• When participants consistently return to engage with learning opportunities, it signifies a sustainable cycle of value creation for both the learners and the organizing institution. This cyclic engagement fosters a sense of ongoing development, deeper knowledge integration, and a supportive community of learners.

• A significant number of trainees show repeated interest. However, the absence of a structured alumni network is evident, largely due to the divergence in interests and professional networks.

3.9 Technical Support

• Participants in the sampled training activities appear to have good access to the necessary technical devices and tools. 88,6% of respondents mentioned that they could freely choose and use different devices like PCs, laptops, mobile phones, and tablets for their learning needs.

• Participants found it easy to access and navigate within the online learning system eCampus. When participants had difficulties accessing the courses via mobile devices, it was mainly attributed to general connectivity issues that were not specific to ITCILO's eCampus services. 

• Technical issues in participating in (especially) online training activities on a regular basis remain a problem that improved compared to 2021. The distribution of responses was not equal across different continents. Participants from Africa and Asia encountered technical issues much more often than their counterparts from other continents.

• Survey responses indicate that Internet connectivity remains an issue. In areas with low bandwidth and unstable connections, asynchronous communication and content delivery tools are preferred. The mean scores for the frequency of use of asynchronous and synchronous tools indicate an average balance. However, synchronous conferencing tools were reported to be slightly overused compared to asynchronous conferencing.

• Respondents were overall very positive about the manner in which the Centre supported them in resolving technical issues.

3.10 Comparative Analysis of Modes of Delivery

• Compared to the previous year's evaluation, blended learning remains the top choice, even though it slightly decreased. There is a decrease in demand for fully online learning, and an increase in preference for face-to-face training. There is a strong belief that blended training can combine the flexibility, learning resources provision, structured learning progression and lower costs of the online modality with the personal contact and immersion of face-to-face modality. 

• Preferences about course modality differ depending on the type of course the respondents completed. Blended option is the most popular one both for the participants that completed blended courses and those who completed online courses. Those who completed face-to-face training had the same preferences for their future courses, with the face-to-face option being the most popular, followed by the blended option.

• When analysing course modality preferences over the last three years, the impact of COVID-19 pandemic has to be taken into account. The educational landscape witnessed an upheaval in 2020 with the outbreak of the pandemic, prompting a swift transition towards flexible and remote learning solutions. Due to its previous experiences with e-learning and high degree of flexibility, the ITCILO adapted to these circumstances highly successfully and experienced a notable surge in the demand for online training activities, aligning with global trends that saw virtual learning platforms becoming the primary mode of education. 

• Preferences differed with the geographical distribution of respondents. Blended modality is the most popular one for participants from Africa and Europe, although nearly as many respondents prefer face-to-face modality, while fully online option is most preferred for around one fifth of the respondents. 

• Participants' assessment of course relevance and applicability depending on the course modality was also analysed. This analysis showed that there are no significant differences in participants' view of course relevance and applicability between different modalities, although the blended modality did achieve slightly higher results than the online and face-to-­face option. 

3.11 Diploma Track

• In the context of ITCILO diplomas, participants are offered the autonomy to select three or four training activities that resonate with their learning preferences. This flexibility extends to the format of the training activities, encompassing both online and in-person sessions, and reflecting ITCILO's commitment to accommodating diverse learning preferences. The diverse geographical and linguistic contexts in which the Centre operates are also taken into account. Essential eligibility criteria stipulate that the chosen training activities should culminate in a Certificate of Achievement.

• The diploma framework, marked by its structured course selection, capstone project, and meticulous assessment process, underscores ITCILO's commitment to delivering a holistic and effective learning experience. 

• The ability to mix and match training activities from distinct programmes would contribute to a more holistic learning journey. This approach could lead to even greater participant satisfaction and fulfilment, as they tailor their educational experience to match their specific interests and professional goals. 

• ITCILO's training offer is closely aligned with the strategy of the ITCILO and the ILO, especially with regard to the strategic objectives related to broadening outreach and ensuring that the training activities meet the needs and requirements of the ITCILO beneficiaries, primarily the ILO constituents. 

• It is especially positive that the ITCILO manages to ensure a high level of diversity in terms of gender, age, socioeconomic background, geographical distribution and organisation type, which means that the Centre is successful in providing relevant training opportunities to a broad base of its beneficiaries. 

• It is clear that the core ILO values are deeply ingrained in ITCILO's work culture and relations with stakeholders. The ITCILO implements a methodology of close cooperation with beneficiary organisations on designing and delivering tailor-made training activities, which ensures that these activities meet very specific requirements of the target group in question. 

• The validity of the activity design is generally at a high level within the ITCILO's work structure and there are no major deficiencies within the process of course design and revision.

• Participants' feedback points to the ITCILO being highly successful in providing its learners with a top-quality training experience which leads to enhanced level of competences on the part of the learners. This position is further reinforced by the stakeholders' perception of ITCILO as an organisation whose brand guarantees such a high level of quality. The confidence in the ITCILO as a training provider is therefore at an extremely high level. 

• This impressive level of quality is supported by two strong pillars: the internal quality assurance system and proximity to interested beneficiaries. 

• A potential area of improvement of ITCILO's training activities is the development of course objectives related to "soft skills", interpersonal and intercultural connections, and peer learning. ITCILO can also improve harnessing the full potential of e-learning tools, primarily the eCampus.

• Stronger cooperation and interlinks between different project teams can ensure that the learners benefit from a more comprehensive perspective and that the learning experience that they have is a result of the best practices in delivering training activities that are available in the whole of the ITCILO structure. 

• The central support services of the ITCILO could have a more thorough facilitating role in relation to individual programme teams with respect to transversal areas where pooling of expertise and resources would be beneficial.

• Findings point to the ITCILO being in a strong financial position with no immediate threats to its financial stability. 

• ITCILO uses the resources at its disposal to ensure a high level of human and technical learning resources, facilitating a high-quality learning experience.

• Course participants rate their learning experience very highly and their feedback points to strong course applicability and practical orientation across different programmes with significant increase in job performance and competences. A number of practical examples has been provided by beneficiaries (workers, employers, ILO regional offices ... ) where the skills and competences developed through ITCILO training activities have been used in contributing to personal, organisational, or societal goals of course participants and/or beneficiary organisations. 

• An impressive level of knowledge and competences that participants acquire through the ITCILO training activities has been noted, and participants seem to be highly motivated to share this with their immediate environment. A significant number of examples related to such initiatives has been provided. 

• Dissemination and training of trainers are valued elements that are part of some of the training activities and could potentially be expanded. Participants are highly motivated to continue their learning journey, with an overwhelming majority interested in future ITCILO training activities and many of them continuing to use the learning resources even after course completion.

View ITCILO's management response