External evaluation 2021 report brief

External evaluation carried out by: (View full report)

In the 2020-21 biennium, the Centre has seen a massive shift from face-to-face training towards fully online learning, driven by the outbreak of the COVID-19 pandemic. Within two years, the number of enrolments per year in online training activities has increased five-fold.

To provide evidence of the relevance, validity of design, effectiveness, efficiency, impact and sustainability of its fully online training activities, and to derive recommendations for the improvement and further development of online training courses, the Centre has in mid 2021 commissioned an external evaluation. The evaluation criteria were based on the OECD DAC evaluation principles and involved desk research, an online survey of more than 9000 participants, interviews with participants, institutional clients and with staff of the Centre, and the elaboration of case studies.

The evaluation reviewed a sample of twenty training activities offered in 2020 and reaching more than 9000 participants. The sample included a variety of paid and free, open and tailor-made, tutor-supported and self-guided courses that took place using a diverse set of tools and platforms, including eCampus, Solicomm, webinars, and virtual reality.

The evaluation criteria were based on the OECD DAC evaluation principles, and the used methodology combined both quantitative and qualitative analysis of findings.

A significant number of 1.284 responses were collected from an online participants’ survey (Quantitative analysis) that also evaluated the validity of the training design to support a meaningful online learning experience using the Community of Inquiry (CoI) framework.

The model assumes that effective learning and engagement in online learning activities occurs within an online learning community through the interaction of social presence, cognitive presence, and teaching presence.

In-depth interviews were conducted with ITCILO’s staff members (27), institutional clients (2), and participants (7), in addition to a desk-based review of relevant documents, reports, courses design, evaluation, and knowledge acquisition results. Over and above, eCampus was visited to review participants’ engagement (Qualitative analysis).

Based on the interviews, three case studies that capture the positive impacts created by the Centre's online training activities were presented. Each case includes information about how participants or institutional beneficiaries made positive changes in their work experiences or institutional culture through learning new knowledge and skills from the Center's online training activities.

• There is a strong sense of appreciation and recognition shared among the interviewees, that the Center has successfully managed to reach out to its target groups and provide training demanded by its beneficiaries, partners, and donors.

• The Centre has effectively played its role in providing ILO constituents with specialized training on different aspects of the Decent Work Agenda by promptly and effectively transitioning its training activities online.

• The staff maintained supportive and collegial working relationships with their client organizations. They were able to collect a range of positive anecdotal evidence, demonstrating the effectiveness of their teaching activities (e.g., successful knowledge implications, effective organizational changes).

• The Center has successfully helped its client organizations and course participants navigate fast-changing situations and recover from the unexpected challenges that emerged during the pandemic.

• Among the reviewed activities, more than half received higher satisfaction average scores in 2020 than in 2019. The remaining courses, despite the slight decrease in their overall evaluation scores, received high mean scores on the questions related to knowledge application, benefit to organization and overall satisfaction.

• The Centre reached a wider and more diversified audience with online distance learning activities, especially, participants from middle-income countries who took advantage of digital learning solutions avoiding costs for travel and accommodation.

• Participants from 151 different countries responded to the participants’ survey. After the online learning experience, only one quarter of the respondents said that they would prefer to go back to fully face-to-face training; the majority preferred blended learning courses, closely followed by participants who would even prefer fully online courses Thus, three quarters of the participants would prefer a digital format of some kind in the future.

• Participants tend to believe that they are more likely to benefit from exchanging their ideas and experiences with other participants in an international-wide learning community. Many appreciated the opportunity to access the training opportunities and up-to-date expert knowledge through online delivery during the pandemic when international travel was entirely banned. Nevertheless, they missed the social aspects of face-to-face training activities.

• Only in Oceanian countries, the majority of respondents would prefer face-to-face training in the future. In Africa, Latin America and Europe, most participants want blended learning courses, and in Asia and the Middle East, fully online courses would be the first choice.

• The participants of the sample courses seem to be well equipped with technical devices and tools to access ITCILO’s online courses. The analysis also found that access to and navigation in the online learning system, e-Campus, was not an issue.

• Some participants experienced difficulties in accessing the online courses via mobile devices.

• There are stronger concerns over the language barriers (than technology barriers). Given the dramatically increased online participation among the target groups, this concern may deserve more careful attention. 

• The Centre provides an appropriate mix of synchronous and asynchronous information and communication tools. Participants tend to slightly prefer asynchronous content that allows for higher levels of flexibility and accessibility.

• Discussions about how to apply topics and concepts covered in the courses are more effective in tutor-guided courses that also provide opportunities for interaction among participants who can bring in their experiences from their work context.

• Participants tend to perceive online courses, which provide them with structured and guided opportunities to use new skills, to improve their own practice more positively than others.

• Participants reported lower levels of knowledge application in self-guided courses.

• Course designers and facilitators managed to deliver highly engaging, interactive, and supportive online courses that provided opportunities for rich and deep learning experiences.

• The emergent transition has brought the quick emergence of a range of different approaches to the design of online training activities. A lack of a coherent design framework and an existing selection of good practices had made each unit develop its own approaches and strategies for the online course design. Hence, the course design has been done rather intuitively (than systematically) based on their previous face-to-face (or BL) training experiences and knowledge.

• Online training activities with multi-media interactive elements tend to be perceived more effective than other training activities that tend to be more text-based or less interactive.

• The resources invested into the delivery of online training activities have been used economically. The inputs were translated into desired results to meet the demands of beneficiaries, partners, and donors. Except for a small number of training activities previously designed in a blended format, most of the new and pivoted online training activities were developed highly efficiently under the restricted financial and situational conditions (e.g., budget cut, lockdown measures) in 2020. When it comes to looking at the non-human resources and inputs (e.g., funds), the DO costs were small, and the converted outputs were significant. 

• All reviewed activities created revenues, covering direct costs (many with revenue far exceeding the costs). Given the dramatically increased enrollments in those online courses without increasing the inputs, it is a remarkable success. Accordingly, the Centre’s online training activities can be considered highly efficient based on a relatively simple inputs-and-outputs formula.

• The vast majority of participants agreed or strongly agreed that the course was relevant (95.4 %), translated theory into practice (92.3 %), and that they can apply what they learnt in their work setting (94.3 %). 54.6 per cent shared a concrete example of their application of knowledge after the training. Furthermore, participants reported that they made large or very large improvements in terms of their competencies (85.6 %) and job performance (69.0 %) as a result of the training activities.

• Participants expressed their wish to receive training more relevant to their local contexts and knowledge readily applicable in their own organizational contexts rather than general or theoretical knowledge.

• The Centre has successfully served its target groups during 2020 despite the massive disruption caused by the COVID-19 outbreak.

• The relevance of the Centre’s online training activities was unexpectedly high in 2020, effectively meeting the demands and needs of its target groups.

• The Centre was able to massively increase the number of participants via distance learning.

• The Centre’s online training activities have been an extremely effective instrument to strengthen the capacity of ILO constituents and other ILO development partners—especially during the COVID-19 pandemic. As the online transition was primarily driven and influenced by the actual needs and demands of the partners, it is clearly viewed as a great success in achieving the Centre’s mission and strategic plans.

• The reviewed training activities effectively achieved their immediate objectives. Many participants found their engagement with the activities beneficial, contributing to their professional practice and development. The simple formula, good objectives lead to good outcomes, was observed in the review process.

• Finding a balance between access, quality, and costs paves the way for ITCILO to reach and serve its target groups by widening access and scaling up distance training activities.

• Giving the enormous importance for the longer-term impact and scaling-up of ITCILO's distance training activities, the participants’ demand for different modes of delivery as well as for synchronous and asynchronous interaction needs to be explored.

• It is very important to provide technical support. There might be room for improvement regarding the information provided for participants.

• Despite the rapid online transition and associated pedagogical challenges, the effectiveness of the new online activities is proven to be impressively high.

• As the pedagogical preferences and technical limitations significantly vary among different learner groups, the unit-based approach to the course design and decision-making model that emerged during the pandemic is still considered adequate.

• Longer courses tend to be perceived more effective by learners because that they are given more time to acquire new knowledge and skills and apply them in their working contexts. On the other hand, shorter courses provide limited time and, thus, knowledge acquisition and application opportunities. The duration of online course activities can be extended, accommodating the possibilities for the knowledge application.

• Learning materials developed in 2020, compared to the ones produced before 2020, are certainly seen as less aesthetic and graphics-oriented—since most of those video productions were done in-house. Particularly, some online training activities were directly translated from the former face-to-face activities, substituting them with a series of synchronous webinars without systematic re-design and resource-intensive re-development.

• Institutional resources and inputs were rather minimal at the planning stage of the activities. Given the positive outcomes of those courses, both their outreach and perceived effectiveness, we can conclude that the resources invested in the PLAN stage were efficient with a cautionary note that there is much room for improvement in the design and development of those online activities. A systematic and more learner-centered approach to the online course planning is required.

• A critical factor contributing to the successful results of the online training activities in 2020 was the Centre staff’s dedication and resilience, supported and facilitated by the unit level of leadership and collaboration. Most of the Centre staff were fully on board to “get through the crisis”.

• It seems necessary to carefully review the current workload and unit structure from the sustainability perspective and, referring to the PDCA cycle, the Centre may increase the inputs for the Checking and Acting tasks and develop a systematic working model and comprehensive framework for online course design and implementation to maintain the efficiency level.

• The single measure of indicator 2.3 alone is not a valid way to measure knowledge application. Survey respondents may be more likely to respond to items on a scale rather than sharing their stories in an open text window.

The report furnished the recommendations based on the three dimensions of the ITCILO’s strategy framework.

View ITCILO's management response