Non-training QA report brief

 
Report brief
Consolidation of the quality assurance processes for ITCILO non-training capacity development services

Original report by:

O.J. krueck

O.J. Krueck

Managing Director at ECCOS
1. Operational context

 

In line with the 2019 ILO capacity development strategy, the capacity development services of the Centre group into three categories: individual-level capacity development, institutional-level capacity development and system-level capacity development.  

The principal means of the Centre to build individual-level capacity is training, delivered online or face-to-face. To build institutional-level capacity, the Centre offers technical advisory services, training material development and media development support to ILO constituents and other intermediaries with a mandate to promote Decent Work. To strengthen system-level capacity, the Centre facilitates dialogue activities and knowledge management services.   

 

Typology of capacity development services

 

The preparation of the original report fell in the time when the Centre witnessed a shift from its previous main emphasis in the service portfolio on individual-level capacity development with focus on face-to-face training to distance learning because of expansive travel restrictions throughout 2020 due to COVID-19. 

In 2020, the Centre reached more than 50,000 learners from more than 180 countries with training and more than hundred organizations with non-training services.

Furthermore, the Centre has seen a significant change when it comes to delivering services with institutional-level and system-level capacity development services showing an increase in number of assignments and turnover in relation to the delivery of traditional training services on campus or in the field.  

2. Objectives

 

  • The overall objective of the assignment was to contribute towards full compliance of the Centre with the ISO standard 29993 for learning service providers outside formal education, and where applicable ISO 20700 (for consultancy service providers).  
  • The immediate objective was to strengthen the capacity of the Office of the Director of Training to quality-assure institutional-level and system-level capacity development services of the Centre, by way of introducing processes and tools to collect and analyze relevant quality data along the PDCA cycle.
  • The direct outcome of the assignment was the establishment of a robust quality assurance system for non-training capacity development services.

 

 

The PDCA cycle

 

3. Discussion

 

  • The ITCILO already provides for a quality assurance architecture which aims at continuously improving upon quality in the field of training services with a set of appropriate procedures, tools, and practice. 
  • The 3 most common dimensions for defining quality are:  

1. compliance with standards

2. satisfaction of the demands of clients

3. acceptable rate of price and costs

  • There are many definitions for advisory services but two main approaches are: 
with a functional view

Any form of providing help on the content, process or structure of a task or series of tasks, where the service provider is not actually responsible for doing the task itself but is helping those who are 

as a professional service

When any time a professional is trying to change or improve a situation or a system but has no direct control over the implementation

  • Research shows that the  most  important  dimension for advisory services is reliability because it will create customer satisfaction the best: 

1. Reliability involves consistency of performance and dependability

2. Responsiveness concerns the willingness of staff to provide service

3. Competence means required skills and knowledge to perform the service

4. Access involves approachability and ease of contact

5. Courtesy involves politeness, respect, consideration, of contact staff

6. Communication means keeping customers informed

7. Credibility involves trustworthiness, believability, and customer's interests

8. Understanding the client's needs

 

  • The ITCILO benefits from a comparative advantage as it is part of the UN family with well-known procurement regulations and in addition a defined group of well-known constituents as clients for which ITCILO is expected to deliver services as part of the mandate. 
  • Service providers can use several tools and instruments to measure quality of services and customer satisfaction in regard to outcomes and impacts. Most commonly used are customer surveys to measure the satisfaction and identify problematic aspects in the service delivery from planning to implementing.
  • In agile processes, a continuous feedback loop minimizes sub-optimal results of advisory services. However, most clients still prefer a one-dimensional traditional process design with pre-defined objectives, milestones, and timelines. Empirical evidence shows however, that agile processes produce a better sustainability and measurable impact of advisory services. 

Agile process

A proto-typical iterative agile process

 

  • The quality of an advisory service can be measured through an agile continuous feedback loop as follows: 
Discussion
Phase 1: Joint discussion with customer

to understand and define in writing: 

  • Strategic and operational context
  • Stakeholders, enablers, obstacles, challenges
  • Organizational culture
  • Vision, mission, and operational objectives
  • Current strategy to achieve objectives
  • Current organizational structure and processes to achieve objectives
Objectives
Phase 2: Definition of objectives of advisory services
  • Desired contribution of services to strategy and operational objectives
  • System boundaries of advisory services
  • Overall timeframe
  • Decision makers
  • Task managers
  • Other involved specialists
Implementation
Phase 3: Iterative implementation of advisory services:
  • Monthly (at least) project meetings to assess progress of process steps, achievements, and encountered challenges
  • Revision of validity of objectives of advisory services
  • Revision of process steps and decision of required changes
  • Revision of suitability of decision makers, task managers, involved specialists both from customer and provider of advisory services
  • Identification of game changing parameters
  • In contrast to the iterative way of an agile process, the more conventional way to assess the quality of the delivery of advisory services can be assured by using a linear “total or integrated quality management system” with a process standard such as ISO 9001.
  • With the latest modification of ISO 9001:2015 there are seven quality management principles which most QA systems utilize as basis for one of the many modified approaches towards delivering service quality:
  1. Customer focus
  2. Leadership
  3. Engagement of people
  4. Process approach
  5. Improvement
  6. Evidence-based decision making
  7. Relationship management.
  • Accordingly, the advisory service would follow a structured, transparent, linear process approach to:
customer needs
1. Understand the needs of the customer and align the advisory service accordingly

versus selling readily available standardized services and aligning the customer’s needs to the available service(s).

implementation roles
2. Define clear roles for the implementation of the advisory services

versus unclear or frequently changing roles and responsibilities during project implementation.

different staff levels
3. Actively involving different levels and target groups of the customer’s management and staff

versus focusing on the nominated decision maker and/or task manager only during project implementation.

milestones
4. Introducing clear and measurable milestones for the overall process

versus delivering results towards the end of the process.

Indicator
5.Introducing quantitative and qualitative indicators to show improvements stemming from the service

versus delivering solely an advisory service without links to the operational objectives of the customer.

Data compilation
6. Introducing a system to compile data throughout the process

as basis for evidence-based decision making for the customer - versus the tendency to use “pet projects/processes” of either customer or provider of the advisory services as basis for decision making.

Customer relationship management
7. Introducing a relationship management system

(“post-sales”) with which a continuous communication with previous customers ensures that the provided advisory services (a) continue to produce the originally intended results (b) required modifications can be addressed swiftly (c) additional services can be identified and turned into new business opportunities.

  • The above process would be documented at the very beginning and verified (in particular achievement of milestones) throughout the implementation process and handed over to the customer. This is typically called a “process documentation handbook”, ensuring for the customer that all process steps have been (a) implemented and (b) can be replicated at any time.
  • Another standard of relevance to be taken into consideration is ISO 20700 which is a guideline for management of consultancy services. For the benefit of ITCILO, it can serve as a basis for the delivery of advisory services, but less so for designing a QA architecture.
4. Global best practices

 

  • Many organizations in the sector of management advisory services have developed total quality management (TQM) systems to provide for a more comprehensive approach which aim at integrating as many quality challenges as possible. There is no internationally agreed standard for TQM, let alone a uniform name for the framework and the underlying improvement process. Digitization of (selected) processes has become a trend for TQM systems.
TQM
Total Quality Management Systems

TQM is a management approach to long-term success through a focus on customer satisfaction. This framework is defined as an incremental improvement process rather than a fixed system.

  • A good number of other organizations are developing and adopting Integrated Management Systems (IMS) which aim at combining all aspects of an organization’s systems, processes and standards into one holistic system. IMS would also utilize digital solutions if and when applicable for reasons of efficiency.
IMS
Integrated Management Systems

An IMS allows an organization to address all elements of the management system as a whole by avoiding multiple management systems. It would avoid having several standards in one organization but aim at streamlining processes along the same standards where applicable. An IMS allows these processes to be combined so that they simultaneously cover all standard-specific requirements.

  • The architectures of TQM and IMS are structurally based on but also go beyond the ISO 20700 standard for providing management consultancy and advisory services. ISO 20700 serves as a basic guideline for the delivery of the services but does not allow for building a comprehensive architecture as such.
5. Conclusions

 

  • The PDCA cycle of ISO 29993 already reflects the general recommendation of ISO 20700 of ongoing evaluations and improvements “to assess and determine the effectiveness of the assignment”. ISO 20700 does not add any more detail or value to the existing QA system at ITCILO with its series of quality checks performed along the PDCA cycle.
  • The design and implementation of the Training Department’s 4-level evaluation methodology with the KAT as center piece is state of the art. In comparison to other training centers this evaluation methodology can even be considered best practice.

  • Constituents of the ILO rightly demand measurable impact towards the decent work agenda. It is clear that the Training Department with its comprehensive and stringent evaluation approach professionally assesses the impact of training activities by Level 3 and Level 4 of the Centre’s evaluation methodology. In turn, the QA process for non-training services would also need to compile qualitative information on the immediate outcome and impact of the Centre’s services.

6. Recommendations
  • It is recommended to follow the existing and tested QA framework of the ITCILO in line with ISO 29993 also for the evaluation process of institutional capacity development services. The corresponding PDCA cycle adequately mirrors the four required stages for providing both individual and institutional capacity development services.
  • ITCILO already uses an online institutional capacity self-assessment through a survey-monkey questionnaire to record the satisfaction with outputs towards desired outcomes and impacts.
  • Similarly, the assignment-specific assessment reports are a suitable way to individually assess and evaluate the service delivery.
  • When it comes to the annual reports with aggregated data on advisory services the question arises if corrective action can derived from aggregated data for rather specific designs for assignments. But overall, the tools are only utilized on an ad-hoc basis or demand basis for non-training services. This is clearly not enough to match the professional approach and set of tools already in place for assuring the quality of training services.
  • The challenge for ITCILO therefore is two-fold, i.e. there are currently not enough tools to assess quality and the ones in place are not being used systematically enough to generate reliable data for the two main purposes of a QA architecture: accountability and quality improvement.

In line with the international trend of applying “holistic” or “integrated” management systems to an organization, the question then arises what elements of the QA architecture ITCILO already has in place for individual level training services could possibly be utilized for assessing quality and impact of institutional level non-training services to avoid parallel or even competing QA systems, tools, and processes.

  • ITCILO is up to now missing suitable tools and instruments to allow for quality assurance for institutional-level capacity development services at least for the stages of planning and checking.
  • An adapted version of the “New World Kirkpatrick Model” can very well serve as an evaluation methodology for non-training services and in turn cover several elements of the PDCA cycle.
  • It is recommended to design and introduce tools that evaluate performance more on a qualitative level than on a quantitative level. Reason being that non-training advisory services provided by ITCILO are usually less standardized.They typically address the needs of a specific client with little opportunities to create and benefit from economies of scale. It is still possible and recommended to introduce standardized tools and instruments to quality check all stages of the PDCA cycle for non-training services.For easy reference the following table compiles specific recommendations for tools along the full PDCA cycle.

Recommended tools on PDCA cycle

 

  • The results obtained from these tools will feed the suggested additional indicator either/or in a quantitative or a qualitative manner. 
  • If ITCILO decides to select a quantitative indicator, e.g. the percentage of customer satisfaction in regard to professional implementation and suitability of provided services to improve performance results from categories 1 - Plan and 2 – DO will feed into the indicator.
  • If ITCILO opts to select a qualitative indicator, e.g. impact of improved services on target groups/beneficiaries of customers, results from categories 3 - CHECK and 4 - ACT would feed into the indicator.

Utilize the set of quality assurance tools and instruments on a regular basis by default instead of ad-hoc basis.

The Centre can set and aim for achieving a key performance indicator (KPI),i.e. the change in the clients institutional performance as an immediate outcome of the delivered service. To measure this, clients should be asked after a 6 to 12 months incubation period for the emerged impacts, for example:

1. To what extent have systems, materials, tools provided through the service been actually been applied and used in the institution

For the data collection, it is recommended to design an online questionnaire in line with the process steps of the agile or linear process design as basic information. The results of this questionnaire will be more of a quantitative nature as replies can be ticked in a yes / no format. The advantage is that only representatives of the customer are required to reply and they can easily and quickly fill such an online questionnaire. The KPI would be designed in way of e.g. “percentage of provided tools used in (daily) operations of customer”.

 

2. To what extent has the performance of the institution improved as a result of their application of the service provided.

For the data collection, a wider perspective is required, i.e. because the target group/beneficiaries of the customer will have to be addressed to actually assess the performance improvement thanks to the advisory services of ITCILO. In turn, a more qualitative methodology is required to produce tangible data, e.g. a focus group discussions or bilateral interviews with beneficiaries.

  • What became apparent during the analysis of documents and when conducting the interviews is a rather interchangeable use of different terms for non-training services. While some Programme representatives might call them advisory services, others might refer to them as non-training services. Also, different documents of the ILO and ITCILO are using different terms. Therefore, before taking a closer look at what non-training service products ITCILO is currently providing a clarification of the terms and its context seems justified.
  • While the typology aims at clarifying the type and level of the capacity development service, an alternative suggestion might be discussed, i.e. the (contractual) relationship of ITCILO with the respective (business) partner in a capacity development service delivery process. The typology could then follow a classification of e.g. Business to Customer (B2C) for individual level services and Business to Business (B2B) for institutional and system level capacity development  services.
  • Going even one step further and selecting the appropriate classification for ITCILO as a non-for-profit organization, one could also think of the more adequate terms of Organization to Participant (O2P) and Organization to Organization (O2O). In either case, the notion of ITCILO providing capacity development services for a “beneficiary” seems a bit outdated carrying the connotation of “handing out a service”. Instead it is recommended to discuss the idea of creating the understanding of a professional relationship between two equally interested parties for supplying and demanding capacity development services.

Introduce at least 4 types and clear definitions of services with two respective sub-categories for Training Material and Media Development and a separation of advisory services into Technical Advisory Services and Project Management Services

For better and more transparent understanding in MAP, aggregated reports, and potential replication of CD services for new clients. For ease of reference as well as internal and external communication it is recommended to introduce a revised categorization of institutional capacity development services.