External Evaluation of Communication and Advocacy Campaigns 2023 Report Brief

External evaluation carried out by:

Download full report

This is an evaluation of the May 2022 digital campaign supporting the 5th Global Conference on the Elimination of Child Labour, centring on six days in Durban.

The conference boasted a number of ‘firsts’ - the first hybrid conference (i.e. face-to-face plus virtual), the first in Africa (of the previous four, two had been in Latin America and two in Europe) and the first to feature the active participation of children.

This campaign was undertaken for partners - ILO Fundamentals, Alliance 8.7, and South Africa’s Department of Labour. The ITCILO was in charge of the campaign design and implementation as well as overseeing the digital aspect of the physical conference.

The purpose of the evaluation was to provide the Centre with evidence of the relevance, validity, coherence, effectiveness, efficiency, impact and sustainability of its  Communication and Advocacy activities, to assess which modalities offered by the Centre are most effective and  efficient, and to extrapolate good practices, lessons learned and recommendations for the improvement or scale-up of Communication and Advocacy activities of the Centre.

The evaluation focused on the six main digital elements of the campaign: Website content and search, Newsletters, Media Relations, Video content, Social media, and Live Streaming.

The analysis of the campaign reviews the ISO standard for Project Management and the OECD DAC Network on Development Evaluation Criteria, and sets out a simplified assessment structure. A number of alignment checks are added of which the most important are Campaign alignment check and Best practices review. 

To simplify this evaluation, these approaches were folded into a timeline of pre-project, project, and post-project activities and the utilized Quality Assurance approaches have been mapped against an uber-framework for digital campaigns to identify their different emphases and key questions on alignment added.

The Questions Grid

The central idea behind this evaluation is campaign alignment with ‘best practice’. A second core part of this evaluation is to look at the priority calls that were made and to assess whether different calls might have generated better results.

At a very basic level, a campaign’s success is determined by its sponsors: did they get what they paid for? In formal interviews the three sponsors gave these responses:

3.1 Campaign Design

• The Turin team contributed ideas and expertise, but was not the ultimate decision-maker.
• There was no explicit mention of a theory of change in any of the documentation, nor was it clear from interviews undertaken that the sponsors had a unified view of the ‘how’ and ‘why’ of the campaign.There was, however, a detailed content strategy from ITCILO, which set out target audiences, and high-level tactics.
• Interviews with the leads from the three sponsoring organisations have yielded the following 3 campaign goals: Awareness, Hybridisation, and accelerating the efforts towards the achievement of SDG 8.7. 
• The target audiences mentioned in those same interviews were: Delegates on the ground in Durban, Virtual delegates using digital media, Decision-makers outside existing networks who could make a difference, including employers, government officials, civil society, National, Regional & Global Media.
• There do not appear to have been any Key Performance Indicators to underpin the audiences and goals that the consortium were targeting.

3.2 Campaign Efficiency

The ITCILO set out its main tactics in its Content Strategy document:

"Our efforts are therefore focused on awareness raising, converting commitments into action, and providing stakeholders with clear guidance on how to keep on implementing their pledges. In parallel, we are implementing a comprehensive content strategy that will engage the global community and wider audience"

The ITCILO committed to create, share and package content via:

3.2.1. The Website

ITCILO built a self-contained conference site, and devised the content. The design was eye-catching and friendly, and content was laid out so that it was easy for users to read. The traffic numbers underline that the site focused on the needs of conference delegates. 

3.2.2. Newsletters

Newsletters were central to engagement with delegates in Durban and online. Segmentation was very strong with separate lists from the Alliance 8.7 distribution list for in-person delegates, online delegates, pledge-makers, workers group, government group, and employers group.

Pre-event research suggested a short and very clear presentation would be optimal. 

Engagement was strong. There were 17 newsletters that went to in-person and online delegates and which were largely a blend of service messaging and event highlights. These generated an average opening rate of 43.7% and a click-through rate of 13.5%. Opening rates were about double the average for Non-profits and the Education & Training sector, click-through rates about 4 times.

The campaign boosted the subscriber list. By the time of the June 12 World Day newsletter, there were 9678 subscribers - nearly 7% more than just before the 5th Global Conference.

3.2.3. Media Relations Strategy

ITCILO had an ambitious media outreach strategy to target all types of media outlets however, budget cuts led to the media strategy being cut right back.

The media relations agency was present at the conference and created nine press releases. But only African media were targeted directly. In addition, only the final press release detailing the Durban Call for Action, was translated.

Despite this, there was extensive coverage - 3,000 mentions of the conference online from more than 300 news outlets. National coverage included the Guardian Nigeria, and regional coverage included CNBC Africa. In addition, SABC, the South African broadcaster carried live streams of the conference on its site.

3.2.4. Video Production

The campaign invested considerable effort into packaged video with more than 20 created and posted on social media with a wide variety of approaches. 

There was selective use of Twitter, Facebook, Instagram, LinkedIn and YouTube accounts from ILO, Alliance 8.7 and ITCILO, and total views are estimated at about 20,000.

Video content was the second most highly rated digital output in the survey of users.

3.2.5. Live Streaming

Almost all the Durban sessions were live-streamed via Zoom for those who registered, and on ITCILO’s YouTube channel for everyone else. The live streams were also aggregated on the event website.

Sponsors were delighted with this core part of the hybrid event, with one remarking on how the streaming had been near flawless.

The sessions were high-level discussions, targeted at a specialist audience, and viewing figures of 2750 Zoom sessions and 4349 YouTube views suggest this content did not reach far beyond the hard-core Alliance 8.7 audience.

Of 200 online users surveyed, a high proportion - 76% - said the live-streams had been useful or extremely useful. The data suggests that the average virtual delegate viewed relatively few sessions.

3.2.6. Social Media

The primary tactic implemented was to post on Alliance 8.7 accounts on Twitter, Facebook, LinkedIn and Instagram, with some posting on ITCILO’s own account.

The biggest category of posts was ‘service messages’ - usually referring to live-streams. Quote cards were the next most common, and then the #RaiseYourHandForKids campaign. 

The only paid promotion of the campaign was enormously significant and accounted for 70% of #RaiseYourHandForKids posts.  

3.3 Campaign Results

There are some clear patterns in the responses to the online survey that was launched in July 2023:

• High levels of satisfaction: a weighted average of 77% saw digital elements as ‘useful’ or ‘extremely useful’.
• Proximity: Overall, ratings were highest from those attending the physical meeting, followed by those who registered for the Zoom livestreams, and then those who support Alliance 8.7 but didn’t register for the event online.
• The website was the most popular element followed by video content.
• The conference app was highly appreciated, but physical delegates ranked it the least useful of the digital tools available to them.
• The daily newsletter was rated least useful by online registrants. However, Alliance 8.7 supporters who didn’t receive daily updates overwhelmingly said they’d have liked them.
• #RaiseYourHandForKids was rated least useful overall. However, this sub-campaign was not designed to be useful to existing supporters of the cause but to extend awareness of the issue to a completely new set of people.

Were the Digital Campaign Goals Met? In terms of campaign goals, the answer to the question is only a qualified ‘yes’:

• Event hybridisation was successfully delivered but the interpretation of what a virtual delegate needs was narrow.
• Awareness-raising: #RaiseYourHandForKids was world-class.
• Acceleration of action was not perceived as a primary campaign goal. Nevertheless, the user survey suggests that decision-makers and individuals were galvanised into turning commitments into actions.

The tactics that worked best - live-streams, newsletters and  #RaiseYourHandForKIds - tended to be the cheapest because they made very good use of what already exists. Meanwhile, the least effective at engagement took most of the resources.

Users and sponsors liked what they saw and ITCILO more than fulfilled the expectations. However, ITCILO is aiming higher in digital advocacy and wants to model best practices in all its work. On this measure, there is much to be done.

4.1 Campaign Design

Campaign design was complicated by a tripartite stakeholder system and communications delays due to COVID-19. ITCILO saw its primary roles as facilitating the hybrid conference and making noise about the event but it did not receive well-defined KPIs.

All successful advocacy campaigns require superb mobilisation of networks, and this element got lost in the planning stage - one of several consequences of campaign sponsors not agreeing on a ‘theory of change’.

4.2 Campaign Efficiency

This was an enormously complex digital event, managed by a tight Turin team. The fact that everything that was promised got produced is a testament to the teamwork and esprit de corps created. However, some of the collective decision-making looks at odds with the campaign goals, and there were some departures from established best practice in digital advocacy.

These misalignments are mostly to do with limited resources. But some could have been avoided by a pre-project alignment check to ensure that audience goals line up with engagement tactics and costs. 

4.3 Campaign Results

The smartness of campaign design largely dictates how successful it will be. This one had many successes but there were some gaps.

While stakeholders and users expressed satisfaction with the outcomes, some modest tweaks to budget allocations would have magnified campaign reach.

1. Insist on coherence in planning

It is essential to get campaign goals, target audiences, engagement tactics, and KPIs fully articulated and well aligned. The way to do this is to insist that sponsors discuss and agree on a theory of change.

2. Act resourceful and adapt rather than build 

No campaign ever has the resources it wants, and all successful ones show huge resourcefulness. The cost-benefit review reveals that building event-specific digital infrastructure was expensive and that the biggest engagement came from harnessing existing platforms.

3. Good storytelling: always put the audience(s) first 

If it’s worth investing in content, then it is worth optimising that content for users. Sessions were rich with insights, case studies and perspectives. People don’t have time to watch long sessions - make it easy by packaging up highlights for all audiences.

4. Use all network opportunities

Change-seekers need to muster all the support they can. Such campaigns would need to feature greater promotion by the ILO, the involvement of adjacent organisations like UNICEF and FAO, and the possible support of the major social platforms.

5. Balance production against promotion 

The investment in Facebook promotion had a huge impact on #RaiseYourHandForKids. This underlines the harsh reality that social and search is becoming more ‘pay to play’. Commercial campaigns typically spend 20% on the production of content and 80% on promotion/advertising. That’s too rich for a purpose-driven campaign, but the 5-15% recommended by Getting attention is a useful benchmark.