Learning Analytics Dashboards

The Centre is situated between Step 2 and Step 3 of the learning analytics sophistication model

• Several dashboards are offered to different stakeholders in the organization however, there is little focus on actual learning data.

• Currently, the dashboards mainly serve the purpose of identifying existing KPIs based around financial data and outreach statistics.

• Moving to Step 3, includes providing all stakeholders with a dashboard that gives insight into a fully connected set of data at the micro, meso and macro levels

Learning Analytics Dashboards should

• be based on a clear educational concept, rather than showing all available data
• use well-defined reference frameworks (social, achievement and / or progress) 
• focus on constructivist learning.

Data sources should not only include activity data but also take into account the content of completed assignments

Several groups of indicators relating to learners, actions, content, results, context and social factors, require different kinds of data sources.

Objectives

• Micro dashboards might include metacognitive, as well as the cognitive, behavioral and emotional, progress of the learner.

• Meso dashboards focus on courses and students at a high level of risk and allow drill-down to find out the reason and take appropriate action.

• The ITCILO operates in four fields: strategy, business development, clients and customers. Learning data is not one of them. 

• The systems lack SSO support, which increases the difficulty in collecting user-related data on third-party tools.

Shift the Centre’s paradigm towards a data-driven course-provider platform, focusing more on lifelong learning and continuous professional development

• Repeat participants should be taken into consideration
• More complex indicators based on learning data should be used as KPIs
• Product Owners should be enabled to iterate on the existing dashboards and modify them over time.

Infrastructure

• New technology was recommended for more granular tracking of user activity
• Data collection should be applied to all customer services, including learning activities realized using third-party software
• A data-lake solution might be considered as a way of improving the infrastructure.

Data and indicators

• Increase the focus on collecting knowledge-acquisition data
• Existing indicators that map collected data should be reframed and pulled together into more meaningful and applicable compound indicators.
• On the meso level, the collection of satisfaction and engagement data was not yet made available to dashboard users.

The PDCA Cycle 

Should be used to support future evidence-centered learning design by using the data and insights gathered as an integral part of learning design for new courses or the overhaul of existing ones.

User-centricity

• Future dashboards should emphasize prioritization and avoid over-complication
• Consider the user experience by creating clean LADs that benefits from clear communication and the reduction of visual messages
• Provide training opportunities to achieve higher data literacy and a Code of Practice should be provided to the dashboard users to raise awareness of how to usefully apply dashboards to everyday work.

• The category of data selected is classified in terms of depth of interaction.

• Starting only with metadata and self-reflected satisfaction levels, which are currently used as KPIs, the metrics become increasingly involved. 

• Depending on the perspective of the user, the green or purple data might be shown.

• Compounding these metrics is a delicate procedure since it is the deciding factor in delivering relevant and accurate indications. The alternative would require an extraordinary degree of data literacy and time investment on the part of all dashboard users.

• Certain users at the Centre should always be allowed a deeper look behind the compound indicators to enable them to improve the indicators and better understand the data. An “advanced” view will therefore be considered, giving graded access to more layers of complexity.

• It is essential that the automatic compounding process give access to detailed in-depth data from the data sources themselves, rather than only data aggregated for the prior logical step. 

• Relative reporting implies an immediate context in the user’s mind, while absolute reporting broadens the focus. This can be achieved by showing last year’s data or data from similar courses above any given indicator. The less a user is involved with the data, the more this approach supports decision making in a desired direction. 

• The deliberate structuring of dashboards by including or omitting data related to desired behavioral change, and displaying relative comparisons as a means to provoke action in the user’s mind, is an apt tool for expediting the standardization and formalization of online courses in an informal way.

• Another relative dimension of use is the comparison of cohorts, which can be applied to A/B test a new tool or didactic approach.

• To determine similarity, topical relations should be highly prioritized. However, there are other attributes to be taken into consideration: a similar course can be found by examining the metadata and learning history of participating learners.

• This indicator takes into account metadata from MAP.

• It provides an overview of different attributes of the participant dimensions. 

• This data is already well-established and represented in IBI. Similar approaches can be applied here.

• Simple messages be derived from this data to help activity managers and their teams gain an overview of whom they are facilitating.

• It is detrimental to show graphs of this indicator in the first instance. Only predefined messages with the respective ratio or metric should be shown.

• When interacting with the indicator by hovering or clicking, a useful graph can optionally be shown to allow a deep dive into the data, whenever necessary.

• It gives quick access to the data collected from satisfaction surveys.

• A list of the five most deviating items - in positive or negative direction - can be shown and optionally expanded.

• In pursuit of Objective I (measure early, measure often), it would be prudent to retrieve rudimentary satisfaction data from participants as they are progressing through the course. 

• If collected, the ongoing satisfaction data could be shown as a distribution over time. They should not be mixed with the survey results.

• It is an indicator of the current state of activity on the course.

• Its main purpose is to give an overview of whether and how many participants are properly on-boarded and have started actively participating in the course.

• It takes into account the aggregated data from the presence-, engagement- and activity-level layers, as well as the metadata collected from the MAP.

• Participants are separated into different states of enrollment: Registered, Engaged, Active, Successful, Certified, Dropped-out.

• Much of this data is already shown on the current activity-manager dashboard on the eCampus. This view can be expanded to show more detailed status indications for each participant.

• It is essential to go into detail and understand what exactly is happening at the micro level and have the opportunity to make a tangent analysis at the meso level. 

• When participation levels are viewed by activity type, problems can be tracked down to their real reasons.

• This indicator only takes into account interactive users and displays data over time. 

• This indicator is concerned with generating an aggregated participation score and distribution across different activity types.

• This indicator aggregates data from the activity, quality-of-results and effectiveness layers.

• It shows quiz scores and assignment grades, and also takes into account the completion rate. As a next step, analysis of the written forum entries can be included.

• The pre-KAT and post-KAT evaluations, which are mandatory for all courses, can be used as a reliable after-the-fact measure of success. The improvement from pre- to post-KAT can be used as a goodness-of-fit test for the success data collected on the ongoing course.

• Especially on the macro level, a bird’s-eye view of all participants is necessary. 

• A stacked bar chart is well suited to display both the total number of active participants over time within the average course and their respective depths of involvement. A given bar represents the nth week of a course. Towards the end of the course, one should see the share of successful participation increase, while the share of participants who are merely active decreases.

• Users should be allowed to make distinctions by mode of training. 

• Depending on the objective, it should also possible to use aggregated numbers of participants instead of weekly measurements, or add the certification dimension as a fourth color, at the same time extending the timeline to after the official end of the course.

• This meta-indicator is an important step towards achieving better data quality and provides an outside evaluation of how well a course can be measured in terms of the indicators used.

• The insights gained from this indicator can also be used in the dashboard to show the level of confidence in other indicators. 

• The data-availability KPI is well approximated by the ratio of activities on a course which provide data vs those which do not. All activities can be extracted via the Moodle API, which also serves the type of the activity.