Question: Evaluating learning and development

Examines approaches to evaluating learning and offers insights for practitioners to consider in their own context

Coverage of learning and development evaluation

Our learning cultures research gives advice on evaluating the learning environment across the whole organisation, with particular teams and at an individual level.

Whilst the majority of organisations carry out some evaluation of learning activities, in 2019 our Professionalising learning and development report showed about a quarter of respondents struggle to understand L&Ds impact. Our 2021 Learning and skills at work report showed that one in four respondents don't systematically evaluate L&D initiatives.

Evaluation activities can include:

Impact - where L&D can work with the organisation to show how the learning interventions have impacted on performance - these can include links to key performance indicators (financial and operational).

Transfer - where L&D can work with the organisation to show how any learning undertaken on L&D events has been transferred back into the employee's role and work area - these can include performance goals and how new skills and knowledge have been used.

Engagement - where L&D can demonstrate how stakeholders are engaged with learning, this can be at an organisational level where a positive learning environment is the goal, at team levels or at an individual level (the ‘happy sheet' is an individual reaction to an individual event).

The seminal model for L&D evaluation, first published in the 1950s by US academic Don Kirkpatrick remains influential today. However, research conducted by Thalheimer indicates this model was first introduced by Raymond Katzell.

It outlines four levels for evaluating learning or training:

Reactions - reaction to a learning intervention that could include ‘liking or feelings for a programme'.

Learning - ‘principles, facts etc absorbed'. Behaviour - ‘using learning gained on the job'. Results - ‘increased production, reduced costs, etc'.

This was helpful guidance when launched. However, in the 1980s Alliger and Janak found that the relationships between the levels were weak because each level is not always linked positively to the next.

Various surveys from the Association for Talent Development have found that most attention is focused on evaluation of learning at the reactions level because of the difficulties and time costs of measuring the other three levels. Thalheimer suggests eight recognised levels of learning evaluation, including some listed above, but he argues that some of these are highly ineffective.

Brinkerhoff success case method
A key criticism of Kirkpatrick's evaluation model is that changes to performance cannot solely be linked to learning. The Brinkerhoff success case method (SCM) addresses this challenge by proposing a wider focus on systems.

Firstly, an SCM evaluation involves finding likely ‘success cases' where individuals or teams have benefited from the learning. These typically come from a survey, performance reports, organisational data or the ‘information grapevine'. Those representing potential ‘success cases' are interviewed and ‘screened' to find out if they genuinely represent verifiable success with corroborating evidence from other parties. Factors that contribute to success beyond the learning intervention are also explored.

Secondly, an SCM evaluation looks at ‘non-success cases' to discover those who have found little or no value from the learning. Exploring the reasons why can be very illuminating.
The approach asks four questions:
How well is an organisation using learning to improve performance?
What organisational processes/resources are in place to support performance improvement?
What needs to be improved?
What organisational barriers stand in the way of performance improvement?
Following analysis, the success and non-success ‘stories' are shared.

SCM should not be seen as comprehensive evaluation method because of the nature of the sampling, but it offers a manageable, cost-effective approach to determine success insights and areas for improvement.

Relevance: How existing or planned learning provision will meet new opportunities and challenges for the business.

Alignment: If the L&D strategy takes an integrated blended approach, it's critical for L&D practitioners to work with stakeholders about what their performance needs and how to achieve them. Aligning with broader organisational strategy gives focus, purpose and relevance to L&D.