Experts Weigh in on Measuring Learning Success

Experts Weigh in on Measuring Learning Success
Thought Industries
November 5, 2021

As learning professionals, one question comes up again and again: How can we measure and showcase the impact of our learning programs? This is a huge topic, and at COGNITION 2021, we put it firmly under the spotlight with a customer roundtable hosted and moderated by Kerry Kaczorowski from Vertex Inc.

In Measuring the Success of Learning Programs, Kerry opened the floor to many fellow customer education professionals and asked, “what metrics do you use to measure the impact of your training, and what’s standing in the way of creating the story that you want to tell?” Here are some of the best practices that came out of the session!

Align your outcomes with the business at the start

Kerry started out by commenting that “the word “impact” in and of itself is a pretty neutral term.” Anything will have an impact, but it’s up to us as learning professionals to figure out how we’re going to tell the story of what that impact is and why it matters. That story will often be split into two elements: the impact on the learners, and the impact on the business.

You can get to the end of a project and feel very strongly that it’s been a home-run, but then when you check in with managers or even executives, you realize you’ve missed the mark. This often comes from misalignment at the start, where you’re looking to measure the impact on the learner or of the course – while executives higher up the chain are looking for big-picture impact metrics; how your training has affected the business unit or the organizational bottom line. In this way, impact is relative to context. You can show a massive improvement in customer satisfaction – but if the business goal was to bring on new customers, your impact doesn’t mean that much.

Takeaway: Set the right expectations for the learning program from the start, defining impact both in terms of learning objectives for the end user, and how that ties into underlying goals for the company.

Measure what matters

Once you have a clear idea of what successful impact would look like for you, start gathering the right metrics to see whether your training is moving the right needle. Start by being clear about what you need to be measuring. There are four tiers of impact that you could be tracking:

  • Learner: Here’s where you ask, did learning happen? To what extent? Has learning been completed? 
  • Course: Now – how effective was the course? How engaged were your learners? Do we need to make improvements? Did it deliver on the learning objectives?
  • Business unit: Think bigger. Are you meeting your departmental goals? Does your education have an impact on departmental goals, like support ticket deflection or product adoption? 
  • Bottom line: Finally, and critically, what are you contributing to the organization? If you’re a profit center, are you increasing revenue? If you’re a cost-center, have you decreased expenses? How are you feeding into overall strategy?

While many organizations are great at measuring at a learner or a course level through qualitative feedback or via an LMS, it’s easy to come unstuck when you start moving deeper through the levels. Make sure that you ascertain whether there are multiple tiers and therefore metrics which you need to be tracking, for example completion rates at the learner level, and then product adoption or churn for aligning with the company’s overarching strategy.

Takeaway: Don’t get lost in meaningless data, look to measure what matters at each tier. Remember, if you aren’t sure what matters on a company-wide level – ask!

Have a strategy in place to harness the right data

If you feel like you’re not “measuring what matters” yet, you’re far from alone. In fact, 77% of our audience commented that they aren’t yet measuring the right data in order to show the impact of learning in their organization, and 66% feel they can’t access the right metrics from the data that they have.

Sometimes this can feel like a moving target, as company goals change quickly, and what you were measuring a year ago might not be the right metric to be following now. 

Here’s where you can utilize the four tiers of impact from learner to bottom line, and decide what metrics would tell each story best. For example, if your company’s goal is to improve product adoption, then you might be looking at reducing Time-to-value. In contrast, if it’s about freeing up support reps, then you’ll want to collect data on ticket deflection. At the same time, if you want to track engagement on a learner level to prove that you’re meeting your own targets, you’ll likely continue collecting data on course churn or completion rates, or qualitative feedback on satisfaction.

One of the big challenges for our audience was aggregating and warehousing this data into a single source of truth, especially when multiple systems were involved, such as an LMS, a CRM, repository for support data, and more. As the customer education field grows, tools that allow for simple integration with a complex range of data sources will be increasingly beneficial, so that your system can grow alongside you over time. 

Takeaway: Carefully consider what metrics will align best with the story that you want to tell, and look for data that displays results on each tier to meet your requirements.

Think about how you’re going to present the data that you collect

Ultimately, the success of measuring the impact will come down to how well you report back. That doesn’t mean that you need to have an overwhelmingly positive result. In fact, sometimes a negative impact can explain where there are gaps or opportunity areas. Try to eliminate your opinion from the data you’re presenting, and focus on the story that the information is telling.

Think about ways to make the data you’re using more relatable. Don’t simply present stats and percentages, explain to your audience what they actually mean by comparing this data to last year’s numbers, or to a specific target or hypothesis that you started out with.

It’s also important to consider who you’re talking to, because while measuring impact across all of the tiers that we’ve mentioned is useful, different data might be more or less useful for different audiences. For example, while your course data might be great for a conversation with your Director of Customer Education, executives will be more likely to want to know how you’re impacting big ticket items such as churn, retention, adoption and more. Kerry made us laugh when she commented that “You get invited to a lot more of the interesting meetings if you can tie your impact back to the business!”

Takeaway: Consider who you’re talking to when you present your data, and steer away from opinion – focusing instead on the facts.

Want more information on measuring the impact of your learning programs? Watch the full session to learn:

  • What metrics today’s customer education professionals are using, and how impactful they are in real-world settings
  • How to integrate each of the tiers, from learner success to strategic impact, into the Kirkpatrick model of evaluation
  • And, the challenges that customer education professionals are facing, including setting attainable milestones and deciphering multiple layers of data.

For your invitation to this roundtable, plus an all-access pass to all of the on-demand recordings from COGNITION, register here.

post dave webinar

Identifying Upcoming Customer Education Revolutions for 2024

Read More

Thought Industries Named #1 Global Learning System for Second Consecutive Year

Read More