Myth #10: An evaluation conversation is only necessary for high-profile initiatives

Misalignment of instructional design and training evaluation practices is a major barrier to transferring learning and improving workplace performance. Therefore, every training project should include evaluation-focused conversations to help guide the purpose, design, and impact of your instructional activities.

Also see: Myth #3: Assessments are not essential to effective training

Reality

Asking your stakeholders the following questions can help ensure your training activities are aligned with a clear evaluation strategy. 

Evaluation-focused questions

    • What are the top business goals and metrics we want impacted by this training solution? (Tier 8)

    • What are the top human performance metrics that should be impacted by this training solution? (Tier 7)

    • Do we want learners to effectively DO the critical tasks (includes decision-making and knowledge) to achieve the performance metrics? (Tier 6)

    • Do we want learners to just show that they can make better decisions based on what they learned? (Tier 5)

    • Do we want learners to just comprehend specific foundational concepts and guidelines? (Tier 4)

  • Do we want to show clear and isolated impact of the training program? If yes, then requires an A/B test (control-group vs. treatment group) where we withhold or delay the intervention from the control group and measure the difference in impact to task competence and workplace performance.

Following are the tiers of Will Thalheimer’s Learning-Transfer Evaluation Model (LTEM). The tiers are ordered from top to bottom, higher validity of success (Tier 8) to lower validity of success (Tier 1). Tiers 4B to Tier 1 are inadequate measures to validate training effectiveness

Evaluation tiers

  • If learners were successful in transferring their learning (improving their workplace performance), what effects is that having on key business metrics?

    Things to keep in mind:

    • Typical business goals include improving service, increasing revenue, avoiding costs, and lowering turnover.

    • This type of analysis is typically performed for only strategically-important initiatives unless the organization already has a mature data pipeline mechanism in place.

    • This can also involve a cost-benefit analysis or ROI study.

  • Are they successful in applying what they’ve learned back to the job?

    Things to keep in mind:

    • This should be the central goal of training.

    • Example human performance metrics include Average Handle Time (AHT), First Call Resolution (FCR), Sales Rate, Quality Score, and more.

    • Can be evaluated as either assisted transfer (transfer learning with support via coaching, nesting, or shadowing) or full transfer (without need for support).

  • Can they effectively DO the tasks or capabilities they were taught?

    Measured by performing realistic tasks within practice/feedback activities and performance-based assessments. Includes application, decision-making, use of systems, and leverage of resources.

    Things to keep in mind:

    • For more accurate results, assessments should be delayed several days after the learning experience

  • Do they know what decision to make?

    Things to keep in mind:

    • Participants practice and receive feedback in making realistic decisions about what to say or do given relevant work scenarios.

    • Scenarios can be text based or can involve video and audio. 

  • Have they fully comprehended the knowledge (foundational concepts and principles) central to work decisions?

    Things to keep in mind:

    • Knowledge recitation - Short-term remembering. Involves testing within the same day in which learning took place.

    • Knowledge retention - Long-term remembering. Involves delays of three or more days after learning took place.

  • Can they regurgitate basic facts or terminology NOT central to work decisions? (e.g., recall the history of the company)

  • Using performance-focused survey questions, participants are asked whether they feel prepared to perform on the job, comprehend the concepts, well supported with on the job resources, etc.

    Note: Can hint at learning success (subjective)

  • Using satisfaction-focused survey questions, participants are asked whether they like, are satisfied, or willing to recommend the learning experience.

  • Participants pay attention and engage in classroom activities, online discussions, and in accessing online resources.

  • Participants enroll, start, or complete the training (i.e., enrollment or attendance records).

  • Can the training team build or facilitate with less time or resources?

    Instruction is efficient to the degree that effectiveness (based on evaluation purpose) is achieved in the least amount of instructional time necessary. 

Resources