May 16, 2013By Lance Baily

How to Evaluate Effectiveness of Clinical Simulation

Evaluating the effectiveness of your medical simulations is a crucial component of building a successful long-term program.  On numerous occasions, the raw data the healthcare simulation team captured at the Clinical Simulation Center of Las Vegas over a period of many years eventually became critically important for arguments when requesting operational changes our lead administrators.  Not only this, but the use of a tool to gauge how simulated scenarios have performed is a necessary step in molding and shaping future experiential learning experiences.

Simulation scenarios should not ever be considered “final drafts” for the following reasons:

1) New clinical procedures or information may shift scenario goals.
2) New faculty may wish to modify existing templates to better angle their learning objectives.
3) Annual review by a simulation committee may provide for refined scenario designs which further empower learners.

Sponsored Content:

The last point is best explained through the Japanese philosophy of Kaizen, which means “never-ending improvement”, a passion for making things better, the drive for perfection. In his book, “Kaizen”, Masaaki Imai says, “It means continuing improvement in personal life, home life, social life, and working life. When applied to the workplace Kaizen means continuing improvement involving everyone–managers and workers alike.” Healthcare is a dynamic process, one that is always changing and reaching for improvement – and so should our simulation scenarios!

“Healthcare Kaizen focuses on the principles and methods of daily continuous improvement, or Kaizen, for healthcare professionals and organizations. Kaizen is a Japanese word that means “change for the better,” as popularized by Masaaki Imai in his 1986 book Kaizen: The Key to Japan’s Competitive Success and through the books of Norman Bodek, both of whom contributed introductory material for this book.

In 1989, Dr. Donald M. Berwick, founder of the Institute for Healthcare Improvement and former administrator of the Centers for Medicare & Medicaid Services, endorsed the principles of Kaizen in the New England Journal of Medicine, describing it as “the continuous search for opportunities for all processes to get better.” This book shows how to make this goal a reality.”

Sponsored Content:

So how to evaluate our healthcare simulations? CAE Healthcare is hosting this free reliable and validated Simulation Effectiveness Tool – Modified (SET-M) which was updated in 2015 by Kim Leighton and Team: Leighton, K., Ravert, P., Mudra, V., Macintosh, C. (2015). Updating the Simulation Effectiveness Tool: Item modifications and reevaluation of psychometric properties. Nursing Education Perspectives, 36(5), 317-323. Doi:10.5480/1 5-1671.

simulation effectiveness tool

Evaluation Tools like this that are handed out to your concluding simulation participants can help track the effectiveness of the engagement from the learners perspective. Use the Simulation Effectiveness Tool – Modified as a guide to building your own surveys which reflect the specific evolutionary needs of your program.

The site offers an online repository of free tools created by a collection of healthcare simulation educators and researchers designed specifically for evaluating various aspects of SBE. Today, simulationists must go beyond measuring satisfaction and participant confidence, to instead engage in comprehensive program evaluation using tools with proven validity and reliability. Evaluating Healthcare Simulation currently features five such tools for use in evaluating the following aspects of a Simulation Program:

  • Facilitator Competency Rubric (FCR) to evaluate simulation facilitators based on a novice-to-expert competency scale,
  • Simulation Effectiveness Tool – Modified (SET-M) to evaluate the simulated clinical experience,
  • Simulation Culture Organizational Readiness Survey (SCORS) to evaluate leadership and organizational culture readiness to integrate simulation into the curriculum,
  • Clinical Learning Environment Comparison Survey (CLECS) to evaluate how well learning needs are met in the traditional and simulation undergraduate clinical environments, and the
  • ISBAR Interprofessional Communication Rubric (IICR), to evaluate student’s communication with a physician; evaluating and measuring students’ communications.

To really measure effectiveness, however, your program will need to invest in a learning management or audiovisual system that integrates real-time performance analysis software.  Most of the major simulation product companies provide for some level of performance tracking in their av recording systems, but unless your simulation educators are actively recording this data while it occurs there will be no long-term program effectiveness evaluation. Administrators MUST understand the importance of capturing data through their learning management systems and audiovisual debriefing systems as much as possible in their simulation programs.

Not only should the effectiveness of your clinical simulation learning programs be evaluated, but also the use of your manikins, av systems, sim labs, surgical simulators, prebriefing discussions, simulation technician operational hours, and anything else connected with your simulation program. Such metrics will enable your simulated program to evolve over time. In what ways exactly, may only reveal themselves in the future when the need is finally identified, and having the data will enable for the best possible decision making!

Additional Articles Regarding the Effectiveness of Medical Simulation:

Do you have another way to evaluate medical simulation effectiveness? Leave a comment at the bottom of this post’s webpage!

Sponsored Content: