How to Evaluate Effectiveness of Simulation
Evaluating the effectiveness of your medical simulations is a crucial component of building a successful long-term program. On numerous occasions, the raw data we captured at the Clinical Simulation Center of Las Vegas over a period of many years eventually became critically important for arguments when requesting operational changes our lead administrators. Not only this, but the use of a tool to gauge how simulations have performed is a necessary step in molding and shaping future experiential learning experiences.
Simulation scenarios should not ever be considered “final drafts” for the following reasons:
1) New clinical procedures or information may shift scenario goals.
2) New faculty may wish to modify existing templates to better angle their learning objectives.
3) Annual review by a simulation committee may provide for refined scenario designs which further empower learners.
The last point is best explained through the Japanese philosophy of Kaizen, which means “never-ending improvement”, a passion for making things better, the drive for perfection. In his book, “Kaizen”, Masaaki Imai says, “It means continuing improvement in personal life, homelife, social life, and working life. When applied to the workplace Kaizen means continuing improvement involving everyone–managers and workers alike.” Healthcare is a dynamic process, one that is always changing and reaching for improvement – and so should our simulation scenarios!
Check out this book I’m reading on Healthcare Kaizen:
“Healthcare Kaizen focuses on the principles and methods of daily continuous improvement, or Kaizen, for healthcare professionals and organizations. Kaizen is a Japanese word that means “change for the better,” as popularized by Masaaki Imai in his 1986 book Kaizen: The Key to Japan’s Competitive Success and through the books of Norman Bodek, both of whom contributed introductory material for this book.
In 1989, Dr. Donald M. Berwick, founder of the Institute for Healthcare Improvement and former administrator of the Centers for Medicare & Medicaid Services, endorsed the principles of Kaizen in the New England Journal of Medicine, describing it as “the continuous search for opportunities for all processes to get better.” This book shows how to make this goal a reality.”
So how to evaluate our healthcare simulations? CAE Healthcare is hosting this free reliable and validated Simulation Effectiveness Tool – Modified (SET-M) which was updated in 2015 by Kim Leighton and Team: Leighton, K., Ravert, P., Mudra, V., Macintosh, C. (2015). Updating the Simulation Effectiveness Tool: Item modifications and reevaluation of psychometric properties. Nursing Education Perspectives, 36(5), 317-323. Doi:10.5480/1 5-1671.
Evaluation Tools like this that are handed out to your concluding simulation participants can help track the effectiveness of the engagement from the learners perspective. Use the Simulation Effectiveness Tool – Modified as a guide to building your own surveys which reflect the specific evolutionary needs of your program.
To really measure effectiveness, however, your program will need to invest in a learning management or audiovisual system that integrates real-time performance analysis software. Most of the major simulation product companies provide for some level of performance tracking in their recording systems, but unless your simulation educators are actively recording this data while it occurs there will be no long-term program effectiveness evaluation. Administrators MUST understand the importance of capturing data as much as possible with everything possible in their simulation programs.
The types of data and how to capture them were covered really well during the 1.5 hour “Research” session of HealthySimAdmin – an HD-recorded two day event which sought to provide solutions for the “how-to” of managing a simulation program. You can still watch this and the remaining 13 hours of administrative content straight to your office or home. Get instant access to the opening session by subscribing to HealthySimulation.com’s free newsletter, and learn more about how HealthySimAdmin can guide your program today!
Do you have another way to evaluate medical simulation effectiveness? Leave a comment at the bottom of this post’s webpage!