|
Best Poster Award: It’s a TIE!
Alison Freeland
Efficacy and Success of Transition to Online eLearning of an Integrated Medical Psychiatry Collaborative Care Training Program
Authors: Alison Freeland, MD, FRCPC, Sanjeev Sockalingam MD, MHPE, FRCPC, Gurpreet Grewal, BEd (ADED), David Wiljer, PhD
Institution: Trillium Health Partners
Purpose/Problem Statement/Scope of Inquiry
Initially designed as an interactive in-person interdisciplinary learning experience, the Medical Psychiatry Collaborative Care Certificate (MP3C) program had to make an accelerated jump to a virtual learning environment as the result of the COVID-19 pandemic. This evaluation explored whether online participants report similar course evaluation scores compared to in-person participants.
Approach(es)/Research Method(s)/Educational Design
Evaluation data, using a standardized tool, was gathered after each online module was completed, and results were compared to those similarly obtained from in-person participants.
Evaluation/Outcomes/Discussion
315 in-person participant and 33 online participant evaluations were received. In both groups, 90% stated modules were a valuable use of their time, 88% strongly agreed/agreed that the courses enabled them to describe and use evidence based assessment tools and 88% would recommend to a colleague.
The MP3C program has made a successful transition to a sustainable, scalable online model. Evaluation feedback to date has demonstrated equivalent levels of satisfaction with the virtual learning experience as compared to those attending in person.
This evaluation reports quantitative findings of participants shortly after completion of each learning module, but does not yet include qualitative feedback provided by participants (currently being synthesized).
Key Learnings for CME/CPD Practice
As the COVID-19 pandemic settles, there will be an expressed desire for many to return to in-person learning. The findings of this evaluation highlight the benefits that maintaining online learning can bring, and support the potential for using CPD strategies which may provide more equitable access for a larger number of diverse health care professionals to acquire new knowledge and skills to impact needed health system change.
James Macaskill
Assessing Outcomes of Saskatchewan’s Practice Enhancement Program
Authors: James Macaskill, Bsc, Andries Muller, MBChB, PhD
Institution: University of Saskatchewan
Purpose/Problem Statement/Scope of Inquiry
This research synthesizes outcomes of Saskatchewan’s Practice Enhancement Program (PEP) over the last 24 years. The PEP is designed to provide physicians with patient and peer feedback on their clinical practice. In doing so, the program serves as an important means of promoting reflection and continued medical education. While the PEP has been delivering this service since 1994, a comprehensive program evaluation has never been done.
Approach(es)/Research Method(s)/Educational Design
We conduct a retrospective review of 825 practice assessment reports and over 4241 individual recommendations over the last 23 years, from 1997 to 2020, following the structure of PEP’s assessments. While the PEP evaluates the practices of various medical specialties, this project focused on the evaluations of family physicians. The PEP organizes its reports into three main categories:
A physician is ranked ordinally in each of these three categories based on the PEP’s in-person assessment. In addition to the categories described above, almost every PEP report includes a list of personalized recommendations for the physician under evaluation. We use an inductive approach with thematic analysis to create subcategories to group the 4241 recommendations. Data from these assessments are collected, organized, and analyzed to determine: (i) What areas need improvement? (ii) Is there any other information that results from the data?
Evaluation/Outcomes/Discussion
We first assess the descriptive statistics of our dataset and find that roughly 38 percent of the physicians considered by the PEP are graduates from a Canadian medical school. The remaining 62 percent are international medical graduates now practicing in Saskatchewan, who account for most of the province's rural physicians. Similarly, urban practices – defined as any clinic located in Saskatoon, Regina, or Prince Albert – account for approximately 56 percent of clinical assessments. One of the most considerable demographic disparities is sex, with male physicians making up 68 percent of total assessments done to date (2020). Approximately one-third of clinics incorporated a walk-in component to their practice. The majority of assessed physicians work more than 40 hours per week and see an average of 32 patients per day or 160 per week.
We next consider the broader outcomes of the PEP’s assessments as reported in its three main categories: Quality of Care, Structure and Organization of Medical Records, and Physical Facilities and Practice Organization. Structure and Organization of Medical Records contained the greatest proportion of improvement required and corresponds to the PEP’s total recommendations overall. However, this number has not remained stagnant over time. From 1997 to 2009, most improvements required came from deficiencies in medical record-keeping, accounting at its peak for 71% of all clinics assessed to be below average. Quality of Care consistently trends upwards, eventually overtaking Structure and Organization of Medical Records after 2010 as the greatest proportion of recommended improvements.
Several themes are identified as common reasons for recommendations. The top recommendation is around the need for improvements in documentation. Greater chronic disease management is the second most common recommendation, with 521 reports citing the need for improvement, followed by better cumulative patient profile updating, at 404. Thus, improvements in the medical record account for two of the three most common recommendations made in PEP reports. However, the prevalence of these two recommendation categories – documentation improvement and the use of cumulative patient profiles – has fallen over time. This drop may be due to the adoption of electronic medical records (EMRs), which significantly improve both legibility and accessibility of patient documentation. This change makes sense; PEP's recommendation to transition to or review management of the EMR increased slightly in the latter half of its reporting while the proportion of medical record recommendations fell, from 71% of all substandard assessments in 2000-2004 to 26% in 2015-2019. The rise of the EMR also aids in explaining the adverse effect of age and years of experience on documentation quality, with technology potentially serving as a barrier to adoption.
Other recommendations that made the top ten list include: medication prescription, emergency preparedness, laboratory investigations, objective measurements of pulmonary function, patient privacy, patient safety and depression counselling and care.
This research utilizes Chi-squared and Student's T-tests to assess the significant determinants of Quality of Care, Structure and Organization of Medical Records, and Physical Facilities and Practice Organization when controlling for data collected from PEP’s practice reporting over time. Student’s T-test is used for any continuous variables of interest, with Chi-squared cross-tabulating categorical variables in the dataset to test for any significant correlations. Significance is set at the ninety-ninth percent confidence threshold.
In general, female physicians are less frequently deemed to need improvement than their male counterparts. We also find a difference between assessment performance and other demographic indicators, such as age and clinic arrangements. We show that physicians' age and years in practice correlate with most of the deficiencies identified by PEP assessments. Furthermore, physicians in group practice generally perform better among accessed recommendation categories than those who are not. However, this is not always the case. Those in group practice fared worse than those in solo practice in chronic disease management, objective measurements of pulmonary function, depression counselling and care – three of the ten most common recommendations made by PEP.
Like practice arrangements, there exists a trend between faculty appointments or teaching involvement and the likelihood that a physician received improvement recommendations in each evaluated category. Physicians holding faculty positions perform better generally than colleagues who did not. However, in the categories of objective measurements of pulmonary function and depression counselling and care, those who occupy faculty positions are more likely to receive recommendations than those who do not. A more robust predictor of performance appears to be physician teaching. Those active in undergraduate or resident education always perform better when a significant correlation between recommendations is found. In other words, we find no instances of teaching being detrimental to assessment outcomes.
Key Learnings for CME/CPD Practice
Saskatchewan’s primary care is generally very good; most physicians assessed by the program either met or exceeded practice standards. While EMR adoption has aided in documentation quality, deficiencies in medical recordkeeping account for two of the three most common recommendations made by PEP and is still an area in need of improvement. Lastly, a physician’s practice profile influenced their PEP assessment performance, with teaching, faculty positions, and group involvement improving outcomes, on average.
Best Oral Abstract Presentation Category
Best Practice
Sanjeev Sockalingam
Adapting the Deteriorating Patient Simulation Method: Notes on the feasibility and acceptability of a virtual mental health simulation
Authors: Sanjeev Sockalingam, MD, MHPE, Noah Brierley, BA, Amanda Gambin, PhD, Thiyake Rajaratnam, MSc, Anne Kirvan, MSW, Michael Mak, MD, Chantalle Clarkin, PhD, Stephanie Sliekers, MEd, Fabienne Hargreaves, MA, Sophie Soklaridis, PhD, Allison Crawford, MD, PhD
Institution: The Centre for Addiction and Mental Health
Purpose
Continuing professional development (CPD) programs in mental health are critical to building primary care teams’ capacity in mental health care. However, healthcare professionals often report barriers such as access to CPD and cost which limit uptake. Deteriorating Patient Simulation (DPS) is a team-based simulation activity that aims to mimic real-life medical situations that deteriorate over time. It involves an instructor acting out a patient scenario that deteriorates as the simulation progresses, and requires the learners to determine the appropriate steps to stabilize the patient. Throughout the process, the instructor facilitates the deterioration while simultaneously offering content knowledge. DPS was originally developed to support medical trainees in emergency medicine; the overall goal of this simulation activity is to promote learning by eliciting evidence based decision-making in a realistic scenario. DPS has not been used in a virtual setting or within a mental health context; as such, this study aimed to evaluate the feasibility and acceptability of a virtual mental health DPS educational activity with primary care teams.
Approach
The study was conducted in two phases: Phase 1 focused on the development of patient scenarios and a competency checklist as well as the feasibility and acceptability of delivering the simulation virtually. Phase 2 sought to refine the checklist and evaluate inter-rater reliability. During both phases, post-DPS focus groups were conducted to understand participants’ perceptions of the activity.
Two patient scenarios were developed using an interprofessional collaborative approach; evidence-based best practice guidelines were used to develop scenarios focused on a major depressive episode and suicide risk assessment. Each scenario had a unique algorithm for deterioration based on the teams’ decision-making. The competency checklist was developed by content expert review including the primary investigators and mapped to case scenario goals and objectives.
Participants were previous participants of ECHO Ontario Mental Health at CAMH and University of Toronto. Generally, participant groups consisted of at least 1 prescribing professional (i.e. MD, NP) and 2 other healthcare professionals. DPS sessions were delivered via Zoom and were typically 2 hours in duration: the 60-minute DPS activity was followed by a facilitator-led debrief (30 minute) and a focus group (30-minute). Participant performance was scored by a third party rater using the competency checklist developed by the team.
Evaluation
Seventeen (17) healthcare providers (3 MD, 3 NP, 5 SW, 2 RN, 2 other) participated across 6 DPS sessions. Moderate inter-rater reliability was established for the DPS checklist (66% agreement). Focus group data highlighted barriers and facilitators to the activity. Participants responded positively to the structure and facilitation of the activity, and emphasized the simulation-based feedback from the post-activity debrief as integral to their learning. The DPS activity was described as valuable and appropriate for skill building, and the team-based nature of the activity provided an important interprofessional lens to patient management. Participants noted time constraints, as well as the formation of “artificial teams” specifically for this activity as challenges.
Key Learnings for CME/CPD Practice
This pilot study provides evidence to support the potential use of virtual DPS as an accessible, low barrier, CPD activity to increase provider competency in mental health clinical decision-making. Further development of this activity could include use as an educational assessment tool, and to increase mental healthcare capacity in primary care teams.
Sophie Soklaridis
Pivoting continuing professional development during the COVID-19 pandemic: A scoping review of emerging and amplified trends and innovations
Authors: Sophie Soklaridis, PhD, Mushfika Chowdhury, Paul Mazmanian, MD, Martin Tremblay, PhD, Mary G. Turco, EdD, Betsy Williams, PhD, Reena Besa, MLIS, Sanjeev Sockalingam, MD
Institutions: Centre for Addiction and Mental Health, Toronto, Ontario, Canada; Department of Medicine, Dartmouth-Hitchcock Medical Centre/Geisel School of Medicine at Dartmouth, Lebanon, NH, USA; Fédération des médecins spécialistes du Québec, Montréal, Québec, Canada; Department of Preventive Medicine and Community Health, Virginia Commonwealth University, Richmond, Virginia, USA; Professional Renewal Centre, Lawrence, KS, USA.
Purpose/Problem Statement/Scope of Inquiry
With limitations on travel and physical distancing, a burgeoning of new information about the management of COVID-19 and an increase in clinical workload, there has been a swift effort to accelerate the development of virtual teaching and learning environments to continue offering CPD throughout the pandemic1. The urgency for rapid and novel adaptations to CPD resulted in both an opportunity for innovations and a deep contemplation on which trends will continue after the most acute phases of the pandemic. In this narrative scoping review, we explored the adaptations and innovations in CPD, strengthened or newly created, due to the COVID-19 pandemic. The purpose of our narrative review was to use a systematic approach to synthesize the findings of studies from March 2020 to July 2021 to examine how this unprecedented time led to adaptations and novel ways of thinking that now define the landscape of CPD.
Approach(es)/Research Method(s)/Educational Design
We used the Arksey and O’Malley’s scoping review framework2 and the PRISMA extension for scoping reviews to ensure the reporting and synthesizing of the evidence met the current standards for reporting on knowledge synthesis research. Our review added to both Gordon et al.’s (2020)3 systematic review of the developments in medical education in response to the COVID-19 pandemic (to May 2020) and the updated publication of this review by Daniel et al (2021)4 (to September 2020). We included all article types such as opinion, commentaries, and editorials- which were excluded in the previous two reviews- because these types of articles provided perspectives from experts in the field regarding current issues and where the field of CPD may evolve in the near future. Our research addressed the following question: What do studies in CPD show us about trends and innovations as a result of COVID-19? Our scoping review answered the following questions: 1) what types of and adaptations to CPD innovations are described? 2) what are the future innovations in CPD?
Evaluation/Outcomes/Discussion
We identified 190 articles to include in our final analysis. Of the total 190 articles included, less than one fourth offered a theoretical or evaluation framework. We identified 123 articles that described three types of adaptation to CPD innovations: 1) The creation of new on-line resources, 2) The increased use of existing online platforms/software to deliver CPD, and 3) The use of simulation for teaching and learning. We identified 65 articles that described or explained five future innovations that will shape CPD initiatives beyond the pandemic: 1) empirical research on effectiveness of virtual learning; 2) novel roles and ways of thinking; 3) learning from other health disciplines and moving beyond medicine; 4) formation of a global perspective; and 5) emerging wellness initiatives for health professionals.
Key Learnings for CME/CPD Practice
1. Thakur A, Soklaridis S, Crawford A, Mulsant B, Sockalingam S. Using Rapid Design Thinking to Overcome COVID-19 Challenges in Medical Education. Academic medicine : journal of the Association of American Medical Colleges. 2020;01(PG-). doi:http://dx.doi.org/10.1097/ACM.0000000000003718.
2. Arksey H, O’Malley L. Scoping studies: towards a methodological framework. International Journal of Social Research Methodology. 2005;8(1):19-32. doi:10.1080/1364557032000119616.
3. Gordon M, Patricio M, Horne L, et al. Developments in medical education in response to the COVID-19 pandemic: A rapid BEME systematic review: BEME Guide No. 63. Medical teacher. 2020;42(11):1202-1215. doi:10.1080/0142159X.2020.1807484.
4. Daniel M, Gordon M, Patricio M, et al. An update on developments in medical education in response to the COVID-19 pandemic: A BEME scoping review: BEME Guide No. 64. Medical teacher. 2021;43(3):253-271. doi:10.1080/0142159X.2020.1864310.