Passer au contenu de la page principale

Evaluating the quality of assessment in one Canadian ophthalmology residency program as an early adopter of CBME

Thème:
Santé publique et ophtalmologie mondiale
Quoi:
Paper Presentation | Présentation d'article
Quand:
11:50 AM, Dimanche 12 Juin 2022 (6 minutes)
Où:
Comment:

Authors: RachelCurtis, Christine Moon, Tessa Hanmore,Wilma Hopman, StephanieBaxter.

Author Disclosure Block: R. Curtis:None. C. Moon: None. T. Hanmore:None. W. Hopman: None. S. Baxter: None.

Abstract Body:

Purpose: To investigate the quality of feedback in ophthalmology resident trainee assessments since the introduction of competency-based medical education (CBME) at Queen’s University. Specifically, this study explored the utility of the Quality of Assessment for Learning (QuAL) scoring method of evaluating narrative feedback. This scoring tool was applied to assessments before and after faculty development initiatives to demonstrate which interventions resulted in feedback improvement, and if the quality of feedback improved over time. The QuAL score was also used to determine if evaluator comments associated with the Canadian Ophthalmology Assessment Tool for Surgery (COATS) procedural assessment forms were of higher value than other Entrustable Professional Activity (EPA) forms.

Study Design: Retrospective cohort study.

Methods: Ophthalmology resident assessment data from July 2017-December 2020 was retrieved from Elentra (Integrated Teaching and Learning Platform) and anonymized. Each assessment was organized by form type and stage-specific EPA. Written feedback was assigned a QuAL score out of 5 based on the previously validated rubric. All individual assessments were scored by an ophthalmology faculty member, and a randomized sample of 10% was independently rescored by an ophthalmology resident. Intra-class correlations were used to determine inter-rater agreement. A Linear-by-Linear test was applied to determine QuAL score improvement over time. Independent samples t-tests were used to compare pre- versus post- feedback interventions, and the COATS versus EPA forms.

Results: A total of 2617 individual assessments were graded. The ICC for the two independent graders was 0.90 (95% CI 0.88-0.92, p<0.001) for the total QuAL scores. The Linear-by-Linear test indicated a significant change over time (p<0.001). QuAL scores significantly increased after a targeted departmental grand rounds presentation aimed at improving coaching behaviors in feedback delivery, with pre- and post-intervention means of 1.64 and 2.61, p<0.001 respectively. COATS forms (n= 483) demonstrated a significantly higher QuAL score versus EPA forms (n=2134), with mean values of 3.48 vs. 2.26, p<0.001.

Conclusions: This study demonstrates that the QuAL score is a reliable and useful tool in evaluating narrative feedback quality in the context of CBME, with the potential to inform about the impact of specific faculty development initiatives. Procedural COATS forms with structured feedback prompts may be more effective in guiding evaluators to deliver comments on resident performance.

Conférencier.ère
Conférenciere invitée
Detail de session
Pour chaque session, permet aux participants d'écrire un court texte de feedback qui sera envoyé à l'organisateur. Ce texte n'est pas envoyé aux présentateurs.
Afin de respecter les règles de gestion des données privées, cette option affiche uniquement les profils des personnes qui ont accepté de partager leur profil publiquement.

Les changements ici affecteront toutes les pages de détails des sessions