End of Rotation Exams Moving to Scale Score Reports
Year after year, more programs administer these exams to their students as one set of objective, validated evaluations of the medical knowledge gained in seven core clinical areas. (Those areas are family medicine, internal medicine, emergency medicine, general surgery, women’s health, pediatrics, and psychiatry/behavioral medicine.) Today, 152 programs collectively administer approximately 50,000 End of Rotation exams a year.
End of Rotation exams have at least two forms, and national comparative data are available for each form. Each form is designed to measure the same content, but contains a different set of questions. While they are also built to be equivalent in difficulty, one form may be slightly more or less difficult than another. Despite our efforts to build equivalent forms, we recognized that it may be hard to use the raw “number correct” score for fair comparisons of students’ performance on different forms of the same exam version (for example, internal medicine version 5 forms 1 and 2).
Ideally, to make consistent and fair decisions based on End of Rotation exam results, the scores reported from different forms need to be equated. Due to the popularity of the exams and an extremely healthy adoption rate, we are now positioned to use a statistical process called “equating” to compensate for small variations in difficulty between exam forms.
For the publication of version 6, we will equate all forms by placing them on a single scale. The new scale will be called a “scale score” and students’ scores will range from 300 to 500. This is a big change. It will mean that instead of seeing a raw score, or number correct, in ExamDriver, you and your students will see a scale score.
Why the change? There are numerous benefits to scale scores, all of which make exam score analysis and interpretation easier and more accurate. Scale scores allow you to compare student scores between different forms of the same exam version. Additionally, scale scores will allow you to compare cohort-level data year-over-year.
We know that for many of you this change will occur in the middle of a clinical cohort — so when the change begins, we will have look-up tables available to help. These look-up tables will allow you to transform raw scores on previous exams to scale scores, and transform scale scores to raw scores for those cohorts currently in progress. Doing so will result in robust data that will help you assess your students from year to year.
Change is never easy, so we are here to help you through the transition to the new scale score reports. To help you understand the new reports, we will be hosting a webinar on Wednesday, May 16, at 1:00 p.m. ET and again on Tuesday, June 5, at 2:00 p.m. ET.
The goal of the webinar is to explain what scale scores are, how to interpret them, and what the new scoring method means for students, faculty, and program directors. Both webinars will cover the same content, and you can register for the one that fits your schedule here.
After their completion, the webinar content will be created into a module that will be easily accessible on our online learning management system. (For information on how to sign in to the learning management system, check out this article.)
If you have any questions in the meantime, please reach out to us at exams@PAEAonline.org.
