PAEA End of Rotation™ Update
This time most years, PAEA is announcing an upcoming publication date for a new version of End of Rotation™ exams. However, last year we temporarily waived our requirement for high-security in person or remote proctoring due to COVID-19. Based on data from the PAEA COVID-19 Rapid Response Report 3, more than a third of programs using the End of Rotation exams still report administering our exams at home without a proctor or via web conferencing technology. These two proctoring methods do not meet our pre-COVID high-security proctoring policy. Therefore, proctoring policies for End of Rotation exams will remain waived and Version 7, the current live version, will remain as the active, available version. Version 7 will remain live until we are ready to release new system features and policies that allow for secure administration across all programs again. All Version 7 forms are undergoing periodic review for content and scoring analysis.
Given this we want to review common questions regarding the PAEA End of Rotation exam, surrounding COVID-19 pandemic.
WHERE DOES THE NATIONAL DATA FOR THE PAEA END OF ROTATIONS EXAMS COME FROM?
PAEA End of Rotation exam national comparative data remain stable year-round. National comparative data are based on the performance of a reference population of student test-takers, not current student test-takers. That reference population took End of Rotation exams in past years with either live in-person or live remote proctors.
Under the current, temporarily waived proctoring policies for the End of Rotation exams, national data will not be influenced. This is because the national comparative data, found on our website and on End of Rotation exam score reports, are from a reference population. Therefore, it will remain unchanged.
HOW HAS STUDENT PERFORMANCE CHANGED DURING COVID-19?
Based on Figure 1 below, we do see a minimal increase between pre and post COVID-19 mean student test-taker scores.
We could postulate that the increase in scores are related to exposed content (i.e. unproctored administrations) and/or students having increased time to study and prepare for exams (while home from clinical rotations). However, when comparing Version 6 to Version 7 within the same time frames, there are also similar, minimal increases in student test-taker mean scores during the same Version 6 administration intervals. While we cannot be certain the reason for the increases in exam scores for both Version 6 and Version 7 over time, initial analysis of the data shows that waiving the exam administration policies had a negligible impact on Version 7 scores overall.
SHOULD WE CHANGE HOW WE INTERPRET SCORES AT THE LOCAL LEVEL?
Waived proctoring policies has resulted in a spectrum of local exam administration practices. Some of these local practices continue to involve high security, while some involve less security. If your program has elected exam administration practices that involve little-to-no security, you may not want to compare your students to the national comparative data, which was gathered from a reference population, who took the exams under live-proctored controlled environments. You also may want to use caution when comparing pre covid cohorts to current cohorts, as the pre covid cohorts took the exam in a live-proctored environment.
If you are currently administering exams in an unproctored and/or non-secured environment, you may want to consider factors that can influence test scores such as a student using a textbook during the exam. If this is the environment that is best suited to your program’s current needs, considerations should be made to compare exam scores at the local level to previous local cohorts or to the national comparative data. Programs can still compare scores, but they must take into consideration that the test environment is different than previous End of Rotation exam administrations, which may have a positive or negative impact on scores.
Aside from proctoring, there are other factors that can influence scores. Almost everyone, whether they are aware of the feeling or not, are in varying states of stress and burnout due to the pandemic that has being going on for more than a year. High-stake exams, in typical environments, are stressful for students. Combined with the daily stressors students may be facing related to COVID-19, their performance could be impacted. Consideration should be given to students and the impact current events may have on their scores during this time.
SUMMARY
As with all our Assessment products, we will continue not only to ensure they are of high quality and rigor but communicate with our members regularly on any changes that may be forthcoming. To date, 80% of our member programs are utilizing the PAEA End of Rotation exams, and we continue to see that number grow. As the circumstances around us continue to evolve, we hope to have Version 8 available to our members as soon as feasibly possible, and we will give you advance notice as changes occur.
