We urge PA faculty to interpret and use the national comparison data for End of Rotation exams carefully and thoughtfully, as a variety of factors can influence individual and cohort-level outcomes.
Scale Score Reports
Starting in 2018, PAEA has moved to reporting scores using a scale rather than the number of correct responses to allow different test forms within the same year and across years to be comparable.
Multiple Forms
There are at least two forms available for each of PAEA’s seven End of Rotation exams. Extra forms offer programs the ability to retest students after remediation. Each form is designed to measure the same content; however, each form contains different sets of test questions. Forms are built to be equivalent in difficulty, but one exam form may be slightly more or less difficult than another, leading to potentially inappropriate comparisons.
Enhanced Process
To make consistent and fair decisions based on assessment results, the scores reported from different forms need to be comparable. Our psychometric team has employed a statistical process called “equating,” which compensates for small variations in difficulty between sets of test questions by placing all scores on a single scale. For more information on this change, see the following resources:
- Introductory article
- FAQ with sample reports
- Calculating Z scores video tutorial
- Case studies on creating grades
Interpreting National Cohort Data
PAEA does not set standards for End of Rotation exams, nor do we advise programs where to set their performance bars for passing scores. These need to be established by each program at the local level through statistical standard setting or other processes in accordance with program policies, learning objectives, and requirements. End of Rotation exams should be part of a broader supervised clinical practice experience (SCPE) evaluation; the weight and passing score of these exams may differ based on other components that your program employs. PAEA offers the following recommendations for approaching and interpreting End of Rotation exam score reports.
Look at Trends
For each exam, scale scores (and raw scores converted to the scale) can be compared between exam forms and across cohort years. Note that scores can only be compared within each specialty exam, not between specialty exams. We recommend that you look at your cohort data, examine your cohort trends year over year, and compare them to the national exam data and national trends year over year.
Interpret with Caution
Data on subscales should be interpreted with caution because they have a small number of questions with varying levels of difficulty. However, they have value if used to assess a content section on multiple exams. For example, if subscores in endocrinology are consistently low across many of the End of Rotation exams, you may want to reflect on your curriculum and determine if endocrinology is covered in sufficient detail during the didactic phase of the program.
Make Data-Driven Decisions
There are lots of ways to incorporate cohort level data into program decision-making. One example is if you have a cohort of students that is consistently performing poorly on the family medicine exam where all the students have been to the same clinical site. An action plan may be to reach out to the site and determine if the site is covering a variety of disease conditions.
Student Performance Data
Individual student score reports have a number of important features.
Performance Scores
In addition to receiving a total score, in the “Feedback by Content Area,” students will see their performance on the test questions in each content area. Like the total score, the performance in each content area ranges from 300–500. Students will also see a national performance average by content area. This is provided as a point of reference so students can compare their performance to their peers.
Keyword Feedback
Score reports provide keyword feedback to help students identify individual areas of weakness. These keywords are formative feedback that can be used to develop targeted remediation and study plans. A keyword is provided for each question a student misses; this feedback includes the content area, task area, and diagnosis—for example, if a student missed a question on urinary retention on a family medicine exam, their keyword feed might look like this: Urology/Renal: Clinical Intervention, Cystitis.
Important Considerations
A range of circumstantial factors may influence student performance on End of Rotation exams, leading to differences in performance within or among programs. Some of these factors apply to any examination, while others may apply more specifically to PAEA End of Rotation exams and comparisons to national data throughout the course of clinical education. Consider:
- Student motivation. Student motivation could be affected by factors that vary at the local program level, including weight of the exam relative to the student’s overall grade and potential penalties articulated for poor performance.
- Length and timing of the clinical experience. Student clinical training experiences in the discipline may vary from 4 weeks to 8 weeks depending on the program and overall length of time the student has been engaged in clinical training—is this their first clinical rotation or their seventh?
- Sequence issues. Students tend to do better on exams later in their clinical year after they have acquired more knowledge from practice, regardless of specialty or setting. When did the testing occur in the overall course of the curriculum? Which didactic experiences and content preceded the testing? Some PA programs may have advanced didactic education later in the curriculum after some (or all) of the clinical year.
- History. Students will sometimes prepare (mentally and emotionally) for End of Rotation exams based upon prior processes and procedures at the program. It is important that different testing methods (computer-based vs. paper and pen) and expectations are clearly communicated. In addition, written learning objectives that match the exam blueprint should be provided well in advance so that students have time to focus their study during the clinical rotation.
- Environmental factors. It is well known that knowledge is reinforced through application. Students will have different clinical experiences based on patient volume, acuity, and disease patterns, which fluctuate by site and season. For example, students in a pediatrics rotation during the summer are less likely to encounter patients with influenza. In addition, the degree of preceptor engagement in student clinical knowledge development through oral quizzing and assignments may help reinforce retention of content that might be on the exam.