FRCS Section 1 Exam July 2022 Following the most recent diet of the Section 1 FRCS Tr&Orth Examination, some concerns were raised that the pass rate was unusually low. This feeling was shared by the Intercollegiate Surgical Board (ISB) and as a result a thorough review of the exam and the two single best answer papers was conducted. When the papers are generated for the exam, a strict formula is used which ensures that no more than 20% of questions in the exam can be “new” questions which have not been previously in the question bank. These are combined with a proportion of the highest performing questions (i.e. questions where the best performing candidates consistently scored highly, and more poorly performing candidates consistently scored poorly) and additional questions from the bank to generate a paper which can be standardized to previous papers using the outcomes in those high performing questions. Once the exam has been completed, an extensive battery of psychometric testing is undertaken to assess the performance of candidates and questions within the exam. Initially an Angoff procedure is used to calculate the cut score for the examination and a single standard error measurement (SEM) is then applied to calculate the pass mark. The addition of the SEM is a GMC requirement to ensure that the pass mark is above the score which could be achieved by a candidate randomly guessing answers throughout the paper. Questions which fail to perform as expected are then subject to further analysis and if there are concerns about the wording or accuracy of the question or if the question is particularly ambiguous, then that question is removed from the examination. A Cronbach’s Alpha is calculated to determine the reliability of the questions (target r>0.8) and a calculation of the SEM as a percentage of the pass mark is also calculated (target <3%). Candidate performance is then analysed based on ethnicity, gender, first language, first time candidate vs re-sitting candidate, and training status. For this exam specifically, additional checks on candidate scores, individual question scores, and the Angoff cut score and SEM were carried out manually. Both papers for the exam were reviewed by the lead question writer for Section 1 and by the chair of the ISB. They checked every question to ensure that the correct answer matched the question. The exam file which was used by Pearson Vue was also analysed to ensure that the scoring system was working as expected. Once analysis was complete, it was felt that the questions in the exam had performed as expected. There was no unusual patterns of results based on ethnicity, gender, language, first time vs re-sitting candidates, or training status and broadly these trends were in line with previous exams. The ISB and BOTA are content that every effort has been made to ensure the validity of the most recent Section 1 exam and are happy that results were accurate and consistent.
Graham Finlayson (BOTA SAC representative)