Report from the Intercollegiate Specialty Examination in T&O Standard Setting Meeting
I was pleased to attend the above meeting and was warmly welcomed by the panel members. The purpose of the day is to establish the pass mark for the written test (Section 1) of the FRCS(Tr&Orth) examination.
The first stage of the process involves ensuring that the questions in the examination are discriminating appropriately between candidates and rewarding candidates for the right response. This involves reviewing questions which have been flagged up because of statistical anomalies. A board of ten orthopaedic surgeons with a range of subspecialist interests, many experienced examiners, discussed each such question. These were usually identified as anomalous because the majority of candidates selected the wrong answer or there were two equally popular answers. The overall performance of candidates who gave each answer is considered. Interestingly, when two answers were equally popular, it was often the case that those candidates providing the correct answer performed better on the rest of the test. This demonstrates such a question to be a good discriminator. In the interest of fairness, and on the basis of this discussion, any questions which are deemed to have inadequate wording, content or clarity are removed from the papers before the scores are finalised.
The Section 1 is judged by a criterion-referenced method, meaning that there is no fixed pass rate or mark (as opposed to a norm-referenced method where a candidate’s success depends upon their performance in relation to others). Rather, a pass mark is established for each examination diet based on the Angoff method. Each question is given a rating which reflects its perceived difficulty, equating to the likelihood that a borderline candidate (a “minimally competent practitioner”) would get the question right. A pass mark is then generated based on the overall difficulty of this particular set of questions. This method allows the pass mark to reflect the difficulty of the paper, ensuring that the conceptual level of competence required for a pass (the level of a day one consultant in the generality of the specialty) remains the same regardless of which questions were used in each diet.
At the standard setting meeting, the members of the panel independently rated each question that had not previously been given a score. I was invited to perform the process as well, although my scores did not contribute to the final rating. Despite this, the panel were very interested to see how my judgment compared to theirs. Analysis showed that my assessment of the difficulty of both the SBA and EMI papers fell in the middle of the other panel members. The chairman was pleased to see that there was similarity between my assessment as a trainee (who had recently passed the exam) and the standard setting panel. It was an interesting exercise, particularly due to the difficulty in distinguishing between what a borderline candidate would know and what one felt they should know.
The Angoff method therefore generates a pass mark that the minimally competent practitioner will achieve. One standard error of measurement is then added to the score, in line with GMC guidelines for high stakes examinations, to make sure that successful candidates can be considered safe clinicians. The pass mark came out nearly identical to the last sitting in June, which had seen an unusually low pass rate. The pass rate was much higher this sitting and, as was the case in June, this closely reflected the proportion of candidates who were on a recognised training scheme. The chance of success at the Section 2 is often shown to be linked with training grade status, which suggests a good alignment between training, clinical practice, and the skills required for success in the examination. While this association is sometimes used to suggest bias or discrimination in the Section 2, the fact that a similar relationship is seen in the written exam provides reassuring evidence, if it were required, to the contrary.
I left the meeting satisfied and reassured that the marking system is rigorous, fair and reproducible and look forward to continuing my role as BOTA representative to the examination board.
12th November 2015
Daniel Westacott FRCS(Tr&Orth)
BOTA representative to the ISB & JCIE