Report from JCIE Part 1 Standard Setting Meeting

Royal College of Surgeons of Edinburgh, 10th November 2016

I was invited to attend the Standard setting meeting for the FRCS Part 1 that candidates sat at the beginning of November.  This was an interesting meeting to attend as I could see the process of how the examiners ensure that the exam is fair and set the pass mark.  I was also given the opportunity to participate in the Angoff method, although my scores were not included in those that set the pass mark (which is a good thing, as I discovered that I am towards the harsher end of the scale).

Whilst there are parts of this process that I cannot detail due to the confidentiality of the exam, I can report that what I witnessed leaves me in no doubt that the process of the exam is fair and robust.  Any questions that are flagged as possibly causing problems are discussed and if there is consensus among the examiners that they are problematic they are removed from the exam.  This occurs prior to the marks being set.

Mr David Limb gives an excellent talk about how the Angoff procedure works and I would advise that if you are approaching the exam that you should try to attend the BOA Instructional course to try to hear it.  However, I can dispel some of the common myths:

  1. We did not see any candidate data throughout the process. At no point did we know how the mark we were setting would influence how many people passed or who would pass or fail.

  2. There is no set pass mark. It is set based on each exam and therefore harder exams will have lower pass marks and vice versa.  Additionally, there is no set number of people who can pass or fail.  Everyone could pass if they met the required standard.

  3. The exam is set by a computer using questions that are tagged to areas of the curriculum. There is therefore a balance on subject areas.  However, some questions may be tagged in more than one way; e.g. a biomechanics basic science question could discuss foot biomechanics which may make it appear like a foot and ankle question but it is testing basic science knowledge.

Lawrence Moulton, SAC