Following the Valentine’s Day glitch that dogged this year’s online exam, the RACP has been advised to stick to paper.
The RACP has scrapped its digital exam format until further notice, with divisional written exams now to be conducted in a paper format and held twice a year.
The move follows a disastrous exam session in February that saw around 120 candidates initially unable to log on to the system.
The announcement to RACP members earlier today came with the release of an “outcomes report”, developed by KPMG and entitled Investigation into the Divisional Written Examinations held on 14 February 2022.
The college said while there were benefits to delivering exams through computer-based testing, this would not proceed until further review of the options in light of the KPMG report. The college added it would remain receptive to feedback on the issue.
The board’s position is that divisional written examinations will be delivered in a paper-based format twice a year for the present.
On 14 February 2022, around 120 candidates from Auckland, Melbourne, Brisbane and Perth experienced delays in being able to log on to the system for their computer-based exam and were unable to complete their session in the allocated time.
Some candidates were able to continue on to complete their session at the end of the day. As an alternative, affected candidates were offered the option to sit one or both exams on the back-up date of 8 March. The back-up was a paper-based only format.
Around 1150 candidates in total sat the exam.
The RACP isn’t the only college to be plagued with technical problems at exam time. In October 2020, a number of GP trainees who tried to attempt their key feature problem exam experienced a technical failure, causing delays and uncertainty as to when, or even whether, the test would resume.
The college cancelled the exam after they were unable to resolve the system outage with an external provider.
Meanwhile, in the RACP’s October 2021 exam, eight candidates experienced response data loss although the college said it didn’t believe the 14 February incident was due to the same problem.
The login delays experienced during that exam weren’t encountered during the two CBT exams run in 2021, the college said. Neither were they identified during the exam system testing program implemented by the exam provider and college staff before the February 14 sessions.
The 14 February and backup 8 March exams were not counted as an official attempt by those candidates who were unsuccessful.
The college’s first foray into computer-based testing, in February 2018, was also a debacle, with a design oversight triggering an IT meltdown half-way through the exam.
The KPMG report, released to members today, included eight recommendations, ranging from ensuring that exam server CPU resource allocations are set with sufficient margins through to assessment of the merits of offline versus online examinations.
The RACP board has accepted all of the recommendations made.
The report has already been shared with the Australian Medical Council and the Medical Council of New Zealand (MCNZ) “to ensure transparency and openness about the outcomes of the investigation”, the college said in a statement to members.
“I acknowledge that for many trainees and supervisors, the [14 February] exam was a very distressing experience, and that reading this report may also be upsetting,” RACP president Dr Jacqueline Small said in the statement.
“I acknowledge the great amount of work and sacrifice to prepare for examinations, and we apologise for the extra stress this has caused for our college community.”
KPMG was tasked with independently investigating the 14 February exam and identifying the issues that affected delivery of the exam and the causes of those issues. The firm also reviewed the risk-mitigation strategies and contingency plans put in place and the effectiveness of their deployment.
