Testing Today’s and Tomorrow’s Surgeon

Even when orthopedists emerge from fellowship, they are still in training. Actually, some say the most rewarding aspect of being an orthopedist is the vast amount of learning opportunities in the field. They get accustomed to being lifelong learners in part because of the Orthopaedic In-Training Examination (OITE), a carefully formulated and validated test taken annually by residents around the globe.

Dr. Lawrence Marsh, Chair of the American Academy of Orthopaedic Surgeons (AAOS) Evaluation Committee, gives a history lesson: “The OITE was the first of its kind—namely an annual test in a specialty—and was first administered in 1963. The ability to have an exam that documents and guides the process of learning during graduate training was increasingly seen as important to the field. This effort was led by a predecessor of the evaluation committee under the auspices of AAOS.”

So what happens behind the scenes to bring this exam, which encompasses 12 orthopedic knowledge domains and 275 questions, to fruition? Dr. Marsh notes,

There are 32 committee members, all of whom sub specialize and thus develop content in their areas of expertise. Other areas include rehabilitation, medically related issues, and professionalism. Each member proceeds to develop questions based on their interest, the literature, and activities they observe in their practices.

They don’t go to an island or make anyone stand up and sing…but they do stay in a big room for two days. Dr. Marsh states, “The questions are then submitted to the evaluation committee where they undergo an extensive review. Afterwards, they are sent back to the committee members who take this preliminary test and comment on all of the questions. Their feedback is compiled into a book and distributed to all of the question writers; then we hold a field test meeting—a full day event where committee members use the field test book and have interactive discussions, thus allowing the questions to be brought to their final form. Each question is then ranked for quality and importance, and is delivered to the committee chair. They usually provide me with a surfeit of questions in the event that one or two questions are going to be unusable.”

Photographer: Andrew HuthOne surgeon who is accustomed to the questioning of his questions is Dr. John Richmond, Professor of Orthopaedic Surgery at Tufts University School of Medicine. “If I write a question it is almost always rewritten because it may not be worded just right. Or there may be an issue of the illustration or MRI not showing what you want to depict.”

Examples of questions that won’t win any beauty contests? Dr. Richmond: “A bad question can be one that is too contrived. For example, ‘When is a double bundle ACL indicated?’ There are no clear indications for these procedures. Also, you can’t put the focus on the negative, i.e., ‘Which of the following are not appropriate treatments for XYZ.’ We want the focus to be on what you should do.”

To clarify things for those charged with testing future generations of doctors, the committee is providing more pointed guidance. Dr. Marsh: “We are compiling an instruction manual to help committee members understand what AAOS thinks is the appropriate way of writing multiple choice questions. When someone starts a term on the committee they usually write the first few items in a style that is not appropriate for the test (true or false questions, multiple correct answers, etc.)”

The OITE spans three levels of sophistication in an attempt to ensure that residents are not only memorizing, but also learning how to integrate knowledge in a useful manner. Dr. Marsh states, “Questions that meet the above parameters can still be unsatisfactory based on content. A question like, ‘What is the bone between the hip and the knee?’ is just too easy to be educational. There are three levels of questions: taxonomy one, where we ask for a fact; taxonomy two, where questions require an interpretation; taxonomy three, where the questions are aimed at problem solving, i.e., ‘Given the imaging, which management strategy would you recommend?’”

The last committee members to leave the field test are likely those struggling with taxonomy three. Dr. Marsh explains, “While we prefer that a reasonable percentage of the test be taxonomy three questions, these items are the ones that can stir up controversy. For example, you have an Xray of a 62-year-old woman who has fractured her right hand. The question writer says, ‘I would manage her with an open reduction and fixation and I can write two different nonoperative distracters that are wrong and two operative strategies that are wrong.’ But when someone from another practice looks at it he or she may say, ‘I may not have operated on that case.’ In the field test process they sit as a group and modify it to the point where everyone can agree…for example, someone may say, ‘If you made her a 25-year-old then I would definitely operate.’”

Despite these painstaking efforts, there are calls and emails from perturbed test takers. Dr. Marsh: “Some of the most frequent complaints are that the quality of the images is poor and that we ask obscure questions. We go through a detailed process using statistical methods to assess how the questions perform. Later, we review the questions that didn’t perform well and decide whether to exclude them.”

Dr. Richmond adds, “In any given year the residents complain that there is too much basic science. And what is relevant or interesting for one specialty area could make someone in another area yawn. ‘Gee, there are no questions on knee ligament injuries but six questions on hand anatomy. I’m in sports medicine so I don’t need these questions.’”

When you create something you run the risk of having it used for unintended purposes. This, says Dr. Marsh, is a real possibility with the OITE.

There is more than a little bit of disagreement regarding the purpose of the exam. The perspective from AAOS and the evaluation committee is that it is an educational tool…essentially a metric for individual residents to identify their strengths and weaknesses. It also can serve to motivate residents, as well as allow for the monitoring of their progress. Another important use of the OITE is as a tool for the programs to assess their educational program (based on their residents’ performance on this exam).

He continues, “There is the potential that the exam would be used as a metric of an individual resident’s success. For example, this may be a factor in deciding whether someone should be awarded a competitive fellowship or not. At times it is also used by programs to advertise their strengths. Those people who feel that using the exam in these ways is important want AAOS to exert more influence on how the exam is taken.”

Speaking of control, to what degree is AAOS able to insist that the test be taken in the manner it deems appropriate? Dr. Marsh notes, “The Academy has guidelines, including that it should be proctored, be given at a defined time, and that the proctor not allow it to be an open book exam. While AAOS can recommend those things, they can’t control them. But those who feel that control of this exam is important are concerned that it’s unfair that another program (which may allow open book, for example), would achieve scores that give the impression of being better than theirs.”

Perhaps most exciting, says Dr. Marsh, is that the exam continues marching into the future…via computer. “I am proud to say that as of fall 2009 orthopedics is the first graduate medical education specialty to produce an electronic in-training exam. In one year, we did several pilot tests and converted residents in over 200 programs around the world to an electronic format on DVD-ROM…that’s 4, 300 residents. The testing environments ranged from controlled medical school computer labs to rented laptops.”

And that variability can inject new questions into the process. Dr. Marsh notes, “Of course we asked, ‘What prevents residents from searching the Internet or looking at notes on their desktop?’ The answer was to build in a feature such that if residents left the exam environment then the program crashed. Although successful, the shutdown feature was overly sensitive, leading to excessive shutdowns for several residents. Proctors were given a code that allowed them to restart the program. We found that there were a small number of legitimate, computer-related issues that resulted in the shutdown of several residents’ exams…meaning that their answer files were lost.”

Overall, says Dr. Marsh, the residents have been positive about this transition. “In future OITE exams we will use videos to make the testing experience more real, especially when it comes to arthroscopy and operative approaches. Also, now that the imaging studies are in an electronic format, MRI and CT questions will be presented such that residents can scroll through multi-slice images. This makes the experience more similar to clinic. We are continuing to solicit feedback from residents and plan ongoing assessments of these new formats.”