Meeting of the International Standard-Setting at Student-Level
February 8-11, 2004
White Plains, New YorkInternational agreement on what constitutes a competent medical graduate was defined at the International Standard Setting Group Meeting on February 8-10, 2004. This Committee, chaired by Dr. David Stern, consisted of eleven professionals experienced in student assessment representing all regions of the world as well as a variety of basic science and clinical specialties. Participants were recommended by IIME Core Committee members, and assisted by a group of standard-setting experts. They were split into two sub-groups to focus on setting a "cut-score" on each element of the exam, by examination method and "GMER" domain. All individuals were sent materials in advance of the meeting, including sample examination materials from the three exam instruments.
 |
In the case of MCQ an Angoff standard setting exercise was completed separately for each of the two panels of judges. After completion of the Angoff judgments, panelists were asked to follow the Hofstee method to make judgments about the maximum, minimum, and ideal percentage cut score and maximum, minimum, and ideal percentage of students passing the examination. In case of OSCE stations exam the modified Angoff procedure (Cizek, GJ 1966) was employed. The concept of borderline examinee was reviewed by all participants in reference to their own country, thus a comprehensive international description of borderline examinees was generated. In each OSCE station, students were rated by an observer on a 5-point scale assessing their performance on three dimensions: Interviewing skills, Physical examination and Communication skills. In the case of Faculty Observation a holistic criterion-referenced Standard Setting approach was employed for 3 domains: Professionalism, Communication Skills and Scientific approach. Consequently a standard was set on each domain. The student actual performance data of the Chinese distribution was not shown to the panelists after the first rating to avoid its influence on the international standard.
 |
The results of parallel panel process revealed nearly identical standards from the two groups. This added confidence to our belief that the standards were valid and that the methods used to assess them were as good as possible in today's world. These "cut-scores" ranged from a low of 41% to a high of 78%.
The success of this process helps support the idea that an international standard for medical student performance can not only be identified, but also closely agreed upon by divergent faculty internationally.
The assessment of global competencies of medical graduates is in its infancy. With evaluation of schools in other countries and validation of the principles of assessment, the coming years promise a more consistent agreement on the expected standards for medical education.
( Details in a paper currently in preparation for submission)
Institute for International Medical Education.
Unauthorized reproduction strictly prohibited.