Setting Performance Standards: Foundations, Methods, and Innovations
Common Elements in Standard Setting Practice 5. Egan, M.
- Cut-scores revisited: feasibility of a new method for group standard setting!
- Everyday Satori (A Gathering of Voices Book 2)!
- Online assessment standard setting for multiple choice questions;
Christina Schneider, and Steve Ferrara 6. Skorupski 8. Reckase and Jing Chen 9. Cizek III. Standard Setting Methods Cizek Kingston and Gail C. Tiemann Lewis, Howard Mitzel, Ricardo L. Mercado, and E. Matthew Schulz Lewis Haertel, Jennifer Biemers, and Julie Miles Phillips Contemporary Issues in Standard Setting Bunch Growth, Standards, and Accountability Damian Betebenner Cizek and Charlotte A. Agger Buckendahl and Susan L.
Invigorating methods of measurement and building better theory - UNC School of Education
Davis-Becker Our Awards Booktopia's Charities. Item Added:.
Are you sure you would like to remove these items from your wishlist? This is an Open Access article distributed under the terms of the Creative Commons Attribution License which permits unrestricted use of work provided the original work is properly cited. Standard setting of assessment questions has matured as a psychometric practice since the s 1 and is now a common practice in Medical Education.
Nevertheless, in order to be defensible standard-setting must be procedurally credible.
This test-centered, criterion referenced method of standard setting is one of the most studied and widely used procedures for high stakes examinations. At the CMHS, following a period of question vetting, the procedure was initially conducted in face-to-face meetings of appropriate faculty members with subject expertise. In this exercise judges make independent estimates of the proportion of minimally competent candidates that would be expected to answer each item correctly.
Where there is a large variation in estimates, judges then discussed discrepancies and revised estimates.
Depending on length of discussion and necessity to refine previously vetted questions, the procedure often took an average of three to six hours to standard set an examination of MCQs. In the case of high stakes examinations, the procedure could take several times that amount of time. Although this process had its benefits in further vetting of questions, useful discussion and feedback between faculty members about their understanding of the task, issues surrounding the minimally competent candidate, etc.
Availability of new technologies have however facilitated evolution of methods and processes of test administration and setting cut scores to meet challenges posed by real and perceived inadequacies of existing processes. The purpose of this paper is to describe how we implemented an online standard setting procedure which significantly reduces the amount of time needed to standard set examinations.
We set out to automate the process of standard setting at CMHS. Relying on an online, secure Assessment Management System AMS 8 where the questions are stored safely by the Medical Education Department, vetted and standard set by faculty, and delivered securely to students.
The system involves two types of users, judges who are faculty members able to make a judgment of the appropriate cutoff mark for each question, and coordinators who are in charge of courses and associated examinations. The system was used at CMHS as a pilot for several course final examinations and is now used for all less high-stakes examinations with judges making judgments using their office computers while entering comments on questions that warrant discussion.
Foundations of the NBCOT Certification Examinations
It is also used for high-stakes examinations in face-to-face meetings with judges using handheld devices to access the questions, have any needed discussion, and then enter their judgments in their handheld devices. The resulting cutoff average, standard deviation, and histogram are displayed on the handheld device and an overhead projector and modification of cutoffs can be performed if necessary. The judges can read the question clearly, make a judgment on the percent of minimally competent students who should answer the question correctly, enter that percent easily, and add a comment about the question if they wish.
Judges can be assigned, removed or emailed by the coordinator.
Citations per year
Completion percent and average cutoff for each judge are clearly shown and comments can be displayed. The assessment analysis shows, question by question, percentage of options chosen, point biserials for correct options, and the difference between students' correct option percentage and judges' cutoffs. This provides an indication regarding the accuracy of judges' cutoff estimations. We feel that we have uncovered some novel ideas in Medical Education standard setting which warrant further research.
Finally, similar to other researchers, we are confident that in time online standard setting methods will proliferate, but believe this will only happen when these new ideas are addressed effectively. Furthermore, as in the case of most standard setting research, we cannot state with confidence that the procedure described here is widely generalizable.
- Standard-setting study.
- International Organization for Standardization.
- Dont Hit My Mama: Overcoming the Effects of Childhood Domestic Violence.
- Julie Stephanis Ultimate Scrapbook Guide (More Than Memories).
- ISO repository of standards and innovation!
- Portent, A Ravensborough Novella (The Ravensborough Saga)?
- Illustrated Guide to Home Biology Experiments: All Lab, No Lecture (DIY Science).
We can nevertheless, attest to the overwhelming positive feedback received from standard setters in our institution and recommend that other institutions faced with similar constraints necessitating consideration of an alternative approach, at least pilot online standard setting.