Writing Effective Multiple Choice Questions

View Printable Writing Effective Multiple Choice Questions (PDF)

Introduction

Mastering the skill of writing effective multiple choice items is truly equivalent to mastering an art form. The beauty of being able to assess higher order thinking skills in a reliable and valid format often comes with some trial and error and revision. Multiple choice items can measure a range of different types of knowledge and ability, and the ease of grading makes multiple choice items a popular assessment tool. Analyses of multiple choice exams can also allow the instructor to determine what concepts were most clear as well as those that were not clear to students.

There are guides for constructing multiple choice items (McMillan, 2001; Suskie, 2010;Tarrant & Ware, 2012), and current literature focuses on connecting the items to learning outcomes and constructing items that assess higher order cognition based on Bloom’s Taxonomy (Scully, 2017). Offering faculty opportunities to develop the skill of crafting multiple choice items has been shown to improve the quality of writing multiple choice items. For example, Abdulghani et al (2015) found that there was an increase in the number of items assessing higher cognitive level of Bloom's Taxonomy in the post training test items for faculty who completed a faculty development program that focused on improving the quality of multiple choice item writing.

Implementation

1. Align the questions with the course learning outcomes that you have established.
2. Consider a mixture of questions that align with Bloom’s Taxonomy with introductory sections and graduate level courses having a proportionately different mix.
3. Write questions keeping in mind the following checklist (McMillan, 2001):
   1. Is the stem stated as clearly, directly, and simply as possible?
   2. Is the problem self-contained in the stem?
   3. Is the stem stated positively?
   4. Is there only one correct answer?
   5. Are all the alternatives parallel with respect to grammatical structure, length, and complexity?
   6. Are irrelevant clues avoided?
   7. Are the options short?
   8. Are complex options avoided?
   9. Are options placed in logical order?
   10. Are the distractors plausible to students who do not know the correct answer?
   11. Are correct answers spread equally among all the choices?
4. Evaluating the exam items
   1. Below is an example of an initial exam item, the revised question and the rationale source: http://crlt.umich.edu/olws/6/practicequestions
   2. Practice: Evaluating Exam Questions
   3. Consider what attributes of the question might decrease its validity or reliability. You can see a revised version of the question along with a rationale for the revision.
      1. The example is drawn from pharmacy, but the principles apply to any multiple choice questions, and in many cases, more broadly to various types of exam questions. The correct answer is in bold font.
5. All of the following are common adverse effects associated with drug overdoses of thyroid hormone replacement therapy except:
   1. Cardiac palpitation
   2. Arrhythmias
   3. Tachycardia
   4. Weight gain
   5. Heat intolerance
6. Revised question: Which of the following adverse effects would be associated with a drug overdose of thyroid hormone replacement therapy?
   1. Cardiac palpitation
   2. Dry skin
   3. Weight gain
   4. Dehydration
   5. Unusual odor
7. Rationale: Questions should ask students to identify the correct answer, rather than an incorrect answer. Identifying what is wrong does not mean the student knows what is correct, so asking students to find an incorrect answer decreases reliability

Frequently Asked Questions

I took a great deal of time to construct multiple choice items and test scores were much lower than I had expected. How do I know if the questions were constructed poorly or the students did not understand the concepts presented to them before the test?

• Analyze the results to see if there are trends or frequently missed questions.
• Consult guidelines for constructing multiple choice items to see if there are any glaring omissions of the guidelines for the items you have constructed.
• Ask a colleague to look over the frequently missed items to offer insight.
• Be patient. Constructing multiple choice items does not always work perfectly the first time. Analyze, gather student feedback, and revise.

I know this content so well that it is sometimes difficult for me to know how to construct choices that are not obvious distractors?

• Ask a colleague or friend who is not an expert in your discipline to take your test and identify any questions that seem to have obviously incorrect choices.

I would like to construct questions related to concepts discussed in class. What is an effective strategy for writing multiple choice questions that relate to content covered in class discussion?

• An important element is capturing accurately the information discussed in class such as the specific terminology used that day. At the end of class, take a few moments to write down those concepts and terms.

Other Resources

Carnegie Mellon University

Vanderbilt University Brame, C., (2013) Writing good multiple choice test questions. 

Virginia Commonwealth University 

University of Michigan, CRLT

Piontek, M.E. (2008)

University of Waterloo, Designing multiple-choice questions. Centre for Teaching Excellence.

References

Abdulghani, H.M., Ahmad, F., Irshad, M., Khalil, M.S., Al-Shaikh, G.K., Syed, S., Aldrees, A.A., Alrowais,N., & Haque, S. (2015) Faculty development programs improve the quality of Multiple Choice Questions items' writing.
Sci Rep. 2015; 5: 9556.Published online 2015 Apr 1. doi: 10.1038/srep09556 PMCID: PMC4381327

McMillan, J. H. (2001). Classroom assessment: Principles and practice for effective instruction. Boston: Allyn and Bacon.

Scully, Darina (2017). Constructing Multiple-Choice Items to Measure Higher-Order Thinking. Practical Assessment, Research & Evaluation, 22(4). Available online: http://pareonline.net/getvn.asp?v=22&n=4

Suskie, L. (2010). Assessing Student Learning: A Common Sense Guide (2nd ed.). San Francisco: Jossey-Bass.

Tarrant M. & Ware J. A. Framework for improving the quality of multiple-choice assessments. Nurse Educ. 37, 98–104 (2012). [PubMed]

Cite this resource: Lightner, J. (2017). Assessing Student Learning – Writing Effective Multiple Choice Questions. Kent State University Center for Teaching and Learning. Retrieved [today's date] from (HYPERLINK).