Thoughts On How to Write an Exam

Choose exam questions with your course goals and grading plans in mind. What are the important themes of the course that all students should have some facility with? Is the time students spend on each question appropriate relative to the number of points assigned to the question (or the importance of the topic)? Remember, the exam should assess a student's understanding of statistical concepts, not his/her mathematical abilities or knowledge of content from other fields (biology, economics, etc.).

Although writing calibrated exams takes some effort, I believe that using absolute scores such as A = 90% and higher, B = 80% to 90%, etc. rather than relative scores (a "curve") leads to meaningful grades and a sense of fairness among your students. After all, if an instructor has not determined what a grade of A "means," then how can he/she provide meaningful course grades to students. Others feel even more strongly

"... that grading on the curve is educationally dysfuntional. If possible, your grades should, both in the students' eyes and in actuality, be more nearly based on absolute standards than on relative standing in this particular class." --Wilbert McKeachie (McKeachie's Teaching Tips 2002, p. 111)

McKeachie, W. (2002). McKeachie's Teaching Tips: Strategies, Research, and Theory for College and University Teachers, 11th edition, Boston: Houghton Mifflin.

Nevertheless, even in this document, I will present the current course grade distribution guidelines for our introductory courses to foster some consistency across quarters and years.

In order to calibrate your grading to an absolute scale, you will need to have an absolute standard in mind as you create your quizzes and exams (and homeworks).

Travers (1950) proposed this set of absolute standards:
A: All major and minor goals achieved.
B: All major goals achieved, some minor ones not.
C: All major goals achieved, many minor ones not.
D: A few major goals achieved, but student is not prepared for advanced work.
F: None of the major goals achieved.

Travers, R. M. W. (1950). How to Make Achievement Tests, New York: Odyssey Press.

You can see that in order to implement such a standard, you must have defined for yourself the major and minor goals for the course. For example, for the less-quantitative Stat 20000 course, a major goal may be to interpret a p-value in non-technical terms and use it to make a decision in the context of the situation. A minor goal may be to actually calculate a p-value correctly for a specific situation.

Then, writing an exam that is calibrated according to this standard, consider including questions such that 75% of the score consists of completing major goals and 25% minor goals. Then, a student who completes all major goals (with minor errors), but has trouble with some minor goals might score 70% + 15% = 85% or so (a B grade). Similarly, a student who completes the major goals, but struggles with the minor goals might score 70% + 5% = 75% or so (a C grade).

Note: After writing and grading many exams, I have recently come to the conclusion that a 90%=A, 80%=B, etc. scale may be too restrictive for calibrating exams well. A small misjudgement in the point allocation on my part and the exam average becomes too high (near 90) or too low (near 75). (A target average for our introductory statistics courses is about 80-83%, a B-). In the future, I might try a broader scale: 80%=A, 60%=B, 40%=C, etc. If you do try this, just be sure to very clearly communicate your grading scale to your students.

To write the exam, create an outline of the major and minor goals, assign points to these goals, and finally develop questions to address the goals. If the exam is too long, remove some questions, and adjust the points as necessary to keep the balance of major and minor goals as needed.

As an example, I have included a STAT 20000 quiz given in the Spring, 2005 quarter. This quiz has 25 points total. Here is a breakdown of the points and topics for this quiz:

6 points (24% of quiz)
These are straightforward probability questions similar to many asked in the homework assignments.
3 points (12% of quiz)
This is a fairly deep question that requires solid understanding of the box model used in the Freedman, Pisani and Purves textbook.
16 points total
1. 3 points (12% of quiz)
  This is a straightforward question about normal distribution probabilities similar to many homework questions.
2. 3 points (12% of quiz)
  This is a straightforward question about regression prediction similar to many homework questions.
3. 4 points (16% of quiz)
  This is a fairly mathematical question about conditional distributions, but is similar to many homework questions.
4. 3 points (12% of quiz)
  This is a fairly deep question that asks for students to convince us that they understand the previous calculations. This question is not a common homework question and is expected to require students to show that they so thoroughly understand the main topics that they can connect ideas on their own. A question like this should separate the “A” students from the “B” students.
5. 3 points (12% of quiz)
  This is a straightforward question to calculate the regression slope similar to many homework questions. The interpretation (1 point = 4% of quiz) might be challenging for some students.

So, about 24-28% of the exam requires fairly deep understanding of course topics and the remaining 72-76% should be very familiar for those that completed the homework assignments. Indeed, students able to complete 90% of the straightforward work and get 1/2 credit for the challenging problems would score about 80% (B-). To earn an A grade, a student must be able to get quite far along on the challenging problems. Since homework scores tend to be quite high (average above 90%), have very little variance, and often have “extra credit” built in (drop lowest homework or divide by n-1 instead of n), there is some room to make up for unfortunate errors by good students on a quiz or exam. Indeed, of the 25 students scoring B+ (88%) or above on this particular mid-course quiz, 20 students went on to earn at least a B+ in the course overall.

After you have so carefully calibrated an exam according to your absolute standards, it would be a shame if most students were unable to demonstrate their skills with the various major and minor goals because the exam was much too long for the time given. I determine an exam is the right length if I (the writer of the exam) can read each question through and actually write out complete answers in about one-third of the alloted examination time. This is not wasted time at all and often leads me to discover errors of omission, typing errors, and unfortunate numerical outcomes. (For example, z-values = 15, p-values near 0.05, slope = 25,000,000, etc.).

Most students should finish or at least have time to try each problem. You do not have to ask a question about every course topic. Ask a CA or fellow instructor to look over your exam before you administer it. Are your questions clear? Could they be misinterpreted?

Be careful that it is quite easy to make a midterm exam too easy since the amount of material to be covered is so much less than and not as difficult as for the final exam. McKeachie (2002, p. 109) cautions that "being more generous in assigning grades to test[s] ... than in the final distribution of grades guarantees visits from aggrieved students."

Consider using 3-4 quizzes instead of a midterm in order to broaden the topics examined prior to the final exam, increase the number of "data points" on each student, and to "force" students to stay current with the course. Indeed, quizzes are one way to get all students thinking individually throughout the course (rather than relying too much on their friends as is often the case on homework assignments). You could still have a midterm exam, and have the quizzes worth less so that they can be used as good feedback prior to the midterm and final exams. Let students ask about the quizzes in office hours and discussion sessions. You could make quizzes self-graded and not part of the assessment of the course grade. If you do want to have in-class or recitation quizzes for credit, your students would appreciate knowing about it in advance (ideally in the syllabus).

Last modified: September 28, 2008 2:40PM