Partial Scoring for Multiple Choice Tests

I read some time ago an article by Terence Tao on how to assign partial credit scores for multiple choice tests – the article can be found here.

Terence’s solution focused on True/False questions, but could be easily generalized to multiple choice questions, but involved as part of the solution giving negative marks to students, including negative infinity marks. Clearly this is an infeasible solution for the classroom, and one wonders if there is a function f:[0,1] \to [0,1] such that f(0)=0 and f(1)=1 which satisfy similar conditions as those outlined in the article. The main condition that needs to be satisfied is that the students expected score should be maximised at their subjective probability of the particular answer being correct.

We can use some of Terence’s work here, to cheat slightly. Let us institute a rule that a person cannot bid 1 or 0 for a particular answer, and secondly that they can only bid a value q\in\{0.01,0.02,...,0.99\}. Now, we can use the form f(p)=Alog(p)+B with the constraints that f(0.99)=1 and f(0.01)=0. This yields the equation f(p) = 0.2176log(p) + 1.0022.

Let us check the required conditions:

f(0.01)=0: f(0.01)=0.2176\times log(0.01) + 1.0022 = 0

f(0.99)=1: f(0.99)=0.2176\times log(0.99) + 1.0022=1

E[mark] = p(0.2176log(q)+1.0022)+(1-p)(0.2176log(1-q)+1.0022)

\frac{\partial E[mark]}{\partial q}=0.2176\frac{p}{q}-0.2176\frac{1-p}{1-q}=0 so that the expected mark is maximised when p=q.

So we have found a function which practically maps to [0,1], with the minor caveat that a student can never pick either 0 nor 1 for their answer.

There is, however, a large flaw with this function, which is a student can lock in a score of 87% simply by choosing 0.5 for both options. Clearly further work is required here!

Advertisements