Re: [Phys-L] evasion-resistant multiple choice

[John Denker <jsd@av8n.com>]

On 04/12/2017 04:10 PM, Philip Keller wrote:

> There is also a middle ground where students don't show work but can't work
> back from answers: the last handful of questions in each SAT section are
> "student produced responses".  The students have to bubble in the numeric
> answers.  There can be more than one right answer. In that format, I think
> you could require algebra:

Is that referring to the so-called "grid-ins"?

> The line through the points (2,5) and (7, -1) is expressed in the form Ax +
> By = C where A, B and C are integers.  Find a possible value of  C.   [1]
>
> I think you would have to do the algebra. Or be so familiar with this kind
> of algebra that you can skip a bunch of steps -- but that's not the same
> thing as evading the algebra entirely.

A feature of question [1] -- and grid-ins generally -- is that they resist
being solved backwards.  They cannot be solved simply by testing each of
the multiple choices.

Another amusing feature of question [1] is that the answer is non-unique.
This poses a challenge to the grading software.
 
> But this raises another question in my mind: this format has been around
> for a while on the SAT -- more than a decade.  But they rarely choose to
> use it to make students do algebra.  I suspect that somewhere in the inner
> core of where SAT policy is created, there are still some people

I reckon "some people" is an understatement.

> who like the fact that the test rewards clever work-arounds.

That suspicion may be reinforced by the fact that a great many steps
can be skipped in the question [1] quoted above.  Hint:  Cramer's
rule.  If we have

	        | 2  5 |
	C = det |      |
	        | 7 -1 |

or any multiple thereof then A and B are guaranteed to be integers.
No other calculations need be done.

At this point there are all sorts of questions about what the student
is trying to achieve (decent score?  perfect score?) and yet more
questions about what the test is trying to achieve.

Cramer's rule is widely taught in high schools, although I cannot
imagine why.  This question [1] is literally the first time in the
last 20 years that I have gotten any value out of Cramer's rule.
The other 99.999% of the time I spend yelling at people to stop
fooling around with Cramer's rule and use Gaussian elimination
instead, since for practical applications it is less laborious
*and* more numerically stable.  So in some sense question [1] is
both a trick question and a perverse trivia question, rather than
a practical question.

On the other hand, I suppose it could be argued that if you want a
perfect score on the test, maybe you ought to know Cramer's rule
/and/ have the insight to trot it out in this weird situation.

Or maybe the SAT is banking on the fact that Cramer's rule is widely
taught, so they don't see the question as tricky or perverse.  I
have no idea what this question was supposed to measure, or what
it actually measures.

  The SAT stands in contrast to the typical classroom situation,
  the teacher usually wants to analyze each question to get some
  idea what the student does or doesn't understand.

  Personally I like it when the student comes up with a clever,
  devious solution ... but I reserve the right to ask follow-up
  questions to verify that the basic bread-and-butter methods
  are understood also (e.g. Gaussian elimination).

On the SAT there is tremendous time pressure, and normally nobody
looks at the question-by-question breakdown, so it is doubly hard
to figure out what question [1] is trying to accomplish.  To get
a really good score it does not suffice to answer the question;
you have to answer it *quickly* so you have time to attack other
questions.

Bottom line:  An SAT question that seems (at first glance) like
it is trying to measure one thing is likely measuring something
else entirely, in ways that are highly nonlinear (since each
question interacts with others, via the time pressure).

High school generally doesn't teach people to be devious.  This
may be part of the reason why there is a market for extracurricular
SAT prep, to train people to look for the devious solution.