Re: [Phys-L] Multiple guess, OK?

[John Denker <jsd@av8n.com>]

On 07/24/2013 10:58 AM, Bernard Cleyet wrote:

> Can free-response questions be approximated by multiple-choice
> equivalents? Shih-Yin Lin and Chandralekha Singh
>
> American Journal of Physics -- August 2013 -- Volume 81, Issue 8, pp. 624

In answer to your question, here's my five-word review:
   Not OK.
   Travesty of science.

===============================
Longer version:

Consider the two assertions:

+A)  There exist one or more questions (X) such that X can be represented
     in free-response format AND in multiple-guess format.

+B)  For all questions (X), if X can be represented in free-response
     format then it can also be represented in multiple-guess format.

I assume everybody on this list knows that +A is true and +B is
false.  I might go so far as to say that +A is obviously true and
+B is obviously false.

As a point of formal logic, the negation of +B is:

-B)  There exist one or more questions (X) such that X can be represented
     in free-response format but not in multiple-guess format.

which is obviously true.  Proof by construction.  I hope everybody
on this list can come up with relevant examples.

The paper in question can be found via:
  http://ajp.aapt.org/resource/1/ajpias/v81/i8/p624_s1

It starts by proving the obvious.  It uses two examples to prove 
assertion (+A).

It then *appears* to claim that two examples prove the general case,
i.e. to prove assertion (+B).  Wow, that's quite a leap, from two
examples to the general case.  I emphasize that it *appears* to
prove this, because the English is so non-specific that I cannot
be sure what it is claiming.  The key conclusion is:

> > The findings suggest that research-based MC questions
> > can reasonably reflect the relative performance of students
> > on the free-response questions ....

This claim *appears* to apply to all possible questions, but a
Philadelphia lawyer could argue that the paper doesn't explicitly
say /what/ the conclusions apply to.  Therefore:
 -- If we are generous, the conclusions apply only to two hand-
  selected examples, and the paper is obviously trivial.
 -- If the conclusions are meant to apply more generally, the
  paper is obviously wrong.
 -- In any case, the paper is so badly written that we cannot
  tell whether it is trivial or wrong!

==================================

Sometimes people who ought to know better assume that if
something is published in the peer-reviewed literature, it
must be OK.  This is certainly not true ... especially in
the PER literature.

The publication of papers like this reflects badly not just
on the authors, but also on the reviewers, on the journal, 
and on the entire field.