Re: [Phys-L] Multiple guess, OK?

[Ze'ev Wurman <zeev@ieee.org>]

Clearly not everything that can be captured by open response can be captured by multiple choice. Just consider writing an essay.

That is not to say, however, that in most situations -- with notable exception like writing a composition or identifying an unknown substance in a lab -- MC items are not as good or better than an open response. Certainly cheaper.

Over the last 3-4 decades a lot of research was done by ETS, College Board and the likes. They generally found that for*similar*  kind of knowledge MC is more reliable. They also couldn't find types of knowledge that don't fall in this category on most normal tests like AP.

I am traveling and w/o access to references, but they are relatively easy to find.

Ze'ev
Sent via BlackBerry by AT&T

-----Original Message----- From: John Denker <jsd@av8n.com> Sender: 
"Phys-l" <phys-l-bounces@phys-l.org> Date: Sat, 27 Jul 2013 05:01:35 To: 
<Phys-L@Phys-L.org> Reply-To: Phys-L@Phys-L.org Subject: Re: [Phys-L] 
Multiple guess, OK? On 07/24/2013 10:58 AM, Bernard Cleyet wrote:
> > Can free-response questions be approximated by multiple-choice
> > equivalents? Shih-Yin Lin and Chandralekha Singh
> >
> > American Journal of Physics -- August 2013 -- Volume 81, Issue 8, pp. 624
> In answer to your question, here's my five-word review:
>     Not OK.
>     Travesty of science.
>
> ===============================
> Longer version:
>
> Consider the two assertions:
>
> +A)  There exist one or more questions (X) such that X can be represented
>       in free-response format AND in multiple-guess format.
>
> +B)  For all questions (X), if X can be represented in free-response
>       format then it can also be represented in multiple-guess format.
>
> I assume everybody on this list knows that +A is true and +B is
> false.  I might go so far as to say that +A is obviously true and
> +B is obviously false.
>
> As a point of formal logic, the negation of +B is:
>
> -B)  There exist one or more questions (X) such that X can be represented
>       in free-response format but not in multiple-guess format.
>
> which is obviously true.  Proof by construction.  I hope everybody
> on this list can come up with relevant examples.
>
> The paper in question can be found via:
>    http://ajp.aapt.org/resource/1/ajpias/v81/i8/p624_s1
>
> It starts by proving the obvious.  It uses two examples to prove
> assertion (+A).
>
> It then *appears* to claim that two examples prove the general case,
> i.e. to prove assertion (+B).  Wow, that's quite a leap, from two
> examples to the general case.  I emphasize that it *appears* to
> prove this, because the English is so non-specific that I cannot
> be sure what it is claiming.  The key conclusion is:
>
> > > The findings suggest that research-based MC questions
> > > can reasonably reflect the relative performance of students
> > > on the free-response questions ....
> This claim *appears* to apply to all possible questions, but a
> Philadelphia lawyer could argue that the paper doesn't explicitly
> say /what/ the conclusions apply to.  Therefore:
>   -- If we are generous, the conclusions apply only to two hand-
>    selected examples, and the paper is obviously trivial.
>   -- If the conclusions are meant to apply more generally, the
>    paper is obviously wrong.
>   -- In any case, the paper is so badly written that we cannot
>    tell whether it is trivial or wrong!
>
> ==================================
>
> Sometimes people who ought to know better assume that if
> something is published in the peer-reviewed literature, it
> must be OK.  This is certainly not true ... especially in
> the PER literature.
>
> The publication of papers like this reflects badly not just
> on the authors, but also on the reviewers, on the journal,
> and on the entire field.
> _______________________________________________
> Forum for Physics Educators
> Phys-l@phys-l.org
> http://www.phys-l.org/mailman/listinfo/phys-l