Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-L] Multiple guess, OK?



And where did they say that?????

I did not see anywhere a statement that all free response questions could be
turned into MC. So that would seem to be inferred because of a personal
bias against MC questions.

The one thing ignored in all this is that the typical free response question
may not be graded well, so a research based MC question might actually be
preferable. The researchers can create much better diagnostics than the
typical teacher. Would you want to have the doctor use a home brew test on
you, or would you rather have a research based certified test done in a
certified lab?

The paper essentially showed that it is possible to make some free response
questions into good MC questions, which is valuable because it makes grading
easier and better. But the big caveat is that they need to be research
based MC questions with appropriate grading rubrics. At present most
scantrons currently in use can not grade that way.

Nowhere is the word proof used in the paper. They "suggest" that carefully
designed MC can reflect the benefits of free response questions. So again
the reader bias has projected an unintended meaning on the paper. This is
the type of thing that happens when a paradigm is challenged. The person
makes up arguments and extreme examples to try to refute the evidence. This
paper is real science as scientific research can not generally prove
assertions, but merely verify them.

John M. Clement
Houston, TX




Can free-response questions be approximated by multiple-choice
equivalents? Shih-Yin Lin and Chandralekha Singh

American Journal of Physics -- August 2013 -- Volume 81,
Issue 8, pp.
624

In answer to your question, here's my five-word review:
Not OK.
Travesty of science.

===============================
Longer version:

Consider the two assertions:

+A) There exist one or more questions (X) such that X can be
+represented
in free-response format AND in multiple-guess format.

+B) For all questions (X), if X can be represented in free-response
format then it can also be represented in multiple-guess format.

I assume everybody on this list knows that +A is true and +B
is false. I might go so far as to say that +A is obviously true and
+B is obviously false.

As a point of formal logic, the negation of +B is:

-B) There exist one or more questions (X) such that X can be
represented
in free-response format but not in multiple-guess format.

which is obviously true. Proof by construction. I hope
everybody on this list can come up with relevant examples.

The paper in question can be found via:
http://ajp.aapt.org/resource/1/ajpias/v81/i8/p624_s1

It starts by proving the obvious. It uses two examples to
prove assertion (+A).

It then *appears* to claim that two examples prove the
general case, i.e. to prove assertion (+B). Wow, that's
quite a leap, from two examples to the general case. I
emphasize that it *appears* to prove this, because the
English is so non-specific that I cannot be sure what it is
claiming. The key conclusion is:

The findings suggest that research-based MC questions can
reasonably
reflect the relative performance of students on the free-response
questions ....

This claim *appears* to apply to all possible questions, but
a Philadelphia lawyer could argue that the paper doesn't
explicitly say /what/ the conclusions apply to. Therefore:
-- If we are generous, the conclusions apply only to two hand-
selected examples, and the paper is obviously trivial.
-- If the conclusions are meant to apply more generally, the
paper is obviously wrong.
-- In any case, the paper is so badly written that we cannot
tell whether it is trivial or wrong!

==================================

Sometimes people who ought to know better assume that if
something is published in the peer-reviewed literature, it
must be OK. This is certainly not true ... especially in the
PER literature.

The publication of papers like this reflects badly not just
on the authors, but also on the reviewers, on the journal,
and on the entire field.
_______________________________________________
Forum for Physics Educators
Phys-l@phys-l.org
http://www.phys-l.org/mailman/listinfo/phys-l