Re: Pendulum acceleration

[Hugh Haskell <hhaskell@MINDSPRING.COM>]

At 9:40 -0500 10/24/01, RAUBER, JOEL wrote:

> John Denker wrote:
>  >
> >  Test-makers should not penalize students for analyzing the
> >  system in ways
> >  the test-makers failed to foresee.
> >
> >  =========================
>
> This is of course a bit of a logical conundrum for the test maker
> (particularly a computerized mass distributed multiple choice test).  How
> can they avoid problems that they fail to foresee?
The way out of the conundrum is to pre-test multiple choice questions
and throw out the ones that have very low correct response rates.
This is one thing that ETS is pretty good about. I don't know what
the IB or ACT (do I have their acronym right? I'm referring to the
competitor of the ETS that is located in Chicago and predominates in
the midwest states) folks do. Of course it's not a perfect solution
and occasionally a klunker will get through. I suspect this might
have been one of those cases.

It is always a problem for the people who make up multiple choice
tests to try to make up questions that will 1) not have a significant
number who get the right answer for the wrong reasons, and 2) not get
the wrong answer for the wrong reasons, i.e., those who analyze the
problem in a way not anticipated by the test makers, and therefore
get an answer that is technically correct but not in concordance with
the anticipated response, and so it gets marked wrong.

It is also important that the questions be such that they do not
"anti-correlate" with overall scores. That is, that the question not
be one which is only correctly answered by those who do poorly on the
test as a whole.

The only way I know of to avoid this problem is to pre-test the
questions and removed those that turn out to the bad, even if they
are the ones best loved by the test-makers.

All of this is why I don't like locally prepared M/C tests, or any
M/C tests that have not been carefully prepared and tested for
validity. This is in addition, of course, to those tests that test
only for factual recall, rather than some level of conceptual
understanding.

IMHO, making up good M/C tests is much harder than making up good
problem tests. Both involve a good deal of labor, the choice is
whether you want to put the labor in before the test or after it. At
least if you put the labor in after the test, you get to see
something of how the students thought, rather than how the
test-makers thought.

Hugh
--

Hugh Haskell
<mailto://haskell@ncssm.edu>
<mailto://hhaskell@mindspring.com>

(919) 467-7610

Let's face it. People use a Mac because they want to, Windows because they
have to..
******************************************************