Re: NYS Physics Regents exam

["John S. Denker" <jsd@MONMOUTH.COM>]

There are some interesting lessons to be drawn about
pedagogy, psychology, and test design.

> ... complaints from across the state that the Physics
> Regents exam was unusually difficult,
...
>  The exam was changed this year with stronger focus on the
> practical application of physics, said Roseanne DeFabio,

We can learn something by pondering the refraction
question (#45) on this year's test
  http://www.emsc.nysed.gov/ciai/testing/scire/psphystestju02.pdf
and comparing it with the corresponding question from
10 years ago:
  http://www.sciencejoywagon.com/physicszone/review/04waves/92q43.html

This year's question is clearly a lot harder.  But why?

The physics is the same.  This year the physics is
combined with a small logic puzzle.

It's simple physics.
It's simple logic.
But somehow the physics+logic question is a lot harder
than the physics-alone question.  Why?

0) In the language of test-design theory, there is one
desired answer and three distractors.  This year has a
much more tempting distractor.  But this still leaves
us with the question of _why_ it is so tempting.

1) I suspect part of it has to do with some sort of
foreground/background "masking" effect.  People like to
latch onto simple explanations.  If there is a simple
explanation in the foreground (in this case, the notion
that the first and third media are the same substance)
people latch onto that;  they don't notice that there's
another explanation lurking in the background.  In the
1992 test, this most-simple explanation was not offered,
so people had no choice but to look for the next-most-simple
explanation.

2) Combining item (0) with item (1), we see that mixing
a little logic with the physics allowed construction of
a much more tempting distractor, something that mentioned
the ultra-simple explanation.

3) I suspect another part of it has to do with expectations.
There is a tradition of designing test questions that test
single facts, one at a time.  So people develop a test-taking
strategy:  figure out what fact the test-author is fishing
for, apply it, and move on.

We now seque to the topic of "Teaching to the Test".  TttT
is to some degree inevitable, especially in a course that
is billed as "AP Physics".  TttT isn't necessarily a bad
thing, if you've got a good test.

One drawback of TttT is that there will be transients
if/when you try to change the test -- as we have just seen.
Teachers in the 2001-2002 school year presumably had little
information about the 2002 test.  Things would have gone a
lot smoother if the Regents had put out a practice test or
some such, to let people know what was coming.

I think that emphasizing the application of physics ideas
to nontrivial situations is perfectly reasonable, but I
can also understand that customers will be irked if the
test differs from expectations.