Re: Assessment; evaluation of GRE scores

["John S. Denker" <jsd@MONMOUTH.COM>]

At 09:03 AM 8/23/01 -0400, Jim Peters wrote:

> http://www.pha.jhu.edu/groups/sps/physicsgre.html.

which raises quite a number of interesting issues.  We could discuss it for
a month.

At 11:43 AM 8/23/01 -0400, Michael Edmiston wrote:
> If we abide by the "rules" stipulated by ETS, we faculty members
> are not allowed to look at the exam.

It is bizarre that Michael Edmiston and his students can't benefit from
looking at past exams.  In contrast, according to the URL cited above
        ... in most Chinese physics departments there are books ... about
        an 80-85% chance of having all the questions on the next exam to
        be given.

It seems foolish of the ETS to set "rules" that can't be uniformly
enforced, especially when this gives such a huge advantage to the
rule-breakers.

This is particularly foolish on a test where there is tremendous time
pressure (1.8 minutes per question).  Memorizing the answers to a modest
subset of the questions can greatly increase the time for thinking about
the remaining ones.

Note that it doesn't have to be this way.  For example, the tests designed
by the Federal Aviation Administration are largely immune from this
particular problem, because
  a) they prepare a huge number of questions, typically about 1000, of
which only 50 or 100 are selected to appear on any given test, and
  b) they publish all the questions.

As Mr. Spock would say, this is crude but effective.  It eliminates one
entire category of rule-breaking.  Anybody who wants to memorize the
answers to all 1000 questions is free to do so.

As a corollary, this takes the previously-underground practice of "teaching
to the test" and moves it above ground.

(Of course umpteen other categories of rule-breaking remain.  You can send
in a ringer, you can take extra time, .......)

Anybody who wants to provide a meaningful test, even a pretense of a
meaningful test, needs to do SOMETHING to control the most obvious forms of
rule-breaking.  I'm not fussy about exactly what controls are used, but you
need to do SOMETHING that works.

Testing is important.  Testing is really really important.  It's a shame we
can't have better tests.