Re: test-design strategy

["John S. Denker" <jsd@MONMOUTH.COM>]

Tina Fanetti wrote:
> Not having taken all sorts of fancy education courses or statistics....
>
> Does it mean anything if the grades are clustered together, say around 60?

1) Test design involves a lot more than statistics.
See item (5) below for more on this.

2) The more tightly the scores are clustered, the more it
doesn't look like a statistics question.

3) Let's assume/pretend for clarity there are 10 questions
worth 10 points apiece.

Most likely everybody got the same 6 right and the same
4 wrong.  (Otherwise it would require a miracle or a
conspiracy to explain why the scores were tightly
clustered.)

So the interesting meaning from this "experiment" doesn't
come from the fact that the typical score was 60.  Rather,
the meaning comes from looking at which 6 were easy and
which 4 were hard.

If you want to dress this up in statistics terminology,
there are 10 separate observations (one for each question)
and to extract maximum information you don't want to
average (or sum) them together prematurely.

In a small class and/or if you do your own grading,
you'll know which questions were hard and which were
easy.  In a large class and/or if your assistants
are doing the grading, make sure they report the
scores for each question separately.

In either case, keep records.  Keep a copy of the test,
with notes about what made each question hard or easy.
This will put you in a stronger position next year.

4) One principle of good experiment design is that
each question should be meaningful.  A super-easy
question that everybody gets right is not very
meaningful.  A super-hard question that everybody
gets wrong is not very meaningful.

But ....

5) A classroom test is not the same as a lab experiment.
Human beings are being tested, and they are very
complicated compared to lab apparatus.  They have
motivational issues.  For this reason, among others,
classroom tests typically contain a number of easy
questions that are not very meaningful from the
experiment-design viewpoint, but are useful as
confidence-builders.