What the CB does is to maintain a file of questions that have been
field-tested on large numbers of students. They keep track of the
fraction of people who answer each question, by the option they
choose (they are all 5-choice M/C questions), so they know what
fraction get it right and what fraction choose each of the possible
wrong answers. They gather all sorts of statistics about who takes
the test and use all this information to rate each question as to
it's degree of difficulty and what the spread of answers is, much of
which I don't understand. They sort the questions by topic and
difficulty (with a fairly fine mesh, so they can cover any give broad
topic to the same depth on different tests), and using these
criteria, they can construct a test that will give whatever range of
scores they are looking for. The important thing is that they have
data on each individual question--no question is used in a test that
counts before it has been seen by lots of "test" students and they
have gathered lots of statistics on that question.
So they assemble their test using the difficulty of each question to
rate the difficulty of the test as a whole. And they make sure that
the material covered is the same from test to test, so different
subjects are not emphasized differently on different tests.
There are several types of questions that they try to avoid: ones
that students who score high on the test often miss but that students
who score low on the test often get; ones for which the answers
correlate with each other, that is, ones that if most everybody gets
one, they also get the other, or if most everybody misses one they
miss the other, usually with the same wrong answer (and they are on
the same topic), as well as ones that everyone gets or everyone
misses, and a few others.
I don't have a lot more detail, since I didn't actually do this work
and it has been several years since I was exposed to it, but perhaps
you get the idea. It's done on a question by question basis.
Never ask someone what computer they use. If they use a Mac, they
will tell you. If not, why embarrass them?
--Douglas Adams
******************************************************