In his 4/26/01 PhysLrnR post "Re: FCI as a measure of student
understanding," Paul Camp wrote:
"One issue that is known to be a problem (I believe I've seen
Hestenes comment on it somewhere) is that FCI questions also make
good exam questions so students can be coached, even if not
deliberately, to know the answers without necessarily understanding
them. That is why they control distribution these days."
Well, not really. The 1992 version is still available in the open
literature; the 1995 version (which contains many of the same
questions as the 1992 version), although password protected at the
ASU site, is in Mazur's book; and Tarara STILL pushes his online
animated version of the FCI!
Ron Greene responded to the above excerpt from Camp:
"I was surprised . . . (by the quote from Camp) . . . since when I
raised some concern . . . . .(10/21/00 PhysLrnR posts "Re: On-line
FCI") . . . . about having the FCI on-line, no one seemed to be
concerned about it. (Truthfully, I would be more worried about
overuse of the FCI at a given site than its on-line availability.)"
Then Paul Camp responded:
"I vaguely recall this discussion. I seem to remember that the
arguments mostly revolved on the unreliability (or even existence!)
of student memories of what they have seen . . . . (no need to
"vaguely recall" and "seem to remember" - just type "on-line FCI"
into the subject slot at the marvelous archive search engine
<http://listserv.boisestate.edu/archives/physlrnr.html> to obtain 16
hits) . . . To me, the real issue has to do with use of the FCI as a
research instrument. Whether or not students recall the material is
pretty irrelevant. If the possibility exists and has neither been
controlled or measured, then any research results based on the FCI
would be necessarily suspect. It would be akin to simply hoping that
the temperature doesn't affect your resistance measurement."
I agree with Paul Camp that "any research results based on the FCI
would be necessarily suspect" . . . (If students had access to the
test and merely memorized the answers).
An example of this danger was provided by Dan MacIsaac in his
10/22/00 post "Re: from Rao Chekuri Re: on-line FCI": ". . . at
Guelph I spoke to a well-known faculty member who commented that her
students one semester creamed the FCI, scoring incredibly well.
Turns out one of her rather keen students found the FCI article in
the original journal article (TPT?) and studied it with her friends."
I have previously expressed concern (mostly to deaf ears) regarding
the security of the FCI:
A. In footnote #48 of ref. 1, I wrote:
"It is unfortunate that the national-assessment value of arduously
constructed and validated standardized tests such as the FCI and the
MB. . . (Mechanics Baseline). . . is gradually being eroded by
distribution of answers to students at some institutions. The danger
of question leakage is especially severe if the posttest FCI/MB
scores are used to determine part of the final course grade. At
Indiana, the FCI test is always given and referred to as a
"diagnostic mechanics exam" in an attempt to shield ref. 9a. We
collect all pre-and posttests from students and none is returned. The
pre- and post-tests scores are posted by ID, but questions and
answers are neither posted, disseminated, nor shown as computer
animations. After the posttest, instructors are quite willing to
discuss FCI/MB questions privately with any student, but answer keys
are not posted. Because there are many sources (ref. 17a) of good
conceptual questions, there is little need to draw on the
standardized tests for questions to be used for ordinary class
discussion and testing. Indiana students understand that the FCI must
be treated just as the MCAT, and there is little dissatisfaction.
Because of the above mentioned dispersal of answers at some
institutions, and the fact that the FCI and MB tests were published
in the open literature, their useful lives may not extend for more
than another year. New and better tests (treated with the
confidentially of the MCAT) are sorely needed in time for a
calibration against the original or revised FCI. The necessary steps
in the laborious process of constructing valid and reliable
multiple-choice physics tests have been discussed in refs. 1a, 9a,
and 50."
B. In ref. 2 I wrote:
As discussed in ref. 5a, in my view, the present survey is a step in
the right direction but improvements in future assessments might be
achieved through (in approximate order of ease of implementation) . .
. . (7) development and use of new and improved versions of the FCI
and MB tests, treated with the confidentiality of the MCAT. . . ."
C. More recently I wrote(3):
"The lengthy and arduous process of constructing valid and reliable
multiple choice tests has been discussed by Halloun & Hestenes
(1985a), Hestenes et al. (1992), Beichner (1994), Aubrecht (1991),
and McKeachie (1999). In my opinion such hard-won Diagnostic Tests
that cover important parts of common introductory courses are
national assets whose confidentiality should be as well protected as
the MCAT (Medical College Admission Test). Otherwise the test
questions may migrate to student files and thereby undermine
education research that relies upon the validity of such tests.
Suggestions for both administering Diagnostic Tests and reporting
their results so as to preserve confidentiality and enhance
assessment value have been given by Hake (2001b)"
Unfortunately, the latter suggestions on administration and reporting
of test results has yet to be placed on the web.
In his 9/25/00 PhysLrnR post "Re: On-line FCI," Mike Zeilik wrote:
"We struggled with these same issues . . .(test security). . . when
developing the Astronomy Diagnostic Test (ADT). We decided to allow
free access, so that faculty who are browsing do not have to take the
extra step of getting a password. We see no reason for an intro astro
student to 'cheat' by trying to find the ADT on the Web."
Nevertheless, IMHO, as long as the ADT is freely accessible on the
web, any research results based on the ADT will be suspect.
I should like to suggest that physics-education research groups
consider devoting some attention to the "development and use of new
and improved versions of the FCI and MB tests, treated with the
confidentiality of the MCAT," since as time goes on, research results
based on the FCI and MB will become more and more doubtful.
Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<rrhake@earthlink.net>
<http://www.physics.indiana.edu/~hake>
REFERENCES
1. R.R. Hake, "Interactive-engagement vs traditional methods: A
six-thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66, 64-74 (1998); on the Web at
<http://www.physics.indiana.edu/~sdi/>.
2. R.R. Hake, "Interactive-engagement methods in introductory
mechanics courses," on the Web at
<http://www.physics.indiana.edu/~sdi/> and
submitted on 6/19/98 to the "Physics Education Research Supplement to
AJP"(PERS).
3. R.R. Hake, "Lessons from the Physics Education Reform Effort,"
submitted on 3/28/01 to "Conservation Ecology"
<http://www.consecol.org/Journal/>, a "peer-reviewed journal of
integrative science and fundamental policy research." On the web as
ref. 10 at <http://www.physics.indiana.edu/~hake>
[ConEc-Hake-O32601a.pdf, 3/26/01, 172K) (179 references, 98
hot-linked URL's). 164K).