Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: Reliability and Validity of FCI



Hi all-
I'll repeat my own take on this issue (below).
Regards,
Jack

On Wed, 3 Oct 2001, Dewey Dykstra, Jr. wrote:

I'm looking for studies that investigate the reliability and validity of the
FCI. Does anyone know any articles or research that's been done about this?

-ML


--
Michael Lach
Chicago Public Schools

Both articles introducing the FCI (original in AJP in 1985 and most
recent TPT 1998) have extensive validation efforts and Hake's article
does even more.

Refs for these are in Antti's note.
Dewey

____________________________________________________________________
Posting on Wed, 2 May 2001 18:32:52
Re: Us vs Them - Doubters Club #4

I generally take exception to postings of the kind that go:
"If you had only read .........., you wouldn't have said what you
did." The trouble is that if I go ahead and read ....... I will
probably have no idea of what you had in mind when you asked me to
read ........, so it is unlikely that I will even consider the issue
that you had in mind.
So I don't really know what Dick had in mind when he suggested
the readings, but I did my homework and here are the conclusions.

Recall that the issue is, what is it that we test when we
give the FCI to students. Dick asserts, as I understand him, that we
are testing for a "minimal competency in mechanics." I responded that
the test had never been validated, and the response was in the context
of the stated purpose. Dick then accused me of being ignorant of the
relevant literature, which he cited.
The relevant papers are the first and last. AJP 53 (1985) 1043
does, in fact, claim that the FCI was "validated". Ah, but that does
not end the inquiry; validated for what purpose?
Well, there is a section of the paper headed by words including
"Validity and Reliability". It turns out that "validity" means that
everyone could agree on the correct answers to the questions. Reliability
means that the answers were stable in the sense that a student would
continue giving the same answer if the test were repeated (without
intervening instruction). So the question remains, what was the test
testing?
That's where the TPT paper comes in. It is a response by
Hestenes and Halloun to an earlier paper by Heller and Huffman, and
a rejoinder by the latter authors (TPT 33 (1995) 503). Heller and
Huffman demonstrate, convincingly in my opinion, that the FCI does
not test a student's ability to apply physical concepts. So the obvious
inference is that it does not test for "a minimal competency in
mechanics."
Heller and Huffman make their point by means of factor
analysis. Briefly, this means that they look at different questions
that test the same concept and ask how well the answers are correlated.
If the student understands the concept, or has a particular
misunderstanding about the concept, the correlation should be high.
In other words, Heller & Huffman were checking to see if the student is
applying a concept consistently, rightly or wrongly. The result was that
the correlations were very low. Conclusion: the FCI does not test
student concepts.
Hestenes, et al. contend that the FCI is an accurate predictor
of a student's readiness to undertake the study of Newtonian physics,
a score of 60% denoting the entry threshold. I have no problem with
this as an empirical result, applicable to a particular institution.
But the question still remains, what is it that the FCI is testing?
Regards,
Jack








--
Franz Kafka's novels and novella's are so Kafkaesque that one has to
wonder at the enormity of coincidence required to have produced a writer
named Kafka to write them.
Greg Nagan from "The Metamorphosis" in
<The 5-MINUTE ILIAD and Other Classics>