Re: Reliability and Validity of FCI

["Dewey Dykstra, Jr." <dykstrad@EMAIL.BOISESTATE.EDU>]

Several things are worth noting in Jack's comments:

1.  Jack says:

> "Heller and
> Huffman demonstrate, convincingly in my opinion, that the FCI does
> not test a student's ability to apply physical concepts. So the obvious
> inference is that it does not test for "a minimal competency in
mechanics."

There are two things in this passage that are important (1) what does
it mean "student's ability to apply physical concepts"?  and (2) what
does it mean "minimal competency in mechanics"?

It strikes me that H & H were questioning whether or not the FCI was
detecting the presence of a unitary concept or not.  It is very clear
from the research that students ALWAYS apply physical concepts.  The
question is just what is the nature of the physical concepts they are
applying.  It is equally clear that the physical concepts they apply
are not always those that resemble what we take to be the physicists'
physical concepts.  As I recall the "minimal competency" (in
actuality a conceptual threshold) in the articles by Hestenes et al
is defined by another test the MBT-Mechanics Baseline Test and how
the FCI scores correlate with the MBT scores and that this was done
for students at several institutions I can think of ASU and Harvard
to name two.  I suspect that given these two and Hake's paper one
could reasonably argue that this threshold could be expected to apply
to many institutions not just a particular one.  So whether one
agrees with H&H or Hestenes et al, the inference is not so obvious
that the FCI does not test for a minimal competency in mechanics
because the Hestenes et al did not base their claim for this
competency on whether or not there was some unitary factor.

It's also the case from other work that the evidence supports an
argument that during transition from one view to another students
will pick choices consistent with one view for certain circumstances
and the other view under other circumstances.  Such students choices
would not be consistently one view or the other but consistently
either--low correlation with either view.  How would the factor
analysis come out then?

I do not know where the quotations come from in Jack's note.  I
presume from other e-mail notes from other folks.  But, in the H et
al, '92 TPT paper, they are not talking "minimal competency in
mechanics" or "ability to undertake the study of Newtonian physics",
what they do say is that "Our data suggest that there exists a kind
of conceptual threshold near 60% on the Inventory.  Below this
threshold, a student's grasp of Newtonian concepts is insufficient
for effective problem solving."   They suggest that problem solving
is a kind of sine qua non that students scoring below 60% on the FCI
appear to need more conceptual work before expectation of
problem-solving is reasonable.

For the record Huffman and Heller say in their March, 95, TPT
article:  "Based upon the results of this factor analysis, it appears
the inventory can still be used as a diagnostic tool and as a means
for evaluating instruction, keeping in mind that the inventory may be
measuring small bits and pieces of students' knowledge rather than a
central force concept, and may also be measuring students'
familiarity with the contest rather than understanding of a concept."

Finally, as I look more closely at the articles involved I notice
that Huffman and Heller say in their March 95 TPT article:

"The knowledge-in-pieces perspective and the p-prims can be used to
explain the apparent lack of relationship between the items on the
FCI."

Nowhere do they give any indication of the kinds of scores the
students in their sample had.  If their scores are below 50% which is
not unreasonable a guess given the results we see in Hake, then it is
reasonable to imagine that their view of forces would show up as
factors very different than Newtonian and in fact if the PER is any
good then the factors OUGHT to show up in a manner consistent with
the k-i-p view and p-prims.  Hestenes et al in their response say:

"...they would need a certified Newtonian population, such as a group
of physics professors.  In that case, we guarantee that they would
find near perfect clustering of their data about a single factor,
every question correlating almost perfectly with every other one."

So it seems to me that Hestenes et al are saying the FCI is for them
a measure of the students' force concept.  In effect the degree to
which the score departs from 100% is the degree to which their
concept differs from a Newtonian one.  They also seem to make the
claim that scores above about 85% are what they would call mastery of
the Newtonian force concept (conceptual mastery not mastery of
mechanics) and below about 60% students are not yet conceptually
ready to be successful at problem-solving in mechanics.  I think they
would say that the FCI DOES measure student concepts and that to
start with they are usually profoundly different, incommensurate,
with the Newtonian view of force.  They are also pointing out that
the Huffman and Heller data actually reveal some of the structure of
the alternative view of force in the alternative factors that showed
up in their data.

If the students started out with no idea and moved to a newtonian
view then we might expect FCI results to be essentially factor-less
to start with and as a class' understanding of the newtonian view
increase the newtonian factors would begin to appear in the data on
repeated samples over time.  What would be extremely interesting is
to look at the factor structure for different score ranges say 0 - 20
%, 20% - 40%,etc. and to look at large numbers of these from
different settings as in the Hake study.  If what both groups are
saying, consistent with PER, is the case then one might expect to see
some sort of factor structure from the lowest scores on up.  This
factor structure would change, possibly in identifiable ways
consistent with qualitative studies of student conceptions ending up
looking like a Newtonian factor structure.  It appears to me that
this is entirely possible given the content of the articles on this
topic so far.

Dewey
PS:  Jack, you might not agree with my conclusions, but thanks for
pushing me into looking back at all this.

--
(As of 12/21/01 the mailserver email.boisestate.edu will cease to
exist.  My new address is:  dewey@mac.boisestate.edu)
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Dewey I. Dykstra, Jr.                 Phone: (208)426-3105
Professor of Physics                  Dept:  (208)426-3775
Department of Physics/MCF421/418      Fax: (208)426-4330
Boise State University            dykstrad@email.boisestate.edu
1910 University Drive                 Boise Highlanders
Boise, ID 83725-1570                  novice piper: GHB, Uilleann

"As a result of modern research in physics, the ambition and hope,
still cherished by most authorities of the last century, that physical
science could offer a photographic picture and true image of reality
had to be abandoned."  --M. Jammer in Concepts of Force, 1957.

"If what we regard as real depends on our theory, how can we make
reality the basis of our philosophy? ...But we cannot distinguish
what is real about the universe without a theory...it makes no sense
to ask if it corresponds to reality, because we do not know what
reality is independent of a theory."--S. Hawking in Black Holes
and Baby Universes, 1993.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++