Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

[Phys-L] Keep Physics Diagnostic Tests Off the Web! - PART 1



PART 1
Those who dislike long posts (19 kB), references, or have no interest
in Physics Diagnostic Tests, are urged to hit the DELETE button. And
if you reply PLEASE DON'T HIT THE REPLY BUTTON unless you prune the
copy of this post that may appear in your reply down to a few
relevant lines, otherwise the entire already archived post may be
needlessly resent to subscribers.

An interesting Physhare thread titled "Force Concept Inventory," and
containing 9-posts (as of 3 Oct 2005 17:46:00-0700), is accessible in
the October Physhare archives
<http://lists.psu.edu/cgi-bin/wa?A1=ind0510&L=physhare>. Among the
posts is Keith Tipton's of 2 Oct 2005 16:46:11-0500. Keith wrote
[bracketed by lines "TTTTTT. . . "; my CAPS]:

TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT
I believe I saw a discussion about the FCI . . . [Hestenes et al.
(1992); Halloun et al. (1995)]. . . some years ago in which it was
pretty clearly stated that it ought to be kept confidential -- NOT
published on the internet. That was the whole point of the
arrangement of getting the information from Jane Jackson . . .[on how
to get a password for downloading the FCI from the Modelling website
at <http://modeling.asu.edu/>]. . .. Students didn't need to be able
to find the questions or answers if I recall correctly. . . . As
Google has gotten better at searching, you can find these previously
invisible PDF documents rather easily. POSSIBLY THEY NEVER SHOULD
HAVE BEEN PUT ONLINE (other than the Physics Teacher article). . .
.[Right On, Keith !!]

On Oct 2, 2005, at 3:56 PM, Michael Harwood . . . [in his Physhare
post]. . . >wrote: > I have found the text of the force concept
inventory at:



TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT

In my opinion, Keith is probably correct in thinking that he saw a
discussion about the FCI. One such was "Re: Force Concept Inventory
for Richard H" [Hake (2005)]. There I attempted to answer an
AP-Physics subscriber's question related to the validity of the FCI
and then wrote:

"Because the FCI originally appeared in the open literature in 1992
and has, in past years EVEN APPEARED ON THE INTERNET !! :-( , one
must take some care in interpreting the results because the test may
have propagated into some student files. Unfortunately, NO ONE SEEMS
WILLING TO EXERT THE CONSIDERABLE EFFORT TO DEVELOP A REPLACEMENT FOR
THE FCI that would be treated with the same confidentiality as the
MCAT."

Furthermore in Section 5 of "Assessment of Physics Teaching Methods,"
[Hake (2002a)] I wrote [bracketed by lines "HHHHHH. . ."]:

HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
V. Suggestions for the Administration and Reporting of Diagnostic Tests
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .
A . Administration of Diagnostic Tests (DT's)

1. When administering DT's to students, refer to the tests by
home-made generic titles rather than the specific titles designated
by the authors; e.g., "Mechanics Familiarity Survey" rather than
"Force Concept Inventory" or "Force Motion Concept Evaluation).
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .

10. DO NOT MAKE DT QUESTIONS OR PROBLEMS AVAILABLE ON THE WEB unless
they are password protected such that only authorized instructors may
gain access. DO NOT PUBLISH DT'S IN THE OPEN LITERATURE, as has been
the common practice. Carefully constructed DT's . . .[requiring
months or years of arduous qualitative and quantitative research and
testing]. . . are international assets whose confidentiality should
be as well protected as the MCAT (The U.S. Medical College Admission
Test).
. . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . .

11. Because of the almost unavoidable slow diffusion of test
questions and answers to student files, replace each DT at
approximately 5- or 10-year intervals, such that it can be
meaningfully calibrated against the previous test(s). [So far this
has NOT been done for the now overused 1992/95 versions of the FCI;
in my opinion, as time goes on, research results based on
the 1992/95 FCI will become more and more doubtful.]
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

In his post on the Physhare thread referred to above, Rick Tarara
(2005), wrote:

TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT
The problem here is that with the publishing of the test in TPT back in
1992, the authors gave permission for free use of the test by teachers.
After that fact, the PER community started to fixate on the FCI as a
diagnostic tool and people using the FCI for various purposes--like testing
new pedagogy--became upset that there was widespread access to the test.
The authors themselves have not withdrawn the original permission nor really
pressured anyone that had web references to the test to remove
such -- although many in the PER community did so -- and often quite rudely.

Today, many (including myself) who had such references have voluntarily
removed or modified such so as to thwart student searches, although I never
had any evidence (and I track my site carefully) that there ever were such.
The main problem remains however, the test is 13 years old (modified only
slightly), tests a rather narrow set of concepts (which IMO are not nearly
so trivial as many pretend), and has been used far too often to try and
prove things for which it is ill suited--such as specific pedagogy. It is
useful to benchmark student background when given as a pre-test, and if you
desire to get post-instruction information about what students have picked
up about Newton's laws. However, no one should be terribly surprised if
students don't do well on this test if you spend the typical 1 week or less
on Newton's laws. The concepts are more subtle than that! ;-)
TTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTTT


Four comments on Tarara's post. He wrote:

111111111111111111111111111111111111111
1. "The problem here is that with the publishing of the test in TPT back in
1992, the authors gave permission for free use of the test by teachers."

I think it's very doubtful that the authors intended "free use" to
include broadcasting the FCI worldwide by placing it on the internet
where it could easily be accessed by students and thereby severely
compromise its integrity.

2. "The authors themselves . . .[Hestenes, Wells, & Swackhamer
(1992)]. . . have not withdrawn the original permission nor really
pressured anyone that had web references to the test to remove such -
although many in the PER community did so - and often quite rudely."

As stated in "1" above, the "permission" almost certainly did NOT
include permission to publish the FCI on the internet. Otherwise why
is it that at the Arizona State University (ASU) Modelling website
<http://modeling.asu.edu/>, the FCI is NOT available without a
password?

333333333333333333333333333333333333
3. "The main problem remains however, the test is 13 years old (modified only
slightly) [and] tests a rather narrow set of concepts (which IMO are
not nearly so trivial as many pretend). . . . no one should be
terribly surprised if students don't do well on this test [after
spending] the typical 1 week or less on Newton's laws. The concepts
are more subtle than that! ;-) "

I agree with Tarara that the "narrow set of concepts" (at the core of
Newtonian mechanics) is NOT trivial and that instructors should not
be surprised if students do poorly on the FCI after spending the
typical 1 week or less on Newton's laws.

The late Arnold Arons (1990, 1997) quotes the historian Herbert
Butterfield (1965, 1997) thusly:

"Of all the intellectual hurdles which the human mind has confronted
and overcome in the last fifteen hundred years the one which seems to
me to have been the most amazing in character and the most stupendous
in scope of its consequences is the one relating to the problem of
motion."

Yet many physics instructors expect their students to pick up in one
week or less what took humankind some 1500 years to achieve!

44444444444444444444444444444444444444
4. [the FCI] has been used far too often to try and prove things for
which it is ill suited - such as specific pedagogy."

What does Tarara mean by "specific pedagogy"? Does he mean:

(a) "traditional (T) pedagogy," consisting of passive student
lectures, recipe labs, and algorithmic problem homework and exams?;

(b) "interactive engagement (IE) pedagogy" consisting of "heads-on
(always) and hands-on (usually) activities which yield immediate
feedback through discussion with peers and/or instructors?" [Hake
(1998a,b)];

(c) curricula such as e.g., "Peer Instruction," "Workshop Physics,"
"Studio Physics," "Socratic Dialogue Inducing Labs," "Scale Up," or
"Interactive Lecture Demonstrations."

If "a" or "b" is what Tarara means by "specific pedagogy," then
Tarara is flat-out wrong. Physics education researchers have found
that (IE) methods CAN result in an almost two-standard deviation over
T methods in the normalized gains on the FCI [or its precursor the
Mechanics Diagnostic test (Halloun & Hestenes (1985a.b)]. For reviews
see Heron & Meltzer (2005) and Hake (1998a,b)] 2002a,b).

If "c" is what Tarara means by "specific pedagogy," then Tarara must
be either unaware of, or else dismissive of, e.g., the many PER
research articles [referenced in e.g., Meltzer (2005), Redish (2004),
Hake (2002b)] in which the merit of certain curricular programs has
been gauged by NORMALIZED gains [Hake (1998a)]on the FCI.


Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<rrhake@earthlink.net>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>

"What we assess is what we value. We get what we assess, and if we
don't assess it, we won't get it."
Lauren Resnick [quoted by Grant Wiggins (1990)]

REFERENCES are in PART 2