In his 1/6/2000 post "Re: data on typical FCI scores," Rick Tarara
repeats his concerns about what he regards as "an 'unhealthy'
reliance on the FCI in determining the effectiveness of instructional
techniques." He writes: "It is not at all clear to me that good
performance on the FCI is a sufficient indicator of conceptual
understanding." To make his case, Tarara raises several points which
I shall consider in order:
A. "I can easily teach to the FCI and especially to the types of
questions on the FCI (Third Law questions as presented in the FCI are
a good example) and get 70-75% performance out of a non-scientist
(low math) class. This does not translate into 75% conceptual
understanding of this material."
I think it fairly obvious that teaching to the test in the sense of
going over experiments, questions, or problems identical or nearly
identical to the test items can artificially raise test scores. But
the crucial question is whether or not the instructors of IE courses
of ref. 1 taught to the test in that sense. In refs. 1 and 2, I
counter the "teaching to the test" criticism (as well as various
other criticisms). To quote from ref. 1:
"Teaching to the Test and Test-Question Leakage:
Considering the elemental nature of the FCI questions, for IE courses
both the average <g> = 0.48 + or - 0.14, and maximum <g> = 0.69 are
disappointingly low, and below those which might be expected if
teaching to the test or test-question leakage (3) were important
influences.
Of the 48 data sets(4) for IE courses (a) 27 were supplied by
respondents to our requests for data, of which 22 (81%) were
accompanied by a completed survey questionnaire, (b) 13 have been
discussed in the literature, and (c) 5 are Indiana University courses
of which I have first-hand knowledge. All survey-form respondents
indicated that they thought they had avoided "teaching to the test"
in answering the question "To what extent do you think you were able
to avoid 'teaching to the test(s)' (i.e., going over experiments,
questions, or problems identical or
nearly identical to the test items)?" Likewise, published reports of
the courses in group "b" and my own knowledge of courses in group "c"
suggests an absence of "teaching to the test" in the restricted sense
indicated in the question. (In the broadest sense, IE courses all
"teach to the test" to some extent if this means teaching so as to
give students some understanding of the basic concepts of Newtonian
mechanics as examined on the FCI/Mechanics Diagnostic tests. However
this is the bias we are attempting to measure.)
There has been no evidence of test-question leakage in the Indiana
posttest results (e.g., significant mismatches for individual
students between FCI scores and other course grades). So far there
has been only one report(5) of such leakage in the literature - as
indicated in ref. 4, the suspect data were excised from the survey."
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
B. "For example (as mentioned before) the Force & Motion Conceptual
Evaluation from Tools For Scientific Thinking, CSMT, Tufts U. can
easily show divergent results from the FCI."
What is the evidence for the above assertion? Although there was an
early and tentative indication of divergence in FCI and FMCE(6) test
results at Monroe Community College(4), direct comparisons of <g>FCI
and <g>FMCE have recently been obtained at Rennselaer by Cummings et
al.(7) Their results show quite good agreement:
"To measure gains, we utilized the FCI and the FMCE ....... we
verified the effectiveness of "Interactive Lecture Demonstrations"
[<g>FCI = 0.35 + or - 0.06(sd) and <g>FMCE = 0.45 + or - 0.03(sd)]
....(sd = standard deviation)...... and "Cooperative Group Problem
Solving" [<g>FCI = 0.36 and <g>FMCE = 0.36]. Further, we have
assessed conceptual learning in the standard Studio Physics course
[<g>FCI = 0.18 + or - 0.12(sd) and <g>FMCE = 0.21 + or - 0.05(sd)]."
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
C. "Interactive Techniques developed in one place or by one person
often don't translate into the same level of performance when used by
someone else."
Very true, but this circumstance does not contravene the conclusion
of ref. 1 that "The conceptual and problem-solving test results
strongly suggest that the classroom use of IE methods can
increase..........(NOT "will increase") ........... mechanics-course
effectiveness well beyond that obtained in traditional practice." In
ref. 4, I write "The use of IE methods appears to be necessary but
not sufficient for marked improvement over traditional methods as
demonstrated by seven courses (N = 717) which utilized IE strategies
but achieved <g>'s ranging from 0.21 to 0.26." Ref. 4 offers some
case-study-generated ideas for increasing the effectiveness of IE
strategies as practiced in the classroom.
D. "The classification of IE and traditional techniques was somewhat
suspect in Hake's original draft (sorry I haven't seen the published
paper) and I suspect, often a subjective call regardless (how many
people really do PURE lecture courses today?)."
Before criticizing ref. 1, I think it would be a good idea to
actually read the published version, as well as the recent research
articles(7-11) which are consistent with those results.
REFERENCES
1. R.R. Hake, "Interactive-engagement vs traditional methods: A
six-thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66, 64-74 (1998) and on the Web at
<http://www.physics.indiana.edu/~sdi>.
2. R.R. Hake, "Interactive-engagement vs Traditional Methods in
Mechanics Instruction," APS Forum on Education Newsletter, Summer
1998, p. 5-7, also at <http://www.physics.indiana.edu/~sdi>. Some
criticisms of ref. 1 and of physics-education reform generally are
countered.
3. It is unfortunate that the national-assessment value of arduously
constructed and validated standardized tests such as the FCI and the
Mechanics Baseline (MB) is gradually being eroded by distribution of
answers to students at some institutions (e.g., Tarara's
Phys-L-advertised free "FCI animations." The danger of question
leakage is especially severe if the posttest FCI/MB scores are used
to determine part of the final course grade.
7. K. Cummings, J Marx, R. Thornton, D. Kuhl, "Evaluating
innovations in studio physics, "Physics Ed. Res., supplement 1 to the
Am. J. Phys. 67(7), S38-S44 (1999).
10. R. Beichner, L. Bernold, E. Burniston, P. Dail, R. Felder, J.
Gastineau, M. Gjertsen, J. Risley, "Case study of the physics
component of an integrated curriculum," Physics Ed. Res., supplement
1 to the Am. J. Phys. 67(7), S16-S24 (1999).