Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: Is the FCI valid?



In his 1/11/2000 Phys-L post "Re: Is the FCI valid?" Robert Cohen
asks three good questions (Q) to which I shall respond (R) in order:

Q1. "Is it true that FCI results are only valid when they are NOT
used as part of the grade?"

R1. In my opinion, NO! Unless there is some significant influence
of the FCI posttest score on the final-grade, some students may be
inclined to exert near-zero effort, even to the extent of responding
to the questions randomly or in mechanical sequence fashion (as
option A, A, A, A; etc. ..... or A, B, C, D, E; A, B, C, D, E;
etc....). Then the average posttest score for the class will not
accurately reflect the average final knowledge state of the students,
and the normalized gain <g> may be artificially lowered.
Nevertheless, FOR THE COURSES SURVEYED IN REF. 1a, I concluded that
whether or not grade credit was given for the posttest was not a
significant factor in <<g>> (the average of the class averages <g>)
for the Interactive Engagement courses of that study. In discussing
possible systematic errors in pre/post testing, I wrote(1a):

"4. Post and Pretest Motivation of Students
......... of the 48 data sets(1b) for Interactive Engagement courses,
27 were supplied by respondents to our requests for data, of which 22
were accompanied by a completed survey questionnaire. Responses to
the question "Did the FCI posttest count as part of the final grade
in your course? If so give the approximate weighting factor" were:
"No" (50% of the 22 courses surveyed); "Not usually" (9%); "Yes,
about 5%" (23%); "Yes, weighting factor under 10%" (9%); No Response,
9%. For the 11 courses for which no grade incentives were offered
<<g>>IE11 = 0.49 ± 0.10sd, close to the average <g> for all the 48 IE
courses of the survey <<g>>IE48 = 0.48 ± 0.14sd. Thus it seems
doubtful a significant factor in determining the normalized gain."

Nevertheless, I would strongly urge instructors NOT to give the
posttest FCI as a "throw-away" zero-credit exam.

John Mallinckrodt gave some advice on FCI posttest strategy in a
5/19/98 PhysLrnR post "Random responses on the FCI (was Re: summary
of references on ...)":

"I give students the equivalent of one perfect quiz score to be averaged
with the others for a "good faith effort" on the FCI and I tell them that
I have a way of determining whether or not they have done so. I have good
reason to believe that I do......

What I do is subject each returned answer sheet to what I call 'the random
guess test' which determines the number of 'rarely given wrong responses.'
I determined the 'rarely given wrong responses' experimentally, but they
match up very well with what common sense (aided by a little experience
with student preconceptions) would suggest. Any student who randomly
fills out the answer sheet should be expected to get about 6 (+/- a few)
answers right (not so uncommon) and also to give about 14 (+/- a few) of
the rarely given wrong responses" (VERY uncommon).

When I find a strong 'random guess' signature, I simply tell the student
privately (and without rancor) that I won't be assigning the reward
quiz score. I haven't yet heard any complaints."

Unfortunately, Mallinckrodt's "random guess test" will not uncover
ALL tests displaying less than students' true final knowledge states.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Q2. If so .....(If it IS true that FCI results are only valid when
they are NOT used as part of the grade?)..... can we really expect
gains of 100% when students know the exam doesn't affect their grade?)

R2. Since it is NOT true, this question is unnecessary. IF it WERE
true, then I would agree that one could not expect AVERAGE normalized
gains (for classes of say, N > 20) of 100%. But one should not
expect such average gains even for posttests where there are grade
incentives (as far as I know, they have never been obtained in
Mazur's Harvard classes - even most professional physicists will
usually flub a question or two on the FCI).

xxxxxxxxxxxxxxxxxxxxxx

Q3. If not, can I use the FCI as an ungraded pre-test and as a graded
post-test? Does this invalidate the gain scores?

R3. In my opinion YES! to the first question and NO! to the second.
As I wrote in ref. 1a:

"As for the pretest, grade credit is, of course, inappropriate but
<g> can be artificially raised if students are not induced (2) to
take the pretest seriously. All surveyed instructors answered "Yes"
to the survey form question "Do you think that your students exerted
serious effort on the FCI pretest?" Likewise, published reports of
the courses not surveyed and my own knowledge of courses at Indiana
suggests that students did take the pretest seriously."

BTW, I would strongly urge instructors NOT to forego pretesting,
since then any comparison with the extensive present national <g>
database (1a,1d) is impossible.

Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<rrhake@earthlink.net>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>
<http://www.physics.indiana.edu/~redcube>

Ludwig Wittgenstein: "It is only the attempt to write down your ideas
that enables them to develop." (Drury, 1982).


REFERENCES
1. R.R. Hake:
(a) "Interactive-engagement vs traditional methods: A
six-thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66, 64-74 (1998);

(b) "Interactive-engagement methods in introductory mechanics
courses," submitted on 6/19/98 to the "Physics Education Research
Supplement to AJP"(PERS).

(c) "Interactive-engagement vs Traditional Methods in Mechanics
Instruction," APS Forum on Education Newsletter, Summer 1998, p. 5-7
(counters some objections to "1a" and physics-education research
generally);

(d) "Towards Paradigm Peace in Physics Education Research," accepted
for presentation at the meeting of the American Educational
Association, New Orleans, April 24-28, 2000; lists recent
physics-education research which is consistent with ref. 1a, and
discusses the sometimes unappreciated complementarity of
quantitative(1a,b) and qualitative research in physics education.

All the above articles are on the web as Adobe Acrobat Portable
Document Files (PDF's) at <http://physics.indiana.edu/~sdi/>.

2. At Arizona State (3) and Indiana University (4) it is explained to
students that their scores on the pretest will not count towards the
course grade but will be confidentially returned to them (see ref. 5)
and will assist both themselves and their instructors to know the
degree and type of effort required for them to understand mechanics.

3. I. Halloun and D. Hestenes, "The initial knowledge state of
college physics students," Am. J. Phys. 53, 1043-1055 (1985).

4. R.R. Hake, "Promoting student crossover to the Newtonian world,"
Am J. Phys. 55, 878-884 (1987).

5. It is unfortunate that the national-assessment value of arduously
constructed and validated standardized tests such as the FCI and the
Mechanics Baseline (MB) is gradually being eroded by distribution of
answers to students at some institutions. The danger of question
leakage is especially severe if the posttest FCI/MB scores are used
to determine part of the final course grade. At Indiana, the FCI
test is always given and referred to as a "diagnostic mechanics exam"
in an attempt to shield ref. 6. We collect all pre-and posttests
from students and none is returned. The pre- and post-tests scores
are posted by ID, but questions and answers are neither posted,
disseminated, nor shown as computer animations. After the
posttest, instructors are quite willing to discuss FCI/MB questions
privately with any student, but answer keys are not posted. Because
there are many sources (1b) of good conceptual questions, there is
little need to draw on the standardized tests for questions to be
used for ordinary class discussion and testing. Indiana students
understand that the FCI must be treated just as the MCAT, and there
is little dissatisfaction. Because of the above mentioned dispersal
of answers at some institutions, and the fact that the FCI and MB
tests were published in the open literature, their useful lives may
not extend for more than another year. New and better tests (treated
with the confidentially of the MCAT) are sorely needed in time for a
calibration against the original or revised FCI. The necessary steps
in the laborious process of constructing valid and reliable
multiple-choice physics tests have been discussed in refs. 3, 6a, and
7.

6. (a) D. Hestenes, M. Wells, and G. Swackhamer, "Force Concept
Inventory," Phys. Teach. 30, 141-158 (1992), (b) I.Halloun, R.R Hake,
E.P. Mosca, D. Hestenes, "Force Concept Inventory (revised, 1995),
password protected at <http://modeling.la.asu.edu/modeling.html>.

7. (a) R.J. Beichner, "Testing student interpretation of kinematics
graphs," Am. J. Phys. 62, 750-762 (1994); (b) S.J. Sobolewski,
"Development of Multiple-Choice Test Items," Phys. Teach. 34, 80-82
(1996); (c) W. Pfeiffenberger, A.M. Zolandz, and L. Jones, "Testing
Physics Achievement: Trends over Time and Place," Phys. Today 44(9),
30-37 (1991); (d) G.J. Aubrecht, "Is There a Connection Between
Testing and Teaching?" J. Coll. Sci. Teach. 20, 152-157 (1991); G.J.
Aubrecht and J.D. Aubrecht,
"Constructing Objective Tests," Am. J. Phys. 51, 613-620 (1983).