In her POD post of 30 Jul 2003 08:46:56-0400 titled "Designing
Pretests" Judith Miller wrote:
"Can anyone recommend a good reference on designing pretests? I am
particularly interested in pretests of content, rather than attitude. Thanks!"
IMHO, if Judith is interested in pretests of CONTENT, then I think
she should consult the disciplinary content experts. In the case of
physics, the classic papers on the design of pre- and post-tests are
by Halloun & Hestenes (1985a,b), as I have pointed out most recently
(Hake 2003a) in a post (Hake 2003a) provocatively titled "NRC's CUSE:
Oblivious of the Advantage of Pre/Post Testing With High Quality
Standardized Tests?"
As I wrote in Hake (2002a):
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
In his recent review of assessment, or lack thereof, in introductory
undergraduate science courses in the U.S., Stokstad (2001) wrote:
"PHYSICISTS ARE OUT IN FRONT IN MEASURING HOW WELL STUDENTS LEARN THE
BASICS, as science educators incorporate hands-on activities in hopes
of making the introductory course a beginning rather than a finale."
(My CAPS.)
This lead position is due in no small part to pre/post testing using
the "Force Concept Inventory" (FCI) [Hestenes et al. (1992), Halloun
et al. (1995)]; the Mechanics Diagnostic (MD) test - precursor to the
FCI [Halloun & Hestenes (1985 a,b)]; the "Force Motion Concept
Evaluation" (FMCE) [Thornton & Sokoloff (1998)]; and the "Conceptual
Survey of Electricity and Magnetism" (CSEM) [Maloney et al. (2001)].
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
The highly successful and extensive physics experience with pre/post
testing [for reviews see, e.g., Hake (2002a,b)] suggests that IF the
test is designed to measure conceptual understanding (as opposed to
rote memorization) in a conceptually difficult areas such as
introductory physics, then whether or not students take the pretest
has little effect on the posttest results [see e.g., Henderson et al.
(1999), Henderson (2002)].
Even despite the apparent lack of interest by the NRC and its expert
committees [McCray et al. (2003); NRC (1997, 1999, 2003); Pelligrino
et al. (2001), Shavelson & Towne (2002)] and even despite the general
disdain of the psychology/education/psychometric (PEP) community
[Hake (2001)]; pre/post testing with high quality standardized tests
is beginning to gain a foothold in undergraduate education. For
references to pre/post testing in astronomy, biology, chemistry,
computer science, economics, and engineering see Hake (2003a,b).
Its not clear to me why the NRC and its committees adamantly refuse
to appreciate the significance of pre/post test evidence from
hundreds of classrooms [Stockstad (2001), Hake (2002b)] that indicate
the substantial superiority of interactive engagement (IE) over
traditional (T) methods of instruction [for operational definitions
of IE and T methods see Hake (1998a)]. In one study [Hake (1998a,b;
2002b)] Cohen's (1988) "effect size" d = 2.43, much higher than d's
listed in the meta-meta-analysis of Lipsey & Wilson (1993).
REFERENCES
Cohen, J. 1988. "Statistical power analysis for the behavioral
sciences." Lawrence Erlbaum, 2nd ed.
Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A
six-thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66: 64-74; online as ref. 24 at
<http://www.physics.indiana.edu/~hake>.
Hake, R.R. 1998b. "Interactive-engagement methods in introductory
mechanics courses," online as ref. 25 at
<http://www.physics.indiana.edu/~hake>. SUBMITTED on 6/19/98 to the
"Physics Education Research Supplement to AJP"(PERS). In this SADLY
UNPUBLISHED (Physics Education Research has NO archival journal!)
crucial companion paper to Hake (1998a): average pre/post test
scores, standard deviations, instructional methods, materials used,
institutions, and instructors for each of the survey courses of Hake
(1998a) are tabulated and referenced. In addition the paper
includes: (a) case histories for the seven IE courses of Hake (1998a)
whose effectiveness as gauged by pre-to-post test gains was close to
those of T courses, (b) advice for implementing IE methods, and (c)
suggestions for further research.
Hake, R.R. 2002a. "Assessment of Physics Teaching Methods,"
Proceedings of the UNESCO-ASPEN Workshop on Active Learning in
Physics, Univ. of Peradeniya, Sri Lanka, 2-4 Dec. 2002; also online
as ref. 29 at <http://www.physics.indiana.edu/~hake/>.
Hake, R.R. 2002b. "Lessons from the physics education reform effort."
Conservation Ecology 5(2): 28; online at
<http://www.consecol.org/vol5/iss2/art28>. "Conservation Ecology," is
a FREE "peer-reviewed journal of integrative science and fundamental
policy research" with about 11,000 subscribers in about 108 countries.
Hake, R.R. 2003a. "NRC's CUSE: Oblivious of the Advantage of Pre/Post
Testing With High Quality Standardized Tests?" post of 25 Jul 2003
13:07:23-0700 to ASSESS, Biopi-L, Chemed-L, EvalTalk, PhysLrnR, and
POD; online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0307&L=pod&O=D&P=17145>.
Halloun, I. & D. Hestenes. 1985a. "The initial knowledge state of
college physics students." Am. J. Phys. 53:1043-1055; online at
<http://modeling.asu.edu/R&E/Research.html>. Contains the "Mechanics
Diagnostic" test. This landmark work is NOT referenced in McCray et
al. (2003); NRC (1997, 1999, 2003); or the NRC volumes Pelligrino et
al. (2001) and Shavelson & Towne (2002).
Halloun, I., R.R. Hake, E.P Mosca, D. Hestenes. 1995. Force Concept
Inventory (Revised, 1995); online (password protected) at
<http://modeling.asu.edu/R&E/Research.html>. (Available in English,
Spanish, German, Malaysian, Chinese, Finnish, French, Turkish, and
Swedish.) NOT referenced in McCray et al. (2003); NRC (1997, 1999,
2003); or the NRC volumes Pelligrino et al. (2001) and Shavelson &
Towne (2002).
Hestenes, D., M. Wells, & G. Swackhamer, 1992. "Force Concept
Inventory." Phys. Teach. 30: 141-158; online (except for the test
itself) at <http://modeling.asu.edu/R&E/Research.html>. NOT
referenced in McCray et al. (2003); NRC (1997, 1999, 2003); or the
NRC volumes Pelligrino et al. (2001) and Shavelson & Towne (2002).
For the 1995 update see Halloun et al. (1995).
Lipsey, M.W. & D.B. Wilson. 1993. "The Efficacy of Psychological,
Educational, and Behavioral Treatment: Confirmation From
Meta-Analysis," American Psychologist 48(12): 1181-1209; online at
<http://www.vanderbilt.edu/cerm/MWL_web_bib.htm#methods%20and%20issues>
under "Program Evaluation, Methods for Intervention Research." See
also Wilson & Lipsey (2001).
Maloney, D., T.L. O'Kuma, C.J. Hieggelke, & A. Van Heuvelen. 2001.
"Surveying students' conceptual knowledge of electricity and
magnetism," Physics Education Research Supplement to Am. J. Phys
69(7): S12-S23.
McCray, R.A., R.L. DeHaan, J.A. Schuck, eds. 2003. "Improving
Undergraduate Instruction in Science, Technology, Engineering, and
Mathematics: Report of a Workshop" Committee on Undergraduate STEM
Instruction," National Research Council, National Academy Press;
online at <http://www.nap.edu/catalog/10711.html>. Physicists
attending the workshop were Paula Herron, Priscilla Laws, John
Lehman, Ramon Lopez, Richard McCray, Lillian McDermott, Carl Wieman,
and Jack Wilson.
NRC. 1997. "Science Teaching Reconsidered: A Handbook," National
Research Council, Committee on Undergraduate Science Education,
National Academy Press; online at
<http://www.nap.edu/catalog/5287.html>.
NRC. 1999. "Transforming Undergraduate Education in Science,
Mathematics,Engineering, and Technology," National Research Council,
Committee on Undergraduate Science Education, National Academy Press;
online at <http://www.nap.edu/catalog/6453.html>.
NRC. 2003. "Evaluating and Improving Undergraduate Teaching in
Science and Technology, Engineering, and Mathematics," ed. by M.A.
Fox & N. Hackerman, National Research Council, Committee on
Undergraduate Science Education, National Academy Press; online at
<http://www.nap.edu/catalog/10024.html>.
Pelligrino, J.W., N. Chudowsky, R. Glaser, eds. 2001. "Knowing What
Students Know: The Science and Design of Educational Assessment,"
National Academy Press; online at
<http://www.nap.edu/catalog/10019.html>.
Stokstad, E. 2001. "Reintroducing the Intro Course." Science 293:
1608-1610, 31 August 2001: online at <http://www.sciencemag.org/>.
Non-AAAS members may access the article by taking a few minutes to
complete a free limited-access registration.
Thornton, R.K. & D.R. Sokoloff. 1998. "Assessing student learning of
Newton's Laws: The force and motion conceptual evaluation and the
evaluation of active learning laboratory and lecture curricula," Am.
J. Phys. 66(4): 338-352. See also Thornton (1995).
Thornton, R.K. 1995. "Conceptual Dynamics: Changing Student Views of
Force and Motion," in "Thinking Physics for Teaching," Bernardini, C,
C. Tarsitani, & M. Vicentini, eds. Plenum.
Wilson, D.B. & M.W Lipsey. 2001. "The Role of Method in Treatment
Effectiveness Research: Evidence from Meta-Analysis." Psychological
Methods 6(4): 413-429; online at
<http://www.vanderbilt.edu/cerm/MWL_web_bib.htm#methods%20and%20issues>
under "Program Evaluation, Methods for Intervention Research."