In my previous post "Re: Measuring Content Knowledge" [Hake (2004)] I wrote:
"Many in the Psychology/Education/Psychometric (PEP) community are .
. . . attracted to INDIRECT measures of student learning such as the
Reformed Teaching Observation Protocol RTOP [Lawson (2003), MacIsaac
(2003), MacIsaac & Falconer (2002)]."
Unfortunately, I inadvertently omitted Lawson (2003) from the list of
references in Hake (2004), but have corrected that omission below.
Beth Kubitskey responded in a PhysLrnR/ASSESS post of 14 Mar 2004
20:24:33-0500:
"My experience with RTOP is that it is a means of investigating the level of
inquiry used in a classroom. Studies have been done showing correlations
between RTOP scores and scores on various tests mentioned in the
original post(for instance the FCI). I did not read the research
using RTOP to suggest it a measure of student learning. I did not
see the reference of Lawson (2003) listed and could not check, but my
reading of MacIsaac's work did not lead me to believe they offered
the RTOP as a measure of student learning and in the training I
received in the use of the RTOP by MacIsaac and Falconer they did not
make that claim."
Beth seems to have erected a straw man. In my post I did NOT state
that RTOP was claimed to be a "measure of student learning." Rather I
stated that it is attractive to many as an INDIRECT measure of
student learning.
What is the rational for the last sentence? It is, as Beth points
out, that "studies have been done showing correlations between RTOP
scores and scores on various tests mentioned in the original post(for
instance the FCI)."
For example:
A. MacIsaac & Falconer (2002) wrote.
"In the evaluation of ACEPT, RTOP scores were found to strongly
correlate with student. . . [NORMALIZED !!]. . . conceptual gains
(Fig. 1), showing that reformed teaching . . .[operationally defined
by the authors "to mean those classroom practices that result in a
high RTOP score"] . . . is also effective teaching. Because the
correlation coefficients between RTOP and student achievement gains
were so high (correlation coefficients in the 0.70-0.95 range were
typical), it occurred to us that the items in the instrument might
provide teachers as well as researchers with a window into
understanding reformed teaching.."
"University Physics 1: Mechanics (PHY 121) is an introductory course
designed for physics majors that focuses on mechanics. A course
evaluation was conducted using three experimental sections of PHY 121
(i.e., sections taught by ACEPT-influenced instructors). Two
experimental sections were taught at ASU and one was taught at a
community college. A non-ACEPT-influenced instructor taught the
control section at a community college. A diagnostic test of
mechanics concepts called the Force Concept Inventory (Halloun and
Hestenes, 1985) was administered to all sections to assess pre- to
posttest gains. Instructors' mean RTOP scores and normalized gains
were compared and a strong positive correlation was found (r = 0.97,
p < 0.01). Once again, this indicates that reformed teaching methods
promote student achievement."
Unfortunately Lawson (2003):
1. Fails to reference the half-century-old "normalized gain,"
independently utilized by Hovland (1949), Gery (1972), and Hake
(1998a); but virtually unknown in the
psychology/education/psychometric community.
2. References the Force Concept Inventory (FCI) as Halloun and
Hestenes (1985a), rather than the correct Hestenes et al. (1992).
Halloun and Hestenes (1985a) is the reference for the landmark
"Mechanics Diagnostic Test" the precursor to the FCI;
3. Gives an ambiguous definition in the second paragraph of page 91
<http://books.nap.edu/books/0309089298/html/91.html#pagetop> of
"normalized gain" as "percent gain/percent gain possible" instead of
the rigorous definition [Hake (1998a)] of the average normalized gain
<g> as the actual gain [<%post> - <%pre>] divided by the maximum
possible gain [100% - <%pre>] where the angle brackets indicate the
class averages.
4. Neglects to mention that although the "reformed teaching methods"
utilized in Mechanics (PHY 121), presumably Modeling [Halloun &
Hestenes (1987), Hestenes (1987, 1992, 1997) Wells et al. (1995)],
may yield <g>'s that correlate highly with RTOP scores, it remains to
be shown whether or not similar high correlations exist for
"interactive engagement" methods [see e.g., Hake (1998b)] other than
Modeling.
Note that "interactive engagement" (IE) methods are operationally
defined [Hake (1998a)], even despite the anti-positivist vigilantes
[Phillips (2000)], as:
"those designed at least in part to promote conceptual understanding
through interactive engagement of students in heads-on (always) and
hands-on (usually) activities which yield immediate feedback through
discussion with peers and/or instructors."
Thus IE methods may or may not qualify as "reform methods" :-( under
the restrictive operational definition of MacIsaac & Falconer (2002):
"those classroom practices that result in a high RTOP score."
REFERENCES
Gery, F.W. 1972. "Does mathematics matter?" in A. Welch, ed.,
Research papers in economic education. Joint Council on Economic
Education, pp. 142-157.
Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A
six-thousand-student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66: 64-74; online as ref. 24 at
<http://www.physics.indiana.edu/~hake>. A comparison of the pre- to
post-test average normalized gain <g> for 62 introductory
high-school, college, and university physics courses enrolling a
total 6542 students showed that fourteen "traditional" (T) courses
(N = 2084) which made little or no use of interactive-engagement (IE)
methods achieved an average gain <g>T-ave = 0.23 plus or minus 0.04
(std dev), regardless of the experience, enthusiasm, talents, and
motivation of the lecturers. In sharp contrast, forty-eight courses
(N = 4458) which made substantial use of IE methods achieved an
average gain <g>IE-ave = 0.48 plus or minus 0.14 (std dev), almost
two standard deviations of <g>IE-ave above that of the traditional
courses. Here: (a) the average normalized gain <g> is the actual gain
[<%post> - <%pre>] divided by the maximum possible gain [100% -
<%pre>] where the angle brackets indicate the class averages; (b) IE
courses are operationally defined as those designed at least in part
to promote conceptual understanding through interactive engagement of
students in heads-on (always) and hands-on (usually) activities which
yield immediate feedback through discussion with peers and/or
instructors; (c) T courses are operationally defined courses as those
reported by instructors to make little or no use of IE methods,
relying primarily on passive-student lectures, recipe labs, and
algorithmic problem exams. MORE RECENTLY, NORMALIZED GAIN DIFFERENCES
BETWEEN T AND IE COURSES THAT ARE CONSISTENT WITH THE WORK OF HAKE
(1998a,b) HAVE BEEN REPORTED BY MANY OTHER PHYSICS EDUCATION RESEARCH
GROUPS AS REFERENCED IN HAKE (2002a,b).
Hake, R.R. 1998b. "Interactive-engagement methods in introductory
mechanics courses," online as ref. 25 at < http://www.physics.indiana.edu/~hake >. Submitted on 6/19/98 to the
Physics Education Research Supplement to AJP (PERS)." In this sadly
unpublished (Physics Education Research has no archival journal!)
crucial companion paper to Hake (1998a): average pre/post test
scores, standard deviations, instructional methods, materials used,
institutions, and instructors for each of the survey courses of Hake
(1998a) are tabulated and referenced. In addition the paper
includes: (a) case histories for the seven IE courses of Hake (1998a)
whose effectiveness as gauged by pre-to-post test gains was close to
those of T courses, (b) advice for implementing IE methods, and (c)
suggestions for further research.
Hake, R.R. 2002a. "Lessons from the physics education reform effort,"
Conservation Ecology 5(2): 28; online at
<http://www.consecol.org/vol5/iss2/art28>. Conservation Ecology is a
free "peer-reviewed journal of integrative science and fundamental
policy research" with about 11,000 subscribers in about 108 countries.
Hake, R.R. 2002b. "Assessment of Physics Teaching Methods,
Proceedings of the UNESCO-ASPEN Workshop on Active Learning in
Physics, Univ. of Peradeniya, Sri Lanka, 2-4 Dec. 2002; also online
as ref. 29 at < http://www.physics.indiana.edu/~hake/ >.
Halloun, I. & D. Hestenes. 1985a. "The initial knowledge state of
college physics students." Am. J. Phys. 53: 1043-1055; online at
<http://modeling.asu.edu/R&E/Research.html>. Contains the landmark "Mechanics
Diagnostic" test, precursor to the much used "Force Concept
Inventory" [Hestenes et al. (1992)].
Hestenes, D. 1997. "Modeling Methodology for Physics Teachers," in
E. Redish & J. Rigden. eds. "The changing role of the physics
department in modern universities," American Institute of Physics,
Part II. p. 935-957; online at
<http://modeling.asu.edu/R&E/Research.html>.
Hestenes, D., M. Wells, & G. Swackhamer. 1992. "Force Concept
Inventory," Phys. Teach. 30: 141-158; online (except for the test
itself) at <http://modeling.asu.edu/R&E/Research.html>. For the 1995
versions see Halloun et al. (1995).
Hovland, C. I., A. A. Lumsdaine, and F. D. Sheffield. 1949. "A
baseline for measurement of percentage change." In C. I. Hovland, A.
A. Lumsdaine, and F. D. Sheffield, eds. 1965. Experiments on mass
communication. Wiley (first published in 1949).) Reprinted as pages
77-82 in P. F. Lazarsfeld and M. Rosenberg, eds. 1955. The language
of social research: a reader in the methodology of social Research."
Free Press.
Lawson, A.E. 2003. "Using the RTOP to Evaluate Reformed Science and
Mathematics Instruction," in McCray et al. (2003): 89-100. In my view
it's ironic that the McCray et al. (2003) provide a complete
"commissioned paper" by Lawson on the RTOP, an INDIRECT measure of
student learning at best, while totally neglecting the landmark
DIRECT measure of student learning by Halloun & Hestenes (1985a) and
the pre/post testing movement that it initiated.
McCray, R.A., R.L. DeHaan, J.A. Schuck, eds. 2003. "Improving
Undergraduate Instruction in Science, Technology, Engineering, and
Mathematics: Report of a Workshop" Committee on Undergraduate STEM
Instruction," National Research Council, National Academy Press;
online at <http://www.nap.edu/catalog/10711.html>. Physicists and
astronomers attending the workshop were Paula Heron, Priscilla Laws,
John Layman, Ramon Lopez, Richard McCray, Lillian McDermott, Carl
Wieman, Jack Wilson, and (believe it or not) pre/post champion Mike
Zelik.
Phillips, D.C. 2000. "Expanded social scientist's bestiary: a guide
to fabled threats to, and defenses of, naturalistic social science."
Rowman & Littlefield.
Wells, M., D. Hestenes, G. Swackhamer. 1995. "A modeling method for
high school physics instruction," Am. J. Phys. 63(7):606-619; online
at <http://modeling.asu.edu/R&E/Research.html>.