If you reply to this long (19 kB) post please don't hit the reply
button unless you prune copy of this post that may appear in your
reply down to a few relevant lines, otherwise the entire already
archived post may be needlessly resent to subscribers.
***********************************************
ABSTRACT: Gina Hiatt (2008) of the POD list has called attention to
Kevin Carey's (2008) provocative "Inside Higher Ed" report "Where's
the Data?" Carey wrote:
". . . .when it comes to the central enterprise of higher education -
teaching students - we don't know if the reigning professional
qualification system works, or how many professors we actually need.
And this is true for all kinds of other basic elements of college
teaching and learning - curricula, training, pedagogy, and much more.
. . . . Why doesn't anyone ever study how much learning varies
between [courses], and why?"
Carey, along with most of academia, appears to be either unaware or
dismissive of the fact that formative pre/post testing is being
successfully employed to enhance student learning in many science,
math, and engineering, and economics (but not psychology!) courses.
For introductory physics courses it's been found that
pre-to-post-test average normalized learning gains for "interactive
engagement" courses are about two standard deviations greater than
those for traditional passive-student courses.
***********************************************
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
. . . . . when it comes to the central enterprise of higher education
- teaching students - we don't know if the reigning professional
qualification system works, or how many professors we actually need.
And this is true for all kinds of other basic elements of college
teaching and learning - curricula, training, pedagogy, and much more.
. . . . . Transcript studies . . . .[ Adelman (2004)]. . . .indicate
that 20 percent of all course credits earned by college graduates
come in just 13 introductory courses like English Composition,
Calculus, and Introduction to Economics. Seventy-one percent of all
college graduates take some version of Psychology 101. Calculus is
pretty much calculus wherever you go (or should be). And even in
cases where curricula vary between institutions, larger universities
routinely teach many sections of the same course every semester. Why
doesn't anyone ever study how much learning varies between them, and
why?
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Carey is evidently either unaware or dismissive of fact ever since
the ground-breaking research of Halloun & Hestenes (1985a,b), physics
education researchers have been studying how much student learning
varies between various types of introductory physics courses [see
e.g., Stokstad (2001)], but the "Lessons of the Physics Education
Reform Effort" [Hake (2002a, 2007b)] have been largely ignored by
academia, obsessed with its misconception that Student Evaluations
are valid gauges of the cognitive impact of courses - see e.g., "Re:
Problems with Student Evaluations: Is Assessment the Remedy?"[Hake
(2002b)].
Among those lessons are [Hake (2007b) - SEE THAT ARTICLE FOR THE
REFERENCES other than Hake (2002a; 2005a; 2007a; 2008a,b); Halloun
and Hestenes (1985a,b); Hestenes et al. (1992); Wieman & Perkins
(2005); and Wilson & Bertenthal (2005).]
Lesson 1: THE USE OF 'INTERACTIVE ENGAGEMENT' (IE) STRATEGIES CAN
INCREASE THE EFFECTIVENESS OF CONCEPTUALLY DIFFICULT COURSES WELL
BEYOND THAT OBTAINED BY TRADITIONAL (T) METHODS. . . . . . Education
research in chemistry [Krause et al. (2004)]; engineering [Froyd et
al. (2006), Evans et al. (2003)]; and introductory science education
generally [Handelsman et al. (2004)], although neither as extensive
nor as systematic as that in physics [McDermott and Redish (1999);
Redish (1999); Thacker (2003); Heron & Meltzer (2005); Hake (1998a,b;
2002a,b; 2005a; 2006a,b;2007a,b); Wieman & Perkins (2005); Wieman
(2005)] is consistent with the latter in suggesting that, in
conceptually difficult areas, Interactive Engagement (IE) methods are
more effective than traditional T passive-student methods in
enhancing students' understanding. . . . .[ For introductory physics
courses it's been found that pre-to-post-test average normalized
learning gains for "interactive engagement" courses are about two
standard deviations greater than those for traditional
passive-student courses - for references see "Design-Based Research
in Physics Education Research: A Review" [Hake (2008a)]. . . . .
Furthermore, there is some preliminary evidence that learning in IE
physics courses is substantially retained 1 to 3 years after the
courses have ended [Chabay (1997), Francis et al. (1998), Bernhard
(2001)]. I see no reason to doubt that enhanced understanding and
retention would result from greater use of IE methods in other
science, and even non-science, areas, but substantive research on
this issue is sorely needed - see e.g., "The Physics Education Reform
Effort: A Possible Model for Higher Education?" [Hake (2005a)].
Pre/post testing in biology [Klymkowsky et al. (2003), Klymkowsky
(2007)]; and mathematics [Epstein (2005)] is just getting started;
while pre/post test results in astronomy (Brogt et al. (2007) and
geoscience [Libarkin & Anderson (2005)], have not, at this early
stage, shown clear cut correlations between pre-to-posttest gain and
pedagogical method, as has been shown in physics.
Lesson 3: "HIGH-QUALITY STANDARDIZED TESTS OF THE COGNITIVE AND
AFFECTIVE IMPACT OF COURSES ARE ESSENTIAL TO GAUGE THE RELATIVE
EFFECTIVENESS OF NON-TRADITIONAL EDUCATIONAL METHODS. . . . . So
great is the inertia of the educational establishment that three
decades of physics education research [McDermott and Redish (1999)]
demonstrating the futility of the passive-student lecture
in introductory courses was ignored until Halloun and Hestenes
(1985a,b) devised the Mechanics Diagnostic (MD) test of conceptual
understanding of Newtonian mechanics. Among many other virtues, the
MD and the subsequent Force Concept Inventory (FCI) (Hestenes et al.
1992) tests have two major advantages: (a) the multiple-choice format
facilitates relatively easy administration of the tests to thousands
of students; (b) the questions probe for a
conceptual understanding of the basic concepts of Newtonian mechanics
in a way that is understandable to the novice who has never taken a
physics course, yet at the same time are rigorous enough for the
initiate. Thus the questions can be given as an introductory course
pretest in pre/post tests to directly determine course-induced gain
in conceptual understanding. . . .[for a recent listing of "Formative
Pre/post Tests For Various Disciplines" see Hake (2008b) -
"formative" is used here in the sense defined by JCSEE (1994):
"Formative evaluation is evaluation designed and used to improve an
object, especially when it is still being developed." ]. . . . . In
my opinion such DIRECT gain measurements of higher-order student
learning are far superior to the INDIRECT (and therefore in my view
problematic) gauges have been developed; e.g., Reformed Teaching
Observation Protocol (RTOP), National Survey Of Student Engagement
(NSSE), Student Assessment of Learning Gains (SALG), and Knowledge
Surveys (KS's) [Nuhfer & Knipp (2003)]. For a discussion and
references for all but the last see Hake (2005b.)
BUT WAIT!
1. Can multiple choice tests gauge higher level cognitive outcomes
such as the conceptual understanding of Newtonian mechanics? Wilson &
Bertenthal (2005) think so, writing (p. 94):
"Performance assessment is an approach that offers great potential
for assessing complex thinking and learning abilities, but multiple
choice items also have their strengths. For example, although many
people recognize that multiple-choice items are an efficient and
effective way of determining how well students have acquired basic
content knowledge, many do not recognize that they can also be used
to measure complex cognitive processes. For example, the Force
Concept Inventory . . . [Hestenes, Wells, & Swackhamer, 1992] . . .
is an assessment that uses multiple-choice items to tap into
higher-level cognitive processes."
2. Considering the canonical arguments regarding the invalidity of
pre/post testing evidence, should not all pre-to-post test gains
cited above be viewed with grave suspicion? The dour appraisal of
pre/post testing by Cronbach & Furby (1970) has echoed down though
the literature to present day texts on assessment such as that by
Suskie (2004)]. In my opinion, such pre/post paranoia and its
attendant rejection of pre/post testing in evaluation, as used so
successfully in physics education reform [McDermott and Redish
(1999); Redish (1999); Thacker (2003); Heron & Meltzer (2005);
Wieman & Perkins (2005), Wieman (2005)] is one reason for the glacial
progress of educational research [Lagemann (2000)] and reform [Bok
(2005)].
Fortunately formative pre/post testing is gradually gaining a
foothold in undergraduate astronomy, biology, chemistry, economics,
geoscience, and engineering, in addition to physics. For references
see Hake (2004, 2007c, 2007d).
"The academic area is one of the most difficult areas to change in
our society. We continue to use the same methods of instruction,
particularly lectures, that have been used for hundreds of years.
Little scientific research is done to test new approaches, and little
systematic attention is given to the development of new methods.
Universities that study many aspects of the world ignore the
educational function in which they are engaging and from which a
large part of their revenues are earned." - Richard M. Cyert, former
president of Carnegie Mellon Univ. in "Problem Solving and Education:
Issues in Teaching and Research," ed. by D.T. Tuma and F. Reif
(Lawrence Erlbaum, 1980).
Hake, R.R. 2002a. "Lessons from the physics education reform effort,"
Ecology and Society 5(2): 28; online at
<http://www.ecologyandsociety.org/vol5/iss2/art28/>. Ecology and
Society (formerly Conservation Ecology) is a free online
"peer-reviewed journal of integrative science and fundamental policy
research" with about 11,000 subscribers in about 108 countries. For
an update on six of the fourteen lessons see Hake (2007b).
Hake, R.R. 2007b. "Six Lessons From the Physics Education Reform
Effort," Latin American Journal of Physics Education 1(1), September;
online at <http://journal.lapen.org.mx/sep07/HAKE%20Final.pdf> (124
kB).
Hake, R.R. 2008a. "Design-Based Research in Physics Education
Research: A Review," in "Handbook of Design Research Methods in
Education: Innovations in Science, Technology, Engineering, and
Mathematics Learning and Teaching" [Kelly, Lesh, & Baek (2008)] -
publisher's information at <http://tinyurl.com/4eazqs>; a
pre-publication version of Hake's chapter is online at
<http://www.physics.indiana.edu/~hake/DBR-Physics3.pdf> (1.1 MB).
Hake, R.R. 2008b. "Formative Pre/post Tests For Various Disciplines,"
online as it appears on the OPEN AERA-D archives at
<http://tinyurl.com/6cyemf> where URL's in my post are properly
HOT-LINKED (unlike those on EdResMeth, EdStat, EvalTalk, and POD).
Post of 7 and 8 July to AERA-D, ASSESS, Biopi-L, Chemed-L, DrEd,
EdResMeth, EdStat, EvalTalk, IFETS, Net-Gold, PhysLrnR, POD, RUME, &
WBTOLL-L.
Halloun, I. & D. Hestenes. 1985a. "The initial knowledge state of
college physics students." Am. J. Phys. 53: 1043-1055; online at
<http://modeling.asu.edu/R&E/Research.html>. The print version
contains the Mechanics Diagnostic test, precursor to the Force
Concept Inventory [Hestenes et al. (1992)].
Hestenes, D., M. Wells, & G. Swackhamer. 1992. "Force Concept
Inventory," Phys. Teach. 30(3): 141-158, March; online (except for
the test itself) at <http://modeling.asu.edu/R&E/Research.html>. The
1995 revision by Halloun, Hake, Mosca, & Hestenes is online (password
protected) at the same URL, and is currently available in 15
languages: Chinese, Czech, English, Finnish, German, Greek, Italian,
Malaysian, Persian, Portuguese, Russian, Slovak, Spanish, Swedish, &
Turkish. A French version should soon be available.
Hiatt, G. 2008. "Article from Inside Higher Ed," POD post of 9 Aug
2008 14:26:26 -0400; online at <http://tinyurl.com/5m3nqn>.
JCSEE. 1994. Joint Committee on Standards for Educational Evaluation,
The Program Evaluation Standards, 2nd ed., Sage. A glossary of
evaluation terms from this publication is online at
<http://ec.wmich.edu/glossary/prog-glossary.htf>. For a four-quadrant
delineation of the formative/summative and public/private dimensions
of assessment/evaluation see Hake (2003).
Stokstad, E. 2001. "Reintroducing the Intro Course," Science 293:
1608-1610, 31 August;
online at <http://tinyurl.com/266973>. Stokstad wrote: "Physicists
are out in front in measuring how well students learn the basics, as
science educators incorporate hands-on activities in hopes of making
the introductory course a beginning rather than a finale."
Wieman, C. & K. Perkins. 2005. "Transforming Physics Education,"
Phys. Today 58(11): 36-41; online as a 292 kB pdf at
<http://tinyurl.com/4py56v>. [Wieman is a 2001 Physics Nobelist.]