PART 1
If you reply to this LONG (19kB) post please DON'T HIT THE REPLY
BUTTON (the bane of discussion lists) and thereby inflict it yet
again on suffering subscribers.
In his EvalTalk post of 5 Dec 2003 Werner Wittmann (2003a) gave some
reasons for his opposition to the American Evaluation Association
(AEA) statement [Krueger (2003)] on scientifically based evaluation,
among them being (slightly edited; bracketed by lines "WWWWWW. . . .
. ."):
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
e) Looking at education we rarely find randomized experiments, Tom Cook. . .
[<http://www.northwestern.edu/ipr/people/tcook.html>, see e.g., Cook
(2000?, 2001, 2002). . . reviewed the evidence and fairly discussed
the sources of resistance.
g) The evidence about the low percentage of RCT's . . . (Randomized
Control group Trials). . . in education hints to serious biases
against the "Doctor's Method" working in education. To give
incentives to consider more of them as a strategy by DOE is a step in
the right direction!
h) The statement by those leading AEA and in charge of managing the
organization at the moment in my eyes is biased against experiments
["experiments" is a psychologists' code word for RCT's] and I'm
happy that Len Bickman, Bob Boruch, Tom Cook, Dave Cordray, Gary
Henry, Mark Lipsey, Peter Rossi, and Lee Sechrest have attempted to
correct it . . . .[see Lipsey (2003a)."
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
I agree with Wittmann that the DOE's [2003a,b)] emphasis on
"scientifically-based research" (SBR) [see Eisenhart & Towne (2003)
for a guide to the bewildering array of directives from Washingon as
to what constitutes SBR and comparisons of those directives with the
careful NRC study by Shavelson & Towne (2002)] may be a step in the
right direction but the DOE's prescription for perfection of SBR in
"Randomized Control-Group Trials" leaves most physicists cold.
Physicists have made progress in BOTH traditional and educational
research [McDermott & Redish (1999)], not by prescribing rigid
"scientific" research procedures, but by simply "doing their damndest
with their minds, no holds barred" [Bridgman (1947)] in an attempt to
build a "community map" [Redish (1999), Ziman (2000)].
With regard to pre-post studies, so valuable in physics education
research, DOE (2003b) states on page:
v. "Types of studies that do not comprise "possible" evidence
[include] pre-post studies (defined on page 2)."
2. "'Pre-post' study designs often produce erroneous results . . . .
. A "pre-post" study examines whether participants in an intervention
improve or regress during the course of the intervention, and then
attributes any such improvement or regression to the intervention.
The problem with this type of study is that, without reference to a
control group, it cannot answer whether the participants' improvement
or decline would have occurred anyway, even without the intervention.
This often leads to erroneous conclusions about the effectiveness of
the intervention."
17. "Studies that do not meet the threshold for 'possible' evidence
of effectiveness include: (i) pre-post studies (p. 2); . . . ."
There is evidently neither favorable mention nor even recognition in
DOE (2003b) of pre-post studies carried out WITH A CONTROL GROUP
(PPWCG), and such non-RCT studies are ignored by the sources listed
in "Where To Find Evidence-Based Interventions" shown in the
APPENDICES of DOE (2003b) and of the present post. This omission
despite the fact that PPWCG research has demonstrated that in both
high schools and undergraduate classes "interactive engagement"
methods [designed "to promote conceptual understanding through
interactive engagement of students in heads-on (always) and hands-on
(usually) activities which yield immediate feedback through
discussion with peers and/or instructors"] can be about TWO STANDARD
DEVIATIONS MORE EFFECTIVE in enhancing the conceptual understanding
of Newtonian mechanics than are traditional (T) methods that rely on
passive student lectures, algorithmic problem exams, and recipe labs
(Hake 2002 and references therein to the many physics-education
research groups whose work is consistent with the above
interpretation).
In the words of Werner Wittmann (2003) - one of the few psychologists
in the World who is aware physics education research - [slightly
edited; my CAPS; see the article for the references]: "A movement,
ALMOST UNNOTICED BY THE PSYCHOLOGICAL AND EDUCATIONAL RESEARCH
COMMUNITY, in physics called physics education research (PER) has
concentrated on changing the typical learning environments in physics
departments. Dissatisfied with the large lecture hall instructions
with its focus on passive reception and rote learning, 'interactive
engagement methods' have been developed as an alternative [e.g.,
Harvard's Crouch & Mazur (2001), Mazur (1997)]. The latter focus on
conceptual learning by 'Concept Tests' as questions and use
content-valid assessment instruments like the 'Force Concept
Inventory' [Hestenes, Wells, & Swackhamer (1992)]. The Harvard
group's basic philosophy is to force the student into the role of the
teacher and thus capitalize on the old saying that it is the teacher,
who profits most from the learning interaction. The effect sizes for
these and other interactive engagement methods are impressive
compared to traditional instructions [Hake (1998)]. Computing Cohen's
d for these new interactive engagement methods shows EFFECT SIZE d
LARGER THAN 2 STANDARD DEVIATIONS. . .[see Hake (2002)]. . . The new
methods reach a broader cohort of students than the traditional ones
and thus obviously give former disadvantaged groups of students new
chances to learn."