If you reply to this long (15 kB) post please don't hit the reply
button unless you prune copy of this post that may appear in your
reply down to a few relevant lines, otherwise the entire already
archived post may be needlessly resent to subscribers.
***************************************
ABSTRACT: In a recent post "Can Pre-to-posttest Gains Gauge Course
Effectiveness? #2," I wrote: "These [pre/post studies] have been
carried out on many different instructors, in many different
institutions, using many different texts, and working with many
different types of student populations from rural high schools to
Harvard." In response, AERA-D's Jeremy Miles asked "WERE THESE
RANDOMIZED TRIALS?" The short answer is "NO." The long answer
explains that: (a) randomized control trials (RCT's) are almost
impossible to carry out in undergraduate physics education research,
and (b) careful non-RCT research can establish causality to a
reasonable degree - as argued by Shadish, Cook, & Campbell; Shavelson
& Towne; Schneider, Carnoy, Kilpatrick, Schmidt, & Shavelson; and
Michael Scriven.
***************************************
In my post of 19 October 2008 titled "Can Pre-to-posttest Gains Gauge
Course Effectiveness? #2" [Hake (2008a)], I wrote:
"These investigations. . . . .[Hake (1998a,b) and about 25 other
pre/post studies referenced in Hake (2008b)]. . . . have been
carried out on many different instructors, in many different
institutions, using many different texts, and working with many
different types of student populations from rural high schools to
Harvard."
In response, AERA-D's Jeremy Miles (2008a) asked: "WERE THESE
RANDOMIZED TRIALS?"
The short answer is "NO!"
The long answer is:
In undergraduate physics education research the use of randomized
control trials (RCT's) is nearly impossible. In "Re: Should
Randomized Control Trials Be the Gold Standard of Educational
Research? " [Hake (2005a)] I wrote:
"Could physics education researchers (PER's) whose work is
predominately in UNDERGRADUATE education utilize RCT's? PER's deal
with populations of UP (university professors) and US (Undergraduate
Students). Most UP's demand autonomy in the way they teach courses
since they obviously know best how to lecture. Most of the US's (or
their parents) paid good money to be lectured at. No one that I know
of has been insane enough to even suggest that subjects from
populations UP and US be randomly assigned to different curricula in
a RCT, especially if one curriculum de-emphasizes lectures. Also the
average UP, thrown into an IE course would be a total disaster. If
anyone has some ideas on how to accomplish an RTC among UP's and US's
while avoiding dismissal or execution please let me know. Of course
one could PAY the subjects, but this might bias the results towards
the greedy and/or impecunious."
That Miles is asking a leading question might be inferred from the
end of Section 2.2: "The structure of experiments" under "Collecting
Data - Experimental Design" in his valuable "Psychology Research
Methods Wiki" [Miles (2008b)]. There Miles gives the canonical
argument for randomized control trials:
"One of the most important aspects of experimental design is random
assignment of different participants to different conditions - if you
do not randomly assign participants to conditions, you cannot be sure
that and differences in the conditions are due to the independent
variable." [Nor can you be sure even if there IS random assignment! -
see the signature quote by Thomas Cook.]
But Miles (2008b) moderates the above in the "digression box" in
"Cause and Effect: The Advantage of Experiments" in Section 3.1:
"You need to be aware that some people take a very strict line, and
say that without an experiment . . . .[i.e., "Randomized Control
Trial]. . . , statements about cause and effect can never be justly
made. Others take a less strict line, and say that statements about
cause and effect relationships can be made occasionally, if care is
taken (yes, you do not get any less strict than that). I hold that
causal statements can be made, if care is taken and appropriate
techniques are used, but your tutors may not agree."
Among "tutors" who DO agree are:
a. Shadish, Cook, & Campbell (2002) in "Experimental and
Quasi-Experimental Designs for Generalized Causal Inference";
b. Shavelson & Towne (2002) in "Scientific Research in Education,"
National Academy Press;
c. Schneider et al. (2007) in "Estimating Causal Effects Using
Experimental and Observational Designs";
d. Michael Scriven (2007) in "The Logic Of Causal Investigations."
"In some quarters, particularly medical ones, the randomized
experiment is considered the causal 'gold standard.' It is clearly
not that in educational contexts, given the difficulties with
implementing and maintaining randomly created groups, with the
sometimes incomplete implementation of treatment particulars, with
the borrowing of some treatment particulars by control group units,
and with the limitations to external validity that often follow from
how the random assignment is achieved."
Thomas Cook and Monique Payne in "Evidence Matters" [Mosteller &
Boruch (2002)]
REFERENCES [Tiny URL's courtesy <http://tinyurl.com/create.php>.]
Cook, T.D. & M.R. Payne. 2002. "Objecting to the Objections to Using
Random Assignment in Educational Research," in Mosteller & Boruch
(2002). See also Cook (2001).
Cook, T.D. 2001. "A critical appraisal of the case against using
experiments to assess school (or community) effects" Education Next,
No. 3 (Hoover Institute, Stanford, 2001), online at
<http://media.hoover.org/documents/ednext20013unabridged_cook.pdf>
(128 kB). CAUTION: a severely truncated popularization of this
scholarly paper appears under the provocative title "Sciencephobia:
Why education rejects randomized experiments," Education Next 1(3):
62-68 (2001), online at
<http://www.hoover.org/publications/ednext/3384446.html>. One can
progress to the more academic "A critical appraisal of the case
against using experiments to assess school (or community) effects" by
clicking on "Unabridged PDF" in the right-hand column. For a
discussion of this paper see Scriven (2007).
Hake, R.R. 1998a. "Interactive-engagement vs traditional methods: A
six thousand- student survey of mechanics test data for introductory
physics courses," Am. J. Phys. 66(1): 64-74; online as an 84 kB pdf
at <http://tinyurl.com/3xuyqe> . See also Hake (1998b).
Hake, R.R. 1998b. "Interactive- engagement methods in introductory
mechanics courses," online as a 108 kB pdf at
<http://tinyurl.com/2tg5d9> - a crucial companion paper to Hake
(1998a).
Hake, R.R. 2005a. Re: Should Randomized Control Trials Be the Gold
Standard of Educational Research? online on the OPEN! AERA-D archives
at <http://tinyurl.com/55deua>. Post of 17/18 Apr 2005 to AERA-C,
AERA-D, AERA-G, AERA-H, AERA-J, AERA-K, AERA-L, AP-Physics, ASSESS,
Biopi-L, Chemed-L, EvalTalk, Math-Learn, Phys-L, PhysLrnR, Physhare,
POD, STLHE-L, & TIPS. For a guide to discussion lists see Hake
(2008c). For a defense of cross-posting see Hake (2005b).
Hake, R.R. 2005b. "Cross-Posting - Synergistic or Sinful?" Post of 1
Nov 2005 08:37:12-0800 to ITFORUM and AERA-L; online at at
<http://tinyurl.com/2m59v4>.
Hake, R.R. 2008a. "Can Pre-to-posttest Gains Gauge Course
Effectiveness? #2" online on the OPEN! AERA-D archives at
<http://tinyurl.com/5dx45q>. Post of 19 Oct 2008 16:08:08-0700 to
AERA-D, ASSESS, EdResMeth, EvalTalk, PhysLrnR, and POD.
Hake, R.R. 2008b. "Design-Based Research in Physics Education
Research: A Review," in "Handbook of Design Research Methods in
Education: Innovations in Science, Technology, Engineering, and
Mathematics Learning and Teaching" [Kelly, Lesh, & Baek (2008)] -
publisher's information at <http://tinyurl.com/4eazqs>; a
pre-publication version of Hake's chapter is online at
<http://www.physics.indiana.edu/~hake/DBR-Physics3.pdf> (1.1 MB).
Miles, J. 2001. " Research Methods and Statistics in Psychology:
Success in Your Psychology Degree," Crucial Publishers. Amazon.com
information at <http://tinyurl.com/6ouepa>.
Miles, J. 2008a. "Re: Can Pre-to-posttest Gains Gauge Course
Effectiveness? #2" on the OPEN! AERA-D archives at
<http://tinyurl.com/6quapc>.
Mosteller, F. & R. Boruch, eds. 2002. "Evidence Matters: Randomized
Trials in Education Research." Brookings Institution. Amazon.com
information at <http://tinyurl.com/59gp6o>.
Schneider, B.M. Carnoy, J. Kilpatrick, W.H. Schmidt, R.J. Shavelson.
2007. "Estimating Causal Effects Using Experimental and Observational
Designs." AERA, publisher's information and FREE download at
<http://www.aera.net/publications/Default.aspx?menu_id=46&id=3360>.
Scriven, M. 2007. "The Logic Of Causal Investigations," probably in
press; online at <http://homepages.wmich.edu/~mscriven/>; especially
Quasi-Experimental Designs (QXTs), pp. 21-23. The reference to Cook
(2000) should be Cook (2001) - see above. Scriven is now at Claremont
Graduate University <http://www.cgu.edu/pages/4745.asp>. See also
Scriven (2008).
Scriven, M. 2008. Re: A Primer on Medical Studies, EvalTalk post of
15 Oct 2008 10:25:41-0700; online at <http://tinyurl.com/56ttjl>.
Scriven writes: ". . . . many alleged RCTs have given incorrect
conclusions (some of them at least because of the Hawthorne effect,
present in virtually all so-called RCTs in social cases under IRBs),
and that's still true even of many truly double-blind RCTs; see the
placebo literature (eg, but not only for this reason, because they're
not triple-blind). The use of a 'third arm' that is, another group
that receives no treatment at all, placebo or experimental, is
increasingly popular in order to avoid the above epidemic of mistaken
interpretations, and even then, the measurement needs to be
unobtrusive or you merely have a second placebo group. Now that's not
too hard to do with drug studies, but it's extremely hard with
studies of social interventions."
Shadish, W.R., T.D. Cook, & D.T. Campbell. 2002."Experimental and
Quasi-Experimental Designs for Generalized Causal Inference."
Houghton Mifflin. Amazon.com information at
<http://tinyurl.com/6kel78>. Note the"Search Inside" feature. A
goldmine of references on social science research.
Shavelson, R.J. & L. Towne, eds. 2002. "Scientific Research in
Education," National Academy Press; online at
<http://www.nap.edu/catalog/10236.html>.