If you object to cross-posting as a way to tunnel through inter- and
intra-disciplinary barriers, please hit "delete" now. And if you
respond to this long (14 kB) post, please don't hit the reply button
unless you prune the original message normally contained in your
reply down to a few lines, otherwise you may inflict this entire post
yet again on suffering list subscribers.
In response to my post "Should Randomized Control Trials Be the Gold
Standard of Educational Research?" [Hake (2005a)], Brian Whatcott
(2005) wrote, [bracketed by lines "WWWWWWW. . . ."]:
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
I am surprised that anyone here would oppose this [Randomized Control
Trial (RCT)] methodology, which is to my mind at least, a minimal
level of ascent to a scientific basis for comparison of methods and
appears to be the basis for upholding interactive participation as
the only tutorial method that gives superior results consistently,
over chalk n talk [also known as Direct Instruction]
WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW
BRIAN, PLEASE GIVE US THE REFERENCE TO THE ABOVE CITED RESEARCH
showing that "interactive participation as the only tutorial method
that gives superior results consistently, over chalk n talk." I've
never heard of it.
Consistent with the information from the Brooklyn Medical Research
Library website (probably <http://library.downstate.edu/>) quoted by
Whatcott, I think most people would agree that Randomized Control
Trials (RCT's) - when they are ethically feasible - are indeed the
"gold standard" of MEDICAL research.
But IMHO RCT's are often NOT the "gold standard" of EDUCATION
research, even despite their enthronement as such by the U.S. Dept.
of Education (USDE). In Hake (2004) I wrote [bracketed by lines
"HHHHHH. . . ."; SEE THAT ARTICLE FOR THE REFERENCES]:
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
The "Coalition for Evidence-Based Policy," CEBP (2004), under the
aegis of the U.S. Department of Education's "Institute for Education
Sciences" [headed by Grover Whitehust (2003)] has produced
"Identifying and Implementing Educational Practices Supported by
Rigorous Evidence: A User Friendly Guide" (IES 2004). The CEBP's
board of advisors
<http://www.excelgov.org/displayContent.asp?Keyword=prppcAdvisory> include
luminaries such as famed RCT authority Robert Boruch (University of
Pennsylvania), political economist David Ellwood (Harvard);
statistician Robert Boruch (Univ. of Pennsylvania); former FDA
commissioner David Kessler (Univ. of California - San Francisco);
past American Psychological Association president Martin Seligman
(University of Pennsylvania); psychologist Robert Slavin (Johns
Hopkins); economics Nobelist Robert Solow (MIT); and
progressive-education critic Diane Ravitch.
Unfortunately, no physical scientists, mathematicians, philosophers,
or K-12 teachers are members of the CEBP. The CEBP's "Guide" is
addressed to K-12 education, but its recommendations could influence
funding for educational research at the postsecondary level - of
primary interest to many PER's (Physics Education Researchers).
According to the "Guide": "Well-designed and implemented randomized
control trials are considered the 'gold standard' for evaluating an
intervention's effectiveness, in fields such as medicine, welfare and
employment policy, and
psychology . . . . randomized control trials are studies that
randomly assign individuals to an intervention group or to a control
group, in order to measure the effects of the intervention. . . .
There is persuasive evidence that the randomized controlled trial,
when properly designed and implemented is superior to other study
designs in measuring an intervention's true effect . . . ."
That a single research method should be designated as the "gold
standard" for evaluating an intervention's effectiveness appears
antithetical to the report of the NRC's "Committee on Scientific
Principles for Education Research" [Shavelson & Towne (2000) - ST].
ST state that scientific research should "pose significant questions
that can be investigated empirically," and "use methods that permit
direct investigation of the questions."
CEBP seems to imply that most questions regarding the effectiveness
of an intervention can be answered by RCT's. This may be the case
AFTER a teaching method has been researched and engineered to its
full potential and is ready for full scale deployment. But crucial
questions in the early stages of an intervention require exploratory
research methods that do NOT necessarily involve RCT's and for which
RCT's might actually be counterproductive.
A case in point: some RCT enthusiasts might suggest that RCT's would
serve to adjudicate the K-8 California science education wars [see
e.g., Hake (2004b)] between "direct" vs "hands-on" instruction. But
since many K-8 teachers are scientifically illiterate (thanks in part
to the failure of society to reward teachers commensurate with their
vital societal contribution, and the failure
of universities to properly educated them), RCT's might well favor
"direct instruction" (DI).
The reason is that DI requires little conceptual understanding of
science on the part of teachers, while "hands-on" lessons guided by
scientifically unprepared teachers can be even worse than DI insofar
as advancing students' understanding of science is concerned. Thus
RTC's could "prove" the superiority of DI and thereby stifle K-8
science education reform in California.
[According to the report by Bhattacharjee (2005), similar
quality-of-implementation factors may confound the analysis of data
from the USDE's current attempt to research the effectiveness of
"Cognitive Tutor" by means of a RCT.]
Judging from the modest effect sizes listed Section 3.5.1 in Table 1
of Lipsey & Wilson (1993), K-8 science education is sorely in need of
non-RCT EXPLORATORY research of the caliber of recent PER research at
the high-school and undergraduate level. That research is consistent
with the recommendations of Shavelson & Towne (2000) - ST. In the
words of Eisenhardt & Towne (2003), "[ST] argued for a postpositivist
approach to scientifically based research in education, including a
range of research designs (experimental, case study, ethnographic,
survey) and mixed methods (qualitative and quantitative) *depending
on the research questions under investigation.*
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
Could physics education researchers (PER's) whose work is
predominately in UNDERGRADUATE education utilize RCT's? PER's deal
with populations of UP (university professors) and US (Undergraduate
Students). Most UP's demand autonomy in the way they teach courses
since they obviously know best how to lecture. Most of the US's (or
their parents) paid good money to be lectured at. No one that I know
of has been insane enough to even suggest that subjects from
populations UP and US be randomly assigned to different curricula in
a RCT, especially if one curriculum deemphasizes lectures. Also the
average UP, thrown into an IE course would be a total disaster. If
anyone has some ideas on how to accomplish an RTC among UP's and US's
while avoiding dismissal or execution please let me know. Of course
one could PAY the subjects, but this might bias the results towards
the greedy and/or impecunious.
The question is "if not RCT's what?" In Hake (2005b) I wrote
[bracketed by lines "HHHHHHHH. . . . ."; see that article for
references other than Hake (2002a,b; 2004)]:
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
Such pre/post testing [as done in PER] does not meet the U.S. Dept.
of Education's (USDE's) much contested [for a review see Hake
(2004)], "gold standard" of randomized control trials, but would
nevertheless probably pass muster at the USDE's "What Works Clearing
House" <http://www.w-w-c.org/> as "quasi-experimental studies
[Shadish et al. (2002)] of especially strong design" [see
<http://www.w-w-c.org/reviewprocess/standards.html>]. The strength of
the design derives partly from:
(a) the use of reasonably well-matched control groups offered by the
"traditional" courses, and
(b) corroboration of the results by many different research groups
[see Hake (2002a,b)].
Despite rampant pre/post paranoia [Hake (2004e,f; 2005d)], pre/post
assessments of student learning are being more and more utilized in
fields such as astronomy, economics, biology, chemistry, computer
science, physics, and engineering [see Hake (2002a,b; 2004g,h)].
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
Hake, R.R. 2002a. "Lessons from the physics education reform effort,"
Ecology and Society 5(2): 28; online at
<http://www.ecologyandsociety.org/vol5/iss2/art28/>. Ecology and
Society
(formerly Conservation Ecology) is a free online "peer-reviewed
journal of integrative science and fundamental policy research" with
about 11,000 subscribers in about 108 countries.
Hake, R.R. 2005a. "Should Randomized Control Trials Be the Gold
Standard of Educational Research?" online at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0504&L=aera-l&T=0&O=D&P=1945>. Post of
15 Apr 2005 to AERA-C, AERA-D, AERA-G, AERA-H, AERA-J, AERA-K, AERA-L,
AP-Physics, ASSESS, Biopi-L, Chemed-L, EvalTalk, Math-Learn, Phys-L,
Physhare, POD, STLHE-L, & TIPS.