[Phys-L] Should Randomized Control Trials Be the Gold Standard of Educational Research ?

[Richard Hake <rrhake@EARTHLINK.NET>]

If you object to cross-posting as a way to tunnel through inter- and
intra-disciplinary barriers, please hit "delete" now. And if you
respond to this long (8 kB) post, please don't hit the reply button
unless you prune the original message normally contained in your
reply down to a few lines, otherwise you may inflict this entire post
yet again on suffering list subscribers.

In "Will the No Child Left Behind Act Promote Direct Instruction of
Science?" [Hake (2005)], I gave, as one of the seven reasons why
Direct Science Instruction threatens to predominate nationally under
the aegis of the No Child Left Behind Act, the following [bracketed
by lines "HHHHHH. . . ."]:

HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
MOST INTERACTIVE ENGAGEMENT AND GUIDED INQUIRY METHODS HAVE NOT BEEN
TESTED IN RANDOMIZED CONTROL TRIALS (RCT'S), THE "GOLD STANDARD" OF
THE U.S. DEPT. OF EDUCATION (USDE)

That a single research method should be designated as the "gold standard" for
evaluating an intervention's effectiveness appears antithetical to
the report of the NRC's Committee on Scientific Principles for
Education Research [Shavelson & Towne (2002) - ST]. ST state that
scientific research should "pose significant questions that can be
investigated empirically," and "use methods that permit direct
investigation of the questions."

Furthermore, the USDE's RCT gold standard is considered problematic by a wide
array of scholars. Taking issue with the RTC gold standard are
philosophers Dennis Phillips [Shavelson, Phillips, Towne, & Feuer
(2003)] and Michael Scrivin (2004); mathematicians Burkhardt &
Schoenfeld (2003); engineer Woodie Flowers [Zaritsky, Kelly, Flowers,
Rogers, Patrick (2003)]; and physicist Andre deSessa [Cobb, Confey,
diSessa, Lehrer, & Schauble (2003)].

In addition, the following organizations oppose the RTC gold standard:
(a) American Evaluation Association (AEA)
      <http://www.eval.org/doestatement.htm>,

(b) American Education Research Association (AERA)
      <http://www.eval.org/doeaera.htm>, and

(c) National Education Association
      <http://www.eval.org/doe.nearesponse.pdf> (88 kB).
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH

Two recent articles [Bhattacharjee (2005), Stipek (2005)] discuss the
pros and cons of RCT's and may be of interest to subscribers. I thank
Larry Woolf for bringing the Bhattacharjee reference to my attention.

Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<rrhake@earthlink.net>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>


REFERENCES
Bhattacharjee, Y. 2005. "Can Randomized Trials Answer The Question of
What Works?: A $120 million federal initiative to improve secondary
math education hopes to draw on an approach some researchers say may
not be ready for the classroom," Science 307: 1861-1863, 25 March,
currently online at
(a) <http://www.sciencemag.org/cgi/content/full/307/5717/1861>,
(b) <http://www.sciencemag.org/cgi/reprint/307/5717/1861.pdf> (208 kB), and
(c) the archives of AERA-L (Politics and Policy in Education) with
academic referencing by R. Hake at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0504&L=aera-l&T=0&O=D&P=1708>.

Burkhardt, H. & A.H. Schoenfeld. 2003. "Improving Educational
Research: Toward a More Useful, More Influential, and Better-Funded
Enterprise," Educational Researcher 32(9): 3-14; online at
<http://www.aera.net/publications/?id=401>.

Cobb, P., J. Confey, A. diSessa, R. Lehrer, L. Schauble. 2003.
"Design Experiments in Educational Research," Educational Researcher
32(1): 9-13; online at <http://www.aera.net/publications/?id=393>.

Hake, R.R. 2005. "Will the No Child Left Behind Act Promote Direct
Instruction of Science?" Am. Phys. Soc. 50: 851 (2005); APS March
Meeting, Los Angles, CA. 21-25 March; online as ref. 36 at
<http://www.physics.indiana.edu/~hake>, or download directly by
clicking on
<http://www.physics.indiana.edu/~hake/WillNCLBPromoteDSI-3.pdf> (256
kB).

Scriven, M. 2004. "Causation," unpublished; see, e.g. "Scriven on
randomized control groups," online at
<http://interversity.org/lists/arn-l/archives/Nov2003/msg00309.html>.

Shavelson, R.J. & L. Towne, eds. 2002. "Scientific Research in
Education," National Academy Press; online at
<http://www.nap.edu/catalog/10236.html>.

Shavelson, R.J., D.C. Phillips, L. Towne, and M.J. Feuer. 2003. "On
the Science of Education Design Studies," Educational Researcher
32(1): 25-28; online at <http://www.aera.net/publications/?id=393>.

Stipek, D. 2005. "Scientifically Based Practice," Education Week
24(28): 33-34; currently online at
(a) <http://www.edweek.org/ew/articles/2005/03/23/28stipek.h24.html>;
(b) as a Math-Learn post of 2 April 2005 10:56 am by Jerry Becker; online at
<http://groups.yahoo.com/group/math-learn/message/7659>
(unfortunately one must subscribe to Math-Learn
<http://groups.yahoo.com/group/math-learn/> in order to access its archives);
(c) as a post with a one-paragraph comment by Susan Ohanian at
<http://www.susanohanian.org/show_atrocities.html?id=3914>;
(d) the archives of AERA-L (Politics and Policy in Education) with
academic referencing by R. Hake at
<http://lists.asu.edu/cgi-bin/wa?A2=ind0504&L=aera-l&T=0&O=D&P=1840>.

Zaritsky, R., A.E. Kelly, W. Flowers, E. Rogers, & P. Patrick. 2003.
"Clinical Design Sciences: A View From Sister Design Efforts,"
Educational Researcher 32(1), 33-34; online at
<http://www.aera.net/publications/?id=393>.
_______________________________________________
Phys-L mailing list
Phys-L@electron.physics.buffalo.edu
https://www.physics.buffalo.edu/mailman/listinfo/phys-l