Some subscribers to Phys-L and Physoc might be
interested in a recent post "The Cult of
Statistical Significance" [Hake (2010)]. The
abstract reads:
******************************************
ABSTRACT: Math-Teach's Domenico Rosa has called
attention to the review of Ziliak & McCloskey
(2008) by Olle Häggström (2010). According to
Häggström, Ziliak & McCloskey's major point is
that "many researchers are so obsessed with
statistical significance that they neglect to ask
themselves whether the detected discrepancies are
large enough to be of any subject-matter
significance."
Consistent with that outlook, in "Lessons from
the Physics Education Reform Effort" [Hake
(2002)] I cited the position of many
psychologists and biologists that the "effect
size" is a preferred alternative (or at least
addition) to the usually inappropriate t-tests
and p values associated with Null Hypothesis
Statistical Significance Testing (NHSST).
Nevertheless, many educational researchers (even
some physicists) still utilize *only* NHSST to
gauge the significance of their research results.
*****************************************
"After 4 decades of severe criticism, the ritual
of null hypothesis significance testing -
mechanical dichotomous decisions around a sacred
0.05 criterion - still persists. This article
reviews the problems with this practice,
including its near-universal misinterpretation of
p as the probability that Ho . . . .[[the null
hypothesis]]. . . . is false, the
misinterpretation that its complement is the
probability of successful replication, and the
mistaken assumption that if one rejects Ho one
thereby affirms the theory that led to the test.
Exploratory data analysis and the use of graphic
methods, a steady improvement in and a movement
toward standardization in measurement, and
emphasis on effect sizes using confidence
intervals, ands the informed use of available
statistical methods is suggested. FOR
GENERALIZATION, PSYCHOLOGISTS MUST FINALLY RELY,
AS HAS BEEN DONE IN THE OLDER SCIENCES, ON
REPLICATION." [My CAPS.]
-Jacob Cohen (1994) in "The earth is round (p < .05)"
REFERENCES [All URL's accessed on 10 October
2010; some URL's shortened by <http://bit.ly/>.]
Cohen, J. 1994. "The earth is round (p < .05)."
American Psychologist 49: 997-1003; online as a
1.2 MB pdf at <http://bit.ly/a45I2t>, thanks to
Christopher Green <http://www.yorku.ca/christo/>.
Hake, R.R. 2010. "The Cult of Statistical
Significance," online on the OPEN AERA-L archives
at <http://bit.ly/dkTyXP>. Post of 10 Oct 2010
19:34:16-0700 to AERA-L, Math-Teach, & Net-Gold.
The abstract and link to the complete post are
being transmitted to various discussion lists and
are also online on my blog "Hake'sEdStuff" at
<http://bit.ly/96bBCM> with a provision for
comments.