Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

[Phys-L] fudging the data ... or not



Quoting Feynman:
http://calteches.library.caltech.edu/51/2/CargoCult.pdf

It's interesting to look at the history of measurements of the charge
of an electron, after Millikan. If you plot them as a function of
time, you find that one is a little bit bigger than Millikan's, and
the next one's a little bit bigger than that, and the next one's a
little bit bigger than that, until finally they settle down to a
number which is higher.

Why didn't they discover the new number was higher right away? It's a
thing that scientists are ashamed of--this history--because it's
apparent that people did things like this: When they got a number
that was too high above Millikan's, they thought something must be
wrong--and they would look for and find a reason why something might
be wrong. When they got a number close to Millikan's value they
didn't look so hard. And so they eliminated the numbers that were too
far off, and did other things like that. We've learned those tricks
nowadays, and now we don't have that kind of a disease.

Actually we do still have that disease. The temptation to
fudge the data will always be there. Eternal vigilance is
required to hold the problem down to manageable levels.

Nate Silver recently analyzed some polling data and found
significant amounts of "herding" ... which is a common
form of fudging. He gives a nice explanation:
http://fivethirtyeight.com/features/heres-proof-some-pollsters-are-putting-a-thumb-on-the-scale/

I would like to add a couple of points:

1) It is all-too-easy to fudge the data inadvertently --
even if you have the best of intentions -- perhaps by
"pruning" the outliers. Don't do it!

Guarding against this requires serious self-discipline and
eternal vigilance.

2) You can get into serious trouble, because it is may be
hard to distinguish inadvertent fudging from intentional
fudging. If people can't trust your results, it can be
a career-ending problem.

If you want to take herding to the extreme, you don't need
to measure anything at all. Just pretend to do the measurement,
and then publish something based on the consensus of other
people's measurements.

If you fudge the data once, you can probably get away with
it. However, fudging is highly addictive, and you will
get caught sooner or later.
http://www.dailykos.com/story/2010/06/29/880179/-Research-2000-Problems-in-plain-sight
http://www.dailykos.com/story/2010/06/29/880185/-More-on-Research-2000
http://www.dailykos.com/story/2010/06/29/880332/-Update2-Nate-Silver-Backs-Kos-R2K-Poll-Fraud-Claim-R2K-Threatens-to-Sue-Kos
http://en.wikipedia.org/wiki/Research_2000

3) This is directly relevant to the teaching lab and classroom.
If you assign students to measure something, there is an 80%
likelihood that at least one student in a class of 35 will get
an answer that is off by two sigma. If you mark that wrong,
you are training people to fudge the data, training them to
herd the data, training them to prune the outliers.

4) Nate Silver points out that there are advantages as well
as disadvantages to "herded" data.

I would point out that there is a way to have your cake
and eat it, too: Publish two numbers!

a) First, say "here is what we measured" with no herding,
no fudging.

b) Second, say "here is the result of a meta-analysis,
including our measurement along with previous results
in a weighted average."

The latter doesn't count as herding, because you are not
trying to pass it off as something it isn't.