Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-L] peculiar data; was: uncertain to

- This reminds me of the oft heard lament, post unfortunate event, "don't
ever trust assumptions," which is of course wrong. We must of necessity
make a .myriad of assumptions every day.
- One can come up with specific dramatic examples of the folly of throwing
out outliers, but could also come up with examples of the folly of
excessively heeding outliers. To exaggerate, suppose the "enemy" has been
credibly planning to "blow up the world" on Feb 15th, and specific research
on intelligence and weaponry must be done really fast in order to
"surgically" take out the enemy before Feb 15th. In this case one would
more likely be wasting time if heeding outliers.
-:How many battle commanders in the various wars would have prevailed, had
they, having already grasped the strengths of their fighters, weaponry and
information, instead had occupied themselves with battle outliers?
Bill Norwood
U of MD at College Park
On Jan 19, 2015 9:36 AM, "John Denker" <jsd@av8n.com> wrote:

On 01/12/2015 12:40 PM, Paul Lulai wrote:
What I normally do:
.have every kid measure the mass of the same thing, I enter it into a
spreadsheet.
.we find the average, max & min
.throw out any unresasonable values.

Sometimes throwing out "unreasonable" values is necessary,
but it is exceedingly difficult to do this properly. In
an introductory course, it may be easier and safer to just
keep all the data.

For example: Suppose you are running an assay office. A
prospector brings you 1000 samples. The first one has a
negligible amount of gold. Ditto for the second one, and
the third one, et cetera. Oddly enough, one of the samples
has 5 sigma more gold than the average. Should you throw
out that reading on the grounds that it is "unreasonable"?
You could argue that based on simple statistics, there
should be only one chance in 10^12 of finding a sample
like that.

However (!) if you do that, you have completely defeated
the purpose of your job, and the prospector's job. It
turns out that in this world, a lot of things (including
gold) are not Gaussian distributed.

Please do not teach students to throw out data just
because it "looks unreasonable".

As another example: Suppose you have a long-term project
to measure and re-measure the brightness of the stars.
You get around to measuring Cassiopeia every two years,
in mid-November of even-numbered years. The magnitude
of the brightest star is:

Year Apparent
Magnitude
1552 2.24
1554 2.24
1556 2.24
1558 2.24
1560 2.24
1562 2.24
1564 2.24
1566 2.24
1568 2.24
1570 2.24
1572 -4.3
1574 2.24
1576 2.24
1578 2.24
1580 2.24
1582 2.24
1584 2.24
1586 2.24
1588 2.24
1590 2.24
1592 2.24

Are you going to throw out the 1572 result as being
"unreasonable"? If you do that, you are trashing one
of the most important and interesting observations in
the history of astronomy.

Please do not teach students to throw out data just
because it "looks unreasonable".

One more example: If you veto intelligence reports
that say those aluminum tubes are /not/ suitable for
making uranium centrifuges, and veto other reports
that do not support your preconceived agenda, you
will make a mistake that costs trillions of dollars
and kills hundreds of thousands of people in the
short run, and creates problems that are likely to
linger for decades.

It is possible to design experiments with internal
controls ("measure twice, cut once") but doing this
properly requires some rather sophisticated skills.
Doing it properly is AFAICT beyond the scope of the
introductory course ... although if somebody knows of
a simple way of doing it properly I would very much
like to hear about it.

In particular, it is *not* acceptable to identify a
"suspicious" result on purely statistical grounds and
then go remeasure it. In such a case you are quite
likely to see what's known as regression to the mean.
This will seemingly confirm your suspicion, but in
reality such a process is horribly fallacious and
invalid. This is a very common mistake.

If you are going to veto some measurements, you need
to work very hard to make sure that the veto is not
correlated with the thing you are trying to measure.
In particular, if you are measuring the size of eggs,
it might make sense to skip any eggs that got dropped
and broken ... but beware! It may be that the egg-
measuring machine has a tendency to drop the smallest
eggs. In that case, vetoing the broken ones leads to
a gross systematic overestimate of the size.

Possibly constructive suggestion: Here is an example
of an experiment with internal controls. Assign each
student to measure a square. Measure all four sides
/and both diagonals/. This gives a set of six
measurements. Each set has some obvious internal
consistency checks. If it fails these checks, you
can throw out the entire set. Do not simply throw
out the one measurement that looks fishy. Then
average over all surviving sets to get an estimate
of the size of the square. This is not a perfect
example, but it is better than just throwing out
readings on a whim.

Any process that minimizes one kind of error will
exacerbate some other kind of error ... so you
have to be really, really careful with this.

Things that are not tolerated in the real world
should not be encouraged in high school.
-- Publishing data with a few peculiar, possibly
erroneous points ... that's mildly embarrassing.
-- Fudging the data to remove peculiar-looking
points ... that's serious misconduct.

Please impress on students that vetoing data is a
nasty business. It easy to do wrong, and hard to
do right. In most cases, it is better to keep all
the data, including the anomalous data.
_______________________________________________
Forum for Physics Educators
Phys-l@www.phys-l.org
http://www.phys-l.org/mailman/listinfo/phys-l