Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: measure what you care about



At 02:14 PM 10/10/00 -0400, Michael Edmiston posted a lucid note which said
in part:

Having been an instrument designer/builder in a former life, I often wanted
to communicate the resolution of the instrument. It wouldn't do much good
to report the standard error of the mean in this case because I could make
it arbitrarily small by taking sufficient data points.
...
I think whether one reports standard error of the mean or sample standard
deviation depends upon what one is trying to communicate.

Right. That got me thinking.

To extend that line of thought: Sample-standard-deviation and
standard-error-of-the-mean are just two out of an endless set of
possibilities. The general procedure is

1) Figure out what people care about.
2) Measure that.
3) Report that.

Report it with enough detail that people can understand what you did and
can replicate it if necessary.

=============

Here is an illustration. Students may find this illustration somewhat
easier to visualize (easier than imagining themselves to be sophisticated
instrument designers).

Suppose you want to roll a die and bet on the outcome. There are various
things you might want to know about the die, including:

.--- A --- --- B --- --- C --- --- D --- --- E ---
. --value for-- --value for-- --how well--
. --formula-- --fair die-- -- my die -- --known--
.
. 0th moment sum_of p(x) 1 1 exact
.
. 1st moment sum_of x p(x) 3.5 3.5 1 ppm
.
. 2nd moment sum_of x*x p(x) 15.166667 15.166667 1 ppm
. about zero

From this you can calculate the second moment about the mean (i.e. the
variance) and the square root thereof (i.e. the standard deviation,
sigma). In this case sigma comes out to about 1.7.

Note that the value of sigma does not necessarily determine how well I know
the mean. Sigma is frozen at 1.7, but I can determine the mean as
accurately as you like.
-- I could determine the mean by rolling my die a gazillion times.
-- For that matter, I might be able to determine the mean without rolling
my die at all; I might be able to do it by measuring the symmetry of the
construction of the die.

I emphasize this point because all too often, students have only seen one
use for the second moment, namely as a way to get at the uncertainty in the
first moment. They come away thinking that the two concepts are related
more strongly than they actually are.

BTW, a similar pattern of confusion arises when students think about using
a high-Q oscillator as a clock:
-- Yes, when building a high-precision clock it helps to start with a
high-Q oscillator.
-- No, the fractional uncertainty is *not* limited to something like
1/Q. If you've got good signal-to-noise you do vastly better than 1/Q.

=============================

To return to the main point: measure what you care about!

Let's get more specific about the betting game. You pay me a dollar for
each roll of the die. If the die comes up showing "2", I will pay you ten
dollars. You can have as many turns as you want. If the die is fair, you
should expect to make a tidy profit.

The die I am using is to all external appearances ordinary; it has a
representation of the numbers 1 through 6 on the faces in the usual way. I
guarantee that the first and second moment are as specified above, to
within 1 part per million.

But you had better be careful. I didn't actually tell you that p(2) =
1/6. Actually, the die in question is loaded so that p(2) = 0, and anybody
who bets against me is going to lose spectacularly.

For homework: Figure out a set of six probabilities p(1) ... p(6) with the
properties stated above.

Bottom line: If you care about p(2), measure p(2). Measure what you care
about!