Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-l] calibration



This is an important topic, and it deserves more careful discussion
than it has so far received. Some constructive suggestions appear
interspersed below.

On 08/07/2007 12:09 AM, Jeffrey Schnick wrote:
Jeff,
Thanks for the shared reference:

Thank you for introducing such a worthwhile topic.

http://www.chem.utoronto.ca/coursenotes/analsci/StatsTutorial/
ErrRegr.html

I like it, except for one detail. The predicting variable y (what is
measured is fluorescence) was plotted vertically while the inferred
variable (concentration) is plotted horizontally. Their formula to
calculate standard deviation of the inferred variable contains the
slope of the line, b.

My predicting (independent) variable, r, is plotted horizontally while
the inferred (dependent) variable is plotted vertically. That seems to
be more logical.

In creating the calibration curve, it makes more sense to me to plot p
horizontally and r vertically. One prepares six samples of an alloy,
each with a fixed value of p. The resistivity of a sample depends on
the percentage p of zinc in the sample. Having prepared each sample,
one then measures the resistivity. You can tell by the regular spacings
of the values of p that it was the values of p that were chosen
independently by the experimenter. The experimenter did not, for each
data point, decide on a value of resistivity and then add zinc to the
copper until the sample had that resistivity.

1) As a super-important rule, applicable in general (not just to
this example):
Unstated uncertainty does not mean zero uncertainty!

As a corollary, just because the data appears to be evenly spaced
does not mean it is exactly evenly spaced. Consider the timing
of US presidential elections, for example; they are not exactly
4 years apart. Additional examples are a dime a dozen.

In this zinc calibration example,
http://pages.csam.montclair.edu/~kowalski/cf/331calib.html
just because AFAICT the experimenter has not yet told us the uncertainty
in the concentration does not mean the concentration is known exactly.
Indeed, given what I know about the volatility of zinc, I would expect
rather substantial uncertainty in the concentration.

To make point (1) the other way, here is a constructive suggestion:
when you are writing a report, state the uncertainty on all relevant
variables. If you think the uncertainty is negligible, say so ...
or (better) give an estimate or at least a bound on the uncertainty.
If at all possible, briefly explain how the estimated uncertainty
was obtained.

It would seem that this
is an important point in that the actual value of p, not just its
standard deviation, depends on which variable is treated as the
independent variable and which variable is treated as the dependent
variable.

Yes, it is an important point ... but no, that is not the right
way to deal with the problem.

2a) Calibrations can be used in various directions:

*) Sometimes the results will be used to predict concentration
based on resistance, and/or
*) sometimes the results will be used to predict resistance
based on concentration.

On the zinc pages
http://pages.csam.montclair.edu/~kowalski/cf/331calib.html
a particular direction is stated. That's fine, but keep in
mind that it is not the most general case.

The analysis should not be based on undocumented assumptions.
The analysis should not be restricted to one direction or the
other, unless you have really good reasons for ruling out the
other direction.

2b) More generally: In most situations, the distinction between
"independent" variable and "dependent" variable is worthless
... or entirely illusory.

On 08/07/2007 03:20 AM, Folkerts, Timothy J wrote:
although the original samples may have been made with specific
compositions, the operation of the meter will take a known
resistivity and predict the composition. Hence resistivity really is
the independent variable, and composition is the dependent variable.

You don't want to go down that rabbit-hole. Yeah, I know people
have been taught since third grade to distinguish "dependent"
from "independent" variables, but that doesn't make it right.
This is a distinction found in schoolbooks but not in the real
world.

In particular, any feedback loop makes a mockery of the usual
definitions of "independent" and "dependent" variables. And
feedback loops are very common in real-world measurement
apparatus.

Constructive suggestion: The general way to formulate all such
problems is in terms of probability. In particular, writing
p(x,y) is vastly more general than writing y=f(x). This allows
us to understand that y=f(x) is a special case of p(y|x) i.e. a
special case of knowing the probability of y given x, namely the
case where p(y|x) is a delta function of y. If instead y has
error bars, then the probability is not a delta function of y.

I know that incoming students don't have a sufficient
background in probability. Often they have more misconceptions
than correct preconceptions about probability. That just
means you have to take the time to lay the groundwork. Trying
to do curve fitting (or "calibrations") without probability
is like trying to swim without getting wet.

++ In all cases, given p(x,y) we can calculate p(x|y).
++ In all cases, given p(x,y) we can calculate p(y|x).
Hint: Integrate.
++ Alas p(y|x) does not generally suffice to calculate p(x|y) or p(x,y).
++ Alas p(x|y) does not generally suffice to calculate p(y|x) or p(x,y).
Hint: Bayes inversion formula.

Sometimes the uncertainty in x dominates the uncertainty in y.
Sometimes the uncertainty in y dominates the uncertainty in x.
Sometimes neither. Don't assume one or the other; you have to
check.

An easy-to-visualize example of what I'm taking about involves
fitting to a step function, e.g.
http://www.av8n.com/physics/img48/step-fit.png
As usual, the treads of the step function have zero slope while
the risers have infinite slope. This means that (depending on
other details) by neglecting the uncertainty in x you might
overestimate the uncertainty in y by an arbitrarily large amount,
possibly approaching an infinite amount. You can see in the
figure that some of the points miss the curve in the y-direction
by a huge amount that is more plausibly explained by x-uncertainty
than by y-uncertainty.

Situations like this -- or even more complicated than this --
are common in physics. Hint: Millikan.

Note that flipping the plot 90 degrees (transposing x <--> y)
does *not* solve the problem.

3) In cases where both variables have non-negligible uncertainty,
it is sometimes easy to plot p(x|y) as a scatter plot and fit
a curve through it by hand. If you are seeking a straight-line
fit, use a ruler. Be sure to use a /transparent/ ruler. If
you don't have one, buy one from the office-supply store or the
art-supply store.

Doing curve fits by computer (instead of by hand) is never simple.
Introductory-level textbooks brutally simplify the math. I don't
mind approximations; I just wish the books would /say/ more clearly
how brutal their approximations are. It seems to me that generation
after generation, new textbooks quote old textbooks, leaving out
more and more detail, until what is left is little more than a
caricature.

It would be preposterous to think that the problem can be captured
in a simple formula, or even in a simple applet. There are general
ways of attacking the problem; for instance, it could be formulated
as an optimization. That is, we write down an objective function
and then extremize it. Often the chi-square is used as the objective
function (which leads to the name "least squares" fitting.) Alas
chi-square is rarely (if ever) a good objective function; people
use it because it is easy, not because it is right. (This is an
example of the proverbial "looking under the lamp post".)

The stair-step example suffices to illustrate some of the problems
with chi-square. Writing down a proper objective function for
this example is not a trivial exercise.

Other problems include the fact that least-squares fitting is
ordinarily formulated as a maximum-likelihood method. People
just /assume/ maximum likelihood is appropriate, but even a
glance at the probabilities tells you this can't be right.
Likelihood is a technical term; it denotes the probability of
the data given the fitting parameters. That's nuts; we need
the probability of the fitting parameters given the data!

The latter is called MAP fitting, where MAP stands for "maximum a
posteriori". There is no way in general to go from the likelihood
p(x|y) by itself to the a-posteriori probability p(y|x); additional
information is needed. Hint: Bayes inversion formula.

4) Let's keep our eyes on the prize. Keep in mind that the whole
point of the exercise is to predict future data,
4a) based on past data, and
4b) based on a-priori restrictions on the form of the data.

Note that the priors (item 4a) are indispensable. Often
"regularization terms" are used in the optimization, which
means that "regularizer" and "prior" are practically synonymous.
The notion of "bias/variance tradeoff" is another name for an
intimately-related idea.

The weird curve in Ludwik's figure 1b is perfectly consistent
with the given data; it is however disfavored by our priors
concerning the electrical properties of alloys.

The role of curve fitting (or "calibration") is as a /proxy/.
That is, rather than predicting future data based directly
on past data and priors, we predict future data based on
the fitted function. The overall dependence is:

PREDICTIONS
based on
FITTED FUNCTION
based on
PAST DATA and PRIORS.

So you can see that the fitted function is a way of "bundling"
what we know about the past data and the priors, bundling it
into a convenient form.