Re: [Phys-L] maximum likelihood versus minimum cost

[John Denker <jsd@av8n.com>]

On 05/10/2012 06:12 PM, Donald Polvani wrote:

> As for John Denker's example A(t) = A(0)exp^(kt).  In the spirit
> expressed above, I did linearize this as ln(A(t)) = kt + ln(A(0)) and
> solved for k and A(0) using linear least squares.

That's the wrong thing to do.  Guess how I know.

I know it's wrong, because I did the same thing, back when I was 
in grad school.  I went so far as to write the code to linearize
the data and then fit to it.  I had about 1000 fits to do, and I
did one of them that way before realizing how totally wrong it was.

Here's an easy qualitative way to see why it's wrong:  Suppose you 
have a decaying exponential with well-behaved additive noise.  You 
are smart enough to run the data out a good long long ways, so as 
to get a good baseline.  That means the exponential gets very small.
That in turn means that the additive noise takes the raw signal below 
zero.  What happens when you take the logarithm of a negative number?
It's nonsense.  What happens if you take the logarithm of some number 
that is not quite zero, but very very close?  It is very nearly 
negative infinity, which leads to grossly wrong (biased) results 
from the fit.

Linearizing the fitted function in such a situation is just jumping 
out of the frying pan into the fire.
  Before linearizing the function:  Frying pan:  You can't do linear 
    least squares, because the fitted function is not an LCBF i.e. 
    linear combination of basis functions.
  After linearizing the function:  Fire:  You can't do linear least
    squares, because the error bars are not well behaved.

So here is the theoretical way to see what's wrong:  Linear least
squares has several provisos that are required for validity.  Before 
linearizing this problem violates one of the provisos, and after 
linearizing it violates a different proviso.

If it "looks like" linear least squares is working, you haven't run
the baseline out far enough.

The obvious way to solve the problem correctly is to do a nonlinear
least-squares fit.  There may be other correct approaches, but this
is the only correct approach I know of.  Represent the fitted function
directly as a decaying exponential with additive noise, and do the
fit.  The nonlinear fitting routine will converge very rapidly if
given reasonable starting parameters.  Use your knowledge of the
physical situation and/or some quick-and-dirty pre-estimator to
obtain suitable starting parameters.  It's OK if the pre-estimator
is noisy and/or biased, so long as it is in the right ballpark.

  The direct, obvious approach is sometimes the correct approach.

  People say I have a keen grasp of the obvious.  I take it as a
  compliment, even when it was not intended as such.