Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-L] LSF Slope Versus Average of Single Point Slopes



On 05/07/2012 01:04 PM, Bill Nettles asked:

"Is the LSF method developed based on the distribution of errors being
a Gaussian distribution?"

And, on 05/07/2012 at 5:28 PM, John Denker answered:

"Yes.

The thing you are minimizing is the _log improbability_. It works like this:

As always, the joint probability is the product of the pointwise probabilities (provided they are independent ... which is not always a safe assumption, but usually gets assumed anyway).

Hence the joint log improbability is the sum of the pointwise log improbabilities.

Hence the joint log improbability for a bunch of gaussians is a sum of squares.

Hence maximizing the joint probability corresponds to least squares.
That's all there is to it."

I searched my references and came up with an expanded answer versus John Denker's above. Google turned out to have the easiest to understand explanation that I found (see http://en.wikipedia.org/wiki/Least_squares). To briefly summarize the Google article, least squares seeks to minimize the sum of squared residuals where a residual is the difference between an observed value and a fitted value of the model. This can be done for linear and non-linear models. We are all familiar with the simpler linear case, and that's what I was discussing in my original post on this thread. Non-linear least squares can be done although (unlike the linear case where a closed form solution exists) the estimates are obtained by successive approximation.

From here on, I'm only talking about linear (or "ordinary") least squares. The usual assumptions are:

1) No errors in the x_i.
2) The y errors are uncorrelated with the x_i.
2) The y errors are zero mean and have equal variances.

(If you are interested in cases which don't satisfy these assumptions, see the classic paper by R.H. Bacon, "The "Best" Straight Line among the Points", Am. J. Phys. 21, 428 (1953).)

Given assumptions 1 - 3, the Gauss-Markov theorem says that the least squares estimate with be the best linear unbiased estimator. Best here means minimum variance.

If, IN ADDITION, to assumptions 1 - 3, the y errors are Gaussian, then the least squares estimator is equivalent to the maximum likelihood estimator as John Denker has pointed out. However, for non-Gaussian errors, the linear least squares estimator IS STILL the best linear unbiased estimator.

As a necessary (but far from sufficient!) test of the Gauss-Markov theorem (and, in accordance with Bill Nettles comment above), I redid my earlier zero mean Gaussian Monte Carlo model for determining slopes (model was slope m = 1, x_i = 1,2,3,4,5). This time I made the y errors uniform random between -0.5 to +0.5 with the same 0.1 standard deviation as in the zero mean Gaussian case. Here APS refers to the slope obtained from the average of the point slopes and LSF to the LSF slope. Here is a comparison of the Gaussian and uniform random results from the simulation after 500 Monte Carlo trials:

Gaussian Errors

Average of m_APS = 0.9982
Average of m_LSF = 0.9989
s_m_APS = 0.0238
s_m_LSF = 0.0134
s_m_APS/s_m_LSF = 1.78

Uniform Random Errors

Average of m_APS = 1.0010
Average of m_LSF = 0.9999
s_m_APS = 0.0246
s_m_LSF = 0.0137
s_m_APS/s_m_LSF = 1.80

Assuming the standard deviation of the residuals is the about the same for both the average point slope (APS) and LSF results (as seems clear from the above results), the ratio of s_m_APS/s_m_LSF should, in theory, be (1/N)*sqrt(S_n(1/x_n^2))*(sqrt(S_n(x_n^2))) or 1.79 for the x_i chosen here irrespective of the error distributions. This is in good agreement with both the 500 trial Gaussian and uniform random results.

So, for non-Gaussian errors, the linear LSF estimator still has a smaller variance than the APS estimator (as it necessarily must), and we expect in such cases (via the Gauss-Markov theorem) that the least squares estimate will be the best linear unbiased estimate.

Don

Dr. Donald Polvani
Adjunct Faculty, Physics
Anne Arundel Community College
Arnold, MD 21012