Re: resolution vs. precision

[John Denker <jsd@MONMOUTH.COM>]

At 07:26 AM 8/30/99 -0700, Leigh Palmer wrote:
> Please amplify this somewhat. I think I have always treated resolution
> and precision as being nearly identical in meaning. What are your views
> on that topic, which I do consider to be very important?

As applied to a raw measurement, I don't see a big distinction between
resolution and precision.

I guess I think of resolution as applying to numbers, and precision as
applying to measurements.  For a raw measurement, the number is the
measurement.

But now suppose that measurement is subjected to a multi-step calculations.
 It would be smart to carry some guard digits (extra resolution) on the
numbers during the intermediate steps, so there won't be a problem with
accumulation of roundoff errors.  Of course this does not change the
precision of the original measurement.

To give a common example:  If you measure something as ten inches (to the
nearest inch), and then convert to centimeters, you could conceivably write
it down as
        25.4 +- 1.27 cm
which has radically more resolution than precision -- but rounding it off
(to make resolution equal precision) would cause a noticeable loss of
precision.

Clear enough?

Cheers --- jsd