Re: Finding information on the Internet

["John S. Denker" <jsd@MONMOUTH.COM>]

On 04/08/2003 05:47 PM, Tim Folkerts wrote:
> It appears you have to make judgements about when to rely on your
> judgements!

Yes indeed.

> *  Suppose I want an accurate voltage measurement.  I go to pick out a
> voltmeter, and the shelf has a $800 Keithley meter, a $9 Radioshack meter,
> and a 40 yr old analog meter.

This is an excellent instructive example.  I
have learned a few things by thinking about it.

> I typically only become suspicious of the measurement if it leads to
> a contradiction with some other "authoritative" result. I can't
> afford to spend the time checking the calibrations of every piece of
> equipment I use.

Right.

Voltmeters are generally very good these days.
For most day-to-day purposes, they all agree,
so it the question of which one to use is usually
academic.

And speaking of "authoritative" results, I think we
can all agree that the standard volt is extremely
well defined and not a matter of opinion.  It is
defined in terms of frequency in a completely physical
non-arbitrary way (e/h), and frequency is defined in
terms of the Cesium spectrum in a way that is very
well established by convention (CGPM).

> I choose the Keithley.  Not because I have
> checked any of the calibrations, but because I simply "judged a book by its
> cover".  Brand name, cost and newness don't guarantee accuracy, but they do
> correlate strongly with accuracy.

There is a valid thought hiding there, but
correlation isn't the right word to capture the
valid thought.  We need to establish causality,
not just correlation.

If you paid $800 for your meter and I paid $900
for my meter, it doesn't mean mine is better than
yours. It might just mean that you are a more
skillful shopper.

If both meters agree, we can combine this with a
strong Bayesian prior expectation that both of them
are accurate, and no further questions need be asked.
But if they disagree, the priors are swept away.  We
turn to the well-established procedures for calibrating
voltmeters.  If necessary we can chase it all the way
back to a Josephson junction and a Cesium beam.

In any case, we don't settle the issue by checking the
price tags.  Willingness-to-pay is _caused_ by evidence
of accuracy, not the other way around.  For a general
discussion of causality, see
  http://www.monmouth.com/~jsd/physics/causation.htm

=======

The point about taking into account the reputation
of the manufacturer is a good point.

I am struggling with this, because I usually
take a rather strong stand against over-reliance
on credentials.  I have had many, many experiences
where people have tried to win an argument by
waving credentials in my face -- and almost always
it's because they're dead wrong.  If they had a
physically-correct argument, they wouldn't need the
credentials.

You will notice that I try to support _my_ arguments
by saying "if you don't believe me, go do the following
experiment" or some such ... as opposed to saying
"you have to agree with me because I have such-and-such
fancy credentials or such-and-such fancy affiliation."

Also:  I get a lot of mail from people who have no
credentials whatsoever.  I make a big point of paying
respectful attention to them.  I've learned a lot
this way!

When I report the results of my experiments, the
ultimate test is whether the results are reproducible.
Similarly, when Keithley sells a meter, the ultimate
test is whether others can reproducibly verify its
calibration.

As a means to this end, there _is_ a role for credentials
or (more precisely) reputations.  That is, Keithley
claims that its products are reliable.  And if people
generally agree, K gets a good reputation.  Or if
people have bad experiences, K gets a bad reputation.
K has a vested interest in protecting its reputation,
and this creates a causal connection between the
secondary thing (reputation) and the primary thing
(real accuracy).  Reputations are a not-quite-systematic
way of communicating crucial information.

In any case, we must keep in mind that reputations etc.
are secondary.  Relying on secondary characteristics
may be expedient but it involves some risk.  If lives
are at stake, or even if reputations are at stake, I
wouldn't rely on the claims of any manufacturer, no
matter how reputable.  I would find some way to cross-
check the calibration.


> * Suppose I want to look for terrorists trying to blow up airplanes.
> Is it "unscientific" to concentrate attention on a 20 year old Arabic
> male travelling one way carrying a backpack?  These are secondary
> characteristics.

This is a really, really, reeeeeally bad idea.

All you will accomplish by this is
 a) Creating a racist society.
 b) Causing all evildoers to take off their
    kaffiyehs, shave their beards, and buy two-way
    tickets.

This will punish the innocent without significantly
burdening the guilty.  This is diametrically opposite
to what a good security measure should do.

This is, in a backhanded way, a powerful illustration
of the risks associated with relying on secondary
characteristics.