Re: [Phys-L] how research is done : exploring a maze using only local information

[John Denker <jsd@av8n.com>]

On 09/18/2015 07:38 AM, Joseph Bellina wrote:

> It could be that one does the correct thing given their belief
> system, but the result of what they does does not turn out as they
> expect.  I suppose you could say they made a mistake, but really the
> mistake is in their belief of how the world works.

I might say it slightly differently:
  A) They did the right experiment, so that wasn't a mistake;
... however ...
  B) They interpreted it unwisely.  They interpreted it in the
   light of their inexpert expectations, not in the light of
   impartial reality.  This is a mistake insofar as it makes
   them undervalue the experiment, especially insofar as it 
   makes them less eager to do a similar experiment the next 
   time the opportunity arises.

  Let's be clear:  As I see it, the mistake is in the post-experiment
  interpretation, not in the conduct of the experiment or the 
  objective result of the experiment.

Simple yet powerful constructive suggestion:  If they were more 
flexible in their expectations, they would be "wrong" a lot less 
often.  They should consider *all* the plausible hypotheses:  
Maybe the coin will turn up heads, and maybe tails.

>  In that sense,
> when they are wrong, when they expect something that does not occur,
> it is an opportunity to learn. 

That's true and useful as far as it goes, but it misses the larger 
point.  People (experts and non-experts alike) can learn just fine 
*without* having wrong expectations.

  I am quite aware that a large segment of the PER literature recommends
  getting the students to commit to an expectation -- *THE* hypothesis --
  before doing the experiment.  Similarly, "science fair" rules are often
  interpreted to require the same sort of thing.

  IMHO this is a Bad Practice.  It produces horrific misconceptions about
  what science is and how science is done.  Sure, it gets students'
  attention ... but then it makes them feel stupid, for no good reason.
  It makes them feel they were Wrong, for no good reason.  This is a
  big deal for many of the students, for deeply-held religions reasons.
  They think that being Wrong is Wrong with a capital W, just like
  stealing is Wrong.

  My suggestion:  There are better ways to engage students' attention!
  Seriously!  Require them to consider *all* the plausible possibilities.
  Do not ask them to commit to one narrow guess.

  This is required for safety, if nothing else.  Before asking them
  to wire up a couple of batteries, consider the possibility that
  they will do it in such a way as to cause an explosion.  Et cetera.

>  In the professional scientific world,
> one hopes for unexpected results since that provide the entre to new
> investigations.  My sense of doing science is rather like a business
> person who does their day to day work, but are always looking for
> that unexpected piece that they can exploit to make more money if you
> are in business, or learn new things about the world if you are a
> scientist.

That's clearly stated ... but clearly /not/ good practice.  Basic
information theory says that an experiment with N possible outcomes
is maximally informative when all N possibilities have probability
near 1/N.  Sure, one should always be on the lookout for the proverbial
flying unicorn, but one should not design an experiment that depends on
or even "hopes for" such events.  When a flying unicorn shows up, it
means the experiment was not optimally designed; the design was
based on a faulty estimate of the probabilities.

This is not just a matter of opinion.  The relevant formulas are

       surprisal = log (1/P)
       information = expectation value of the surprisal
                   = weighted ensemble average
                   = ∑ Pi log(1/Pi)

The rare event is informative in proportion to log(1/P) [which is 
only moderately large]... while for planning purposes it is devalued
by a factor of Pi [which is immoderately small].  The only way you can
even hope to win at this game is to cast a verrrry wide net, capable 
of catching not just flying unicorns but billions of other kinds of
hypothetical creatures.  Such a net tends to be very expensive, 
usually not worth the trouble.

If somebody gift-wraps a unicorn and drops it on my doorstep, I'll
be appropriately grateful, but I don't plan my life around such
scenarios.

======================================

> Let me make my point a bit clearer.  Being wrong does not mean making a mistake necessarily.

That's a subtle distinction.  I doubt that I fully grasp it.
Based on the context, here's my best guess as to what was
intended;  please let me know if this isn't right:
  *) "Wrong" means in /their/ judgment they made a bad decision.
  *) "Mistake" means in /your/ more-sophisticated judgment
   they made a bad decision.