Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: Design of Experiment

John Denker responded to my original post about design of experiments...
Perhaps an example will enliven this thread:

Suppose you are trying to locate an object O using
passive sonar receivers R1 R2 and R3. The following
layout would be really terrible:

R1 R2 R3 O

since to first order any displacement of the object in
the directions perpendicular to the line of symmetry is
undetectable.

You would be *much* better off arranging the receivers
like this:
R1

R2 O

R3
and even better off arranging them in a *big* triangle
surrounding the object.

This shows the importance of the physical design of the experiment.
Another important question is the design of the "phase space" of the
experiment and this is where DOE excels. Let me give a simple example.

Suppose you want to make paper helicopters that fall as slowly as possible
You identify several factors that might effect the measured time to fall:
* L: the length of the blades
* W: the width of the blades
* P: the person timing the fall
* H: the height from which it falls
* C: the color of the paper
* N: the # of paper clips attached to the bottom.

APPROACH 1: You could build a standard helicopter, say with L = 3", W =
0.5", N = 5. Then build a set of helicopters with only one of the
parameters changed. You might find that increasing L to 5, increasing W to
3" or decreasing N to 2 all help improve the time. So you change all three
and get a poor result! Your experiment design explores only the three axes
of the "parameter phase space".

APPROACH 2: You could adjust L to get a maximum, then adjust W , then
adjust N. Again, you are exploring a very limited amount of the "phase
space". You would probably have to cycle through this process several
times to get close to the true maximum. Again, you are only exploring a
small part of the "phase space".

But perhaps a heavy helicopter works best from great heights and lighter
helicopters work better from lower. Or perhaps person P1 consistently
times 0.2 seconds longer than P2, and P1 did all the long wing helicopters,
so they appear to be better than they really are. Or person P1 has poor
eyesight and can't judge well when black paper is used. Or ...

There are any number of ways to go wrong if you don't think about things
clearly ahead of time.

APPROACH 3: Pick a small number (often two) different choices for each
variable. You can then run all possible trials (2^6 = 64) - known as a
"full factorial design" - in our case. Or better, run a carefully chosen
subset - a "partial factorial" - as illustrated below. The subset of 16
listed below is not unique, but it does have special properties. Each
factor is high 8 times and low 8 times. Furthermore, in the set of 8
trials where any one variable is high, each other variables is high 4 times
and low 4 times. You lose some info by cutting the trials, but you save
time. You discover which variable have large effects and which have small
effects. You also get a good idea about which of the variables are
interacting with others. And note the random order, which cuts down on
unidentified systematic changes.

Run A B C D E F
1 - + - - + +
2 + + - - - +
3 + - - - + -
4 - + + - + -
5 + + + - - -
6 - - - + - -
7 - + - + + +
8 - + + + + -
9 + + - + - +
10 + - + + + +
11 + - - + + -
12 - - - - - -
13 + + + + - -
14 - - + - - +
15 - - + + - +
16 + - + - + +

I'm amazed, too. The ideas change your thinking and improve
your "gut reactions" even if you're not using the full formalism.

Most of us probably have a pretty good gut reaction here, but you could
think of many other cases where there is no good theory or even "gut
reaction" to serve as a guide.

Some of the DOE techniques require statistical sophistication to fully
understand, but you don't need to know all the details to apply the
techniques

* perform trials in random order, rather that stepping through in order

Yes, there are lots of cases where people should be
randomizing the data-taking and they neglect to do so.


But there is more to the story. This is actually tricky.
You couldn't have done the Mercury/Gemini/Apollo missions
in random order, because the later missions exploited
knowledge acquired in earlier missions.

I would consider these as separate experiments, not a single experiment to
find a single result. Often you do one experiment knowing full well that a
follow-up experiment is needed. With our helicopters, we might find that
the area of the blades and the # of clips are the two key factors. Then
you can follow up with an experiment designed to optimize these two
factors.

In general you should not take a bunch of data and then
try to figure out how to analyze it.

Exactly - that is why you choose the trials in advance so that you know
exactly what are going to be able to learn from the experiment


* standard deviation isn't a good measure of accuracy (but you probably
knew that one)
I didn't know that one until I got to grad school and found
myself the proud owner of a bunch of data where various
variables were highly correlated with others.

That, too. I was actually thinking about a much simpler idea - standard
deviation, sigma, vs. standard deviation of the mean, sigma / N^1/2. The
first tells you how far any given measurement might fall from the average.
The second tells you how far the average might fall from the "true" value.

* optimizing consistency is often more important that optimizing
magnitude

i.e. don't turn up the gain if all you're doing is amplifying
the noise.

That's not the only reason! It is better to roll 10/10 pretty strong bolts
off the assembly line than to run 9/10 great bolts and 1/10 duds. The
second set may still have a higher average strength, but I'd buy the first
set any day.


Tim Folkerts


Department of Physics
Fort Hays State University
Hays, KS 67601
785-628-4501

This posting is the position of the writer, not that of SUNY-BSC, NAU or the AAPT.