This topic is rife with statements of the form "XXX is
good" when in fact XXX is good some of the time but not
all of the time. Non-experts have no way of knowing whether
XXX will help them or hurt them.
Let's start with a couple of positive example, illustrating
the value of DoE -- followed by some cautionary tales.
DoE is fundamentally an information-theory game. Most
people's favorite example is the "weighing 12 coins"
puzzle previously discussed in this forum (although nobody
responded to the challenge to find a _parallel_ algorithm,
i.e. one that specifies all the test vectors before any
measurements are carried out).
Next, here's a slight elaboration of the example Tim introduced.
Suppose there are N variables of interest. Imagine N=10 to
make things concrete. For simplicity let them be smoothly
continuous variables (which is a drastic restriction).
You have been hired by the plant owner to optimize some
process that depends on these variables. So, what is your
best strategy for exploring the space?
If the objective-function were a linear function of the
variables, there would be N+1 unknown coefficients. So
you would need 11 measurements. Almost any strategy
will work; you could take one "baseline" measurement
and then wiggle each of the 10 variables separately.
Using fancy DoE-based test vectors won't make the problem
any easier; it's already so easy it can't get any easier.
So, what's the point of DoE?
Answer: most likely the plant owner is not an idiot.
Probably the plant is operating somewhere near the
optimum already, so we must be prepared for the possibility
that the objective function is _not_ a linear function
of the variables. Near the optimum the lowest-order
description must be second-order, with all first-order
terms vanishing. More generally there will be:
1 zeroth-order term
N first-order coefficients
N*(N+1)/2 second-order coefficients
for a total of:
N*(N+3)/2 + 1
i.e 66 unknown coefficients in our example, excluding
third-order and higher terms.
It is usually convenient to write the second-order
coefficients as a matrix with N^2 elements, but
since it is a symmetric matrix we don't need to do
N^2 work, just slightly more than half that.
You can reduce the number of unknows if you know _a
priori_ that some of the given variables are independent,
or (!!) if you know how to perform a change of variables
so that some of the new variables are independent.
It is fairly common to know a few such things, but
let's pretend for now that you don't.
Now, suppose you wiggle each variable separately,
once to the high side and once to the low side.
That (plus the baseline case) will give you 2N+1
i.e. 21 data points. This is not nearly enough
to determine all the unknowns.
At the other extreme, suppose you wiggle each and
every variable to the high side and to the low side,
in all combinations. That involves 3^N measurements,
i.e. 59,000, which is vastly too many.
This is the classic case where you can use DoE to
tell you what test vectors to use.
You want slightly more than N(N+3)/2+1 test vectors.
You want them to be more-or-less evenly distributed
around the N-dimensional hypercube. The vectors you
get by wiggling each variable separately are not
numerous enough and not nearly evenly-enough distributed.
Interestingly, randomly-chosen test vectors (chosen at
random from the points on the hypercube) are nearly
optimal, although you can do slightly better if you
work at it.
By random I mean really random, not just the first
thing that pops into your head. People are not very
good random-number generators.
===
Things get tricky when you have to decide _how much_ to
wiggle each variable. You want something big enough to
give you a good signal-to-noise ratio, but not so big
that it's going to ruin something.
You might want to ask the plant owner about this. He
probably already knows, approximately. If not, you've
got a lot of work to do. Start small and take ever-larger
steps until you get a signal. You might need to do
signal averaging or the like.
You must not separate the data-taking phase from the
analysis phase. You need an iterative, interactive
process, where previous results guide the design of
the next measurement. This complicates the statistics
quite a bit, but that's why you get the big bucks:
paying you to be smart about the DoE is cheaper than
doing ill-designed experiments.
===
Debugging a non-working plant is about a lot harder
than optimizing an already-working plant.
===
As discussed in my previous note, if the topography of
the objective function isn't nice and simple and smooth,
all bets are off.
Recall the previous example: No DoE book is going to
teach you how to write computer programs by "experimenting"
with the spelling of the code.
As perhaps a less-extreme but still effective cautionary
example, consider linear programming. This is a hard
problem, and there's a big demand, so there's a huge
industry devoted to finding approximate solutions. You
might not at first understand why it's so hard, because
the objective function is piecewise linear. If it were
totally linear, the problem would be trivial. But there
are constraints that cut in here and there; the solution
is always found at the corner formed by N constraints.
There are roughly 2^N corners and for large N you don't want
to check them all. Your intuition based on N=2 is misleading;
when N=2 you can check all the corners in an instant. When
N>50 that doesn't work so well.
The "partial factorial analysis" that some people consider
the bread-and-butter of DoE won't help you a bit with linear
programming tasks. There are other techniques that will.
This posting is the position of the writer, not that of Clotho,
Lachesis, or Atropos.
This posting is the position of the writer, not that of SUNY-BSC, NAU or the AAPT.