Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-L] Goodhart's Law



The phenomenon has been rediscovered many times.

Another version is
Be careful what you measure; you might get it.
This is popular in business management circles.

An even more basic version is
Teachers teach to the test.
Students study to the test.
This is either a bad thing or a good thing,
depending on the test.

Another version is:
Données utilisés, données usés.
which is to say
Used data is used-up data.
in the sense that data that is used for training cannot
be used for testing, if you want valid results. This
version is popular in the machine learning community.
The same idea applies to test questions and even homework
questions:
A used question is a used-up question.
Pre-test questions cannot be used as post-test questions,
if you want valid results.

Conceptually this is related to the idea of /overfitting/
the data. If you fit a polynomial of too-high degree to
a too-small data set, you will get an excellent fit to
the training data, but terrible interpolation and worse
extrapolation. A fancier name for this is bias/variance
tradeoff.

Comparing one teaching method to another is basically an
experiment on human subjects, with ridiculously many adjustable
parameters, along with ridiculously many ill-controlled noise
sources, and (usually) no semblance of proper experimental
protocols. Overfitting and bias are to be expected.