Chronology |
Current Month |
Current Thread |
Current Date |

[Year List] [Month List (current year)] | [Date Index] [Thread Index] | [Thread Prev] [Thread Next] | [Date Prev] [Date Next] |

*From*: John Denker <jsd@av8n.com>*Date*: Fri, 10 Nov 2017 21:38:54 -0700

On 11/10/2017 01:58 PM, Bill Norwood via Phys-l wrote: > I need to build an equation by > which correct answers and incorrect answers are tabulated from a sample set > and a value given to the likelihood that a difference can be determined > between two different items. The question as stated is underspecified. Let me take a guess as to what the intent was, and then outline a solution. It seems to me there are /two/ samples involved. In the community that uses tetrad tests, the samples are often called panels. I will use the two terms interchangeably. My guess is that score ("correct answers and incorrect answers") from panel A is to be used to infer the amount of difference between the two categories ("items"). Then that difference is to be used to infer what the score from panel B will be, and to determine how large panel B needs to be so that the score will not have too much random scatter, when we compare one instance of panel B to another (B1, B2, et cetera). I suggest that we before we worry about the size of panel B, we need to worry about the size of panel A, and there it is not obvious that the two should be the same. They play different roles. Please refer to the graph: https://www.av8n.com/physics/img48/tetrad-test-1000.png The black circles show the results of a panel of size 1000. The abscissa is the true difference between categories, and the ordinate is the tetrad score. The magenta line is meant to indicate /roughly/ what the results would be for a panel of infinite size. In statistic, this would be called the /population/. A panel of size N corresponds to drawing a /sample/ of N panelists from this population. As you can see, there is a fair amount of scatter in the finite-panel results. The scatter is even worse for smaller panels, as you can see here: https://www.av8n.com/physics/img48/tetrad-test-100.png If I understand the intent of the question, the first step is to take the results of panel A and interpret them as an amount of difference. If panel A has only 100 panelists, a score of 0.4 could plausibly correspond to any difference from 0 to 0.8 or so. A score of 1.0 could plausibly correspond to any difference between 3.3 or so and infinity. The numbers in the previous paragraph come from reading the graph inverse-wise, i.e. starting with an ordinate and finding the corresponding abscissa. Now for the second part of the job. If/when you have a value for the difference, you can calculate the population tetrad score (i.e. the magenta line). Then it becomes a relatively conventional exercise in sampling theory to predict how much scatter there will be when a sample (i.e. panel) of size N is drawn from this population. I say "relatively" conventional because lots of people who ought to know better routinely get this wrong. I get tired of seeing public-opinion polls where one of the options is polling at 2%, and the alleged margin of error is 4%. Gimme a break. Probabilities can't be negative, so 2% ± 4% is obviously nonsense. It's easy to guess what calculation they are doing to produce that number, and it's the wrong calculation. For the next level of detail on this, see https://www.av8n.com/physics/sampling-intro.htm A spreadsheet was requested: https://www.av8n.com/physics/tetrad-test.gnumeric https://www.av8n.com/physics/tetrad-test.xls It's basically a Monte Carlo simulation of the panel, for a given amount of difference. That is, the spreadsheet makes it easy to compute /one/ point in the graph. You can collect data by changing the amount of difference (in the yellow-highlighted cell) and hitting the F9 ("recalculate") key to get a new set of random numbers. Then save the results as an ordered pair (copy and save-by-value). You could do that, but I'm way to lazy to do that much handwork, so I wrote a perl program to loop over all difference-values. https://www.av8n.com/physics/tetrad-test.pl The magenta curve is an ad_hoc phenomenological two-parameter fit to the data. In other words, a kludge. It seems likely that some statistician has derived an exact expression, but locating it is more work than I feel like doing at the moment. For purposes of understanding what's going on, and for estimating the required sizes of the panels, the kludge is good enough. As a sanity check, it is easy to prove that when the difference is zero, the tetrad score is 1/3 (for the infinite population). Also, obviously, when the difference is huge the score is 1.0. ============= Last but not least: The whole idea of testing an "unspecified attribute" is just begging for trouble. People wouldn't bother using methods like this when the difference is large, so it is safe to assume that we are looking at small, ill-controlled, higher-order correction terms. Assuming that there is only one such term, i.e. assuming that the difference is one-dimensional, seems like a very rash assumption. The items could subtly differ as to size, shape, color, odor, shelf-life, and ten other attributes simultaneously. Now we are neck-deep in human factors issues, because depending on the /framing/, different panelists will focus on different attributes. This introduces horrific uncontrolled variables, and invalidates all predictions made in the standard way using tetrad tests and all similar unspecified-attribute tests.

**References**:**[Phys-L] Tetrad excel equation***From:*Bill Norwood <bnorwood111@gmail.com>

- Prev by Date:
**Re: [Phys-L] Senior Science Educator/Physicist** - Next by Date:
**[Phys-L] Arecibo, dwarfed.** - Previous by thread:
**[Phys-L] Tetrad excel equation** - Next by thread:
**[Phys-L] Arecibo, dwarfed.** - Index(es):