Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-l] What's the point of teaching to the test whenit's graded inaccurately?



JC said;
"This is because while they have good statisticians, they do not have people
who actually do
research in the education in these fields."

Really?

"The ETS Research & Development division provides R & D to support existing
assessments, to develop future products and services, and to advance
educational measurement and policy research worldwide.
In addition, ETS researchers and analysts regularly publish reports that
contribute to the policy debate on pressing educational issues. To ensure
that they remain up-to-date, innovative and flexible, our research teams
collaborate with other research organisations around the world that are
engaged in similar pursuits.
The R&D division at ETS is comprised of dedicated centres for:

a.. Assessment research
b.. Statistical analysis and psychometrics research
c.. Assessment innovations and technology transfer
d.. Data analysis research
e.. Global assessments
f.. Learning and teaching research"

M. Horton

----- Original Message -----
From: "John Clement" <clement@hal-pc.org>
To: "'Forum for Physics Educators'" <phys-l@carnot.physics.buffalo.edu>
Sent: Wednesday, March 25, 2009 4:44 PM
Subject: Re: [Phys-l] What's the point of teaching to the test whenit's
graded inaccurately?



The ETS has a certain arrogance about them that I have never liked. They
claim to be able to test for anything. If there is something you would
like to determine about someone (or a group), they can come up with a test
to find out for sure whether they know it or not. I don't think this can
be done for many things!


ETS has to their credit done some very good metastudies. But they have the
problem that they must sell tests commercially. They are also obviously
excellent statisticians.

But they do try to project the image that their tests have a huge
importance, and that they are accurate. The reality is that the correlation
between the test results and student performance is not has high as one
might think. For example the SAT certainly has some predictive ability in
the freshman year, but much less after that. The GRE is likewise not a very
good predictor of grad student quality. But these tests certainly provide
benchmarks which may be the only information that a school can use. When a
student comes from a very small school, the grades do not tell the school
what they need to know. Grades and class rank obviously are better
predictors from a large school where statistics are easier to correlate.

There is a problem that these tests are culturally biased, so that minority
students will score lower. There was a very interesting study about reading
ability. They gave a standard reading ability test to students and then
ranked them according to ability. Then they gave a second reading ability
test, but used passages about baseball. Students who ranked low in the
original test, but were baseball fans scored in the top rank on the second
test. Most standard reading tests have very unrepresentative prose in the
sense that it is drawn from standard literature, and not from what students
might be interested in.

Then ETS has ignored some very important research findings. They have
students fill out a form where they have to indicate things like race...
This is done before the test is administered. It is now known that this
depresses minority performance. But if the form is filled out after the
test, minorities do better.

While ETS might be able to test "understanding" of the concepts of a
science, in general they have not done this very well. This is because
while they have good statisticians, they do not have people who actually do
research in the education in these fields. Now if they wanted to, they
could use the FCI&FMCE as a baseline, and then develop other questions that
measure the same things. And similarly neither education researchers nor
cognitive scientists have been able to make a large improvement in physics
gain. It takes physicists who collaborate with other researchers to do
this.

They are involved in testing teachers to see if they have competence in the
subject material, but not all of their questions are very good. Also these
tests do not really measure how well the teacher can teach the subject.

Probably ETS originally did what Binet did with his initial "IQ" test.
Binet generated various questions, and looked at how well the students did
in school. Then he cooked the questions and analysis to maximize the
correlation. But this process does not reveal why students fared poorly.
Notice that such an "intelligence" test does nothing of the kind. It merely
tests the ability within a particular culture (French middle/upper class) to
achieve good grades in school.

There is a big problem with many of the math tests. It is easy to make an
end run around many questions. For example rather than solving an equation
it is often easier to just plug in the answers and see which ones are
correct.

ETS certainly makes much better tests than many other agencies such as the
state testing agencies, but that does not mean they are infallible. To make
good tests you need subject knowledge, cognitive theory knowledge, and good
language knowledge, plus probably other skills. So for example you make a
question testing density understanding, but it may actually be testing the
ability of a student to do 2 variable reasoning, which is low in HS
students.

I am not against testing. I am against high stakes testing which distorts
education. Use good tests, and then use the results to help the schools
rather than just punish the teachers who are often overworked, and do not
know what to do to improve the results. TX has probably one of the most
intensively tested school systems, and also the highest dropout rate. They
are finally considering doing away with the current high stakes testing, and
only using testing to grant a state blessed diploma. Above all learn enough
about thinking to be able to understand and make or acquire good tests.

John M. Clement
Houston, TX

_______________________________________________
Forum for Physics Educators
Phys-l@carnot.physics.buffalo.edu
https://carnot.physics.buffalo.edu/mailman/listinfo/phys-l