[Phys-l] assessments

[John Denker <jsd@av8n.com>]

On 02/12/2011 03:51 PM, Richard Tarara wrote:

> What I find confusing is that while we want our students to become less 
> mechanical and less bound by a rigid set of rules in their problem solving 
> and learning techniques, many educational people are striving to make the 
> assessment of that learning ever more mechanical and ever more bound by 
> rigid sets of rules!

Indeed!

That's something we need to discuss.  In particular, how can we 
improve the situation?

I don't mind teaching to the test ... if it's a good test!  We
need better tests.  We need /much/ better tests.  But the tests 
seem to be getting worse not better.  How do we turn this around?

==============

As one small contribution to the discussion, it seems to me that
one malign driving force can be found in the ECLBE law (every
child left behind equally).  This law is up for revision, so now
would be a good time to identify specific changes we would like 
to see.

One provision of the law requires testing, but leaves the details
up to the states.  The states have nothing to gain and plenty to lose
from a test that actually measures anything.  The entirely foreseeable 
consequence is that the firms that prepare the tests (under contract 
to the states) are engaged in a race to the bottom, striving to come 
up with tests that anyone can pass, whether they understand anything
or not.

This is ineffably foolish ... but I don't see any way, politically, 
to get it changed.  National testing would prevent some of the worst
abuses, but would not solve anywhere near all the problems, and would
be politically impossible AFAICT.  People would start waving their
states-rights don't-tread-on-me flags and that would be the end of
the discussion.  I am /usually/ sympathetic to the idea that innovation
is easier at the state and local level than at the federal level, but
recently many of the testing innovations have been going in the wrong 
direction, so this argument, while valid, is not helpful.  On the
other hand, the national testing operations such as ETS have not exactly
covered themselves in glory, either.  For years they have been pushing 
multiple-guess tests that I find far too mechanical as opposed to 
thoughtful or thought-provoking.

One possibility is that the law will not be changed, because it is
working as intended.  The combination of school vouchers and the
punitive provisions of the ECLBE law are evidently intended to destroy 
the public school systems, so that students can be re-segregated along 
racial, partisan, and sectarian lines.

This discussion is currently at the level of David Logan's "stage 2"
tribalism. ("How can these people be so stupid and yet live?"  
"Their process has run roughshod over principle.")  I would like 
to kick it up a stage or two ... but I don't know exactly how.

I am convinced we could do better if we wanted to.  Much better.

As I mentioned above, I don't mind teaching to the test ... if it is
a good test.  As a positive example, I point to the system the FAA
uses for certifying pilots:
  1) There is a written test.  This is not worth much IMHO.  It is
   multiple-guess.  The passing score is 70, but if you get anything
   below the high 90s you should seriously ask yourself what you're
   doing wrong.  The crucial point is that this test is only a very 
   small part of the overall assessment.  See also item (3).
  2) The instructor signs off, certifying that the candidate meets
   all the requirements.
  3) There is a one-on-one oral exam with an FAA inspector or designated
   examiner.  If the candidate did not do well on the written test, you
   can expect extensive, pointed questioning at this stage.
  4) There is a behind-the-wheel practical test.  The examiner will put
   the candidate into unfamiliar situations.  This leaves no doubt that
   the pilot knows the principles and can apply them.

The practical test is conducted in accordance with a published standard
(which often serves as the de-facto syllabus for the training program).
The examiners all go to "examiner standardization" training.  The 
examiners are all highly experienced, far beyond what is required of
ordinary instructors ... and flight instructor certification is nothing
to be sneezed at, either.  Lives are at stake here, so everybody takes 
this seriously.  The certification system contributes a few hundred bucks 
to the cost of training, which is not zero, but not as much as you might
have thought, considering the complexity of the task.

In contrast, a student can spend well over a hundred bucks on the SATs,
for a test that is IMHO incomparably less confidence-inspiring.  Considering
the vast economies of scale that should apply to such a testing program, 
I don't think we are getting good value for our money.

Can you imagine what it would be like if we considered physics a life
and death proposition, with the final exam conducted one-on-one with
a highly-respected inspector in a practical situation?