Re: [Phys-L] Indicators of quality teaching (Was:MOOC: EdxOffers Mechanics course by Prof.Walter Lewin)

["John Clement" <clement@hal-pc.org>]

The FCI is not a test that is mandated by beaurocrats, it is basically used
by the teachers themselves.  The mandated tests are usually not very good
and they are used inappropriately.  The FCI is a research based test with
high validity.

As to testing, teacher made tests are usually not bad, but they can not be
compared with what other teachers are doing, so the FCI and its bretheren
are the ONLY tests that can be used to compare quality of teaching.  Along
with the FCI there are questions that have come from McDermott's group that
are also very good evaluations of specific learning.  The FCI is just the
beginning.

It is difficult to raise scores on the FCI or the FMCE, without doing the
types of things that have been developed in PER.  Does this mean that PER is
achieving the ultimate quality of education.  Certainly not!  But it is the
beginning of achieving better education for most students.  Achieving it for
all students is still elusive.  Perhaps someday we will be able to test
individual students and figure out what each needs to improve their thinking
and understanding.  Reuven Feuerstein has come up with this for low
performing students.  He uses his test, the LPAD, to diagnose specific
mental deficiencies.  Then the Instrumental Enrichment program can be used
in a group setting to help students overcome these deficiencies.  In extreme
cases individual treatment may be necessary.

The problem with most teacher made tests is that they often can be passed by
sheer memorization.  But tests like the Lawson test, the FCI, FMCE... are
difficult to pass by just in class memorization.  There have been a few
cases where the students got copies of these tests and memorized the
answers, but with a bit of security and obfuscation this is rare.  Remember
that Mazur essentially developed his method to maximize FCI gain.  Then he
used his old final exam as a benchmark to see if FCI gain translated into
better scores on his traditional problem solving exam.  And lo it worked.
Essentially the scores at the low end of the exam were drastically reduced.
So there is independent evidence that the FCI maximization helps the
students.  The Heller's did not like the FCI and created a program which is
designed to improve complex problem solving.  But it turns out that their
program also produces high normalized FCI gain.  It also improved the
students' abilities to solve more complex problems.  So high FCI gain seems
to accompany better learning, but low FCI gain seems to be an indicator of
poorly organized physics concepts.  So I submit that FCI gain is a necessary
condition, but not always a sufficient condition for good learning.

Does this mean you shouldn't test other things?  Certainly not.  There are
other issues that are not addressed by the FCI.  For example I always have
students drawing graphs, and this is not on the FCI, but some graphical
analysis is on the FMCE.  But I have them drawing more complicated graphs.
I have them doing tasks where proportional reasoning is important.  I have
them doing drawings and these are graded.  There are always tasks which
resemble some things they have done in class, but require some transfer.
Yes, I also have some problems.

The FCI is merely addressing the lowest order of optimization.  We have yet
to standardize higher levels of educational optimization. Instructors who
think that their in class tests are the best are usually fooling themselves.
You need some external references.  That is how science works.  Until people
begin to look at education as being a subject that can be measured
scientifically it will remain a cottage art and can be perverted by
administrators and politicians who take pseudo-scientific views.

Obviously JD has NEVER used the FCI or tried to get gain on it, or he would
know how difficult that can be.  Judging a test by just inspecting it is not
very good.  That is like judging pills by looking at them.  You only know
the effect by testing them on subjects, human or otherwise.

John M. Clement
Houston, TX



> I have to ask, because of the Subject: line.  Using the FCI 
> as an "indicator of quality teaching" is an incredibly bad idea.
>
> ... not that that stops certain people from doing it..........
--------------------------------------------
> Note the provisos.  The problem is that we have some really 
> lousy standardized tests.
>
>  -- I am not opposed to all tests.  I am opposed to dumb tests.
>  -- Teaching to the test is either a good thing or a bad thing,
>    /depending on the test/.
>
> Again, the problem is that we have some really lousy 
> standardized tests, and a bunch of mindless bureaucrats are 
> putting waaaay too much emphasis on the lousy tests.  This is 
> the sort of thing that gives mindless bureaucracy a bad name.