Guy Ashkenazi, In his PhysLrnR post of 2 May 2004 11:16:18+0300
titled "Re: Is success dependent on technique - Hawthorne Effect,"
wrote:
"I share the same experience [to that reported by John Clement in a
previous post], where the introduction of a new [Interactive
Engagement] teaching method has resulted in negative social feedback,
but positive performance gain. Anyone (John?) has a reference to
similar results?"
Here are two case studies showing an inverse relationship of Student
Evaluation of Teaching (SET) scores to student learning:
I. When I first started teaching "P201," the large-enrollment
non-calculus-based introductory course for science (but not physics)
majors at Indiana University, I was just in off the
industrial-research-lab street. Following the example of the several
faculty in our department who had won university teaching awards, I
taught P201 in a fairly traditional manner - passive student lectures
(but with lots of exciting demos), algorithmic problem exams, recipe
labs, and a relatively easy final exam - a 77% average in my first
P201 course.
In that course I was gratified to receive a Student Evaluation of
Teaching (SET) evaluation point average EPA = 3.38 [on a scale of 1 -
4 (B plus)] for "overall evaluation of professor," with many glowing
student comments about my WONDERFUL teaching. I have little doubt
that had I continued using traditional methods and giving easy exams
I would have risen to become the U.S. Secretary of Education, or at
least president of Indiana University.
Unfortunately for my academic career, I gradually caught on to the
fact that students' conceptual understanding of physics was not
substantively increased by traditional pedagogy. As described in
Hake (1987, 1991, 1992, 2002c) and Tobias & Hake (1988), I converted
to the "Arons Advocated Method" [Hake (2004a)] of "interactive
engagement." This resulted in normalized gains <g> on the "Mechanics
Diagnostic" test or "Force Concept Inventory" that ranged from 0.54
to 0.65 [Hake (1998b), Table 1c] as compared to the <g> of about 0.2
typically obtained in traditional introductory mechanics courses
[Hake (1998a)].
But my EPA's for "overall evaluation of professor," sometimes dipped
to as low as 1.67 (C-), and never returned to the 3.38 high that I
had garnered by using traditional ineffective methods of introductory
physics instruction. My department chair and his executive committee,
convinced by the likes of Peter Cohen (1981, 1990) that SET's are
valid measures of the cognitive impact of introductory courses, took
a very dim view of both my teaching and my educational activities.
II. Consistent with the my own experience with SET's, Crouch & Mazur
(2001) wrote [see that article for the references]:
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Student attitudes to a course taught with PI [Peer Instruction], as
measured by student evaluations and by our interactions with students,
have differed. In the calculus-based course, EM's [Eric Mazur's]
average evaluation score - 4.5 on a scale of 1-5 - did not change
on introducing PI, and written comments on evaluations indicated
that the majority of students appreciated the interactive approach of
the course.
FOR THE ALGEBRA-BASED COURSE, while still good, EM'S AVERAGE
EVALUATION SCORE DROPPED SIGNIFICANTLY, TO 3.4 [my CAPS]; although
most students are satisfied with the course, there are more
dissatisfied students than in the calculus-based course. Some of this
dissatisfaction is not related to PI; the most frequent complaint
about the algebra based course is that it meets at 8:30 a.m. - the
calculus-based course meets at 11 a.m.!. We also surmise that
students in the algebra-based course are on average less interested
in the course and more intimidated by the material, since these
students are primarily non-science majors; the students in the
calculus-based course are mostly honors biology or chemistry majors.
IT IS IMPORTANT TO NOTE THAT STUDENT EVALUATIONS AND ATTITUDE ARE NOT
A MEASURE OF STUDENT LEARNING [my CAPS]; as discussed in Sec. II, we
saw high learning gains for the students in the algebra based course
in spite of lower perceived satisfaction overall. Other instructors
report similar experiences [Jones et al. (2000)]. Furthermore,
research indicates that student evaluations are based heavily on
instructor personality [Ambady & "Rosenthal (1993)] rather than
course effectiveness. We are nevertheless continuing to try to find
strategies that will help motivate more of the students in the
algebra-based course.
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Please forgive me if I put my ignored broken record back on the
turntable. In Hake "Re: Problems with Student Evaluations: Is
Assessment the Remedy?" [Hake (2002a)] I wrote [bracketed by lines
"HHHHHHHH. . . ."; see that post for the references other than Hake
(1998a,b; 2002c); Cohen (1981, 1990); Feldman (1989)]:
HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH
And yet, despite all the evidence that SET's are NOT valid in their
measures of the COGNITIVE impact of courses, they continue to be so
interpreted by administrators in most of the universities of the U.S.
For example, twelve years after the pioneering work of Halloun &
Hestenes (1985a,b), D'Apollonia & Abrami (1997) approvingly quoted
Michael Scriven (1988) as stating that "student ratings are not only
A valid, but often THE ONLY valid, way to get much of the information
needed for most evaluations." (EMPHASIS in the original.)