Setting Straight the Scirven Stance on SET's (was: Is success dependent on technique ?)

[Richard Hake <rrhake@EARTHLINK.NET>]

In response to my post "Re: Is success dependent on technique -
Hawthorne Effect" [Hake (2004)], Michael Scriven (2004), in his
EvalTalk post of 3 May 2004 02:09:42 EDT wrote [bracketed by lines
"SSSSSSSSSSSSS. . . . .":

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Richard Hake, in an otherwise valuable contribution to this thread, ends up
with the following paragraph>

>  For example, twelve years after the pioneering work of Halloun &
>  Hestenes (1985a,b), d'Apollonia & Abrami (1997) approvingly quoted
>  Michael Scriven (1988) as stating that "student ratings are not only
>  A valid, but often THE ONLY valid, way to get much of the information
>  needed for most evaluations." (EMPHASIS in the original.)

This might be a good time to clarify my position, since it's the opposite of
what the reader might suppose.
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

It appears that, unknown to me, my quote of d'Apollonia & Abrami
(1997) misrepresented Scriven's stance on SET's. d'Apollonia &
Abrami's (1997) article was devoted to criticism of Williams & Ceci
(1997). The only mention of Scriven's stance on SET's by d'Apollonia
& Abrami was contained in their final paragraph:

DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
In conclusion, despite the expressive writing style of [Williams &
Ceci] we question that this study contributes much that is new to our
knowledge of the influence of intructor expressivity on students'
ratings of instruction. Thus THE BEST OVERALL CONCLUSION ON THE
VALIDITY OF STUDENT RATINGS REMAINS THAT OF SCRIVEN [my CAPS] that
"student ratings are not only A valid, but often THE ONLY valid, way
to get much of the information needed for most evaluations."
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

I apologize for propagating d'Apollonia & Abrami's quote of Scriven.
To set the record straight, Scriven (2004) wrote in a post to
EvalTalk [herewith distributed to those in the "To" slot with the
permission of Scriven]:

SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
Many years ago, I published my reasons for thinking that EVEN IF
student ratings are a good indicator of learning gains, they would
NOT thereby be valid measures to use in the evaluation of faculty
teaching. This is simply because, like other positive statistical
correlations such as the correlation between skin color and criminal
record, you cannot judge individuals based on group characteristics
WHEN YOU CAN GET BETTER INDICATORS, e.g, track record, or, in the
present case, pre/post gain scores on valid tests, or even, at worst,
text analysis of posttests against grades given.

However [in Scriven (1988)], I carefully identified six other reasons
for using student ratings ON VALID QUESTIONS ABOUT THE PERFORMANCE OF
FACULTY DUTIES to get information that you can't get in other ways,
e.g., punctuality, presence at scheduled office hours, availability
of alleged reserve books, explanations of grades, speed of feedback,
comprehensibility of materials, legibility of overheads, necessity
for expensive texts, etc., etc. Hence I support the use of student
ratings, but only when using a form whose validity depends on
connections to these aspects of faculty teaching performance. This
excludes most forms in common use. Such forms are not the only, and
certainly not a valid way to get evidence about learning that is to
be used in faculty evaluations. Hence the quote from me is NOT
support for THAT use of them, the most common such use and the one
that Richard Hake is talking about throughout that post.

I am in fact the most radical critic of the usual forms, since my attack on
them applies EVEN IF THEY WERE GOOD INDICATORS OF LEARNING GAINS. Let me
underline what a tough position that is. While it seems probable that in many
contexts they are a weak statistical indicator of learning, RH's
detailed examples show that they have fatal flaws in that respect.
But in my view, it's easy to show that EVEN IF THE CORRELATION
BETWEEN S.E.T. SCORES AND LEARNING WAS 1.0, THEY COULD NOT BE USED AS
VALID INDICATORS FOR THE EVALUATION OF TEACHING.

People that find that hard to swallow (most people, in my experience) should
ask themselves if they think civil servants in S. Carolina can
legitimately use evidence showing that women have NEVER ONCE
succeeded in senior admin
positions in that state's civil service (n = 136) should be allowed
to use that fact, a 1.0 correlation, to exclude female candidates
from . (A
hypothetical case.)

Moral of all this. Program evaluation is not the only branch of evaluation
where plenty of evaluators are, as the New Yorker used to say,
"Unclear on the Concept".
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

Two further points:

1. Scriven's approval of SET's as indicators of the performance of
faculty duties agrees with my own approval of SET's for
non-cognitive-impact measures as indicated in Hake (2004):

"I think SETs can be 'valid' in the sense that can be useful for
gauging the AFFECTIVE impact of a course and for providing diagnostic
feedback TO TEACHERS [see, e.g., Hake & Swihart (1979)] to assist
them in making mid-course corrections."

2. I have listed below the Ambady & "Rosenthal (1993) reference in
Crouch & Mazur's statement (quoted in Hake (2004) that:

"Furthermore, research indicates that student evaluations are based
heavily on instructor personality [Ambady & "Rosenthal (1993)] rather
than course effectiveness."

Also referenced are introductions to the "thin-slice judgement"
literature and debate thereon see Hake (2003a,b,c).


Richard Hake, Emeritus Professor of Physics, Indiana University
24245 Hatteras Street, Woodland Hills, CA 91367
<rrhake@earthlink.net>
<http://www.physics.indiana.edu/~hake>
<http://www.physics.indiana.edu/~sdi>

REFERENCES
Ambady, N. & R. Rosenthal. 1992. "Thin Slices of Expressive Behavior
as Predictors of Interpersonal Consequences: A Meta-analysis,
"Psychological Bulletin 111: 256-274. For an introduction to the
"thin-slice judgement" literature and debate thereon see Hake
(2003a,b,c).

d'Apollonia, S. & P.C. Abrami, 1997. "In response . . [to Williams &
Ceci (1997)]. . . Change, September/October 1997.

Hake R.R. & J.C. Swihart. 1979.  "Diagnostic Student Computerized
Evaluation of Multicomponent Courses," Teaching and Learning V(3)
(Indiana University), updated on 11/97; online as ref. 4 at
<http://www.physics.indiana.edu/~hake>, or by simply clicking on
<http://www.physics.indiana.edu/~sdi/DISCOE2.pdf> (20 kB).

Hake, R.R. 2003a. "Thin-Slice Judgments, End-of-Course Evaluations, Grades,
and Student Learning; online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0303&L=pod&P=R18434>. Post
to ASSESS, EvalTalk, PhysLrnR, POD, & STLHE-L of 28 Mar 2003
16:23:25-0800.

Hake, R.R. 2003b. "Thin-Slice Judgments, End-of-Course Evaluations,
Grades, and Student Learning - CORRECTIONS; online at
< http://listserv.nd.edu/cgi-bin/wa?A2=ind0303&L=pod&P=R19378>. Post
to ASSESS, EvalTalk, PhysLrnR, POD, & STLHE-L of 29 Mar 2003
11:45:27-0800.

Hake, R.R. 2003c. "Thin-Slice Judgments, End-of-Course Evaluations, Grades,
and Student Learning; online at
< http://listserv.nd.edu/cgi-bin/wa?A2=ind0303&L=pod&P=R21469>. Post
to ASSESS, EvalTalk, PhysLrnR, POD, & STLHE-L of 31 Mar 2003
12:47:55-0800.

Hake, R.R. 2004. "Re: Is success dependent on technique - Hawthorne
Effect," online at
<http://listserv.nd.edu/cgi-bin/wa?A2=ind0405&L=pod&F=&S=&P=484>.
Post of 2 May 2004 22:08:12-0700 to AERA-J, ASSESS, Biopi-L,
Chemed-L, EvalTalk, Phys-L, PhysLrnR, Physhare, and STLHE-L (later
sent to AERA-D, AERA-K, and Dr-Ed).

Lang, S. 1997."Cornell study finds student ratings soar on all
measures when professor uses more enthusiasm. Study raises concerns
about the validity of student evaluations." Cornell Science News,
Sept.; online at
<http://www.news.cornell.edu/releases/Sept97/student.eval.ssl.html>.

Lewis, R. 1998., "Student Evaluations: Widespread and Controversial,"
The Scientist 12(9):12, Apr. 27, 1998; online at
<http://www.the-scientist.com/yr1998/apr/prof_980427.html>.

Scriven, M. 1988. "The Validity of Student Ratings," Instructional
Evaluation 9: 5-18.

Scriven, M. 2004. "Re: Is success dependent on technique - Hawthorne
Effect," EvalTalk post of 3 May 2004 02:09:42 EDT; online at
<http://bama.ua.edu/cgi-bin/wa?A2=ind0405a&L=evaltalk&T=0&O=D&X=14AD7F14CF0B2527AF&Y=rrhake@earthlink.net&P=963>.
The encyclopedic URL indicates that the EvalTalk archives are open
only to subscribers. However, it takes only a few minutes to
subscribe (and then unsubscribe) by following the simple directions
at <http://bama.ua.edu/archives/evaltalk.html>/ "Join or leave the
list (or change settings)" where "/" means "click on." If you're
busy, then subscribe using the "NOMAIL" option under "Miscellaneous."
Then, as a subscriber, you may access the archives and/or post
messages at any time, while receiving NO MAIL from the list!

Williams, W.M. & S.J. Ceci. 1997 "How'm I Doing," Change,
September/October, pp. 13-23; an article summary is online at
<http://www.aahe.org/change/> / "Online Archive", where "/" means
click on.  Search for "Williams."
Williams & Ceci wrote: "Today, all instructors would be well advised
to ask their students frequently 'How'm I doing?' and listen
carefully to the answer. As in politics, however, the answer may have
more to do with style than substance." See also Lewis (1998) and Lang
(1997).