If you reply to this long (17 kB) post please don't hit the reply
button unless you prune copy of this post that may appear in your
reply down to a few relevant lines, otherwise the entire already
archived post may be needlessly resent to subscribers.
***************************************
ABSTRACT: My post "Formative Pre/post Tests For Various Disciplines"
[Hake (2008a)] has thus far stimulated 9 responses on the Chemed-L
archives <http://mailer.uwf.edu/archives/chemed-l.html>. Most of the
responses are critical of certain questions on a "Chemistry Concept
Test" (CCT) due to two graduate students and not representative of
diagnostic tests developed through arduous qualitative and
quantitative research by disciplinary experts. Nevertheless, I
respond to criticisms by Roy Jensen and Mark Bishop regarding (a)
ambiguity in question wording, (b) poor figures, (c) utility of the
CCT as a first-day "ice-breaker," and (d) the canonically assumed
inadequacy of multiple-choice tests.
***************************************
So far (9 Jul 2008 13:20:00-0700), there have been 9 responses to my
Chemed-L post "Formative Pre/post Tests For Various Disciplines"
[Hake (2008a)] on the archives of Chemed-L at
<http://mailer.uwf.edu/archives/chemed-l.html>. Most of the
responses are devoted to criticism of "Chemistry Concept Test"
questions at <http://www.daisley.net/hellevator/cci/cciv5.pdf> (92
kB), due to David Boyer and Consuelo Rogers, evidently (at the time -
2001) graduate students in a course at ASU.
Three main points "1," "2," and "3":
111111111111111111111111111111111111111111111
1. Roy Jensen (2008a) wrote [bracketed by lines "JJJJJJJJ. . . ."]:
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
". . . .in http://www.daisley.net/hellevator/cci/cciv5.pdf, questions
4 & 5. The figures are poor. I believe the correct answer to 4 is E.
But the correct answer to 5 could be B or D, depending on how much
you know about gases. D - gases disperse throughout the room. B - the
perfume has a high molecular mass, hence a higher density, and will
settle in air.
Several inventories use mass and weight interchangeably. They're not.
JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ
Three comments "a," "b," and "c":
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
a. As is typical in discussion lists - see e.g., "Referenceless
Posts - Like Children Alone at Night In a Big City" [Hake (2008c)],
Roy gives us no clue at to where the test
<http://www.daisley.net/hellevator/cci/cciv5.pdf> (92 kB) originated.
I finally found it ensconced in the reference "Chemistry Concept
Test" [ASU (2004)] located within "Re: Measuring Content Knowledge"
[Hake (2004a)]. To bring the test to your screen first click on
<http://www.daisley.net/hellevator/> and then click on "Evaluations"
near the top of the page.
The test is titled "Chemistry Concept Inventory, is dated 07/20/2001,
and is credited to David Boyer and Consuelo Rogers (PHYS 540
Project). I assume they were, at the time, graduate students
enrolled in a course PHYS 540 at ASU.
Of course, that test should not be regarded as representative of
valid and consistently reliable diagnostic tests developed through
arduous qualitative and quantitative research by disciplinary experts
- see e.g., Halloun & Hestenes (1985a,b).
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
b. Roy commented "Several inventories use mass and weight
interchangeably. They're not."
To which inventories is Roy referring? Surely none authored by
physicists! ;-)
cccccccccccccccccccccccccccccccccccccc
c. Roy wrote:
When I click on the URL AS IT APPEARS ON THE CHEMED-L ARCHIVES at
Jenson (2008a) I get a "Not Found" panel.
Suggestion #9 of my universally ignored "Fourteen Posting
Suggestions" [Hake (2005a)] is (slightly edited):
************************************
9. Give URL's as <http://......>. For most (but not all) server/mail
systems: the "http://"; serves to hot-link URL's, and the angle
brackets <. . .> serve to
(a) preserve hot-linking across line breaks, and
(b) delineate what is and what is not part of the URL, e.g., is the
comma after "pdf" in
"http://www.daisley.net/hellevator/cci/cciv5.pdf, " part of the URL?
Note that unlike <http://www.daisley.net/hellevator/cci/cciv5.pdf>,
its abbreviation <www.daisley.net/hellevator/cci/cciv5.pdf> is NOT
hot-linked.
CLICK ON ALL URL'S IN YOUR MESSAGE TO BE SURE THEY WORK.
************************************
222222222222222222222222222222222222222222222
In a later post Roy Jensen (2008b) wrote:
[The question on the inventories] "would make for an interesting
exercise on the first day. An ice breaker, of sorts."
The use of diagnostic tests such as those indicated in "Formative
Pre/post Tests For Various Disciplines" [Hake (2008a)] should not, in
my opinion, be relegated to mere "interesting exercises and/or
icebreakers for the first day - see e.g. "The Physics Education
Reform Effort: A Possible Model for Higher Education?" [Hake (2005b)].
3333333333333333333333333333333333333333333333
3. Mark Bishop wrote [bracketed by lines "BBBBB. . . . ":
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
I'm reminded of the reasons why I'm no fan of multiple choice. For example, for
A guest at the party dives in the swimming pool holding an inflated
balloon! When she gets to the bottom of the pool, the balloon
(A) becomes smaller.
(B) becomes bigger.
(C) does not change in size.
(D) does not submerge with the diver.
I'd want to write something in the exam margin like, "If she DIVES
into the pool, isn't it most likely that the balloon would pop and
therefore get smaller?" Multiple choice is often an exercise in
figuring out what the author of the question probably meant, trying
to keep from thinking too hard about the subtleties.
BBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
The question is poorly written and not representative of those on
valid and consistently reliable diagnostic tests developed by
disciplinary experts such as Halloun & Hestenes (1985a,b).
Why do most diagnostic tests utilize multiple-choice tests [MCT's]?
In "Should We Measure Change? Yes!"[Hake (2007)] I wrote:
HHHHHHHHHHHHHHHHHHHHHHHHHHHH
Why MCT's? So that the tests can be given to thousands of students in
hundreds of courses under varying conditions in such a manner that
meta-analyses can be performed, thus establishing general causal
relationships in a convincing manner.
Can MCT's measure conceptual understanding and higher-order learning?
[For a cogent discussion of higher-order learning see Shavelson &
Huang (2003).] Wilson & Bertenthal (2005) think so, writing (p. 94):
"Performance assessment is an approach that offers great potential
for assessing complex thinking and learning abilities, but multiple
choice items also have their strengths. For example, although many
people recognize that multiple-choice items are an efficient and
effective way of determining how well students have acquired basic
content knowledge, many do not recognize that they can also be used
to measure complex cognitive processes. For example, the Force
Concept Inventory . . . [Hestenes et al. 1992] . . . is an assessment
that uses multiple-choice items to tap into higher-level cognitive
processes.
HHHHHHHHHHHHHHHHHHHHHHHHHHHH
REFERENCES [As these appear on the Chemed-L (also EvalTalk and POD)
archives, the URL's will probably not be correctly hot linked,
evidently due to a BUG in LISTSERV 15.0 and 15.6. For that reason,
I'm also transmitting this post to the OPEN archives of AERA-D
<http://listserv.aera.net/archives/aera-d.html> where the hot-linking
will probably be preserved by AERA-D's LISTSERV 14.5.
Hake, R.R. 2008b. "Formative Pre/post Tests For Various Disciplines -
ADDENDUM," Chemed-L post of 8 Jul 2008 20:30:45-0700; online at
<http://mailer.uwf.edu/listserv/wa.exe?A2=ind0807&L=chemed-l&D=1&F=&S=&P=3246>.
I call attention to an apparent BUG in Chemed-L's LISTSERV 15.0 that
prevents accurate hot-linking of most URL's as they appear in the
archives.
Halloun, I. & D. Hestenes. 1985a. "The initial knowledge state of
college physics students." Am. J. Phys. 53: 1043-1055; online at
<http://modeling.asu.edu/R&E/Research.html>. The print version
contains the Mechanics Diagnostic test, precursor to the widely used
Force Concept Inventory [Hestenes et al. (1992)].
Hestenes, D., M. Wells, & G. Swackhamer. 1992. "Force Concept
Inventory," Phys. Teach. 30(3): 141-158, March; online (except for
the test itself) at <http://modeling.asu.edu/R&E/Research.html>. The
1995 revision by Halloun, Hake, Mosca, & Hestenes is online (password
protected) at the same URL, and is currently available in 15
languages: Chinese, Czech, English, Finnish, German, Greek, Italian,
Malaysian, Persian, Portuguese, Russian, Slovak, Spanish, Swedish, &
Turkish. A French version should soon be available.
Shavelson, R.J. & L. Huang. 2003. "Responding Responsibly To the
Frenzy to Assess Learning in Higher Education," Change Magazine,
January/February; online at <http://www.stanford.edu/dept/SUSE/SEAL/>
as the first "highlighted" paper.