Chronology Current Month Current Thread Current Date
[Year List] [Month List (current year)] [Date Index] [Thread Index] [Thread Prev] [Thread Next] [Date Prev] [Date Next]

Re: [Phys-L] evasion-resistant multiple choice



There is also a middle ground where students don't show work but can't work
back from answers: the last handful of questions in each SAT section are
"student produced responses". The students have to bubble in the numeric
answers. There can be more than one right answer. In that format, I think
you could require algebra:

The line through the points (2,5) and (7, -1) is expressed in the form Ax +
By = C where A, B and C are integers. Find a possible value of C.

I think you would have to do the algebra. Or be so familiar with this kind
of algebra that you can skip a bunch of steps -- but that's not the same
thing as evading the algebra entirely.

But this raises another question in my mind: this format has been around
for a while on the SAT -- more than a decade. But they rarely choose to
use it to make students do algebra. I suspect that somewhere in the inner
core of where SAT policy is created, there are still some people who like
the fact that the test rewards clever work-arounds.

On Sat, Apr 8, 2017 at 5:41 PM, John Denker via Phys-l <
phys-l@mail.phys-l.org> wrote:

It has been rightly observed that:
"It is not so easy to write multiple-choice questions that
actually force students to use algebra. ETS rarely succeeds."

-- Philip Keller
_The New Math SAT Game Plan_
https://www.amazon.com/The-New-Math-Game-Plan/dp/
098158960X
Reviewed at https://www.av8n.com/physics/
keller-math-sat.htm

This leaves us with a question: Suppose we want to assess actual
algebra skills. Are there systematic was of doing that?

I frame the discussion in terms of math in general and algebra
in particular ... but the same words apply to a great many other
math and physics topics.

One zeroth-order obvious approach would be to do away with multiple
choice questions altogether Instead, ask open-ended questions.
Require students to show the work.
Pro: This is what most (albeit not all) real-world tasks look like.
Typically there is a multi-step /chain/ of reasoning. Maybe one
of the steps can be solved by working backwards, but others cannot.
Pro: If the student makes a mistake, you get to see the nature of
the mistake.
Con: This sort of grading is labor-intensive. Many teachers face
a workload that does not permit this approach.
Possible path forward: Trade-and-grade. Get students to help with
the grading. This leaves less time for covering new material, but
IMHO it is better to truly learn a lesser amount of material than
to trivialize a greater amount.
Also: One can assign some of each. If students score well on
multiple-guess questions but cannot handle open-ended questions,
that's a warning.
Remark: The day will come when open-ended questions can be graded
automatically, almost as easily as fill-in-the-bubble MC questions
are graded today. This will require a combination of optical
character recognition and AI.


Meanwhile, I'm not quite ready to give up on multiple choice. All too
often it reduces to multiple guess, but not quite always. Not quite
necessarily. So the question remains, is there a systematic way of
generating evasion-resistant MC questions?

One reasonably well-known tactic uses two-step questions: The first
step poses the main question. The second step involves the student
answering indirect questions /about/ the part-1 answer, rather than
directly exhibiting the part-1 answer. See examples below.
Pro: Even though part 2 can be worked backwards, part 1 usually must
be worked forwards. So this is reasonably similar to real life.
Con: If the student makes a mistake, the answer-sheet is unlikely
to tell you much about the nature of the mistake. This problem is
inherent in the multiple-choice framework.
Possible mitigation: You can require students to give the MC answer
/and/ show the step-by-step work. You can review the steps in
selected cases.

Example: We are given three points in the plane. We can represent
them as Cartesian pairs, using the following notation:
a = (a_X, a_Y)
b = (b_X, b_Y)
c = (c_X, c_Y)

These can be used to define three lines. We are given the following
information about the lines:
_ab_: Y = 6 - X
_bc_: Y = 3 + 2X
_ca_: Y = 2 + X

Q1: Choose the best statement about the coordinates of the points:
A) a_X < b_X < c_X
B) a_X < c_X < b_X
C) b_X < c_X < a_X
D) b_X < a_X < c_X
E) c_X < a_X < b_X
F) c_X < b_X < a_X

Q2: Choose the best statement about the coordinates of the points:
A) a_Y < b_Y < c_Y
B) a_Y < c_Y < b_Y
C) b_Y < c_Y < a_Y
D) b_Y < a_Y < c_Y
E) c_Y < a_Y < b_Y
F) c_Y < b_Y < a_Y


Remark: The second step here is a /ranking task/. That is a standard
more-or-less systematic way of asking indirect questions about the step-1
answers ... but it is certainly not the only way.

Important point: AFAICT you cannot evade step 1. You cannot solve it
by merely plugging in a handful of MC answers one by one. There are, in
principle, infinitely many points that have to be considered.

Note: Step 1 can be done algebraically (simultaneous linear equations)
or graphically. Either way is fine with me. It is entirely typical in
the real world to have more than one perfectly reasonable line of attack.
By way of analogy: Oftentimes there are multiple paths through the maze,
and you won't know until afterward (if at all!) which one is shortest.
https://www.av8n.com/physics/glorpy-maze.html
https://www.av8n.com/physics/research-maze.htm

Perspective: As usual, all the extremes are wrong.
That should go without saying, but I feel obliged to say it anyway.
Otherwise some folks assume when I argue against one extreme I must
be arguing for the other extreme ... which is explicitly emphatically
*NOT* what I mean.

The fact is:
1) If all the questions can be answered by working backwards, that's
a defect.
2) If none of the steps in any of the questions can be answered by
working backwards, that's a defect also.

Nowadays defect (1) is incomparably more prevalent than defect (2), so one
can afford to push things pretty far in the direction of de-emphasizing
the backwards approach. However, the backwards approach should not be
deprecated outright. It is one tool in the toolbox. Sometimes it can
be used to great effect.
++ George Green wrote a book that blurted out on page 1 the idea of a
Green function, and then spent the rest of the book explaining why it
made sense and what it was good for.
++ Hermann Minkowski wrote a paper that blurted out, almost on line 1,
the idea that time is the fourth dimension, and then spent the rest
of the paper explaining why that made sense and what it was good for.
++ et cetera...........

By way of analogy: Bananas are good for you /in moderation/. However,
if you ate nothing but bananas you would have serious problems before long.


========

Keep in mind that MC is rarely a goal unto itself. A reasonable goal
is to make things easy to grade, which is not quite the same thing.
Any question that leads to a short, unambiguous answer will fill the
bill. Here's a non-MC example

"Cafeteria Multiplication"

Two intelligent, honest students are sitting together at lunch one
day when their math teacher hands them each a card. “Your cards each
have an integer on them,” the teacher tells them. “The product of the
two numbers is either 12, 15 or 18. The first to correctly guess the
number on the other’s card wins.”

The first student looks at her card and says, “I don’t know what your
number is.”

The second student looks at her card and says, “I don’t know what
your number is, either.”

The first student then says, “Now I know your number.”

What number is on the loser’s card?

-- Trevor Ferril, via
https://fivethirtyeight.com/
features/can-you-outsmart-our-elementary-school-math-problems/

You could recast that in MC form, but that's hardly worth the
trouble, because it's super-easy to grade as it stands.

The main point is that although there are only a handful of plausible
answers, working backwards from the answers doesn't buy you very much,
because the problem requires multi-step reasoning. Working backwards
might help with the last step, but that's nowhere near enough. You
still need a systematic approach to the early steps.

I reckon that skill and confidence in dealing with multi-step reasoning
problems is an important goal unto itself ... more important than any
particular domain-specific physics factoid.

===============================

Topics for further discussion include:

a) Does this make sense?

b) Are there other systematic ways of generating easy-to-grade questions
that are evasion resistant, i.e. that cannot be trivialized by working
backwards from a handful of possible answers?

c) Does anybody have any favorite examples (systematic or otherwise)
that exhibit a high degree of evasion resistance?

_______________________________________________
Forum for Physics Educators
Phys-l@mail.phys-l.org
http://www.phys-l.org/mailman/listinfo/phys-l