[Phys-L] Re: Research on Student Response Systems

[John Denker <jsd@AV8N.COM>]

Rick Tarara wrote:
> But how do you even do a meta-study?

It ought to be possible.  As usual, it is important to
ask the right question;  see below.

> In the end, I think this is just a very tough area to apply
> 'scientific' testing to.  We can't do double blind experiments
> in most cases.  No placebos to put out there.  No really
> 'neutral' presenters.  Way, way too many variables.
....
> 'research' here is unlikely to be very meaningful, so in the
> end, I'd look to personal testimonials of people you trust.

Let's not let the perfect be the enemy of the good.  Let's
focus on what's possible.

In particular, one could use standard survey techniques to
test, with fair accuracy, the hypothesis that SR systems
tend to fall into disuse after an initial flurry of
excitement.

Outline of method:  send questionnaires to people at N
different institutions, asking
  *) how many classrooms on campus, total
  *) how many hours per week is each classroom used
  *) how many classrooms have had SRS installed
  *) pick the SRS installation nearest your office and apply
   the following questions to it:
   -- what year was this SRS installed
   -- what make & model
   -- what year was it most-recently used
   -- how many hours per week was this room used in
    the fall '04 semester, total
   -- how many hours per week was this SRS used in
    the fall '04 semester
  *) etc. etc. ... you get the idea

The key here is to design objective questions, i.e.
questions that don't involve an opinion.  This greatly
reduces the need for blinding.  You can survey multiple
observers at each institution, to detect the occasional
grossly unreliable respondent.  Do whatever you have to
do to get a high response rate, so the results are not
biased by whatever factors cause some people to respond
or not respond.  Blah blah blah etc. etc.  There is a
big literature on how to conduct surveys, with reasonable
consensus on what to do and not do.

We agree this isn't like measuring the fine-structure
constant to one part in 10^13, but it's not hopelessly
unscientific, either.   It won't tell you everything
you might want to know, but it will tell you a few
things ... perhaps enough to sway a purchase decision,
and/or perhaps enough to guide the design of a follow-up
investigation.

In general, some information is better than no information.