ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************

From:	Phil Edgerton [mailto:[log in to unmask]]
Sent:	Tuesday, November 21, 2006 1:58 PM
To:	'Bill Watson'
Subject:	RE: evaluations


Hi Bill,
Next time I'll be more forthright! (Ha!). I was being ironic when I called
open responses anecdotal. I actually feel that anecdotal responses have the
most validity in really getting to the crux of the visitors' feelings about
an exhibit. It boggles the mind to think that multiple [non]choice questions
are used because other types are "more time consuming to analyze."  One
wonders what the actual goal of the evaluation must be, if the methodology
is more important than the result. That's been my beef all along. Of course
that's probably because I'm a "get 'er done" kind of a guy rather than an
academic.
My best,
Phil

-----Original Message-----
From:	Bill Watson [mailto:[log in to unmask]]
Sent:	Tuesday, November 21, 2006 10:49 AM
To:	[log in to unmask]; [log in to unmask]
Subject:	RE: evaluations

Gee, Phil, why don't you tell us how you really feel? :)
I'm sure you're aware of this, but I think it's worth reiterating that many
(certainly not all - maybe not even most) multiple choice (or other selected
response) evaluation instruments are the result of very thoughtful
consideration of what visitors/participants might say or do, based on what
other visitors have said or done in the past, as well as pilot testing with
potential audiences. On the other hand, the kinds of questions you prefer,
when done right, really can give the evaluators extremely rich and telling
data. However, those, too, have their challenges: They are difficult to
write in such a way as to really get at the essence of the data that are
desired, and they are more time-consuming to analyze than multiple choice
instruments. They also potentially take more time to complete, so there is
always a danger that you lose some of the richness because people just want
to be done with it.
I do want to clarify something from your email. It sounded like you consider
"essay" questions to be "anecdotal" evidence/data. If the questions are
designed - and answers are collected and analyzed - in a systematic way,
then many consider those data to be as valid (or more valid) than data
collected in a multiple choice format. It's one thing to just ask people
their opinion and report what they say (that's more anecdotal). It's another
thing completely to analyze the responses for trends in light of all the
other answers that were collected. The latter, of course, takes
significantly more time - time that many evaluations/evaluators just don't
have. But if the time is available, I think those responses can be far more
than "anecdotal" evidence of the phenomenon you are studying.
Bill
-----Original Message-----
From:	Informal Science Education Network
[mailto:[log in to unmask]]On Behalf Of Phil Edgerton
Sent:	Tuesday, November 21, 2006 8:51 AM
To:	[log in to unmask]
Subject:	evaluations


ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related
institutions.
****************************************************************************
*

Any evaluation should have a response in each question: "None of the above."
That's the one I would almost always check. I quit participating in anyone's
evaluation years ago. Either the evaluations I was exposed to were skewed to
give the evaluators what they wanted to hear, or they never gave me the
chance to chose a response that was anywhere near what I really felt. The
desire to quantify all human experience is leading us far astray from valid
information. If you really want to know what I feel about something, ask me
"essay questions" instead of "multiple [lack of] choice" questions, or even
interview me. That's the only way you would get anything really useful as a
response. But of course, you'd call it (shudder) "anecdotal", then.
Phil Edgerton

***********************************************************************
More information about the Informal Science Education Network and the
Association of Science-Technology Centers may be found at
http://www.astc.org.
To remove your e-mail address from the ISEN-ASTC-L list, send the message
SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]

***********************************************************************
More information about the Informal Science Education Network and the
Association of Science-Technology Centers may be found at http://www.astc.org.
To remove your e-mail address from the ISEN-ASTC-L list, send the
message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]