ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************

We debated the value of evaluation several times.  But this has been a
good thread, and I have a few thoughts that might be useful.

ON CHARLIE'S CHALLENGE:  Charlie asked if anyone could "identify one
single exhibit or exhibition that has greatly benefited from the tender
hands of evaluation.  And by greatly, I mean been changed from flop  to
star."  I described the development of 4 exhibitions using evaluation,
from a couple of decades of my own experience, in the article "Visitor
Studies:  Convincing the Director," which can be read for free at
www.friedmanconsults.com/publications. Two of the examples fully meet
Charlie's challenge:  they would have been total flops as originally
designed and built, confusing visitors and displeasing them as
demonstrated by formative evaluation.  Using evaluation results the
developers turned them into exhibitions which have run for decades and
been replicated by other science centers who have been happy they did.
Summative evaluation showed that people enjoyed the final results and
actually learned important things from these exhibitions.  In the third
example, the exhibit was dramatically improved in terms of visitors'
understanding what it was about, and increased visitors' pleasure at using
it, although I can't say if that made it a "star" or not.  In the fourth
example, the original idea was eventually scrapped because formative
evaluation clearly demonstrated it would be a flop, and we never did
figure out a way to make it work any better.  I think that's a valuable
outcome, because it saved us from installing an expensive, permanent
exhibit that wouldn't have made anything clearer or been much fun for
visitors.

I can also describe much more recent exhibitions that have been similarly
improved by front-end, formative, and remedial evaluation, and whose
effectiveness as demonstrated by summative evaluation resulted in
replications and more grants.  I haven't written those up yet, but am
happy to share off-line.  Martin Weiss and Eric Siegel worked on several
of these.

ON THE STEVE JOBS EXCEPTION:  Yes, Jobs created amazingly successful
products without the use of market testing or other forms of front-end or
formative evaluation.  He was unique at being able to do this consistently
over decades.  I spoke to someone who was a senior executive at Apple and
then at Intel, and who knows the high tech field very well.  She tells me
nobody else in the industry could do what Jobs could, and all the other
successful players use market testing (equivalent to what we call
front-end and formative evaluation) to vet ideas and refine products.
They all do summative evaluation, but solely in terms of profits
generated.  

Mr. Jobs was a genius.  I happily accept that geniuses may not need
evaluation.  But for those of us who are not geniuses, evaluation sure is
a great aid to doing what we do better.

ON EVALUATION AS A TOOL:  Tools are very effective in some hands, and not
so much in others'.  Wood chisels don't produce anything beautiful when I
attack a piece of wood, but in others' hands they create smooth, useful,
beautiful objects. And some kinds of wood just are not well suited for
this tool.  Same for evaluators, evaluations, and exhibitions, IMHO.

ON THE CONSEQUENCES FOR OUR FIELD OF NOT EMBRACING EVALUATION AND ALSO,
SEPARATELY, RESEARCH:  Have you noticed that the current funding for ISE
at NSF is about half of what it was at its peak?  And that a number of big
private foundations and corporations which used to be mainstays of science
center exhibitions no longer fund in this area?  There are multiple
reasons for these trends, but one frequently cited reason is that these
funders want some hard evidence that what we do really works, and works
better than other options whose proponents are also pounding on funders'
doors for grants.  I can testify to this personally, since I now serve on
the boards of several grant-making foundations, and I am having a hard
time getting some of them to support science center exhibitions.  Hard
data on impact really does sway funders, and I don't find that at all
unreasonable.  The ISE field has always had proponents of evaluation and
research, and their tribe is growing, but we've been reluctant for so long
that we now have a deficit we've got to overcome if we want funding to
return or even to continue.

Alan      
________________________________________
Alan J. Friedman, Ph.D.
Consultant for Museum Development and Science Communication
29 West 10th Street
New York, New York 10011 USA
T  +1 917 882-6671
E   [log in to unmask] <mailto:[log in to unmask]>
W www.FriedmanConsults.com <http://www.friedmanconsults.com/>
 
a member of The Museum Group
www.museumgroup.com <http://www.museumgroup.com/>

***********************************************************************
For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.

Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.

The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
http://www.lsoft.com/LISTSERV-powered.html.

To remove your e-mail address from the ISEN-ASTC-L list, send the
message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]