ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************

Not the right question, generally speaking.

So yes, I did put out the challenge.  I thank Alan for responding so fully and specifically.  His essay is a useful read.  And while I agree that formative assessment and evaluation has lead to exhibit modification and "improvement."  I remain un-convinced that evaluator assessment and suggestion would have necessarily changed the end product. I am not writing this to dismiss the evaluative process nor people engaged in it.  It's tough stuff, generally.  These are very noisy experiments with typically small N's.  The studies described by Alan, are very useful, good and likely impactful on particular exhibits, but do not in themselves constitute proof.  The process is expensive and there are benefits but not guarantees.  

As a counter recent example, I cite Body Worlds and the various knock offs.  Immensely popular devoid of formal evaluation (front end, formative, and summative) as far as I know.  Money settled the question.  And as I  presented at ASTC many popularly visitor remembered exhibits at the Exploratorium have had no formal formative input.  So there is little correlation between between formative and summative.

Alan's cited examples are mostly poorly constructed experiments.  Again not to denigrate the field of evaluation and visitor study nor Alan.  We are all in this together. Human's are notoriously difficult experimental subjects, and this stuff is tough to do.  This is not to say that the effort wasn't worth it either.  I firmly advocate for a critical, communicative eye, be it in the hands of the exhibit creator or evaluator.  The variable evidence suggests that evaluation by a professional evaluator is not a necessary nor essential component however.  As much as some may want, for every example, a counter example can be cited.  Both are tools and talents.  But such exchanges and conversations do highlight and sharpen the focus on the role of museums writ large.

Dave Ucko pointed out that the NSF has invested approximately 1 billion dollars in museum exhibits over the past ten years (if memory serves me correctly), and that taxpayers (you and me included) deserve to know the social benefit.  (Again, let me stress, this is not an attack on a particular field of endeavor, nor museums.  I love them, find them valuable, and don't want a world without them.)  

However, in my view, we have engaged in intensive efforts on evaluation, and sought to determine aspects of visitor use and learning, but we have in a sense failed to see the forest for the trees.

The real question being asked by tax payers and other funders is what's the effect of all this effort?  We have answered with something like "The efforts we're making are good and we're making it better all the time, but we don't real have any idea what our broad effects have been."  In most respects, it doesn't really make a difference if people get a concept, or recite a fact immediately after a visit.  And I say this from personal experience, it's how we do and compete in the world at large on average?  These types of evaluations don't answer that.

In this area our efforts are problematic. Such effects might be answered by evidence of social change and broader cognitive improvements.  We have not freed the world from racism, made the sexes equal, removed poverty or likely changed the course of science, math and engineering education.  We have likely had some impacts, but they are by definition small as far as I can tell, but they may be bigger.

My point is that while we believe, we have not always convinced others, particularly the funders.  No amount of formative evaluation is likely to change that interpretation.  It's really about stuff like popularity, perceived benefits, improved test scores in school, how much did my child and I succeed.  These are directly measurable benefits and social effects.  In this broad social domain, our record's not so good.  There's lots of anecdotal evidence, but quantifiable valuation is much harder.

We know that many people vote with their feet.  Museum attendance is unquestionably great.  Folks visit and they visit to learn.  They resoundingly say this in every marketing survey we've conducted at the Exploratorium.

Ironically, in the United States, we have high museum attendance, but year after year the educational STEM edge diminishes.   Admittedly , it's hard to tease these effects from the broader social context.  This makes it hard to argue convincing that museums have made a difference.

My point here is not the absolute specifics, but let me point out that there are opportunities for probing and answering these broader questions.  For example, what happens to the students in an museum attendance area when exhibitions change and exhibits change?  What does changing a location or revamping a museum do to local participation in schools?  What happens to a school district's students when a museum comes or goes?  What happens to local public science, math, engineering literacy when a science museum emerges or recedes?  What's the stuff that makes popular science exhibitions popular?  How do exhibits tie to important socio-cultural aspects of human existence to visitors?  What makes exhibits popular?  There are many questions to ask in this domain.  They're not easily answered however.  And I'm sure my list of questions is inadequate, but let us not quibble about such differences.

Can we make exhibits and exhibitions better attended and used, through a series of scientifically driven and controlled processes?  I don't know, but I think it worth an experiment.

I would suggest as a thought experiment with some potential for an actual experiment that it might be useful to firmly establish baseline effects on something that's about to change and compare pre and post.  I absolutely know that an institution with a vested interest in the outcome is not the one to run the study.

I absolutely know that it's difficult to get an objective answer from anyone with a vested interest in the experimental outcomes.  That's human nature.

I absolutely know that there are many studies that could be conducted but they need be separated from vested involvements.

These may be high hurdles, and we may not appreciate the answers, but they would be valuable.   And until we do them, what we do is a matter of speculation.

I am more than happy to rest with the statement that visitors vote with their feet.  I remain unconvinced that we have learn much beyond that.  And would observe that given the vagaries of humans and all their social constructs, museums constitute poorly  controllable environments. Museums in fact may not be go places in which to study learning behaviors.

We would need to do large longitudinal studies to get the answers we desire, all this other stuff is basically noise.  

That's my take, from my personal corner of the world.  Lots of other stuff is about how to make a living.  An essential thing, but it colors the situation and inherently biases the data.

Resolving the real questions will take a series of real experiments.

C

On Oct 29, 2011, at 10:54 AM, Alan Friedman wrote:

> ISEN-ASTC-L is a service of the Association of Science-Technology Centers
> Incorporated, a worldwide network of science museums and related institutions.
> *****************************************************************************
> 
> We debated the value of evaluation several times.  But this has been a
> good thread, and I have a few thoughts that might be useful.
> 
> ON CHARLIE'S CHALLENGE:  Charlie asked if anyone could "identify one
> single exhibit or exhibition that has greatly benefited from the tender
> hands of evaluation.  And by greatly, I mean been changed from flop  to
> star."  I described the development of 4 exhibitions using evaluation,
> from a couple of decades of my own experience, in the article "Visitor
> Studies:  Convincing the Director," which can be read for free at
> www.friedmanconsults.com/publications. Two of the examples fully meet
> Charlie's challenge:  they would have been total flops as originally….

Charles Carlson
Senior Scientist
exploratorium
3601 Lyon St.
San Francisco, CA 94123
[log in to unmask]
Tel:   415-561-0319
Fax:  415-561-0370

Skype: sciskypecharlie
MobileMe: [log in to unmask]
Twitter: charliec53
http://blogs.exploratorium.edu/whyintercept/













***********************************************************************
For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.

Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.

The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
http://www.lsoft.com/LISTSERV-powered.html.

To remove your e-mail address from the ISEN-ASTC-L list, send the
message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]