ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************
I am posting this for a colleague - Kate Haley Goldman - in response to the discussion about formative evaluation, its relationship to usability, and sample size (and in particular Jakob Nielsen's Alertbox article). . .
Applying Nielsen's work in heuristic evaluation of web sites to museums is an interesting question. His work on a variety of subjects has been seminal in the field, and is certainly worth reading. He argues for a very small n, basically on the idea that even a few users can make suggestions that lead to improvements. Although that is certainly true, there are some other cautions.
First, Nielsen's work in the Human Computer Interaction world is controversial. (See the humanfactors.com site, especially <http://www.humanfactors.com/downloads/sep012.htm> http://www.humanfactors.com/downloads/sep012.htm) There are both reliability and validity issues to be addressed, as per Desurvire, H. (1994). Faster, Cheaper!! Are Usability Inspection Methods as Effective as Empirical Testing? which raises serious concerns about the work, such as missing significant numbers of critical errors.
Second, I'm concerned with the notion that usability can replace evaluation. Lynn Dierking and I discuss this in an upcoming book chapter (see reference below) -- it is akin to concentrating solely on whether the door of your science center is unlocked, without paying attention to how visitors are using the resource, whether it could be improved, and whether they are having enjoyable and meaningful experiences.
I have a paper on this at: <http://www.archimuse.com/mw2003/papers/haley/haley.html> http://www.archimuse.com/mw2003/papers/haley/haley.html
All the best--
Kate
[The usability versus evaluation issue will be discussed in: Haley Goldman, K. and L.D. Dierking (in press) Free-Choice Learning Research and the Virtual Science Center: Establishing a Research Agenda. In R. Subramaniam & L.T.W. Hin (Eds.) E-learning and Virtual Science Centers. Idea Group Publishing. Hershey, USA.]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Kate Haley Goldman
Institute for Learning Innovation
166 West Street
Annapolis, MD 21401
[log in to unmask]
410-268-5149
*******************************************
-----Mensaje original-----
De: Eric Gyllenhaal [mailto:[log in to unmask]]
Enviado el: Sábado, 04 de Septiembre de 2004 06:35 AM
Para: [log in to unmask]
Asunto: Subject: Formative evaluation
****************************************************************************
Regarding sample size for formative evaluation, here's an interesting argument for from the field of Web usabiliy testing:
Jakob Nielsen's Alertbox, March 19, 2000:
Why You Only Need to Test With 5 Users http://useit.com/alertbox/20000319.html
"Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford."
The explanation depends on viewing usability/formative testing as discovering uasability problems and includes a mathematical formula that demonstrates the deminishing returns/decreasing numbers of problems discovered as the number of user tests increases.
Eric Gyllenhaal
Selinda Research Associates
> ------------------------------
> Date: Fri, 3 Sep 2004 10:09:43 -0500
> From: =?iso-8859-1?Q?P=E9rez_Castellanos_Leticia?=
> <[log in to unmask]>
> Subject: Formative evaluation
>
> ISEN-ASTC-L is a service of the Association of Science-Technology
> Centers Incorporated, a worldwide network of science museums and
> related institutions.
> **********************************************************************
>
> I'm currently working in a new interactive museum in Mexico City, we =
> just started the process of evaluate some exhibits in the design and =
> production phase (formative). I'm having problems to explain why this
> = kind of evaluation does not need large statistical samples, great
> part = of the team thinks that we need to have representatives samples
> and, I = read in some classical texts (Screven, Griggs, Hein) that
> this is not = necessary nor practical.
>
> Does any one have a good methodological frame to pick good samples for
> = this kind of evaluation? Or does any one could recommend me the =
> "Formative evaluation handbook"? I prefer things I can consult through
> = web because in Mexico is difficult to get those books, although we
> can = buy some basic and interesting texts.
>
> Thanks in advance
>
> Leticia Prez Castellanos
> Research and Evalaution Area
> Espacio Cultural y Educativo Betlemitas
***********************************************************************
More information about the Informal Science Education Network and the
Association of Science-Technology Centers may be found at http://www.astc.org.
To remove your e-mail address from the ISEN-ASTC-L list, send the
message SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]
|
