ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************

Hi Carey,
Thanks for taking the time to discuss this. It's all new to me.  I take "RTC" to mean a randomized, controlled trial. This is the first I have heard of a "Gold Standard" in experimental design. It suggests to me a rather narrow idea of experimental design, a narrowness that surprises me. My education was in biology (so I understand your point about hierarchies within science). I think of rigor as first an attitude of ruthlessness towards data analysis and doggedness in its collection and second a flexibility in thinking about the logic of the argument. I would not have selected one specific method as the "right" way to go about science. That would be like saying that the sonnet is the only true poem.

Instead, I think, the idea is to keep thinking up new ways to disprove something or at least to separate it from other explanations for the observations. Of course, if your goal is to legitimize an idea (or a pharmaceutical), then you are going to be only as rigorous as you are required to be. Also, if you are always testing similar ideas in similar situations, I guess it would be easy to settle on a specific methodology for doing that--as the pharmaceutical industry largely has. And perhaps evaluation?

I know really almost nothing about the evaluation question, though I can see it's a fraught area. What little I know is mostly from reading this list for the last year, but it seems to me that a great deal depends on what a given exhibit is intended to accomplish. It's not clear to me that informal science education should be required to have learning objectives in the way that formal education does. Do art exhibits have learning objectives? (This is not a rhetorical question!) 

Jennie "going out on a limb" Dusheck
Science Writing & Editing
Santa Cruz, CA
[log in to unmask]

On Jun 27, 2012, at 8:03 AM, Carey Tisdal wrote:

> Jennie,
> 
> Yes, ideas of rigor evolve. Patton makes the point that the Gold Standard itself collapsed because of rigidity. In a nut-shell, when findings developed with strict control are "loosed" into the natural world with unlimited numbers of interacting variables which can effect dependent variables, then they don't hold up. They can also produce unexpected side-effects, some minor, but some major ones. The first of the references I posted below is a PowerPoint where Patton discusses these issues. It is a PowerPoint of a presentation he made. The other references are from the Zelik team.
> 
> Actually, Patton notes that drug testing goes through four phases (RCT's are Phase 4) and all four phases are needed for rigor (including the small sample uncontrolled correlational studies where effects are identified). The final phase involves RCTs. Actually, I was the one extending the argument that RCTs don't provide adequate rigor for drugs. I take a drug was put on the market without investigation of what happens when people stop taking it. It is safe to take (well, as far as anyone knows), but the withdrawal effects are quite substantial. There is no commercial impetus for investigation of withdrawal--actually, there are commercial biases that support NOT finding methods of for withdrawal. Patton does not criticize this area of drug trials. I do. I think the arguments for rigor posed by Zelig would have helped prevent a very daily effect of the application of science in my own life. :-)
> 
> If you will look on p. 84 of the first reference I posted you will find the rigor attribute model. This alternative model does not throw out traditional ideas of rigor. But remember when those ideas initially evolved, the wide-spread application of all sorts of research findings not part of the cultural context. Philosophers of science were making claims that the purpose of science is to produce knowledge and practical application should not be considered in judging quality or value. Culturally, in that era and in this one, this claim provides the freedom for exploration and scientific autonomy that researchers need. On the other hand, this is a claim can lead to ideas of rigor that are inadequate for the natural world contexts.
> 
> I really don't want to dismiss the implicit criticism of the earlier (and ongoing) remarks from Charlie and Eric about evaluation. But, I believe that the appropriate response is not, as they suggest, to stop doing evaluation but to get a better concept of rigor and use it-- to really look at our own work.  Those of us who do social science really need to raise questions when methods developed in physics and chemistry and medicine are imposed on us. These are higher prestige areas of science, and this itself is a cultural and historic artifact.  Certainly, scientific research in these areas have salient areas of application, particularly for warfare between nation states and in industry. Yet the methods which developed authentically from social science (including testing multiple-hypotheses against the evidence and  triangulation as ways to reach conclusions) may actually be more rigorous and provide stronger and more reliable evidence for decisions.
> 
> Jennie, I hope that clarifies my statements. A listserv is a difficult place to discuss rigor. But, the topic came up and a discussion of it requires some real thought and consideration. It is NOT a closed question and the concept does continue to evolve. I think this rigor attribute model is somewhat difficult to grasp--maybe that is a disadvantage of it. I am not advocating adopting it, but discussing it as an alternative and using the concepts that help us do better and more useful work.
> 
> Best regards,
> 
> Carey
> 
> Carey Tisdal
> Tisdal Consulting
> [log in to unmask]

***********************************************************************
For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.

Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.

The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
http://www.lsoft.com/LISTSERV-powered.html.

To remove your e-mail address from the ISEN-ASTC-L list, send the
message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]