ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************

Jennie,

Yes, ideas of rigor evolve. Patton makes the point that the Gold 
Standard itself collapsed because of rigidity. In a nut-shell, when 
findings developed with strict control are "loosed" into the natural 
world with unlimited numbers of interacting variables which can effect 
dependent variables, then they don't hold up. They can also produce 
unexpected side-effects, some minor, but some major ones. The first of 
the references I posted below is a PowerPoint where Patton discusses 
these issues. It is a PowerPoint of a presentation he made. The other 
references are from the Zelik team.

Actually, Patton notes that drug testing goes through four phases (RCT's 
are Phase 4) and all four phases are needed for rigor (including the 
small sample uncontrolled correlational studies where effects are 
identified). The final phase involves RCTs. Actually, I was the one 
extending the argument that RCTs don't provide adequate rigor for drugs. 
I take a drug was put on the market without investigation of what 
happens when people stop taking it. It is safe to take (well, as far as 
anyone knows), but the withdrawal effects are quite substantial. There 
is no commercial impetus for investigation of withdrawal--actually, 
there are commercial biases that support NOT finding methods of for 
withdrawal. Patton does not criticize this area of drug trials. I do. I 
think the arguments for rigor posed by Zelig would have helped prevent a 
very daily effect of the application of science in my own life. :-)

If you will look on p. 84 of the first reference I posted you will find 
the rigor attribute model. This alternative model does not throw out 
traditional ideas of rigor. But remember when those ideas initially 
evolved, the wide-spread application of all sorts of research findings 
not part of the cultural context. Philosophers of science were making 
claims that the purpose of science is to produce knowledge and practical 
application should not be considered in judging quality or value. 
Culturally, in that era and in this one, this claim provides the freedom 
for exploration and scientific autonomy that researchers need. On the 
other hand, this is a claim can lead to ideas of rigor that are 
inadequate for the natural world contexts.

I really don't want to dismiss the implicit criticism of the earlier 
(and ongoing) remarks from Charlie and Eric about evaluation. But, I 
believe that the appropriate response is not, as they suggest, to stop 
doing evaluation but to get a better concept of rigor and use it-- to 
really look at our own work.  Those of us who do social science really 
need to raise questions when methods developed in physics and chemistry 
and medicine are imposed on us. These are higher prestige areas of 
science, and this itself is a cultural and historic artifact.  
Certainly, scientific research in these areas have salient areas of 
application, particularly for warfare between nation states and in 
industry. Yet the methods which developed authentically from social 
science (including testing multiple-hypotheses against the evidence and  
triangulation as ways to reach conclusions) may actually be more 
rigorous and provide stronger and more reliable evidence for decisions.

Jennie, I hope that clarifies my statements. A listserv is a difficult 
place to discuss rigor. But, the topic came up and a discussion of it 
requires some real thought and consideration. It is NOT a closed 
question and the concept does continue to evolve. I think this rigor 
attribute model is somewhat difficult to grasp--maybe that is a 
disadvantage of it. I am not advocating adopting it, but discussing it 
as an alternative and using the concepts that help us do better and more 
useful work.

Best regards,

Carey

Carey Tisdal
Tisdal Consulting
[log in to unmask]





On 6/26/2012 6:28 PM, Jennie Dusheck wrote:
> ISEN-ASTC-L is a service of the Association of Science-Technology Centers
> Incorporated, a worldwide network of science museums and related institutions.
> *****************************************************************************
>
> This is a fascinating post, but I'm not sure I understood all of it--in particular the idea that rigor is an idea that is evolving. Carey, are you saying that applied science needs different standards of rigor than other science? Could you give an example of what criteria would be more appropriate to, say, a clinical drug treatment trial than a pre-clinical one?
>
> Thanks,
> Jennie Dusheck
> Science Writing & Editing
> Santa Cruz, CA 95060
>
> On Jun 25, 2012, at 2:23 PM, Carey Tisdal wrote:
>
>> ISEN-ASTC-L is a service of the Association of Science-Technology Centers
>> Incorporated, a worldwide network of science museums and related institutions.
>> *****************************************************************************
>>
>> I think we also need to consider the societal context of these critiques of social science. With overall federal  funding cuts there is lobbying from the physics, engineering, and medical areas which have more political clout to maintain funding in the areas that support their own economic interests. Science as a societal endeavor has always been a value-laden and political process. It isn't surprising that these types of critiques come out at this time when scarcity has been created in the economic system. The prestige hierarchy of university sciences is also well-identified phenomena and when conflated with corporate interests, it is not an inconsequential factor in producing these critiques.
>>
>> On the other hand, rigor is a topic that continues to evolve. Particularly in all applied areas, the traditional criteria for rigor (including repeatability/replication) are being called into question across multiple areas of science. Engineering and drug research have some ongoing issues, both with investigator independence (sponsors influencing results) and the initial focus and design blinding researchers from identifying essential questions.  For example, random control designs may be considered highly rigorous, but the length of the studies (influenced both by patient need and commercial interests)  prevent the identification of serious, sometimes even fatal, side-effects.
>>
>> Michael Quinn Patton recently recommended that evaluators look at the work of Zelik and his collaborators for better criteria for rigor. Zelik (et. al) looked at several situations (e.g. the Challenger Incident) where  where decisions, based of evidence, failed and have recommended criteria for rigor appropriate to applied situation. I think we need to consider this framework for rigor in informal science education evaluation.
> ***********************************************************************
> For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.
>
> Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.
>
> The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
> http://www.lsoft.com/LISTSERV-powered.html.
>
> To remove your e-mail address from the ISEN-ASTC-L list, send the
> message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
> [log in to unmask]
>

***********************************************************************
For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.

Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.

The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
http://www.lsoft.com/LISTSERV-powered.html.

To remove your e-mail address from the ISEN-ASTC-L list, send the
message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]