ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************
Jennie,
Yes, ideas of rigor evolve. Patton makes the point that the Gold
Standard itself collapsed because of rigidity. In a nut-shell, when
findings developed with strict control are "loosed" into the natural
world with unlimited numbers of interacting variables which can effect
dependent variables, then they don't hold up. They can also produce
unexpected side-effects, some minor, but some major ones. The first of
the references I posted below is a PowerPoint where Patton discusses
these issues. It is a PowerPoint of a presentation he made. The other
references are from the Zelik team.
Actually, Patton notes that drug testing goes through four phases (RCT's
are Phase 4) and all four phases are needed for rigor (including the
small sample uncontrolled correlational studies where effects are
identified). The final phase involves RCTs. Actually, I was the one
extending the argument that RCTs don't provide adequate rigor for drugs.
I take a drug was put on the market without investigation of what
happens when people stop taking it. It is safe to take (well, as far as
anyone knows), but the withdrawal effects are quite substantial. There
is no commercial impetus for investigation of withdrawal--actually,
there are commercial biases that support NOT finding methods of for
withdrawal. Patton does not criticize this area of drug trials. I do. I
think the arguments for rigor posed by Zelig would have helped prevent a
very daily effect of the application of science in my own life. :-)
If you will look on p. 84 of the first reference I posted you will find
the rigor attribute model. This alternative model does not throw out
traditional ideas of rigor. But remember when those ideas initially
evolved, the wide-spread application of all sorts of research findings
not part of the cultural context. Philosophers of science were making
claims that the purpose of science is to produce knowledge and practical
application should not be considered in judging quality or value.
Culturally, in that era and in this one, this claim provides the freedom
for exploration and scientific autonomy that researchers need. On the
other hand, this is a claim can lead to ideas of rigor that are
inadequate for the natural world contexts.
I really don't want to dismiss the implicit criticism of the earlier
(and ongoing) remarks from Charlie and Eric about evaluation. But, I
believe that the appropriate response is not, as they suggest, to stop
doing evaluation but to get a better concept of rigor and use it-- to
really look at our own work. Those of us who do social science really
need to raise questions when methods developed in physics and chemistry
and medicine are imposed on us. These are higher prestige areas of
science, and this itself is a cultural and historic artifact.
Certainly, scientific research in these areas have salient areas of
application, particularly for warfare between nation states and in
industry. Yet the methods which developed authentically from social
science (including testing multiple-hypotheses against the evidence and
triangulation as ways to reach conclusions) may actually be more
rigorous and provide stronger and more reliable evidence for decisions.
Jennie, I hope that clarifies my statements. A listserv is a difficult
place to discuss rigor. But, the topic came up and a discussion of it
requires some real thought and consideration. It is NOT a closed
question and the concept does continue to evolve. I think this rigor
attribute model is somewhat difficult to grasp--maybe that is a
disadvantage of it. I am not advocating adopting it, but discussing it
as an alternative and using the concepts that help us do better and more
useful work.
Best regards,
Carey
Carey Tisdal
Tisdal Consulting
[log in to unmask]
On 6/26/2012 6:28 PM, Jennie Dusheck wrote:
> ISEN-ASTC-L is a service of the Association of Science-Technology Centers
> Incorporated, a worldwide network of science museums and related institutions.
> *****************************************************************************
>
> This is a fascinating post, but I'm not sure I understood all of it--in particular the idea that rigor is an idea that is evolving. Carey, are you saying that applied science needs different standards of rigor than other science? Could you give an example of what criteria would be more appropriate to, say, a clinical drug treatment trial than a pre-clinical one?
>
> Thanks,
> Jennie Dusheck
> Science Writing & Editing
> Santa Cruz, CA 95060
>
> On Jun 25, 2012, at 2:23 PM, Carey Tisdal wrote:
>
>> ISEN-ASTC-L is a service of the Association of Science-Technology Centers
>> Incorporated, a worldwide network of science museums and related institutions.
>> *****************************************************************************
>>
>> I think we also need to consider the societal context of these critiques of social science. With overall federal funding cuts there is lobbying from the physics, engineering, and medical areas which have more political clout to maintain funding in the areas that support their own economic interests. Science as a societal endeavor has always been a value-laden and political process. It isn't surprising that these types of critiques come out at this time when scarcity has been created in the economic system. The prestige hierarchy of university sciences is also well-identified phenomena and when conflated with corporate interests, it is not an inconsequential factor in producing these critiques.
>>
>> On the other hand, rigor is a topic that continues to evolve. Particularly in all applied areas, the traditional criteria for rigor (including repeatability/replication) are being called into question across multiple areas of science. Engineering and drug research have some ongoing issues, both with investigator independence (sponsors influencing results) and the initial focus and design blinding researchers from identifying essential questions. For example, random control designs may be considered highly rigorous, but the length of the studies (influenced both by patient need and commercial interests) prevent the identification of serious, sometimes even fatal, side-effects.
>>
>> Michael Quinn Patton recently recommended that evaluators look at the work of Zelik and his collaborators for better criteria for rigor. Zelik (et. al) looked at several situations (e.g. the Challenger Incident) where where decisions, based of evidence, failed and have recommended criteria for rigor appropriate to applied situation. I think we need to consider this framework for rigor in informal science education evaluation.
> ***********************************************************************
> For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.
>
> Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.
>
> The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
> http://www.lsoft.com/LISTSERV-powered.html.
>
> To remove your e-mail address from the ISEN-ASTC-L list, send the
> message SIGNOFF ISEN-ASTC-L in the BODY of a message to
> [log in to unmask]
>
***********************************************************************
For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.
Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.
The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
http://www.lsoft.com/LISTSERV-powered.html.
To remove your e-mail address from the ISEN-ASTC-L list, send the
message SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]
|