ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************
I don't disagree with David Taylor's comment about the lack of evidence showing long-term cognitive change from short-term science center experiences. But, that doesn't mean you can't engage in meaningful evaluation. The first step is to be brutally honest in determining what it is that you want to measure. Often our program goals and aspirations are so lofty that it's downright frightening to try to measure real change. But we have to be honest with ourselves in what change we can really expect to happen as a result of a program and then design an evaluation accordingly.
While setting realistic evaluation goals is the first step, the last step is just as important. That is to be careful how you interpret the data and how you state the findings. A recent evaluation of one of COSI's outreach programs showed statistically significant difference between how a sample of students answered "knowledge" and "attitude" questions on a post-program survey from their pre-program responses, but we are careful not to suggest a long-term impact. We simply don't have the resources to do a long-term follow up, and we won't say that the program achieved a long-term result. We can simply say that the students' knowledge and attitude improved from immediately before the program to immediately after the program because that's all we measured.
A few practical notes about the two studies we've done on our outreach programs. First, we had a little funding to hire a professional evaluation consultant, and an outside eye is very helpful in taking that honest look at measurable goals. As I mentioned already, we used a pre/post test method in surveying students on a series of knowledge and attitude questions. We took advantage of the structure of the formal classroom to have the teachers administer the surveys for us. Two caveats here* One is that it is very difficult for students and teachers to see the surveys as something other than "tests for a grade" and there's a good deal of anxiety about the students not being able to answer the questions, especially on the pre-program test. It's hard for them to see this as something other than student assessment, so you need to do a lot of messaging about how these tests are different from school tests. Secondly, it took much more coaxing, reminding, and general handholding of the teachers than I ever expected to get this done. Teachers are so overwhelmed in today's classrooms that this simple study was burdensome to some. I had better luck when I got the support of the principal first, and used a lot of patience in developing a continuing rapport with the teachers. And remember, you don't want to keep going back to the same teachers that you already know well. A more representative sample will include teachers who are not necessarily "friends" of the science center.
Good luck! 'Would be happy to discuss in more detail off-line.
Rita Deedrick
Senior Director for Program Evaluation
COSI Columbus
Return to the Days of the Dinosaur!
Dinosaur Dynasty: Discoveries from China at COSI Columbus
May 8 - September 6, 2004
GroupWise powered by Time-Warner Road Runner
>>> [log in to unmask] 05/27/04 09:07PM >>>
ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************
Aloha,
I am trying to establish an evaluation procedure to assess the extent to which our outreach programs accomplish our stated goals, specifically in terms of student learning. Does anyone have any advice or recommendations about how to get a true picture of the impact our program had on students? We are interested in both what content students learned and how their attitudes changed because of our program. This is not intended to evaluate how well the students or teacher liked the program. I am mainly considering some type of pre/post test that teachers conduct and return to us, but if anyone has other ideas I'd be happy to hear them. Also, if anyone has evaluation samples they are willing to share, that would be great.
Thanks,
Nancy
****************
Nancy Alima Ali
Science Educator
Bishop Museum
1525 Bernice Street
Honolulu, HI 96817
Phone:(808) 847-8227
Fax: (808) 848-4155
***********************************************************************
More information about the Informal Science Education Network and the
Association of Science-Technology Centers may be found at http://www.astc.org.
To remove your e-mail address from the ISEN-ASTC-L list, send the
message SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]
***********************************************************************
More information about the Informal Science Education Network and the
Association of Science-Technology Centers may be found at http://www.astc.org.
To remove your e-mail address from the ISEN-ASTC-L list, send the
message SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]
|
