ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************

You can look at examples of PISA items here: http://www.oecd.org/pisa/38709385.pdf

The format suggests to me that you can find answers to these questions without having been taught them in school. The developers of PISA items try to ensure that they are not dependent upon a particular curriculum by providing sufficient context within the question to find the answer, so the answers do not depend on factual recall. 
Incidentally, the last spider I saw had three legs. It was having difficulty not walking in circles.

Best wishes, Leonie

-----Original Message-----
From: Informal Science Education Network [mailto:[log in to unmask]] On Behalf Of Charles Carlson
Sent: Friday, 22 March 2013 10:24 AM
To: [log in to unmask]
Subject: Re: Poor ranking on international test misleading about U.S. student performance, Stanford researcher finds

ISEN-ASTC-L is a service of the Association of Science-Technology Centers Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************


Okay let me try this one again!


Hi Dennis,

Thanks, what a great discussion.  Isn't this why the researchers suggested that comparisons needs to reflect SES comparisons across countries and not simply lumping of results by country? 

Agreed the aggregated test results only provide an integration of total learning (both formal and informal), and this is why one looks a learned skills like reading and math rather than questions like how many legs does a spider have?  It's not so important to know how many legs a spider has, but whether or not one can read the question and can evaluate it critically.  How does one determine how many legs a spider has or spiders have? might be a more appropriate question.

Interestingly, the detected decline observed at normally advantaged students is not explained by apparent lack of exposure or access to formal or informal education.  It certainly could reflect social trends beyond the realm of formal.

Now I haven't seen a PISA test, and am operating off the logical aspects of testing, so the above is a bit of speculation, but I assume it has been well considered over the years.

C
On Mar 21, 2013, at 5:46 PM, Dennis Bartels <[log in to unmask]> wrote:

>> Hi Charlie,
>> 
>> My bigger point is that no external-reference exam can ever measure anything but SES by definition.  That's to say, if the test is constructed by others outside of the direct learning environment, e.g., classroom, there is no way to verify that the content on the questions was ever taught in the first place.  Therefore it is measuring something else.  This problem is only exacerbated with international tests.  Only those exams that are directly tied to the taught curriculum, such as the NY Regents end of course exams or the AP tests come to mind--as the test items come 100 percent from the prescribed curriculum and presumably all students had the same opportunities to learn the same material.  If you are being tested on material you never were taught in school, and you get the question right, you either guessed right OR were exposed to the material someplace else, e.g, an out of school environment.  Perhaps you learned it from your parents, or from TV, or from something you read at your leisure, or from the internet, etc you get the point.  So these tests often measure better your outside exposure and experiences which do covary by SES considerably.  That's why I like the "how many legs does a spider have" example.  If you were never taught it in school, and you get the question right, how do you suppose you knew the answer?  There's good evidence elsewhere that the variation isn't as strongly coordinated to even factors as health, nutrition or safety as they are other factors in kids backgrounds, very often the education level of their immediate family.  Which, of course is highly correlated with SES!  Not teachers or schools. 
>> 
>> So the point being that no external reference exam--no matter how well constructed--can rid itself of an SES bias, nor is a good proxy of true learning, let alone teacher "effectiveness."  If you really want to measure learning, the instrument has to be constructed around and tied directly to the course or set of learning experiences actually engaged by the learners.  When this is true, most SES differences decrease dramatically.

***********************************************************************
For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.

Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.

The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
http://www.lsoft.com/LISTSERV-powered.html.

To remove your e-mail address from the ISEN-ASTC-L list, send the
message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]