ISEN-ASTC-L is a service of the Association of Science-Technology Centers
Incorporated, a worldwide network of science museums and related institutions.
*****************************************************************************


Okay let me try this one again!


Hi Dennis,

Thanks, what a great discussion.  Isn't this why the researchers suggested that comparisons needs to reflect SES comparisons across countries and not simply lumping of results by country? 

Agreed the aggregated test results only provide an integration of total learning (both formal and informal), and this is why one looks a learned skills like reading and math rather than questions like how many legs does a spider have?  It's not so important to know how many legs a spider has, but whether or not one can read the question and can evaluate it critically.  How does one determine how many legs a spider has or spiders have? might be a more appropriate question.

Interestingly, the detected decline observed at normally advantaged students is not explained by apparent lack of exposure or access to formal or informal education.  It certainly could reflect social trends beyond the realm of formal.

Now I haven't seen a PISA test, and am operating off the logical aspects of testing, so the above is a bit of speculation, but I assume it has been well considered over the years.

C
On Mar 21, 2013, at 5:46 PM, Dennis Bartels <[log in to unmask]> wrote:

>> Hi Charlie,
>> 
>> My bigger point is that no external-reference exam can ever measure anything but SES by definition.  That's to say, if the test is constructed by others outside of the direct learning environment, e.g., classroom, there is no way to verify that the content on the questions was ever taught in the first place.  Therefore it is measuring something else.  This problem is only exacerbated with international tests.  Only those exams that are directly tied to the taught curriculum, such as the NY Regents end of course exams or the AP tests come to mind--as the test items come 100 percent from the prescribed curriculum and presumably all students had the same opportunities to learn the same material.  If you are being tested on material you never were taught in school, and you get the question right, you either guessed right OR were exposed to the material someplace else, e.g, an out of school environment.  Perhaps you learned it from your parents, or from TV, or from something you read at your leisure, or from the internet, etc you get the point.  So these tests often measure better your outside exposure and experiences which do covary by SES considerably.  That's why I like the "how many legs does a spider have" example.  If you were never taught it in school, and you get the question right, how do you suppose you knew the answer?  There's good evidence elsewhere that the variation isn't as strongly coordinated to even factors as health, nutrition or safety as they are other factors in kids backgrounds, very often the education level of their immediate family.  Which, of course is highly correlated with SES!  Not teachers or schools. 
>> 
>> So the point being that no external reference exam--no matter how well constructed--can rid itself of an SES bias, nor is a good proxy of true learning, let alone teacher "effectiveness."  If you really want to measure learning, the instrument has to be constructed around and tied directly to the course or set of learning experiences actually engaged by the learners.  When this is true, most SES differences decrease dramatically.
>> 
>> 
>> On Thu, Mar 21, 2013 at 2:23 PM, Charles Carlson <[log in to unmask]> wrote:
>> Dennis,
>> Thanks! That's exactly right.  This study detected stratification of test results because of SES.  They statistically demonstrate test result variability as a result of SES,an then some divergence from this finding at wealthier levels of SES for unexplained reasons. 
>> 
>> Optimistically, test results vary and everybody given the right opportunities and freedom from hunger and want learns.  Depressingly, PISA, which I believe was formulated to account for this type of correlation, is shown to be subject to similar biases, which is a point of their comprehensive study.
>> 
>> Sent from Charlie Carlson's iPad
>> [log in to unmask]
>> Cell 510-499-8086
>> Try me on Skype (510) 984-3543
>>> 
>>> On Mar 21, 2013, at 1:43 PM, Dennis Bartels <[log in to unmask]> wrote:
>>> 
>>>> Hi Charlie,
>>>> 
>>>> In fact the findings and research is even more insidious and nuanced.  There is a great deal of research on the subject going back 50 years, and exemplified by the great debates in the 1960s with the New Deal in a series of government studies done by Dr. Jim Coleman (i.e., the Coleman Studies) which essentially established the strongest link between test scores and any other factor, including teachers and schools, was SES.  It's been hotly debated every since.  Here is the insidious part, and then the nuanced part.  
>>>> 
>>>> The insidious part is for many years people treated these infamous studies as if demography is destiny.  In other words, that children from poorer backgrounds cannot learn as well or as fast as kids from more affluent backgrounds.  However, later cognitive research and empirical studies demonstrating many counter examples renewed our optimism and beliefs that indeed every human no matter background has similar learning potentials, excluding serious cognitive disorders.  So SES can and has been overcome many times over.  
>>>> 
>>>> Now the nuanced part.  Why the tight correlation between SES and standardized test scores (which I note here is NOT the same thing as learning or learning potential).  Subsequent analysis of standardized testing has revealed that external-referenced standardized tests (e.g., tests used that are not directly tied to the curriculum taught but created somewhere outside of schools) always have a SES bias, including PISA.  Why?  Well, if the tests are not curriculum dependent (say for sake of contrast like end of course exams or AP exams, which are highly curriculum dependent), who decides what a 3rd grader should know?  
>>>> 
>>>> Why some psychometrician somewhere, of course!  And if the tests are not tied to classroom learning, but external references, such as how many legs does a spider have--if I was never taught that in school, who has a better chance of answering that question correctly, a affluent kids with access to all kinds of media and TV, or a poor kid who might not even have a TV or been to a summer camp or Exploratorium?  
>>>> 
>>>> Of course we all know the answer to that question.  So ALL that standardized tests (read non-curriculum or non-school dependent tests) measure is a perfect proxy FOR SES!  Ta-da!
>>>> 
>>>> It's amazingly circular and I'm stunned more people don't know about this consistent result and that researchers still feel a need to verify it!
>>>> 
>>>> The more important question is why anyone feels like these standardized tests are useful measures of anything BUT SES.
>>>> 
>>>> Dennis
>>>> 
>>>> PS:  If you really want a scandal, read a well-written New Yorker article a few years back on Stanley Kaplan and the founding of the Kaplan Studies program back in the early 20th century.  That will really make you stomach turn!
>>>> 
>>>> 
>>>> 
>>>> Dennis M. Bartels, Ph.D.
>>>> Executive Director
>>>> Piers 15/17
>>>> San Francisco, CA 94111
>>>> [log in to unmask]
>>>> 415-528-4326
> 

Charles Carlson
Senior Scientist | Teacher Institute

http://blogs.exploratorium.edu/whyintercept/
Twitter: @charliec53
email: [log in to unmask] 
Tel:   415-528-4319
Fax:  415-885-6011
exploratorium.edu
facebook.com/exploratorium twitter.com/exploratorium
The Embarcadero, Piers15 & 17
S.F., CA 94111





***********************************************************************
For information about the Association of Science-Technology Centers and the Informal Science Education Network please visit www.astc.org.

Check out the latest case studies and reviews on ExhibitFiles at www.exhibitfiles.org.

The ISEN-ASTC-L email list is powered by LISTSERVR software from L-Soft. To learn more, visit
http://www.lsoft.com/LISTSERV-powered.html.

To remove your e-mail address from the ISEN-ASTC-L list, send the
message  SIGNOFF ISEN-ASTC-L in the BODY of a message to
[log in to unmask]