Dear all:

While I can appreciate Michael Kramer's ability to understand statistics, he seems to have more expertise in associational studies than developing solid designs for larger-scale intervention trials.  I haven't thoroughly read the PROBIT study, but my impression is that it falls prey to many pitfalls that those who have only done associational studies or even clinical trials may not appreciate when they "scale-up".  

Scaling up is hard.  You get different results from a small scale clinical trial than when you do a large-scale intervention trial because there are many implementation issues that crop up.  To use a nonbreastfeeding example -- let's take charter schools.  A recent movie picked five small scale examples, one of which a few subway stops away of charter schools.  You can find a few select examples of charter schools that result in better scores than public schools.  These charter schools typically have far more resources and input from "celebrity" supporters than other schools.  As the Hawthorne studies showed, if you give workers enough attention, no matter if it merely changing the light bulbs, the attention will spur them on to work harder and more productively (at least initially).  Overall, however, the picture is mixed on charter schools with the AVERAGE comparison shows no difference.  No one has scaled them up yet.  Take for example, the Harlem Success Academy.  The teacher and administrator turnover rate is brutal.  You can get away with this in large metropolitan centers such as Manhattan and Chicago because you can dip into a large pool of talented professionals.  The principal of our son's school who did well at his school for two out of her three years, lasted less than 2 months at the Harlem Success Academy.  If, however, you are in a smaller area such as the one where I grew up -- less than 3,000 in the town I lived in and so the school had to accommodate four towns -- I went to school with the same 60 kids in two different classes for 8 years.  If you had fired teachers and principals at the same rate as the Harlem Success Academy, you would quickly run into a problem whereby you wouldn't be able to find anyone to teach or administer the school.  Moreover, if you employ these tactics in a large urban center and then want to expand, you may encounter political resistance.  The Harlem Success Academy's attempt to open up schools in our neighborhood is seen by many parents as a hostile takeover and political resistance is high. So already you encounter political resistance.  Furthermore as this school expands, will it continue to have the financial and celebrity backing to work on a larger scale?

In terms of evaluation, there are pitfalls that crop up because you cannot control large-scale interventions in the same way that you can in clinical trial.  One problem is that as you move from a clinical trial in which you control all the variables, you are measuring what in the epidemiogical training I received is called "efficacy".  When you have a large-scale intervention trial one component is "efficacy" of the specific intervention and another component is the "effectiveness" of implementation.  There are many ways in which implementation can fail when you scale up.  

To give you a nonbreastfeeding, nonschool based example, I evaluated a nongovernmental organization that had three interventions:  immunization, and oral rehydration, and growth monitoring and promotion.  My chore was merely to measure the effectiveness of implementation, not the efficacy of the interventions.  Immunization was implemented well.  Most children were immunized according to their target goals.  Oral rehydration was interesting.  Most parents knew it would help prevent dehydration, the availability of the oral rehydration packets was low, most parents would make up the home solution if their children were sick, most of the parents made it up in the wrong proportions.  Now, what is interesting is that another separate study (as opposed to what I conducted which was program evaluation -- not "research") suggests that this may have been OK, because the seriously dehydrated children usually did get the packets that were more precisely measured and the less seriously dehydrated children were usually treated at home.   The final intervention really never happened.  Hardly anyone had the charts let alone used them.  So therefore, you really cannot draw any conclusions if you really didn't implement the intervention (which was supposed to be targetted counseling when children had a drop in their usual growth). 

As I understand it, the PROBIT study may well have suffered from lack of full implementation.  To conclude that not breastfeeding is not detrimental to health is false on two counts if the breastfeeding promotion was poorly implemented.  First, it is not breastfeeding that was the intervention, it was breastfeeding promotion.  Second, if they didn't fully implement breastfeeding promotion, you cannot conclude that lack of breastfeeding promotion doesn't result in less desirable health.  You can only conclude that it doesn't result in less desirable health if it is implemented (or not) exactly as designed in this study.

The second problem is that you can have a carryover effect.  While this is a desirable thing in public health, it can make evaluation a challenge.  To give you another nonbreastfeeding example, some nongovernmental organization shave implemented very large scale program working gardening programs. In one evaluation, they had a carry over effect to the control villages, whereby those control villages started to implement the same type of gardening.  In this case, you can no longer compare the intervention villages to the control villages because all the villages received the interventions.

As I understand it, the PROBIT study may well have suffered from this as well.  If the breastfeeding promotion (however successfully or unsuccessfully it was implemented) spread to the control population, then you cannot conclude that not breastfeeding is not detrimental to health because in fact, you lost your control group.  Of COURSE you wouldn't see a difference.

Best regards, Susan Burger, MHS, PhD, IBCLC

             ***********************************************

Archives: http://community.lsoft.com/archives/LACTNET.html
To reach list owners: [log in to unmask]
Mail all list management commands to: [log in to unmask]
COMMANDS:
1. To temporarily stop your subscription write in the body of an email: set lactnet nomail
2. To start it again: set lactnet mail
3. To unsubscribe: unsubscribe lactnet
4. To get a comprehensive list of rules and directions: get lactnet welcome