Blogging Lovaas, 6

Science or scientism?

This teaching manual places a major emphasis on describing treatment programs confirmed effective based on methods of scientific inquiry. The final goal of scientific inquiry is to make the treatment procedures and data on treatment outcomes believable and replicable to the scientific community and to parents, teachers, and others who want to apply them.

Perhaps it's just picky, but it's mildly interesting that when framing the point of scientific inquiry Lovaas looks first at its rhetorical advantages - it makes the outcomes believable.

This in turn means that one's investigative efforts are subject to review, commonly referred to as peer review, by other scientists so as to establish the validity of these efforts. The validity and efficacy of behavioural treatment is based on thousands of scientifically sound studies of learning processes investigated for over 100 years and published in journals with competent peer review by a large number of researchers from across this country and abroad.

This claim of scientific status is certainly one of the central planks of the Lovaas phenomenon. Myself, I can see more of what Feynman called cargo cult science;

I think the educational and psychological studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes with lots of good materials, and they want the same thing to happen now. So they've arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head to headphones and bars of bamboo sticking out like antennas--he's the controller--and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land.

In part, Lovaas' is an appeal to method - ABA works with scientific protocols, and so are scientific - and partly an appeal to outcomes - the planes, it is said, have landed. I'll speak more of that later; at the moment, I'd like to quote some more Feynman that suggests that you can have some of methods and some of the outcomes and still not be a science.

There is one feature, I notice, that is generally missing in cargo cult science. It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid--not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked--to make sure the other fellow can tell they have been eliminated. Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can--if you know anything at all wrong, or possibly wrong--to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it.

I can't see any evidence of that attitude in anything produced by Lovaas; anybody who thinks they have found an instance is welcome to draw it to my attention.

This may not be as significant as the next issue Feynman raises.

There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

Feynman says that you collect data to form your hypothesis, and then you go off and collect different data to test your hypothesis. Lovaas came up with the first round with his canonical study in 1987, and spent the next ten years defending it. Feynman would have suggested that yes, it was a good start, but you needed a different data set to test the predictions made by the first one. Lovaas has worked on the assumption that a single experiment can establish his theory. Not, in Feynman's terms, scientific.