“The tragedy of science — the slaying of a beautiful hypothesis by an ugly fact.”
~ Thomas Henry Huxley ~
I frequently receive e-mails and phone calls from people asking what reading programs are “scientifically research-based.” Unfortunately, that’s not an easy question to answer. Before I can answer the question, I have to ask two other questions — “To do what?” and “For which children?”
To do what?
Let’s consider the first question first — When you are conducting research, you must pay attention to both the design of the study and the outcome measures you wish to examine. The design and methodology of the study depend on what questions you want to answer.
For example, let’s say I wanted to test a diet drug. The first question I might want to know is, does taking this drug result in weight loss? That’s fairly easy to study — I randomly assign a sizeable sample of people to treatment conditions, give some of them the drug, give others a placebo, and monitor their weight for a few months while they are taking the drug.
However, I may also want to answer some other questions — are there side-effects from taking this drug? Could this drug cause long-term damage? Does the drug continue to be effective over long periods of time? Those questions are also easy to study, but the first study does not answer those questions. This was illustrated by studies of the mix of diet drugs popularly known as Fen-Phen. Studies showed that Fen-Phen did help people lose weight. However OTHER studies showed that people who took Fen-Phen were more likely to have heart valve disease.
Was Fen-Phen effective? Yes. Was it research-based? Yes. Was it deadly? Apparently for some people, yes.
Getting back to reading, there are some reading programs which seem to offer short-term gains, but which do not seem to offer much in the long-term. One intervention may involve pulling a struggling reader out of his or her regular classroom for 20 minute tutorials by a highly-trained professional. And there may be research studies that show that students make great gains while they are in the program. However, other research may indicate that after the students leave the program, the good habits and skills they learned in the program are not reinforced in the regular classroom, and the student begins to lag behind again. Thus, the program is effective in the short-term, but does not seem to have lasting effects.
Also, it is important to remember that reading is a skill which is built upon two subskills. The ability to read with understanding is the product of the ability to fluently decode text, and the ability to comprehend language (see S is for Simple View of Reading). Research related to a reading program may focus on that program’s ability to help children learn, for example, to decode with fluency. There may be no evidence that the program helps children to develop comprehension skills or reading skills in general — only decoding skills.
For which children?
This leads to the second question — For which children? If you pick a classroom at random, some children in that classroom will already have proficiency in all three of those skills (reading comprehension, listening comprehension, and decoding fluency). Some children will need help learning to decode text. Some children will have some difficulty with comprehension. Some children will have difficulties in both areas. One reading program may be highly appropriate for some children in that classroom, but inappropriate for other children. There are excellent programs available for helping children develop phoneme awareness, and use that awareness to develop fluent decoding skills. However, those programs are wasted on children who are already adept at reading. They may also not be optimal for children who primarily have comprehension difficulties. They are only effective for certain children who need instruction in the areas that program addresses.
Thus, it does not matter if there is a great deal of research showing a reading program is effective for enhancing, for example, decoding skills if none of your students need instruction in decoding. A program can be “research-based,” and still be inappropriate for your students.
How good is good enough?
There is another point I’d like to raise about research — a caveat emptor, if you will. Not all gains are equal. Simply showing that a reading program “improves” reading performance is not adequate. Some studies of reading programs show that they raise scores “on average,” but as a consumer, you should be asking if the gains reported are satisfactory — are they they gains you want for your students? Let’s say I have a reading program that studies have shown moderately enhances reading scores for 25% of the students. It is “research-based” — specifically, the research shows that average scores are raised, but not all that much for any one student. As the consumer, you have to decide if that is adequate. As a consumer, you have to decide if another reading program might be more effective and bring you more “bang for your buck.” It is not enough to ask if a program will raise scores; as consumers, we must also ask, how much gain can be expected from this program?
Quality
All of that said, there is another concern that I have that I’d like to air here. What I have said up to now in this essay assumes that the research in question is of high quality. However, most of the research that I have read related to reading programs has been of extremely poor quality — poorly designed, poorly executed, and poorly reported. The connection between the design of the study and the conclusions drawn from the study must be clear and appropriate, and all too often in reading research, that connection is tenuous at best. Poor-quality studies should not be allowed to see the light of day, but unfortunately, they are often published, and widely referenced.
In high-quality research, there is a peer-review process that must be free from bias, politics, and polemics. If I conduct a study of a reading program, before I am allowed to publish a description of that study in any reputable journal, there should be a panel of experts in the field who review my report to make sure the study was designed well, conducted well, reported well, and that the conclusions that I draw are clearly connected to the findings of the study. If I make generalizations in my report, the peer-review should check to make sure the generalizations are appropriate. The peer-review panel should be skeptical and critical, and yet objective. Peer-review can seem cruel, but it really isn’t. Good researchers are used to it, and they understand that it is a process that effectively elevates the quality of the research that is published.
Another filter that helps to improve the quality of research is the concept of “convergent evidence.” If I conduct one study that suggests that smoking does not cause cancer, you should be skeptical in light of all of the other studies that have been conducted that suggest a causal link between smoking and several types of cancer. No single study is all that convincing, but a collection of studies that are all in agreement about a conclusion is very convincing.
For now, the research in reading and education programs is scant, sketchy, and of poor quality. For 35 years, there has been excellent basic research in the psychology of reading — we know a great deal about how children learn to read in general — but the body of research in reading instruction and reading programs is quite poor. That is changing, and there are signs that the change is here for good. Just as there was a revolution in medicine and psychology research 50 years ago, so too is a revolution afoot in education research. People are adopting more of a healthy skepticism about programs — people are asking vendors to show them the research supporting the program. That’s good. For now, the vendors are trying to use poorly designed studies to support the effectiveness of their programs, and for now, consumers are still buying it. But the consumer is getting smarter, and in the future, the publishers and vendors will have to respond with high-quality research evidence that shows their products and services are effective and worth an investment of time and money.