Evidence-Based Therapy or Not?
Or, When Is an Evidentiary Basis on the Level?
A Report to Advocates for Children in Therapy
April 2006
By Jean Mercer, PhD
Professor of Psychology
Richard Stockton College
Pomona, New Jersey
The idea of evidence-based or evidence-supported treatment (EBT) has been discussed in medical circles for quite a few years, and has more recently been a topic among psychotherapists and counselors. The public, insurance companies, and indeed many practitioners themselves are enthusiastically demanding that mental health interventions be demonstrated to be both effective and safe before they are widely used. The call is for an evidentiary basis — systematically-collected information about the effects of a treatment. In response, we see some practitioners announcing that their work is supported by an evidentiary basis (for example, see materials at The Center for Family Development); their statements may even claim that their techniques are the unique EBTs directed toward a specific problem.
The only problem here is that most consumers of therapy do not really know what an evidentiary basis is — whether it is anything more than testimonials from satisfied clients. Practitioners themselves are not always very clear on this point. And, in fact, there is probably no single definition of an EBT that can be applied in all cases. Like other information, evidence for a treatment’s effectiveness comes in several different sizes or levels, and one size does not fit all situations. Neither does one kind of evidence allow for the same conclusions or claims as other kinds.
Here are some comments on levels of evidence that may or may not support the idea that there is reliable evidence for the efficacy of particular treatments. Families looking for evidence-based treatment should examine practitioners’ materials carefully before accepting the idea that a therapy is supported by systematic evidence.
Class III Evidence: Simple, But Not as Common as You’d Think
The lowest level of evidence for a treatment’s effectiveness, Class III, is just a matter of a detailed description of the treatment and its outcome in one or more specific cases. This type of evidence simply tells what happened with particular applications of the treatment, including both what the therapists did and how the client responded. It provides information showing that the treatment has been employed without any harm being done. It does not provide evidence that a treatment “worked.”
Minimal as it may be, Class III evidence is an important starting place. Without such descriptive material available, no one can interpret more complicated kinds of studies. Class III reports may be published in journals, newsletters, or books, or the technique may be demonstrated in video or audiotapes, or there may be workshops or classes that show how the treatment is carried out. Only published material can be easily accessed, of course, so a workshop may not be a very good way to provide information that can be used in interpreting later studies of a treatment.
For various reasons, more complex research reports often do not include the descriptive material characteristic of Class III studies, so unless the authors have published a description elsewhere, interpretation may be difficult. Practitioners who specialize in proprietary treatments with trademarked names and descriptions generally refuse to release descriptions of their techniques and consider them to be “trade secrets, ”thus creating special difficulties for making any claim of an evidentiary basis. Prospective clients who are considering treatments bearing the ™ symbol should keep this in mind.
One special problem associated with an absence of Class III material has to do with documentation of informed consent. When an adult, or the guardian of a minor, agrees to psychotherapeutic treatment, the person’s agreement should involve an informed consent document. This document is a written statement of the client’s acceptance of the treatment, acceptance which should be based on accurate information describing the nature of the treatment, the probability of its effectiveness, any possible ill effects, alternative treatments that may be used, the confidential nature of the proceedings, and several other points that have been determined by professional groups and governmental agencies. Higher levels of evidence require informed consent that includes the material included in Class III studies.
Class III evidence thus provides only a very limited level of support for a treatment, and should not be presented as supporting EBT status. However, in the absence of Class III evidence, little can be done to establish a better evidentiary basis through more complex studies, and documentation of informed consent would not be possible.
Class II Evidence: Easier to Collect Than to Interpret
Even when reports of the Class III level seem to give a positive picture of a treatment’s effectiveness, that information should not be accepted as a genuine evidentiary basis, and a treatment should not be designated an EBT on such evidence. The effects of a treatment must be compared with the outcome of some other treatment, or with the effect of no treatment at all. This is because emotional disorders spontaneously vary in seriousness, whether or not a person receives treatment. Symptoms may become worse or better without any obvious reason for doing so. Without some form of comparison, it is impossible to know whether an improvement occurred spontaneously, or whether it was probably caused by a specific treatment.
The problem with Class II studies is that it is hard to be sure that the people in the two groups were not different in some way other than the treatment they chose and received. If they were different in other ways, those differences might be responsible for the later differences in their conditions. For example, if some people are chosen for treatment because they seem to be the ones who would most benefit from it, there would be differences present before treatment ever began.
A common way to set up a Class II study is to compare people who came to a clinic and received treatment with others who came to the clinic but did not follow through on the treatment (see Myeroff, Mertlich & Gross, 1999; Becker-Weidman, 2006). But this is not a good way to do a Class II study, and allows only a very weak claim that an evidentiary basis has been demonstrated. Why? Because the reasons for failing to follow through on treatment could be ones with a real effect on the client. For example, with child clients, treatment might not be pursued because the parents disagreed on it, a situation that might worsen the child’s condition; or, other children in the family might have problems that required financial resources and parent energy, again possibly having an ill effect on the first child.
Class I Evidence: Powerful, But Requires Careful Planning
The most powerful evidentiary basis in support of a therapy is established when the clients do not make their own decisions about their treatment, but are instead assigned to receive treatment through some random mechanism like flipping a coin. When this is done, we can feel sure that there is no overall effect of the person’s preference, beliefs, living situation, marital happiness, or finances. All those factors will average out over a large number of people, and the only thing left to cause a difference will be the type of treatment a group receives.
Of course, this may not be a very easy or practical plan to carry out. Unless a person has come into treatment because of a court order, he or she (or his or her parents or guardians) will probably have some preferences about the therapy. One way to approach this problem would be to deal with a group of people who have all decided to seek the same kind of treatment. The researchers would then delay the beginning of treatment of one group, while starting to treat the other group. The conditions of the two groups could be compared at the end of some appropriate period, and the delayed group could then be started in treatment. If the researchers were to randomly assign people to the earlier or the later-starting group, and if they found a significant difference between their conditions after the earlier group was treated for a while, the results would give a very strong reason to claim EBT status.
A Caution About All Classes of EBT Claims
Whatever class of evidence is being collected, the proceedings should be kept as objective as possible. This means that the individuals providing treatment should not be the ones who assess the client’s condition either before or after treatment; someone who knows what treatment has been received may unintentionally evaluate changes to match what is expected to happen. Neither, by the same token, should the evaluators know what type of treatment an individual is going to receive or has received. It is certainly a problem in the assessment of child psychotherapy if parents assess the child and also participate in the treatment. Ideally, researchers testing the outcome of a treatment should be independent of the treatment process.
A second requirement often mentioned with respect to EBT-related work is that studies must be replicated (see Chambless & Hollon, 1998). It is possible for statistical analysis to indicate a significant effect of a treatment in one study, but repeated tests in other studies do not show the same results. Unless replication is possible, it is not safe to say that the evidentiary basis supports the efficacy of a treatment. More than one study is always needed before a treatment can be claimed to be an EBT.
References
Arthur Becker-Weidman. “Treatment for children with trauma-attachment disorders: Dyadic Developmental Psychotherapy,” Child and Adolescent Social Work Journal, online edition, 3 March 2006.
Dianne L. Chambless & Steven D. Hollon. “Defining empirically supported therapies,” Journal of Consulting and Clinical Psychology, February 1998, 66(1):7-18.
Robin L. Myeroff, Gary Mertlich & Jim Gross. “Comparative effectiveness of holding therapy with aggressive children,” Child Psychiatry and Human Development, Summer 1999, 29(4):303-313. [Abstract]