Part 7 in a series.

The first six parts of this series reviewed research designs ranging from experiments to surveys. This material was presented as if everything in research proceeds without a mishap. Well, that's not how it happens in the real world.

However, there is one important thing to remember about the validity of research findings. The way science operates, you can never be absolutely sure about the results, but you can obtain a high degree of confidence. That confidence is achieved by keeping the threats to validity to a minimum.

Notice also the words "internal" and "external." Internal validity refers to findings that can be shown to result only from the effects of the independent variable(s) on a dependent variable-for example, the effects of supportive therapy (independent variable) on the recovery rates (dependent variable) of a sample of clients in your program. When the internal validity is sound, you can have a high level of confidence that no other incidental effects/variables have contaminated the effects on your dependent variable.

This is the major problem associated with many claims of treatment success in our field: They have poor internal validity.

External validity refers to the confidence of the research findings being generalized beyond the research that was completed. For example, can a valid piece of research conducted in a laboratory be utilized outside the lab? If it can, the external validity is high. That is what many substance use disorders counselors want from research.

There are many threats to validity, too many to cover in one column. In this piece, we will review six common threats to internal validity. Next time, we will review a few more internal threats, and then finish this subject area by reviewing some external threats to validity.

All that follows was taken from Bordens and Abbott (1996); Heppner, Kivlighan, and Wampold (1999); The National Institute on Drug Abuse (1993), and Polit and Hungler (1995).

In the pre-test/post-test designs we reviewed in Part 4 of this series, it was assumed that the time between measurements would be a reasonable interval. This is one way that we can assure ourselves that a planned independent variable is having an effect on the dependent variable (e.g., treatment's effect on outcome). If, however, there is a lot of time between these measurements or observations, we run the risk of unplanned events threatening to confound any results we observe. History is the name of this particular threat to internal validity.

For example, history can have a profound effect on what is thought to happen to clients when they leave treatment and remain abstinent. The longer they are out of treatment, the less we can be sure our treatment program is responsible for the positive effects. The reason is, there is a high likelihood that other variables (employment, family support, improved coping skills, etc.) will have a more powerful influence in the recovery process than the initial treatment.

Maturation is the next internal threat to validity. That is, as people grow older, they generally mature and refrain from old behaviors that might have had negative consequences. You can probably think of a few unproductive things you did years ago that you don't do now. If you don't do those things anymore, and you never entered treatment for the problem, you can probably say with some level of certainty that you matured out of that phase of your life.

In our treatment-outcome example, it can be conjectured that a client sample under investigation might have stopped destructive drinking and/or drugging as a result of maturing. With this particular threat to internal validity, you can never be quite sure if the treatment you offered was the key variable in the cessation of an addiction, or if it was maturity. This particular threat remains one of those unknown influences.

The next threat to internal validity comes about through the effects of testing. That's right, the very process of testing can threaten your outcome. For example, it has been documented that the mere act of collecting information from people changes them. This is especially true of self-reports: Knowing they are under observation, some people modify answers on a questionnaire. This contingency would deny you an accurate reading of a subject, and that inaccuracy would compromise the outcome of your research. This is generally not a problem if one uses standardized tests.

Instrumentation is the next internal validity threat. Here, changes in the way the data is collected between one point of observation and the next point can cause problems. For instance, a researcher may use one form of measure at a pre-test point and a different one at a post-test point. That is not wise, and can generally be avoided through the use of standardized forms of measurement.

People vary in their daily behavior. Some days you may be at a one extreme, and on other days you may be at an opposite extreme, but most of the time you moderate at some level of "normal." This trend to the norm is called statistical regression. If you tested someone on one of those extreme days-for depression, for example-and obtained a certain score, you might be convinced that this is the score you would get all the time. That is not true; we humans vary more than we stay constant. A high score on a depression test one day will often result in a different, more moderate score on another day. This difference, or regression to the norm, must be taken into account when testing subjects. Generally this bias can be handled by comparing test scores to a randomized control group.

Finally, attrition has to do with the number of people who drop out of an experiment. For example, let's say you pre-test 50 subjects in an experimental group and 50 subjects in a control group. Before you administer a post-test, 20 subjects drop out of the experimental group. That dropout group will affect how representative the overall scores are on the post-test, especially if the pre-test scores were of an extreme nature. That is a threat to internal validity. If the dropout rate is high, there may be no recourse but to start over.

Next: More threats to internal validity. This series will continue in Counselor, the Magazine for Addiction Professionals.

Michael J. Taleff, PhD, CAC, MAC, is assistant professor in the Counselor Education Department and project director of Chemical Dependency Programs at Pennsylvania State University. He is also a member of the NAADAC Research Committee and welcomes comments on this series. His e-mail address is This e-mail address is being protected from spam bots, you need JavaScript enabled to view it .

References
Bordens, K.S., & Abbott, B.B. (1996). Research design and methods: A process approach (3rd ed.). Mountain View, CA: Mayfield Publishing Co.

Heppner, P.P., Kivlinghan, D.M., & Wampold, B.E. (1998). Research design in counseling, (2nd ed.). Belmont, CA: Wadsworth.

National Institute on Drug Abuse (1993). A guide to evaluation: How good is your drug abuse treatment program? (NIH Publication No. 93-3609). Washington, DC: U.S. Government Printing Office.

Polit, D.F., & Hungler, B.P. (1995). Nursing research: Principles and methods (5th ed.). Philadelphia: J.B. Lippincott.

Posavac, R.J., & Carey, R.G. (1997). Program evaluation: Methods and case studies (5th ed.). Upper Saddle River, NJ: Prentice-Hall.

Sexton, T.L., Whiston, S.C., Bleuer, J. C., & Walz, G.R. (1997). Integrating outcome research into counseling practice and training. Alexandria, VA: American Counseling Association.

Shontz, F.C. (1986). Fundamentals of research in the behavioral sciences: Principles and practice. Washington, DC: American Psychiatric Press.

Utts, J.M. (1999). Seeing through statistics (2nd ed.). Pacific Grove, CA: Duxbury Press.

---

Comments

Add New Search RSS

Write comment
Name:
Email:	do not notifynotify
Title:
UBBCode:	-color-aquablackbluefuchsiagraygreenlimemaroonnavyolivepurpleredsilvertealwhiteyellow -size-tinysmallmediumlargehuge
	Please input the anti-spam code that you can read in the image.

Powered by !JoomlaComment 3.25

3.25 Copyright (C) 2007 Alain Georgette / Copyright (C) 2006 Frantisek Hliva. All rights reserved."