Ninth in a series designed to help clinicians, prevention specialists, supervisors and administrators use and understand research. Parts 1 through 8 are available online at Counselor Magazine Online.

In the last two parts of this series, we reviewed internal threats to validity. They covered what to do to diminish competing explanations for the causes and effects in an experiment. Now it is time to meet the external threats.

Just as there are many internal threats to validity, there also are many external ones. We will cover a few to give you a taste of these hazards.

The first threat to external validity that we will examine is called interaction of selection and treatment. Here, we refer to the ability of a study to generalize across different people. However, people are distinct. If you are thinking about applying the results of certain drug and alcohol treatment research to a population, you have to consider if it will apply to different genders, as well as different cultures. In addition, there are different levels of intelligence to bear in mind, as well as degrees of dysfunction, life and pathology experience, sexual orientation, and personalities, among other things.

For instance, say I have done a randomized study of the effects of a certain treatment on men, it was significant, and now I want to apply this treatment to women. I cannot just say it is effective with women; I have to study the effectiveness with women. If it proves to be equally successful, then my therapy can be generalized; if not, I cannot claim effectiveness with women.

In situations such as these, you can strengthen the external validity by examining the relationship variables while considering the different categories of people. (Bordens and Abbott (1996); Heppner, Kivlighan and Wampold (1998); and Polit and Hungler 1995).

Another external threat to validity is called interaction of setting and environment. The issue here is: Can the research be generalized across settings? For example, can results of good research from a hospital setting be applied to private practice? What about generalizing research that was found to be effective in a standard drug and alcohol inpatient program to an inpatient dual-disorders program? I am guessing that most of you would say no, you couldn't generalize such programs. If you thought that, you would be right.

As in the first example, you can improve the validity of this external threat by accounting for different settings in your research.

The third external threat to validity is called interaction of history and treatment, which is the ability of research to generalize across time. For example, would the treatment research of the 1960s make sense today? Probably not. The intoxicating levels of many drugs have changed, the people have changed and certainly the programs have changed. These variables and the new research of today require us to modify the way we treat our clients. Practitioners who use the same treatment research today as they did in the '60s are going to have significant problems with external threats to validity. The way to avoid this threat is to try to replicate studies across time, and analyze prior research across various periods.

Another spin on this interaction of history and treatment is the effect of outside elements on an experiment. If you would repeat the experiment, and the outside effect was not present, the results of the experiment might be different.

For example, say you conduct an experiment about a treatment you are studying and that experimental treatment coincides with a negative media blitz. If the subjects of your experiment heard about this investigation, that would certainly affect your outcome. If there were no media blitz, your results would be different. You just have to be careful of outside effects.

The next threat to external validity will be familiar to anyone who has taken a general psychology course: it is the Hawthorne effect. This threat arises when the subjects of an investigation behave in a manner that is not in keeping with their normal behavior. That is, they may act in a certain way during the research, and when the investigation or research is complete, they return to their everyday behavior. If they do this, it will be difficult to generalize such behavior seen in the research to a more natural setting. So researchers sometimes use double blind experiments — in which neither the subjects nor researchers know who is in which group.

People tend to act in either an enthusiastic or a skeptical manner about doing new things. If such behavior begins to occur during an experiment, that altered behavior will not generalize well to the natural world. This is called a reaction to novelty effects. Once the novelty wears off, you are more likely to get accurate results from your experiment.

I am sure many of you who tried a new procedure in a program found that it worked well many times before the novelty wore off, and then the results were different. This is what happens with the novelty effect.

If you can have novelty effects that will interfere with generalizing to other people and settings, you can also get experimenter effects that will do the same thing. In this threat to external validity, the characteristics of the researcher may alter the outcome of the study. Contrary to popular belief, researchers have feelings; that is, they can have an emotional investment in any hypothesis they put forth. If that is the case, the investigator may produce a bias in the observation of subjects, altering the outcome of the research. As noted, a way out of this predicament is do a double-bind experiment.

The last external threat to validity that we will examine is called measurement effects. Here, if a researcher uses a certain test or instrument to collect data, who is to say that the method of collection would apply to another group? The new series of rapid assessment that addresses women and African American women came about to decrease this threat to validity.

Finally, it is difficult to reduce both internal and external threats to validity. Usually you have to sacrifice some parts of one for another. It is usually preferable to reduce the internal threats, because it doesn't make sense to try to externalize findings of an experiment if the findings are not internally valid.

With all this information in hand, you can now probably determine the good experiments from the rubbish.

Next: Part 10 examines qualitative research, which is different from quantitative research and reflects how many counselors think.

Michael J. Taleff, PhD, CAC, MAC, is assistant professor in the Counselor Education Department and project director of Chemical Dependency Programs at Pennsylvania State University. He is also a member of the NAADAC Research Committee.

References
Bordens, K.S., & Abbott, B.B. (1996). Research Design and Methods: A Process Approach. Mountain View, CA: Mayfield.

Heppner, P.P., Kivlighan, D.M., & Wampold, B.E. (1998). Research Design in Counseling (2nd ed.). Belmont, CA: Wadsworth.

Huck, S.W., Cormier, W.H., & Bounds, W.G. (1974). Reading Statistics and Research. New York: Harper & Row.

Polit, D.F. & Hungler, B.P. (1995). Nursing Research: Principles and Methods (5th ed.). Philadelphia: J.B. Lippincott.

Comments

Add New Search RSS

Write comment
Name:
Email:	do not notifynotify
Title:
	Please input the anti-spam code that you can read in the image.

Powered by !JoomlaComment 3.26

3.26 Copyright (C) 2008 Compojoom.com / Copyright (C) 2007 Alain Georgette / Copyright (C) 2006 Frantisek Hliva. All rights reserved."

---