In this Issue
Qualitative Methods and Philosophies
Qualitative Methods and Applications
Qualitative Methods in Research
Other Original Research
The Use Of The Solomon Four-Group Design In Nursing Research
Thayer W. McGahee PhD, RN
Martha S. Tingen, PhD, RN
Purpose/Methods: One of the most rigorous designs that can be used in quantitative studies is the Solomon four-group design, yet there is a paucity of its use in nursing research. This article presents the advantages of this design, obstacles to its use and how they can be overcome, methods for statistical analysis of data generated from this design, and an example of research using this design.
This article summarizes and illustrates the usage of this design in nursing literature, and presents an example of its use in a smoking prevention study with fifth-grade children. Various statistical methods for analyzing data utilizing this design are presented.
Implications: The benefits of using the Solomon Four group design far out-weigh the obstacles. This rigorous design provides the researcher confidence in the significance of study results since it guards against both threats to internal and external validity. The use of the Solomon four-group design is recommended for consideration of use by nurse researchers as we strive to expand our knowledge and science, through advancement in the use of rigorous research methodology.
Keywords: Solomon-four group design, methodology
The Use Of The Solomon Four-Group Design In Nursing Research
As the breadth and depth of nursing research have expanded over the past several decades, so has nursing science and knowledge. There has been an increase in applied research, including experimental research that has grown out of excellent basic research. Nurse researchers have become more educated about methodology and statistical analysis for quantitative studies. One of the most rigorous designs that can be used in quantitative studies is the Solomon four-group design,12 yet there is a paucity of its use in nursing and other health care literature. This article presents the advantages of this design, the obstacles to its use and how they can be overcome, methods for statistical analysis of data generated from this design, and an example of research using this design.
The Solomon four-group design has historically been used more in experimental studies conducted in laboratories where variables can be carefully manipulated and controlled. Most nursing research involves human subjects, and this can make the use of this design more challenging. Although the use of this design has been used in published nursing research,1,9,11,14,15 it is rare.
Advantages of the Solomon Four-Group Design: The Solomon four-group design is one of the three primary designs recommended for use with true experimental research, but can also be used in quasi-experimental studies. This design is a combination of the pretest-posttest control group design and the posttest only design.10,13 It has the advantages of these other two designs with the additional advantage of being able to test and control for instrument reactivity. Table 1 provides a schema of the design. Researchers using this design are in a position to examine both the main effects of testing and the interaction of testing and treatment. The researcher is also in a position to examine the combined effect of maturation and history by comparing O6, (the posttest only control group) and O3 (the pretest control group). Using the Solomon four-group design, subjects are randomly assigned to one of four different groups. Two of the groups receive the treatment (i.e. intervention) and two do not (i.e. control). Only one of the treatment groups is administered the pretest; however, all four groups are post-tested. Some researchers also use a modified version of this design in which there is only one control group instead of two, and that group is given a pretest and a posttest.6
This design is the only type of experimental design that is able to assess the presence of pretest sensitization.7,13 In other words, the posttest measure may be affected not only by the treatment or intervention, but could also be distorted by exposure to the pretest.12 Braver and Braver3 discuss the general lack of concern investigators have for pretest sensitization, but this interaction may significantly limit the generalizability of the research. The Solomon four-group design allows the researcher to test for this and separate out the effects of the pretesting and the treatment. Use of this design enables the researcher to have more confidence in inferring causal relationships because experimental designs have a higher degree of internal validity than other methods. There is also a reduction in between-subject variation, which increases the power of the study to determine true treatment effects.4
Obstacles and How to Overcome Them: Despite the advantages in strengthening both internal and external validity of research, the Solomon-four group design is seldom used. While its rigor is an advantage, it may deter researchers who are not aware of how it can be utilized in various and creative ways.
One of the major barriers to its use is the large number of subjects needed. Although this design does require twice the number of groups as others, it does not necessarily require twice the number of subjects. Braver and Braver3 showed that it is possible to have the same number of subjects that other designs would employ, and simply cut the size of each group in half. Their approach demonstrated that by doing this, the statistical power was still adequate, and even greater than the power of the posttest-only control group design.
Another barrier in using the Solomon-four design is the difficulty in introducing the treatment simultaneously for all groups.4 If possible, this is desirable, in order to avoid extraneous temporal effects. To have the treatment or intervention administered at exactly the same time for both groups requires a collaborative team effort with multiple researchers and staff. A potential approach for this obstacle would be to have the same researcher conduct the intervention to both groups within a narrow time frame. This would eliminate differences and potential bias in the delivery of the intervention.
Another reason cited for this design being used infrequently is the difficulty of randomizing subjects to one of the four groups. This may be approached by the researcher by changing the design from true experimental to quasi-experimental. Convenience sampling could be used to obtain the subjects, and then they could be randomly assigned to each of the four groups. Sometimes it is not feasible to randomly assign individual subjects to groups, but it may be possible to randomly assign groups of subjects to treatment arms. McGahee & Tingen9 did this by obtaining a convenience sample of schools, and then randomly assigning the schools to each of the four groups.
Spector13 asserts that the basic premise of the Solomon four-group design could also be extended to larger and smaller designs. For example, in the single-group pretest-posttest design, it could be used to control for instrument reactivity by omitting the pretests of half of the subjects. Another way it could be utilized in studies with more than two groups is by adding pretests for half of the subjects.
A last barrier to the use of this design is often cited as being the complex statistical analysis needed.3,4,12,13 There is more than one method for analyzing data gathered using this design, and the methods used are dependent on the questions under investigation.
Methods for Statistical Analysis: Campbell & Stanley5 were among the first to explore the statistical analysis of the Solomon four-group design, after deeming Solomon’s suggestions unacceptable. Because of the asymmetry of the design, they ruled out using analysis of variance (ANOVA) of gain scores. Their 2 X 2 analysis of variance design is illustrated in Table 2. The pretests are basically disregarded, except as another “treatment” coordinate with X. From the column means, estimates are made of the main effect of the treatment; from row means, estimates are made of the effect of pretesting. The interaction of pretesting with the treatment is made from the cell means.
Spector13 addresses this issue by recognizing that using ANOVA would result in missing data for half of the subjects. His suggestion is to conduct the analysis in stages. The first stage involves comparing the two treatment groups on pretests as this assesses for pretest sensitization. The second stage is the analysis of all of the posttest scores using a 2 X 2 factorial design. Treatment level is one factor and the presence or absence of the pretest is the other factor.
Braver & Braver3 wrote an article on the statistical treatment of this design using meta-analysis. Although this type of analysis is usually applied to the results of several different studies, they used hypothetical data to illustrate its application to several different tests of the same effect within one single study. They demonstrated superior power using meta-analytic techniques instead of customary analysis using ANOVA.
An Example of Research Using the Solomon-Four Group Design: Michel & Haight11 applied the algorithm presented by Braver & Braver3 to a data set involving a five year intervention study utilizing the Solomon four-group design. They found the algorithm of meta-analytic techniques to be clear, concise, and fully interpretable.
McGahee8 and McGahee and Tingen9 utilized the Solomon four-group design to study the efficacy of a smoking prevention curriculum in fifth-grade children. Several different types of analysis were used, depending on the specific research question being explored. Babbie’s2 suggestion of examining four comparisons involving posttest scores was utilized. Figure 1 presents these comparisons schematically. These comparisons helped to determine whether or not the smoking intervention made a difference, whether the difference was unaccounted for by problems of internal validity, and whether there was an interaction between testing and the intervention. A t-test for paired samples was used to analyze these comparisons.
McGahee’s8 study involved several different research questions. The first question examined the effectiveness of the intervention in decreasing the subjects’ intention to smoke. The second question explored the impact of the intervention on several different dependent variables.9 Analysis of variance was used to answer these questions. A third question in McGahee’s8 study aimed to identify predictors of a child’s intention to smoke, and multiple regression analysis was the statistical technique used for this question. This study illustrates the variety of questions that can be examined using the Solomon four-group design and the combination of statistical analyses utilized.
Conclusion: Although the Solomon four-group design is considered to be one of the most rigorous and prestigious designs, and gives explicit consideration of external validity of the treatment effect, it is rarely utilized in nursing research. Although there are several reasons for this paucity of use, many reasons may be easily overcome by the flexibility of the design. Specifically, it can be utilized in both true experimental and quasi-experimental studies. Also, adjustments can be made in how subjects are assigned or grouped. There are also a variety of statistical methods that can be employed for data analysis. Thus, the benefits of using this design far out-weigh the obstacles. This rigorous design can give the researcher confidence in the significance of the study results since it guards against both threats to internal and external validity. The use of the Solomon four-group design is recommended for consideration of use by nurse researchers as we strive to expand our knowledge and science, and through advancement in the use of rigorous research methodology.
Table 1: Solomon Four-Group Design
Table 2: Campbell & Stanley 2x2 Design Analysis