39.4 The Status of Experimentation in Educational Technology

39.4.1 Uses and Abuses of Experiments

The behavioral roots of educational technology and its parent disciplines have fostered usage of experimentation as the predominant mode of research (see 2.3). As we will show in a later section, experiments comprise the overwhelming proportion of studies published in the research section of Educational Technology Research and Development (ETR&D). The representation of alternative paradigms, while gradually increasing, remains quite small.

39.4.1.1. The Historical Predominance of Experimentation. Why is the experimental paradigm so dominant? According to Hannafin (1986), aside from the impetus provided from behavioral psychology, there are three reasons. One is that experimentation has been traditionally viewed as the definition of "acceptable" research in the field. Researchers have developed the mentality that a study is of higher quality if it is experimental in design. Positivistic views have reinforced beliefs about the importance of scientific rigor, control, statistical verification, and hypothesis testing as the "correct" approaches to research in the field. Qualitative researchers have challenged this way of thinking, but until recently, acceptance of alternative paradigms has been reluctant and of minimal consequence (see Guba, 198 1).

Second, Hannafin (1986) proposes that promotion and tenure criteria at colleges and universities have been strongly biased toward experimental studies. If this bias occurs, it is probably attributable mainly to the more respected journals having been more likely to publish experimental designs (see next paragraph). In any case, such practices are perpetuated by creating standards that are naturally favored by faculty and passed down to their graduate students.

Third, the research journals have published proportionately more experimental studies than alternative types. This factor also creates a self-perpetuating situation in which increased exposure to experimental studies increases the likelihood that beginning researchers will also favor the experimental method in their research.

As will be discussed in later sections, however, practices are changing in the direction of greater acceptance of alternative methodologies, such as qualitative methods. The pendulum may swing far enough to make the highly controlled experiment with low external validity less valued than eclectic orientations that use a variety of strategies to balance internal and external validity.

39.4.1.2. When to Experiment. The purpose of this chapter is neither to promote nor criticize the experimental method, but rather to provide direction for its effective usage in educational technology research. On the one hand, it is fair to say that, probably for the reasons just described, experimentation has been overused by educational technology researchers. The result has frequently been "force fitting" the experiment in situations where research questions could have been much more meaningfully answered using an alternative design or a combination of several designs.

I For example, we recall a study on learner control that was submitted about a year ago to ETR&D for review. The major research question concerned the benefits of allowing learners to select practice items and review questions as they proceeded through a self-paced lesson. The results showed no effects for the learner control strategy compared to conventional instruction on either an achievement test or attitude survey. Despite the study's being well designed, competently conducted, and well described, the decision was not to accept the manuscript for publication. In the manner of pure scientists, the authors had carefully measured outcomes but totally omitted any observation or recording of how the subjects used learner control. Nor did they bother to question the learners on their usage and reactions toward the learner control options. The experiment thus showed that learner control did not "work," but failed to provide any insights into why (see 24.11).

On the other hand, we disagree with the sentiments expressed by some writers that experimental research conflicts with the goal of improving instruction (Guba, 1969; Heinich, 1984). The fact that carpentry tools, if used improperly, can potentially damage a bookcase does not detract from the value of such tools to skilled carpenters who know how to use them appropriately to build bookcases. Unfortunately, the experimental method has frequently been applied in a very strict, formal way that has blinded the experimenter from looking past the testing of the null hypothesis to inquire why a particular outcome occurs. In this chapter, we take the view that the experiment is simply another valuable way, no more or less sacrosanct than any other, of increasing understanding about methods and applications of educational technology.

39.4.1.3. Experiments in Evaluation Research. In applied instructional design contexts, experiments could potentially offer practitioners much useful information about their products, but will typically be impractical to perform. Consider, for example, an instructional designer who develops an innovative way of using an interactive medium to teach principles of chemistry. Systematic evaluation of this instructional method (and of the unit in particular) would comprise an important component of the design process (Dick & Carey, 1985; Kemp, Morrison & Ross, 1994; Ross & Morrison, 1993a; Tennyson, 1978; see also 42.1, 42.2). Of major interest in the evaluation would certainly be how effectively the new method supports instructional objectives as compared to conventional teaching procedures. Under normal conditions, it would be difficult logistically to address this question via a true experiment. But if conditions permitted random assignment of students to "treatments?" without compromising the integrity (external validity) of the instruction, a true experimental design would likely provide the most meaningful test. If random assignment were not viable, but two comparable groups of learners were available to experience the instructional alternatives, a quasi-experiniental design might well be the next best choice. The results of either category of experiment would provide useful information for the evaluator, particularly when combined with outcomes from other measures, for either judging the method's effectiveness (summative evaluation) or making recommendations to improve it (formative evaluation). Only a very narrow, shortsighted approach would use the experimental results as isolated evidence for "proving" or "disapproving" program effects.

In the concluding sections of this chapter, we will further examine applications and potentialities of "applied research" experiments as sources of information for understanding and improving instruction. First, to provide a better sense of historical practices in the field, we'll turn to an analysis of how often and in what ways experiments have been employed in educational technology research.

39.4.2 Experimental Methods in Educational Technology Research

To determine practices and trends in experimental research on educational technology, we decided to examine comprehensively the studies published in a single journal. The journal, which is now called Educational Technology Research and Development (ETR&D), is published quarterly by the Association for Educational Communications and Technology (AECT). ETR&D is AECT's only research journal, is distributed internationally, and is generally considered a leading research publication in educational technology. The journal started in 1953 as AV Communication Review and was renamed Educational Communication and Technology Journal (ECTJ) in 1978. ETR&D was established in 1989 to combine ECTJ (AECT's research journal) with the Journal of Instructional Development (AECF's design/development journal) by including a research section and a development section. The research section, which is of present interest, solicits manuscripts dealing with "research in educational technology and related topics." Nearly all published articles are blind refereed, with the exception of infrequent solicited manuscripts as part of special issues.

39.4.2.1. Analysis Procedure. The present analysis began with the Volume I issue of AVCR (1953) and ended with Volume 40 (1992) of ETR&D. All research studies in these issues were examined and classified in terms of the following categories:

Experimental Studies

      true-experiment

      Quasi-experiment

      Time series (single subject, repeated measures)

      Nonexperimental (Descriptive) Studies

      Correlational

      Ex post facto

      Survey

      Observational/ethnographic

A total of 303 articles were classified into one of these seven categories. Experimental studies were then classified according to two additional criteria:

Stimulus Materials

Actual content. Stimulus materials classified in this category were based on actual content taught in a course from which the subjects were drawn. For example, Tennyson, Welsh, Christensen, and HaJovy (1985) worked with a high school English teacher to develop stimulus materials that were based on content covered in the English class.

Realistic content. Studies classified in this category used stimulus materials that were factually correct and potentially usable in an actual teaching situation. For example, Beck (1984) used a unit on carnivorous plants with fourthgraders that was not part of the curriculum.

Contrived content. This stimulus material category included both nonsense words (Morrison, 1986) and fictional material. For example, Feliciano, Powers, and Kearl (1963) constructed fictitious agricultural data to test different formats for presenting statistical data. Studies in this category generally used stimulus materials with little if any relevance to subjects' knowledge base or interests.

Experimental Setting

Actual setting. Studies in this category were conducted in either the regular classroom, computer lab, or other room used by the subjects for real-life instruction.

Realistic setting. This category consisted of new environments designed to simulate a realistic situation. For example, if elementary students from a university lab school were tested in one of the university's computer labs, the study was classified in this category.

Contrived setting. Studies requiring special equipment or environments were classified in this study. For example, Niekamp's (1981) eye movement study required special equipment that was in-lab designed especially for the data collection.

The final analysis yielded 266 articles classified as experimental (88%) and 37 classified as descriptive (12%). In instances where more than one approach was used, a decision was made by the authors as to which individual approach was predominant. The study was then classified into the latter category. The authors were able to classify all studies into individual design categories. Articles that appeared as reports and lacked the rigor of other articles in the volume were not included in the list of 303 studies. The results of the analysis are described below.

39.4.2.2. Utilization of Varied Experimental Designs. Of the 266 articles classified as experimental, 182 (60%) were classified as true-experiments using random assignment of subjects; 74 (24%) of the studies were classified as using quasiexperimental designs; and 10 (3%) were classified as employing time-series designs. Thus, following the traditions of the physical sciences and behavioral psychology, usage of true-experimental designs has predominated educational technology research.

Figure 39-1. Experimental design trends.

An analysis of the publications by decade (e.g., 19531962, 1963-1972, etc.) revealed the increased use of true experimental designs and decreased use of quasiexperimental designs since 1953 (see Fig. 39-1). In the first 10 years of the journal (1953-1962), there was a total of only six experimental studies and three descriptive studies. The experimental studies included two true-experiments and four quasiexperimental designs. During the next 30 years, there was an increase in the number of true-experimental articles. Table 39-2 presents the number of articles published with each design in each of the 4 decades. It's interesting to note that quasi-experimental designs reached a peak during the 1963-1972 period with 43 articles and then decreased to only 7 articles in the 1983-1992 time period.

39.4.2.3. Utilization of Stimulus Materials. An additional focus of our analysis was the types of stimulus materials used in the studies. For example, did researchers use actual materials that were either part of the curriculum or derived from the curriculum? Such materials would have high external validity and provide additional incentive for the subjects to engage in learning process. Figure 39-2 illustrates the three classifications of materials used by the various studies published during the past 40 years. In the 1963 to 1972 period, actual materials were clearly used more often than realistic or contrived. Then, starting in 1972, the use of actual materials began a rapid decline while use of realistic materials tended to increase. There are two possible explanations for this shift from actual to realistic materials. First is the increasing availability of technology and improved media production techniques. During the 1963-1972 time frame, the primary subject of study was film instruction (actual materials). The increased utilization of realistic materials during the 1973-1982 period may have been the result of the availability of other media, increased media production capabilities, and a growing interest in instructional design as opposed to message design. Similarly, in the 1983-1992 time frame, the high utilization of realistic materials may have been due to the increase in experimenter-designed CBI materials using topics appropriate for the subjects, but not necessarily based on curriculum objectives.

39.4.2.4. Utilization of Settings. The third question concerns the settings used to conduct the studies. As shown in Figure 39-3, actual classrooms have remained the most preferred locations for researchers, although the classroom site has decreased in popularity during the last 10 years, while usage of realistic labs has increased. One possible explanation for the increased use of the realistic lab during the 1983-1992 period may be the increased number of CBI studies conducted in university computer labs not normally accessible to the subjects (hence, the classification as a realistic lab). Utilization of contrived settings, as was the case for contrived materials (see Fig. 39-2), has remained fairly constant over time.

39.4.2.5. Interaction Between Usage Variables. Extending the preceding analyses is the question of which types of stimulus materials are more or less likely to be used in different designs. As shown in Figure 39-4, realistic materials were more, likely to be used in true-experimental designs (48%), while actual materials were used most frequently in quasi-expenmental designs. Further, as shown in Figure 39-5, classroom settings were more likely to be chosen for studies using a quasi-experimental (82%) than for true-experimental designs (44%). These relationships are predictable, since naturalistic contexts would generally favor quasi-experimental designs over  true-experimental designs given the difficulty of making the random assignments needed for the latter.   The nature of educational technology research seems to create preferences for realistic as opposed to contrived applications. Yet, the trend over time has been to emphasize true-experimental designs and fewer classroom applications. Internal validity thus appears to have acquired relatively greater value than external validity in the design of many studies. Changes in publishing conventions and standards in favor of high experimental control have certainly been influential. Affecting future patterns will be the growing usage and acceptance of qualitative methods in educational technology research. Although qualitative methods are currently popular, that pattern was not noticeable in the studies reviewed in the present analyses, all of which were published by 1992.