Kulhavy and Stock (1989) have proposed a model of feedback in written instruction that attempts to clarify and explain previous findings in the literature. Their model also goes beyond these basic explanations to make testable Predictions undergirded by theoretical rationales. The model has been scrutinized (Bangert-Drowns et al., 1991; Dempsey, Driscoll & Swindell, 1993; Mory, 1991, 1992, 1994) and tested by current researchers (Kulhavy & Stock, 1989; Kulhavy, Stock, Hancock, Swindell & Hanumich, 1990; Kulhavy, Stock, Thornton, Winston & Behrens, 1990; Mory, 1991, 1994; Swindell, 1991, 1992; Swindell, Peterson & Greenway, 1992). It is cited as the most comprehensive treatment of feedback in facilitating learning from written instruction (Dempsey, Driscoll & Swindell, 1993), since it integrates the factors of learner confidence, feedback complexity, and error correction, and has been investigated under different modes of presentation and timing. (Note that each of these components will be discussed individually and in depth later.)

Kulhavy and Stock (1989) assert that much of the prior research on feedback is conceptually flawed. For one thing, researchers always treated responses as being absolutely right or wrong, a dichotomy that virtually ignored the complexity of learning behavior. Consider that a correct answer may be just a lucky guess, or that a wrong answer may be anything from a careless mistake to a total miscomprehension of the material. Even more puzzling were studies that resulted in initial correct answers being changed to wrong responses on a posttest, and instances in which initial errors were never corrected, in spite of what was included in the feedback (Lhyle & Kulhavy, 1987; Peeck, van den Bosch & Kreupeling 1985).

The model proposes that the feedback process is made of three cycles that constitute each instructional episode. In cycle 1, the learner is presented with a task to which he or she needs to respond. In cycle II, feedback is presented based on the input from the learner in cycle 1. In cycle III, the original task is presented again as a test item to which the learner again responds. Within each cycle, a common series of steps ensues. Put succinctly, each cycle involves an input from the task at hand to the learner, a comparison of the input to some sort of reference standard that then results in an output. The degree of mismatch between the perceived stimulus and reference standard results in a measure of error. The discrepancy between these two entities causes the system to exert effort to reduce the discrepancy. Dempsey and his colleagues (Dempsey, Driscoll & Swindell, 1993) have graphically represented the Kulhavy and Stock model, as seen in Figure 32-1.

During each cycle, the learner engages in mental activity aimed at processing the input and preparing an appropriate response. The model emphasizes the learner's level of certainty (termed response certitude) between the demands of the instructional task in cycle I and his or her prior knowledge and current understanding of that task. If this perceived match is good, the learner will select a response with a high level of certainty or confidence. The worse the match, the lower the learner's confidence level will be. In cycle II, when the learner receives feedback on his or her response, the feedback acts as verification to allow the learner to compare the response to the information contained in the feedback. When this verification is combined with the learner's initial response confidence level, a discrepancy value results. If learners receive verification of a correct answer when they are certain they were correct, there is no discrepancy. Conversely, learners who are informed that their answer was wrong when they were confident that their answer was correct will produce a high level of discrepancy.

Kulhavy has represented this discrepancy value in the equation

f, X c=d

where fv is the verification component, c is the initial certitude level, and d is discrepancy. 'Me verification component f, is set to equal (- I)-, where m = 0 for initial error responses and m = I for initial corrects. This is explained as having the effect of assigning an algebraic sign to d, where [(-1) 0 = +11 for errors and [(-I)' = -1] for correct responses. The response certitude variable, c, usually employing a 5-point Likert-type scale, results in a discrepancy, d, from (-5) to (+5) (Kulhavy & Stock, 1989; Kulhavy, Stock, Hancock et al., 1990).

In this model, it is predicted that the level of discrepancy is a major factor influencing how much time and effort a student will naturally expend in error correction. In the case of a high-certitude correct answer (low discrepancy), the student has little need for extensive or elaborated feedback. But when students think an answer is correct but was in reality an incorrect response (high discrepancy), they will exert much effort to find out what was remiss in their thinking. In the case of low-certitude responses, regardless of whether the student's answer is correct or wrong, the student likely does not understand the information and would likely benefit from feedback that acts as new instruction. Even in Kulhavy's (1977) prior research, we see that highconfidence correct answers yield the shortest feedback study times, high-confidence errors yield the longest time, and low-confidence responses fall somewhere in between (Kulhavy, White, Topp, Chan & Adams, 1985; Kulhavy, Yekovich & Dyer, 1976, 1979). Obviously, discrepancy must mediate effects of different types of feedback in terms of their complexity or elaboration. Further, according to the model, prescriptions can be made as to how much and what type of information to include in feedback for the varying levels of discrepancy.

Figure 32-1. Representation of Kulhavy and Stock's (1989) certitude model of text-based feedback. (From Interactive Instruction and Feedback, p. 42, by J. V. Dempsey & G. C. Sales, eds., 1993, Englewood Cliffs, NJ, Educational Technology.) Copyright 1993 by Educational Technology Publications. Reprinted with permission.

Kulhavy and Stock's (1989) predictions have been shown to prevail in a number of conditions, thus suggesting its robustness. In testing the model, they performed three studies relating to discrepancy and feedback times and the durability of correct answers under low discrepancy (Kulhavy & Stock, 1989). As predicted by the model, learners who thought they answered correctly when in fact they were in error (high discrepancy) spent more time studying feedback. To further test this finding, students in a second study (Kulhavy & Stock, 1989) were told that an answer was wrong when it was in fact correct, and vice versa. Because the students thought their answer was wrong when they had assumed they were correct (even though in actuality the answer was correct), they indeed spent more time studying the feedback. Again, these results support the model. And in their third study (Kulhavy & Stock, 1989), they demonstrated that the probability of a correct posttest response increased with the initial response certainty level, particularly when practice responses were also correct. In this way, feedback served to increase the durability of initially correct responses.

Several other studies have also supported the model. Kulhavy and his associates (Kulhavy, Stock, Hancock et al., 1990) found that in the absence of feedback, response confidence and the probability of a correct posttest response are positively related. The model suggests that feedback elaboration should be useful in correcting particularly high-certitude errors, a prediction that a study by Swindell (1991) supports. One problem in the Swindell study, however, is that feedback elaboration consisted of presenting the stem and all of the alternatives listed with the correct alternative designated by an asterisk. As will be discussed later, feedback elaborations usually provide more information than was operationalized in the Swindell (1991) study, usually informing the learner of why an answer is incorrect or re-presenting a portion of the original instruction.

The prediction that there is a direct relationship between increases in discrepancy and increased study effort is supported by another study by Swindell (1992). In this study, she also constrained the time that students were allowed to study feedback, expecting that as feedback reading time became increasingly constrained, the probability of a correct posttest response would decrease. This was generally true, but for groups receiving feedback at both slow and average presentations speeds, high certitudes resulted in lower probabilities of correct responses, and lower certitude resulted in higher probabilities. She explains this through interference theory, suggesting that in the case of errors, certitude may reflect response competition that results in an inaccurate perception of comprehension. Her study. was not able to support a durability hypothesis that high-certitude response alternatives would be better remembered and carry over to a posttest, and that low certitude judgments are more likely to be forgotten over time and are less likely to be chosen again on a posttest. No systematic relationship could be determined from her study.

Swindell and her colleagues (Swindell, Peterson et al., 1992) also have attempted to extend the model to younger learners, since the original model was developed from a research base of adult learners. Certainly the developmental stage a child is at will likely determine whether or not the child is able to assess accurately his or her own learning confidence. The results of the study suggest that fifthgraders demonstrated the pattern that high-confidence errors (maximum discrepancy) were more likely to be corrected on a posttest than were low-confidence errors. However, third-graders in the study demonstrated an Opposite Pattern: high-confidence errors were less likely to be corrected than those of low confidence. Further, fifthgraders were more likely to correct high-confidence errors than were the third-graders.

Dempsey and his associates (Dempsey, Driscoll & Swindell, 1993) point out that the Kulhavy and Stock (1989) model also provides a useful framework for past research results. The durability hypothesis explaining why initially correct responses are better remembered than errors, assuming that learners are more likely to make higher-confidence judgments for correct responses than for incorrect responses, is supported by Peeck & Tillema (1979) and Peeck et al. (1985). Measures of response certitude and durability should be positively related because high confidence should represent better comprehension and will therefore be better remembered. Further, the model supports the finding that learners were not only more likely to recall initially correct responses, they were also more likely to correct initial errors if they could recall their initial response. And a recent study (Swindell, Kulhavy & Stock, 1992) found similar response patterns for durability as well.

Although the Kulhavy and Stock (1989) model of feedback is the most comprehensive to date, it does have some problematic aspects. For one thing, response certitude is a self-report measure. While response certitude judgments do provide some useful information about the cognitive status of the leamer (Kulhavy et al., 1976; Metcalfe, 1986; Nelson, Leonesio, Landwehr & Narens, 1986), the nature of determining certitude has some underlying problems. The idea behind response certainty lies in the learner's metacognitive process of predicting his or her criterion performance on a task. This process can be related to "feeling of knowing" research (Butterfield, Nelson & Peck, 1988; Metcalfe, 1986; Nelson, 1988; Nelson et al., 1986). Feeling-of-knowing has been shown to be accurately predicted for memory recognition tasks and has been found to exist over all age groups, and the reliability of feeling of knowing has been found to be generally excellent. However, the stability of an individual's feeling-of-knowing accuracy has been found to fluctuate significantly (Nelson, 1988). In Nelson's (1988) findings, when a subject gives a higher feeling- of knowing rating to one item over another, there is perfect retest reliability in that the same outcome occurs if the person subsequently makes feeling-of-knowing responses on those same items (Nelson et al., 1986). Conversely, individuals having a relatively high level of feeling-of-knowing accuracy at one time do not also have a relatively high level of feeling-of-knowing accuracy at another time (see Nelson, 1988). Since individual differences of feeling-of-knowing accuracy may be inconsistent, it raises the question of whether or not a response certitude estimate is valid for prescribing feedback, if certitude statements may not be a stable measure of an individual's true knowledge. Perhaps if a variable or variables could be identified that influence these changes, researchers would have more insight into the process. For example, learners' general level of selfesteem or motivation might be influencing the learners' perceptions of certainty.

Further inconsistency predominates when comparing the levels of tasks involved in feeling-of-knowing research. Learners were able to predict accurately their feeling-of-knowing in memory tasks~ but overestimated their likelihood of success on problem-solving tasks or problems requiring insight (Metcalfe, 1986). Other researchers (Driscoll, 1990) have found a contrary finding, that students learning concepts tended to underestimate their feelings of answer correctness. These cases of over- and underestimation show that students generally possess an inaccurate perception of their own knowledge. Of further concern, most feedback studies using response certitude have employed verbal information tasks only; in fact, the model itself was built on a vast well of studies that involved rote memorization of verbal information. As researchers are discovering (Dempsey & Driscoll, 1994; Mory, 1991, 1994), tasks of learning intellectual skills may produce different results, especially in light of the prior findings, suggesting that subjects tend to wrongly estimate their feeling of knowing during studies using higher-level tasks. Indeed, this was the case in a recent study (Mory, 1994) that used response certitude estimates as part of the feedback cycle for both verbal information and concept-learning tasks. Students tended to have a high level of certitude for concept questions, regardless of actual answer correctness. Thus, low-certitude feedback designed to give the most information was not encountered when it was truly needed. Learners simply were not able to give accurate assessments of their own abilities to classify a particular concept.

Another issue that regards the application of response certitude estimates within an instructional situation is that of efficiency. Corrective efficiency results from taking the total number of correct answers on a posttest and dividing it by the amount of time spent during an instructional task. Kulhavy and his associates (1985) examined efficiency using two separate measures. One measure isolated the amount of time spent reading the instruction, thus accounting for the efficiency of only the instruction or "text" portion of the lesson. When this measure was tested across varying feedback groups, there were no significant differences found. A second measure used was the amount of time spent just in studying the feedback, since less complex forms of feedback are usually more time efficient in terms of what Kulhavy and his colleagues (1985) call "posttest yield per unit of study time invested" (p. 289). The amount of time a learner spends on feedback is affected by two things: (1) the amount of information included in the feedback message (load) and (2) response certitude levels. Results from the study confirmed that the less-complex forms of feedback were more time efficient, and also that efficiency rose as a function of increases in confidence values. Considering that high-confidence responses should reflect an understanding of subject matter and content, the learner would be more likely to make efficient use of the feedback presented (Kulhavy et al., 1985).

One should note that the Kulhavy study (Kulhavy et al., 1985) examined efficiency in terms of the feedback portion of a lesson only. But the process of giving a response confidence rating for each question could possibly add considerable time and interference to the overall lesson for the student. Mory (1991, 1994) investigated adaptive feedback that was based on levels of discrepancy and prescriptions of the model. The study supports that feedback efficiency can be increased by varying the amount of feedback information according to levels of discrepancy; however, the added time for response certitude evaluations resulted in lower overall lesson efficiency. Further, when a typical nonadaptive feedback sequence was compared with an adaptive one that employed response certitude as part of the cycle, adaptive feedback was significantly less efficient than traditional feedback in terms of overall lesson efficiency (Mory, 1994).

And lastly, one might question the generality of a model that was built around experimental testing environments and usually limited to the use of multiple-choice questions (see Kulik & Kulik, 1988). Many of the studies present brief paragraphs of text information, followed by multiplechoice questions based on the preceding paragraph (Chanond, 1988; Kulhavy et al., 1976, 1979; Lhyle & Kulhavy, 1987). Many of these studies used generic topics with limited relevance to current topics being studied by learners within the experimental groups. And to further confound matters, in several studies students were not given instruction at all, but questions and feedback alone served as "instruction" (Anderson et al., 1971, 1972; Kulhavy & Anderson, 1972; Kulhavy & Stock, 1989; Swindell, 1991). In fact, recent findings (Clariana, Ross & Morrison, 1991) support the notion that feedback effects tend to be stronger in conditions where materials involve no text but use questions and feedback only, than in conditions in which text was used before questions and feedback. This leads to the question of whether or not the model will be supported in "real-world" instructional environments. Researchers (Chanond, 1988; Dempsey, Driscoll & Litchfield, 1993; Mory, 1991, 1992, 1994; Peterson & Swindell, 1991) are beginning to recommend that the model be examined under more typical classroom learning situations.

Researchers interested in exploring the Kulhavy and Stock (1989) model further should consider some of the aforementioned issues, both supportive and problematic. Dempsey, Driscoll, and Swindell (1993) point out that the model has made more precise predictions for high-confidences responses than for low-confidence responses, and that midrange levels of confidence have no such predictions. This means that the entire range of metacognitive judgments should be examined. Further, if response confidence could be linked to a variable other than self-report, the adaptation of feedback might more readily fit the needs of the learner. For example, Dempsey and others (Dempsey, 1988; Dempsey, Driscoll & Litchfield, 1993) used levels of fine and gross discrimination error during a conceptlearning task to adapt feedback to the needs of learners.

Bangert-Drowns and his associates (1991) organize the findings of previous researchers' investigations of textbased feedback into a five-stage model, describing the state of-, the learner as he. or she is going through a feedback cycle. The model emphasizes the construct of mindfulness (Salomon & Globerson, 1987), described as "a reflective process in which the learner explores situational cues and underlying meanings relevant to the task involved" (Dempsey, Driscoll & Swindell, 1993, p. 38). They describe both behavioral and cognitive operations that occur in learning. To direct behavior, a learner needs to be able to monitor physical changes brought about by the behavior. Learners change cognitive operations and, consequently, activity by adapting it to new information and matching it with his or her own expectations about performance (Bangert-Drowns et al., 1991). These researchers emphasize that:

... any theory that depicts learning as a process of mutual influence between learners and their environments must involve feedback implicitly or explicitly because, without feedback mutual influence is by definition impossible. Hence, the feedback construct appears often as an essential element of theories of learning and instruction (p. 214).

The five stages include (1) the learner's initial state, (2) what search and retrieval strategies are activated, (3) the learner's response, (4) the learner's evaluation of the response, and (5) adjustments the learner makes. A graphic representation of the model by Dempsey and colleagues may be viewed in Figure 32-2.

This model emphasizes the construct of mindfulness, in which activities are exactly the opposite of automatic, overlearned responses. Feedback can promote learning if it is received mindfully. However, it also can inhibit learning if it encourages mindlessness, as when the feedback message is made available before learners begin their memory search or if the instruction is too easy or redundant. The inhibition of learning effect relates to research conducted on processes that "kill" learning (Clark, Aster & Hession, 1987) and presearch availability (Anderson et al., 1971, 1972; Kulhavy, 1977).

These researchers (Bangert-Drowns et al., 1991) examined 40 studies using meta-analytic procedures looking at such variables as type of feedback, timing of feedback, error rates, among others, in terms of their various effect sizes. They report generally weak effects of feedback on achievement. Also, feedback indicating only whether an answer was correct or wrong resulted in lower effect sizes than feedback containing the correct answer. Further, using a pretest within a study significantly lowered effect sizes, as did uncontrolled presearch availability of answers.

Dempsey and his colleagues (Dempsey, Driscoll & Swindell, 1993) point out that the emphasis on mindfulness is an important framework for future research involving text-based feedback. While the studies examined by the Bangert-Drowns et al. (1991) meta-analysis "may be too simple or specific" (Bangert-Drowns et al., 199 1, p. 234), it leads us to believe that future studies should examine feedback in more complex environments that involve higher learning outcomes.

Figure32-2. The state of the learnerreceiving feedback, based on Bangert-Drowns et al., 1991. (From Dempsey, Driscoll & Swindell, 1993.) (From Interactive Instruction and Feedback, P. 40, by J. V. Dempsey & G. C. Sales, eds., 1993, Englewood Cliffs, NJ: Educational Technology.) Copyright 1993 by Educational Technology Publications. Reprinted with permission.