29.2 Information -Processing Approach To Human Cognition

29.2.1 Historical Perspectives

Notions such as seeing with our "mind's eye" or "listening" to our inner "voices" portray an ancient metaphor of a mind with sense organs much like the body. The mind feels pain (e.g., "it hurts me when I think about..."), has a sense of taste (e.g., "I want this so bad I can taste it"), and smell (e.g., "die more I think about this the more it smells"), etc. Moreover, our language reflects specific, organ-based memories, as in "I'll never forget the look on his face or the sound of his voice," or "I can still feel (or smell) it after all these years." Yet, the nature of sensory image processing, storage, interpretation, and generation is not nearly as clear (nor as noncontroversial) as our conversational descriptions would imply.

Images are mentioned in Greek scrolls that date as early as 500 BC. A few hundred years later, a building collapsed during an earthquake. Simonodes, a survivor, related his use of mental images to recreate the seating arrangement at the feast he had been attending in the building. The power of the mind to "see" is exemplified, for example, by authors such as St. Augustine (who refers to inner sight or insight) and Descartes (who believed that during dream states the mind could both see and hear during its "travels").

To understand the current views of these historical concepts, however, it is necessary to take a position on how the human memory system works. For simplicity sake, and to make discussions about modality easier, we have selected the model that began the current rise of cognitive psychology: information processing.

29.2.2 Cognitive Overview

The information-processing approach to human cognition (see 1.4.4, 5.2.4, 28.1.2.1, 30.2) relies on the computer as a metaphor. Gardner (1985) states that cognitive science was "officially" recognized at the Symposium on Information Theory held at MIT in 1956. While Broadbent (1958) published the first model, it was Neisser, in his 1967 book Cognitive Psychology, who synthesized earlier attempts to apply information theory and computer analogies to human learning (see, e.g., Bartlett, 1958; Broadbent, 1958; Miller, 1953; Posner, 1964).

The information-processing approach focuses on how the human memory system acquires, transforms, compacts, elaborates, encodes, retrieves, and uses information. The memory system is divided into three main storage structures: sensory registers, short-term memory (STM), and long-term memory (LTM). Each structure is synonymous with a type of processing.

The first stage of processing is registering stimuli in the memory system. The sensory registers (one for each sense) briefly hold information until the stimulus is recognized or lost. Pattern recognition is the matching of stimulus information with previously acquired knowledge. Klatzky (1980) referred to this complex recognition process as assigning meaning to a stimulus. Unlike the sensory registers, STM does not hold information in its naw sensory form, (e.g., visual---~'icon," auditory--"echo") but in its recognized form. For example, the letter A is recognized as a letter rather than as just a group of lines. STM can maintain information longer than the sensory registers through a holding process known as maintenance rehearsal, which recycles material over and over as the system works on it. Without rehearsal, the information would decay and be lost from STM.

Another characteristic of STM is its limited capacity for information. Miller (1956) determined that STM has room for about seven items (chunks) of information. Moreover, STM has a "limited pool of effort" or cognition capacity (see, e.g., Britton, Meyer, Simpson, Holdredge & Curry, 1979; Kahneman, 1973; and Keff, 1973). This limited pool is assumed to affect everything from decision making to the sizes of visual images that can be processed (e.g., Kosslyn, 1975). Klatzky (1980) defined STM as a "work space" in which information may be rehearsed, elaborated, used for decision making, lost, or stored in the third memory structure: long-term memory.

LTM is a complex and permanent storehouse for individuals' knowledge about the world and their experiences in it. LTM processes information to the two other memory structures and in turn receives information from the sensory registers and STM: First, the stimulus is recognized in the sensory registers through comparison with information in LTM. Second, information manipulated in STM can be permanently stored in LTM.

Perception is an interpretive process involving a great deal of unconscious inference (Helmholtz, 1866, as cited in Malone, 1990). An important characteristic of STM for our purposes is that despite the fact that it can apparently manipulate visual information (e.g., Cooper & Shepard, 1973), phonemic coding is the preferred modality (Baddeley, 1966; Conrad, 1964; Sperling, 1960). Related to this phenomena is that STM apparently treats printed text and spoken words the same: acoustically (e.g., Pellegrino, Siegel & Dhawan, 1974, 1976a, b). Basic research studies not only tend to confirm this treatment but also suggest that while people can remember information as being presented by picture or spoken word, printed text is identified as printed (versus spoken) at about a chance level (Burton, 1982; Burton & Bruning, 1982).

To understand how an individual is able to interpret information, the researcher must first focus on decisions made at each memory storage structure. Within the information-processing model, attention and pattern recognition determine the environmental factors that are processed. A large amount of information impinges on the sensory registers, but it is quickly lost if not attended to. Attention, therefore, plays an important role in selecting sensory information.

Early information-processing models viewed attention as a filler or bottleneck (e.g., Broadbent, 1958). For example, an individual could follow an auditory message across many "ears" (headphones) but could attend to only one message; the rest were filtered out. Work by Cherry (1953, 1957), Moray (1959), and Treisman (1960) indicated, however, that information in an unattended channel (same modality) can penetrate this proposed bottleneck. Cur-rent models (e.g., Shifffin & Geisler, 1973) view attention as attenuation with unlimited capacity for recognition of stimuli coming from different channels at the same time. Recognizing a stimulus in one channel does not disturb the process of recognizing a second stimulus in another channel (Bourne, Dominowski, Loftus & Healy, 1986). Attention is conceived of as being a very limited mental resource (Anderson, 1985). It is difficult to perform two demanding tasks at the same time. While all information is registered by the sensory registers, only information attended to and processed to a more permanent form is retained. Bruner, Goodnow, and Auston (1967) stated that a person tends to focus attention on cues that have seemed useful in the past. Pattern recognition enables the individual to organize perceptual features (cues) so that relevant knowledge from LTM is activated. In other words, recognition is attention (Norman, 1969). Pattern recognition integrates information from a complex interaction that uses both bottom-up and top-down processing (Anderson, 1985). Bottom-up processing is the use of sensory information in pattern recognition. Top-down processing is the use of pattern context and general knowledge. In fact, attention is assumed to use both processes; that is, it is interactive (Neisser, 1967). Once relevant information is activated from LTM, the individual focuses attention on the relevant stimulus and brings it into the working memory (STM).

Long term memory contains large quantities of information that have to be organized efficiently so they can be effectively encoded, stored, and retrieved. These three processes are interdependent. For example, the method of presentation determines how information is stored and retrieved (Klatzky, 1980). Encoding is related to the amount of elaboration and rehearsal conducted in STM. Elaboration uses information received from LTM after the stimulus is recognized. As new information is compared to the old and manipulated information, it is either added or subsumed into the existing schema, then encoded in LTM (Anderson, Greeno, Kline & Neves, 198 1). These schema or "set of past experiences" are the cognitive structures that, when related to new information, cause meaning (Mayer, 1983, p. 68). As information is restructured and added, new structures are formed that result in new conceptualizations (Magliaro, 1988). These knowledge structures combine information in an organized manner. Evidence for memory storage indicates that representations can be both meaning based and perception based. Retrieval of information is also an active process. Infomation is accessed by a search of the memory structures. The speed and accuracy of retrieval is directly dependent on how the information was encoded and the attention being given to the stimulus. To be recalled from LTM, information must be activated. The level Of activation seems to depend on the associative strength of the Path. The strength of the activation increases with practice and with the associative properties (Anderson, 1985).

Imagery theorists obviously make a distinction between the codes used for images versus verbal information. Paivio (1971, 1986) developed the dual-code model (see 16.2.1), which stated that the two types of information (verbal and imaginal) are encoded by separate subsystems, one specialized for sensory images and the other specialized for verbal language. The two systems are assumed to be structurally and functionally distinct. Paivio (1986) defined structure as the difference in the nature of representational units and the way in which these units are organized into higher-order systems. Structure, therefore, refers to LTM operations that correlate to perceptually identifiable verbal or visual objects and activities.

It is important to note that Paivio defines his two systems very broadly. An image can be a picture or a sound or even perhaps a taste, while the verbal store, on the other hand, is construed broadly to mean a language store (Burton & Bruning, 1982). In Paivio's (1971) words, image refers to:

concrete imagery, that is, nonverbal memory representations of concrete objects and events, or nonverbal modes of thought (e.g., imagination) in which such representations are actively generated and manipulated by the individual. This will usually be taken to mean visual imagery, although it is clear that other modalities (e.g., auditory) could be involved, and when they are, this must be specified. Imagery, so defined, will be distinguished from verbal symbolic processes, which will be assumed to involve implicit activity in an auditory-motor speech system (p. 12).

Functionally, Paivio's two hypothesized subsystems are independent, meaning that either can operate without the other, or both can work parallel to each other. Even though independent of one another, these two subsystems are interconnected so that a concept represented as an image can also be converted to a verbal label in the other system, or vice versa (Klatzky, 1980). Paivio is very explicit, however, about the power of images: While words that can be imaged may be, images (and presumably all concrete sensory input) that can be translated will be, automatically. Paivio argues that this is why pictures are often remembered better than verbal information (Pressley & Miller, 198j). ,

Dual-code theorists accept that mental images are not exact copies of pictures but instead contain information that was encoded from a sensory event after perceptual analysis and pattern recognition (Klatzky, 1980). It is thought that the images are organized into subunits at the time of perception (Anderson, 1978). Paivio (1986) further explained that mental representations have their developmental beginnings in perceptual, motor, and affective experience and are able to retain these characteristics when being encoded so that the structures and the processes are modality specific. For example, a concrete object such as the ocean would be recognized by more than one modality-by its appearance, sound, smell, and taste. Therefore, a continuity between perception and memory as well as behavioral skills and cognitive skills is implied (Paivio).

There are, however, the same limits on imaginal processing that we see throughout the information-processing model. The concept of limited space was demonstrated by Kosslyn (1975), who asked students to visualize two named objects and then to answer questions about one of the objects. Students were slower to find parts that were next to an elephant than to find those next to a fly. STM for visuals appeared to have a processing limitation. Large objects like elephants (or even very large flies) "fill up" the system and slow it down. Retrieval of visually coded material also differs from other forms of internal representation. As previously stated, information is available simultaneously rather than by a sequential search and can be located by template or by an unlimited-capacity parallel search (Anderson, 1978).

Dual-coding theory can account for our personal impression of having images. The theory is often supported by research studies that conclude that individuals have a continuous and analog ability to judge space from images, in at least some cases (Kosslyn, 1975), and finally for studies that indicate strong visual-memory abilities. Paivio's theory is also able to effectively support the recurrent finding that memory for pictures is better than memory for words (Shepard, 1967), otherwise known as the pictorial superiority effect (Levie, 1987). Imagery theories have been used by researchers to construct and test hypotheses on learning from graphics (Winn, 1987) and seem a fruitful heuristic source for multimodality research in the future.

In terms of simple recognition, text modality detail does not seem to be important. Nelson, Metzler, and Reed (1974), for example, varied visual representations of the same scene from nondetailed drawings to photographs and compared recognition for the visuals versus text descriptions. As we would expect, pictures were superior in recognition tests, but there were no differences among the detail levels used. For recall, however, detail is important in at least two ways. Mandler and Parker (1976) showed that the location of detail elements are best recalled if they are organized in a meaningful way. Thus, for example, graphic elements of classroom items that are placed in their "usual" locations are superior to the same elements when they are not organized in a meaningful manner. Obviously, "meaningful" reflects prior knowledge, including culture. In a related way, specific expertise impacts memory for visuals. Egan and Schwartz (1979) demonstrated that skilled electronics technicians showed superior recall for circuit diagrams compared to novices, as long as the diagrams made "sense," that is, were organized in a meaningful manner.

Images can also be used to organize incoming information. The classic demonstration of this use of visuals to "make sense" of subsequent textual information is Bransford and Johnson's (1972) Balloons passage. In their study, people found text without the visuals (or the visual following the text) to be difficult to comprehend and remember relative to the same text following an organizing visual. A related effect, "priming" (see, e.g., Neely, 1977; Posner & Snyder, 1975), has been demonstrated with text. Basically, a categorical prime, such as a bird, facilitated access to a specific bird, such as a robin. Conversely, an incorrect categorical prime inhibits access. A representative of the category in whatever modality should produce a similar effect (Miller & Burton, 1994).

Theory, basic research, and applied research predict and support the efficacy of images (and instructions to image) in learning and memory. Yet, images are prone to the same processes (and problems) that affect all aspects of the human system: distortions from "reality." We assume that human sensation is about the same for all of us. When confronted with a visual stimulus, we assume that our rod, cones, optic nerves, and so forth react about the same. Perceptually, however, we do not see the same things. We extract (and create) meaning from visual stimuli just as we do from text. Therefore, our prior experience, inferences, expectations, beliefs, physical state, and other factors determine what we see as surely as the stimulus before us. A similar process operates when we recall an image from memory: We reconstruct from our constructed images. Naturally, like memory for text, we forget details (Miller & Burton, 1994).

Finally, where there are gaps, we unconsciously fill them. As you will see in later chapters, images are effective for connecting items to be remembered and, if the level of detail is correct, for learning new facts and relationships. However, these tasks are rather low level and rote. In general, unless images are entrained to the point of pattern recognition, we can assume that the human memory system deals with images as it deals with text: generally or prototypically. The system is great at "gist" or meaning and poor at specifics. Thus, images may work "better" than text in many applications, but they probably do not work differently (Miller & Burton, 1994).