8.8 An Ecological Approach to Understanding Media

As mediated perception extends and substitutes for direct perception, so do the affordance properties of mediated environments extend and substitute for the affordance properties of real environments. End users therefore must be guided by implicit conventions or explicit instructions that help them to select or construct MIROS that can substitute for the missing affordances.

8.8. 1 Analogs for Acting

Every media technology from book to video to computer simulation imposes profound constraints on representation of real or imaginary worlds and requires trade-offs as to which aspects of a world will be represented.

A topographical map, for instance, represents three-dimensional land forms on a two-dimensional surface. Such maps are constructed through electromechanical processes, aided by human interpreters, in which numerous aerial photos taken from different angles are reconciled to yield a single image. This process captures some of the normal affordance properties available to the aerial observer--shadings, textures, angles, and occlusions, for instance--as well as the ways the values for these properties change in response to observer movement. The original affordance information--the climbability and walkability of the terrain, for example--is re-presented as a flat image that indicates elevation through contour intervals and ground cover or other features through color coding. Much of the information detected by the aerial observer is thus available vicariously to map viewers, provided that the viewers can use the affordances of the map--contours, color coding, legends, grids--in concert with their mental models of map viewing to imagine the affordances of the actual terrain. Thus,

Media + MIROS== Realia

Many activities of everyday living are generally intuitive and relatively automatic because perceived affordances can be immediately exploited with minimal mental effort and because consequences can be immediately perceived. This tight linkage of action and perception in real time characterizes the praxis that enables much unschooled learning.

The collapsed affordance structures of many mediated environments, however, transform the means by which

users can exercise their powers of perception, mobility, and agency: Scanning a photo is not the same as scanning a scene, although ecological psychologists will argue that much is similar about the two acts. The lack of certain affordances in mediated representations may even promote reflection by reducing cognitive load: Viewing a scene vicariously through a photo frees one of the need to monitor or respond to immediate events: A topographical map can be read at leisure; there is no need to attend to the immediate passage of land forms under the reconnaissance airplane.

8.8.2 The Importance of Being There (or Not)

Gibson's (1977/1982) partial insights about visual displays remind us that, like other apes, human beings have well-developed faculties for managing information about objects and spaces when that information is derived through locomotor and stereoscopic functions.

8.8.2.1. Depiction. Pictorial representations of complex environments often pose extreme problems for writers of captions or other information about spatial relations. Picture captions also impose on readers task-irrelevant cognitive processing burdens such as referencing figures in the text by cited numbers or hunting through the text of the caption to find relevant descriptions. Inspection of a typical illustration and its caption from Gray's Anatomy (Gray, 1930, p. 334) makes it clear that, lacking information about the hypothetical viewpoint of the artist, and lacking information about the more subtle relationships between the components depicted in the drawing, viewers will be unable to construct a suitable MIROS (Mental-Internal Representation of Situations) to complement the mediated representations (see Fig. 8.3).

Fortunately, anatomists have developed a rich lexicon for describing spatial relationships between viewers of an illustration and the objects portrayed by the illustration. For example, the text description matched to the preceding figure from Gray ~ reads:

The Ligamentum Teres Femoris--The ligamentum teres femoris is a triangular, somewhat flattened band implanted by its apex into the antero-superior part of the fovea capitis femoris; its base is attached by two bands, one into either side of the acetabular notch, and between these bony attachments it blends with the transverse ligament. It is ensheathed by the synovial membrane, and varies greatly in strength in different subjects; occasionally only the synovial fold exists, and in rare cases even this is absent (p. 334).

Using only propositions to tell people about how to construct a MIROS for a three-dimensional structure may be a misappropriation of cognitive resources if better means are feasible--a physical or pictorial model, for instance. The issue is partly a matter of instructional intent: Designers of an anatomy course might decide to use, say, animated 3-D renderings of a situation-with orienting zooms and pans to teach gross structure. If the goal is to teach spatial nomenclature as preparation for dissection through a spatial structure, however, the designers might select a strategy in which there is less emphasis on explicit visual representation of operations and more emphasis on narration. The two approaches are not mutually exclusive.

Figure 8-3. A drawing from Gray's Anatomy (Gray, 1930, p. 334).

8.8.2.2. Photography. Consider the camera as a tool for capturing photographic images: A photograph excludes large quantities of information that would have been available to bystanders at the scene who could have exercised their powers of exploratory action, ranging from gross-motor movements to tiny adjustments in eye lenses. In capturing the image, the photographer chooses to take the picture from a single viewpoint in space and time--a viewpoint that is but one of a number of possible viewpoints that are in principle infinite.

Even though a subsequent user of the photograph might be able to manipulate the position and orientation of the photo itself, take measurements of the objects as they are depicted, and engage in selective visual exploration, such exploration is an imperfect surrogate for ambulatory perception at the original scene. Both the user's perception of the depictions in photographs and the user's interpretation of these depictions require prior knowledge about the conventions of photographic culture as well as knowledge of the ways in which photography distorts situational factors such as orientation, distance, texture, hue, contrast, and shadows. The user's ability to perceive and interpret the photo may be enhanced if he or she can integrate information in the photo with adjunct verbal information such as captions, scales, and dates that, however inadequately, support development of MIROS complementary to the actual situation.

8.8.2.3. Cinematography. Although cinematography can record the transformation of imagery that results from camera movement through multiple viewpoints, cinematographs, like photographs, evoke mediated perceptions in the end user that are fundamentally decoupled from the kind of action that would have been possible in the actual situation. In other words, attention is partially decoupled from intention: The viewer can attend to changes in imagery but is unable to effect changes through exploratory action. Several studies have shown, in fact, that interfering with proprioception and ambulation retards adaptation by mammalian visual systems. For example, when experimenters require human subjects to view their surroundings through an inverting prism apparatus, the subjects adapt to the upside-down imagery after several weeks, achieving a high degree of functionality and reporting that their vision seems "normal" again (Rock, 1984). This adaptation does not occur, however, if the experimenters restrict the subjects' tactile and proprioceptive experience or their ability to engage in self-controlled locomotion.,

In a study more directly related to use of media in education and training, Baggett (1983) found that subjects who were denied an opportunity to explore the parts of a model helicopter were less effective at a parts assembly task than subjects who explored the parts in advance, even though both types of subjects saw a videotape depicting the assembly process before, performing the task.

Conventional cinematography substitutes dynamism for dimensionality by recording the way perspective views of objects transform in response to camera movements, collocating information on a single plane. Cinematic dynamism provides information about perspective, which always implies a single point of view or movement along a path. More importantly, cinema portrays invariant structure, t e environment of many observers,

through such devices as multiple points of view, glimpses of the surrounding of a scene (establishing shots).... It is important in editing a film to splice sequences in such a way that this invariant information is not destroyed by the sequence. For example, one must avoid splicing together two views of the same scene taken from opposite parts of the layout, for this would make the left side of the first sequence suddenly transform--without any information about the observer's path--into the right side. Ecological optics, with its emphasis on flow, might very well provide a scientific basis for the empirical, trial-and-error practices of film editing (Reed, 1988, p. 291).

8.8.3 carousing Multivariate Data

The problems of cinematography reflect the central challenge for authors and designers of most media products: How to collapse multivariate data into flat, two-dimensional displays while optimizing the ability of the end user to exploit the affordances of the displays.

As Tufte explains in Envisioning Information (1992), techniques for collapsing multivariate data involve constraints as well as opportunities. On the one hand,

. . . nearly every escape from flatland demands extensive compromise, trading off one virtue against another; the literature consists of partial, arbitrary, and particularistic solutions; and neither clever idiosyncratic nor conventionally adopted designs solve the inherent general difficulties of dimensional compression. Even our language, like our paper, often lacks immediate capacity to communicate a sense of dimensional complexity. Paul Klee wrote to this point: "It is not easy to arrive at a conception of a whole which is constructed from parts belonging to different dimensions. And not only nature, but also art, her transformed image is such a whole.... For with such a medium of expression, we lack the means of discussing in its constituent parts an image which possesses simultaneously a number of dimensions" ( p. 15).

On the other hand, as Tufte richly illustrates, thetrade-offs necessary to successful compression of a data set with four or five variables, such as a map with an integrated train schedule, can work to the end user's advantage if the sacrificed data would have been confusing or superfluous.

8.8.4 Media and MIROS

To describe the evolutions or the dances of these gods, their juxtapositions and their advances, to tell which came into line and which in opposition, to describe all this without visual models would be labor spent in vain.--Plato, The Timaeus

Regardless of the medium and whether its representational constraints affect spatial and temporal dimensions or other properties such as form, color, and texture, authors of mediated representations must always sacrifice options for exploratory action that would have been available to unimpeded observers or actors in the represented situation. Media cannot represent realia in all their repleteness. Therefore, what is critical is this: that enough information be provided so that users can construct useful actionable mental models according to their needs and goals.

The short film Powers of Ten (C. Eames & R. Eames, 1977/1986) offers another neatly constrained example of language as an aid to interpreting mediated representations. Created by the office of Charles and Ray Eames to help viewers grasp "the relative size of things in the universe," Powers of Ten opens with a viewpoint somewhere in the dark void of intergalactic space, initiating a trip that ends in the nucleus of a carbon atom, 91/2 minutes later in Chicago.

Such a visual experience would be meaningless for most viewers without an audio narration about how to interpret the rapidly changing imagery?which includes diverse depictions ranging from galaxies, to the solar system, to Lake Superior, to a cell nucleus. The book version of Powers of Ten (Philip Morrison & Phylis Morrison, 1982) displays 42 frames from the film, supplemented by elaborative text and supplementary photos. The authors use a set of "rules" to describe the film's representation of situations, including propositions such as:

Rule 1. The traveler moves along a straight line, never leaving it.

Rule 2. One end of that line lies in the darkness of outermost space, while the other is on the Earth in Chicago, within a carbon atom beneath the skin of a man asleep in the sun.

Rule 3. Each square picture along the journey shows the view one would see looking toward the carbon atom's core, views that would encompass wider and wider scenes as the traveler moves further away. Because the journey is along a straight line, every picture contains all the pictures that are between it and the nucleus of the carbon atom....

Rule 4. Although the scenes are all viewed from one direction, the traveler may move in either direction, going inward toward the carbon atom or outward toward the galaxies....

Rule 5. The rule for the distance between viewpoints [is that]... each step is multiplied by a fixed number to produce the size of the next step: The traveler can take small, atom-sized steps near the atom, giant steps across Chicago, and planet-, star-, and galaxy-sized steps within their own realms (pp. 108-10).

Morrison rules might be taken as an invitation to propositional reasoning. Yet the rules can also be usefully construed as instructions for constructing a MIROS that complements and partially overlaps the work of representation carried out by the film. Rule 2, for example, provides a framework for the reader to imagine moving back and forth on the straight line connecting the starting point (outermost space) and ending point (carbon nucleus), thus substituting for the action of the imaginary camera dollying across outer and finally inner space. Rule 3 describes the way in which each square picture encompasses a wider or narrower scene.

Rules 2 and 3 can also be directly perceived in the film itself by attending to the symmetricalness of image flow as various objects and structures stream from a fixed center point and move at equal rates toward the edge of the visual field. The film also indicates movement by depicting changes in the texture gradients of star fields and other structures. Such cues to both movement and direction epitomize the appropriation by filmmakers and other media producers of visual processing capabilities that are widespread among vertebrates, and as common among human beings as a jog on a forest trail or a drive down a two-lane highway.

What cannot be obtained through direct perception from either the film or the photos, however, is information indicating deceleration of the hypothetical camera as it dollys towards Earth. Rule 5, which concerns the logarithm governing the speed of motion, cannot be perceived directly because (a) the camera motion simulates a second-order derivative (deceleration rather than speed), and (b) the objects flowing past the camera are largely unfamiliar in everyday life and therefore have little value as scalars.