24.11 Conclusions

We have presented a strong rationale for the application of cognitive tools in education, described a number of alternative approaches to using cognitive tools, and provided evidence that supports the uses of these cognitive tools in specific contexts. Support for some cognitive tools, such as programming languages, was found to be inconsistent. We also reported that very little research has been done to investigate the premise that cognitive tools based on common software such as databases and spreadsheets have beneficial effects on the development of higher-order thinking skills.

The overall finding that (1) learners develop criticalthinking skills as authors, designers, and constructors of knowledge and (2) learn more in the process than they do as the recipients of knowledge prepackaged in educational communications presents a major challenge for researchers in our field. For starters, this finding throws into question the value of much of the research reported elsewhere in this handbook. The findings of traditional educational communications research, already narrow in scope and limited in generalizability, may ultimately have little or no relevance in a world transformed by constructivist learning theory.

24.1 1.1 Future Research with Cognitive Tools

If nothing else, the research described in this chapter illustrates the need for sustained research agendas regarding the cognitive effects of cognitive tools, such as the studies carried out by Harel (1991), Kafai (1995), and other associates of Seymour Papert at MIT, rather than the isolated short

An increasing number of software designers, cognitive scientists, and educators have come to the view that experience is actively constructed and reconstructed through direct interaction with the world, and that, indeed, knowledge is experience. According to this view, a learner is not an empty vessel to be filled, or a passive listener to be filled in. Knowledge is not a mere commodity to be transmitted from one person to another. It is not an entity to be emitted at one end, encoded, stored, retrieved, and reapplied at the other. The conduit metaphor is progressively fading away, and is being replaced by the more recent toolmaker paradigm.... Children are perceived as the active builders of their own cognitive tools, comprising both mental capacities and external mediations that prolong those mental capacities. Constructivism is in the air ... (pp. 13, 14).

We share in Ackermann's enthusiasm for these changes, and also in the caution she issues later in her paper that "'Hands-on' won't do without 'heads-in"' (p. 15). We view cognitive tools as affording learners unparalleled opportunities for heads-in learning within constructivist learning environments. Cognitive tools, as described in this paper and elsewhere (cf. Jonassen, 1996), represent a vehicle for both educational restructuring and a research agenda that will have significant impact on the restructuring process. We invite others to join us in this quest. To further that quest, we would make some recommendations about methodologies that should be used the:

24.11.1.1. Multiple Assessments. The cognitive processes engaged by cognitive tools are complex and cannot be adequately assessed using a single type of measuring device. We recommend assessing the products of using cognitive tools as evidence of the thinking engaged by them. Criteria for evaluating those outcomes have not been verified, though many are suggested by Jonassen (1996). For instance, for evaluating semantic networks that are produced by students, a researcher might assess the:

• Number of nodes (breadth of the net)
• Number of instances (extent of the net)
• Ratio of instances to concepts (integratedness or embeddedness of concepts)
• Centrality of each node
• Depth (hierarchicalness) of the net
• Number of links (parsimony or economy of connections)
• Consistency in use of links
• Number of "dead-end" nodes (linked to only one other concept)
• Ratio of the number of links to the number of nodes

Other products of learning might include traditional tests (objective form), essays (look at the use of structural knowledge in the essays), speeches and presentations, and, most importantly, measures of transfer of learning. Cognitive tools should benefit learning transfer, so it is important to include assessments of problem solving in different domains or contexts, as well as in the domain being studied. These measures must assess the ability of learners to solve original problems, diagnose situations, draw implications or inferences from problem situations, or predict the result of changes in any problem situation.

In addition to assessing products of learning, it is important to assess the process as well. This can be accomplished by observing students as they work with cognitive tools and assessing variables such as effortful time-on-task, level of collaboration, or creativity. Research focused on how learners construct mental models may be especially fruitful (Jih & Reeves, 1992; Seel & Dinter, 1995). We recommend carefully analyzing the mix of methods employed by Lehrer (1993), Lehrer et al. (1994), and Spoehr (1992, 1993, 1994) in their investigation of learning from producing hypermedia knowledge bases. These are exemplary studies.

24.11.1.2. Qualitative Methodologies. In order to assess the complexities and subtleties of knowledge construction, it is essential to use qualitative as well as quantitative assessment strategies. It is impossible and even inappropriate to hypothesize all of the cognitive outcomes of using cognitive tools. 'Me processes are too rich and unpredictable. Qualitative methodologies are elaborated in Chapters 40 and 41.

24.11.13. Meaningful Contexts and Assignments. Knowledge that is acquired in classrooms is "inert' in part because the purpose and context for learning such knowledge often has no relevance to learners whatsoever. The use of cognitive tools will likely result in greater learning if they are used in the context of solving some kind of problem that is meaningful to the learners. However, the meaningfulness of the context may also provide a researchable variable. Comparing the effects of cognitive tools used in a meaningful context with the use of cognitive tools as an adjunct memorization aid might provide some illuminating results.

24.11.1.4. Multiple Evaluations. In addition to using in multiple-assessment methods, the evaluations made from those assessments should also vary. Comparing the learner's knowledge or interpretations with the expert's or teacher's may be reasonable in some contexts. After all, research has shown that during the process of learning, the learner's knowledge structure increasingly resembles the knowledge structures of the instructors, and the degree of similarity is a good predictor of classroom examination performance (Diekhoff, 1983; Shavelson, 1974; Thro, 1978). Given an instructivist context where the purpose of instruction is to get the learner to think like the teacher, researchers may wish to determine the extent of the learner's knowledge growth and compare it to that of the teacher or expert. In such a context, the products of using cognitive tools can provide evidence in a pretest-posttest fashion of how much the learner has learned.

However, from a constructivist perspective, it is always necessary to assess the learner's unique perspective in addition to making any external comparisons. Understanding the sense that learners make from studying any content domain may be far more informative than comparing the student's knowledge to that of the teacher or expert. With this in mind, it is often useful and illuminating to allow learners to create multiple products (perhaps using different tools) with reference to the same content or problem. Some learners may be better able to express themselves through multimedia, others through more abstract tools such as spreadsheets or databases.