4 The Knowledge Survey: A Tool for All Reasons

The Need for a More Direct Assessment of Learning

Knowledge surveys address learning in specific detail, as opposed to the common global summative item: “Evaluate this course as a learning experience.” Summative student ratings are frequently used to evaluate professors for rank, salary, and tenure. Summative ratings are measures of students’ general satisfaction, which results from a mix of content learning, cognitive development, and affective factors. The most thorough research on the relationship between content learning and summative ratings produced a correlation coefficient derived by meta-analysis of r = 0.43 (Cohen, 1981). That relationship is strong enough to prove that, in general, students learn more from professors whom they rate highly, but it is too weak to allow learning in any single class to be assessed by inference based on student ratings of the class, the individual professors who taught it, or the overall learning experience. Researchers such as Feldman (1998) have shown that educational practices that produce the most learning are not exactly the same as those that produce the highest student ratings. Because student ratings alone cannot reliably reveal learning outcomes produced by individual instructors or courses, good assessment requires a more direct assessment of learning. We believe knowledge surveys, properly used, serve this purpose. Knowledge surveys were created to allow instructors to prove that their courses produced specific changes in students’ learning and to disclose the detailed content of a course to students (Nuhfer, 1993, 1996). Tobias and Raphael (1997) grasped the usefulness of the tool and included it as a best practice in their book. Knowledge surveys do achieve their original purposes, but employment of this tool encourages further instructional improvements that are discussed in this chapter.

Knowledge Surveys that Enhance High-Level Thinking

When instructors develop sophistication with knowledge surveys, they can code each item according to levels of reasoning, such as those of Bloom (1956) (see Tables 4.1 and 4.2). This coding allows an instructor to see the course as a profile of levels of inquiry addressed (Figure 4.2) and verify that the course plan accords with the original intentions of the instructor and the purposes of the course. As noted by Gardiner (1994), most testing pinpoints low-level content knowledge, even though most instructors really aspire to teach some higher-level critical thinking. If an instructor includes critical thinking as her or his course objective, and the detailed knowledge survey reveals overt emphasis on mere memorization, then recognizing this discrepancy is empowering. Such revelations allow course redesign that will ensure the desired critical thinking experiences.

Interpretations of graphs like Figure 4.2 require care. Although 54% of knowledge survey items for this lower division course addressed the lowest Bloom levels, this does not imply that 54% of course time was spent addressing lower levels. In fact, high-level open-ended questions require disproportionately more time to confront than do simple recall knowledge items. Figure 4.2 verifies that students encountered no less than (they actually encountered more) 28 analytical, 16 synthesis, and 16 evaluation challenges in this course. Yet, students can do synthesis and evaluation poorly (see Nuhfer and Pavelich, 2001), thus operating at the lower levels of the Perry (1999) model or the reflective judgment model (King & Kitchener, 1994). If one wants to use such a presentation like Figure 4.2 to prove that students truly mastered some critical thinking in this course, then one must disclose the rubrics used to evaluate students’ answers to representative high-level items. (Rubrics are the disclosed criteria that instructors use for evaluation of a test question or project.) When instructors present rubrics along with evidence like Figure 4.2, the assertion that students addressed high-level challenges with high levels of sophistication is hard to refute. Table 4.2 near the end of this chapter provides rubrics for a few such questions.

Figure 4.2 Levels of Thinking Represented in a Knowledge SurveyNote: Data are from the same environmental geology class and knowledge survey shown in Figure 4.1, but here have been rearranged to present the course outcomes as a profile of levels of educational objectives (Bloom, 1956) encountered in the course. The graph reveals that reduced learning in the final two weeks (Figure 4.1) occurred in material typified by the lower Bloom’s levels.

Does Increased Confidence Reveal Increased Learning?

The video, Thinking Together: Collaborative Learning in Science (Derek Bok Center, 1993), from Harvard University shows a brief paired-learning exercise in Dr. Eric Mazur’s introductory physics class. The students confront a problem, answer a multiple-choice question about the problem, and rate the confidence that their own answer is correct. The students then engage in paired discussion to convince the partner that one’s own answer is correct. Thereafter, the entire class debriefs and summarizes results. In that video, a bar graphic displays a positive relationship between confidence and correctness. Mazur (personal communication, November 29, 2001) revealed that there is no strong correlation initially between correctness and confidence to answer correctly, but overall class confidence rises significantly as discussion of the problem proceeds and greater understanding results.

We have observed overt overconfidence expressed on knowledge surveys given at the start of a class by one or two students enrolled in about 10% of courses we have examined. Such students may appear in any level course, but are more frequently freshmen with undeveloped self-assessment skills. However, the aberrations contributed by occasional individuals never affect a class average in a significant way. To date, every class average we have examined has invariably been a very good representation of that class’s knowledge and abilities. The confidence rating to address content does indeed parallel ability to address it in an exam situation (Figure 4.3), especially when a teacher designs effective teaching/learning experiences for the topic. Learning gaps revealed in post-course surveys reflect strong concurrence among nearly all students of little confidence to address an item. Interviews with professors who use knowledge surveys show that they nearly always know what produced a so-designated gap. Often a gap reveals a topic not covered or one that was inadequately addressed (see caption, Figure 4.1).

The best results occur when survey items clearly frame specific content, and students take the survey home to complete it with plenty of time for self-reflection. Instructors should refer frequently to the survey throughout the course and remind all students to monitor their progress. Like any tool, a user’s ability to be effective with it increases with practice and experience. Based on our work with individual faculty, the first knowledge survey an instructor creates tends to consist almost entirely of recall and comprehension items, even though the instructor aspires to instill critical thinking skills in the course. Some items may not be so clearly stated as to elicit students’ focused reflection upon the intended knowledge or skill. In fact, lack of clarity or direction may contribute to the isolated incidents of overt overconfidence noted above. Initial surveys tend to be constructed without deep reflection on more central unifying concepts, global outcomes, or the function of the course in a larger curriculum. A developer can truly help an instructor to improve a course by making queries about course purpose and instructor’s intentions for outcomes. This helps the instructor to frame such outcomes as survey items and to organize lower-level items as a way to develop larger general outcomes.

Once the content and learning goals have been clearly laid out and organized in writing, the instructor can more clearly concentrate on what kinds of pedagogies, learning experiences, and rubrics (Table 4.4) will best ensure content learning and intellectual development. The overall result of deliberation and practice that begins with knowledge surveys is a course with increasingly sophisticated organization, good development of learning outcomes, and clear conveyance of a plan for learning to students.

Figure 4.3 Comparison of Normalized Reported Knowledge and Final Examination Results from an Astronomy Class(after Knipp, 2001)Note: Students were slightly overconfident about their knowledge on several of the questions from the first part of the course and less confident about their knowledge level of material toward the latter part of the course. The latter portion of the course covered material that was conceptually and mathematically new to the students.

Knowledge Surveys Promote Preparation and Organization

Feldman (1998) used meta-analyses to tease apart the instructional practices that produce learning as opposed to those that produce high ratings of student satisfaction. He discovered that the most important instructional contribution to learning was the instructor’s preparation and organization of the course. However, this practice ranked only sixth in importance in gaining high ratings of satisfaction. Satisfaction depended much more strongly on the professor’s enthusiasm and stimulation of interest. The National Survey for Student Learning (Pascarella, 2001) thoroughly supports Feldman’s findings about student learning. The use of knowledge surveys as a best practice can be justified on the basis of the dominant evidence: Nothing a teacher can do to produce learning matters more than the efforts put into course preparation and organization.

The process of making a knowledge survey—laying out the course in its entirety—considering the concepts, the content, levels of thinking, and questions suitable for testing learning of chosen outcomes is extremely conducive to organization and preparation. Further, the act of disclosing the course in its entirety is akin to providing students with a detailed road map to the course. Students know the content, the sequence in which it will come, and the levels of challenge demanded. Erdle and Murray (1986) confirmed that students perceive disclosure as moderately important in physical science and humanities courses, and more important in social science courses. Simply put, the research shows that if one carefully decides what one is going to teach and conveys this clearly to students, then students are more likely to achieve the learning outcomes desired.

Knowledge Surveys Boost Practice of the Seven Principles

Chickering and Gamson (1987) summarized the outcomes of a Wingspread Conference in which attendees expressed consensus by drafting “Seven Principles of Good Practice” for succeeding with undergraduates. Developers know well that getting some faculty to adopt progressive pedagogical practices can be difficult. Most faculty relate better to content learning than to the practices that might better produce learning. The fact that knowledge surveys reach faculty where they are—with an obvious relationship to the content they value—makes them a good tool developers can use to introduce faculty to thinking toward improved practice. Knowledge surveys predispose a class to making use of all seven principles.

1) Good practice encourages student-faculty contact. One reason that few students ever come to a professor’s office for help is because they are often unaware of what they do not know or understand. Once students confront a knowledge survey and understand its use, students can more clearly see their need to seek help. A pre-course survey can also reveal which students have the most confidence with the material and which do not (Figure 4.4). Such insights permit faculty to know something about each student and make them aware of each student’s possible needs for extra assistance before the class is even underway. Knowledge surveys also indicate which individual students really have the prerequisites needed to engage the challenges forthcoming in the course.

Figure 4.4 A Pre-Course Knowledge Survey from a Music Theory ClassNote: Here the data is used to identify students (names have been changed) by confidence level at the start of the course. Data here are averages of responses to a 50-item survey plus a separately calculated class average (lighter bar).

2) Good practice encourages cooperation among students. Knowledge surveys help to impart several of the five basic elements of cooperative learning (Johnson, Johnson, & Smith, 1991; Millis & Cottell, 1998). Individual accountability includes the critical ability of individuals to be able to accurately assess their own level of preparedness or lack thereof. When a course has detailed disclosure, students more readily know when they have deficiencies, making them more receptive to engaging in positive interdependence, promotive interaction, and group processing to overcome deficiencies. Pre-course knowledge survey results such as the kind shown in Figure 4.4 provide the information needed to form heterogeneous cooperative groups composed of members with known, varied abilities.

3) Good practice encourages active learning. Knowledge surveys can be a powerful prompt for addressing high-level thinking. When students receive both good example items and a copy of Bloom’s taxonomy, they can make up their own new test questions for a unit. Such questions will address the material and appropriately high Bloom levels. A simple assignment could be:

When students know the important concepts and outcomes desired in a unit lesson, groups can become resources, thus structuring peer teaching into a course while assuring quality outcomes.

4) Good practice gives prompt feedback. When a detailed knowledge survey is furnished, it allows students to monitor their progress through the course. One of the first signs that an instructor has produced a survey of good quality is a query from a student: “Will I really be able to learn all these new things?” Prompt feedback delivered by a survey is the students’ own continuous tracking of knowledge gains as the course unfolds. When students can create their original test questions that address the material at a respectable level of thinking, they have reached what probably constitutes adequate preparation and understanding. If any student must be absent, the survey immediately discloses the material missed and reveals to an extent the work required to master it.

5) Good practice emphasizes time on task. Full disclosure at the start of a course allows timely planning and study. A review sheet given out before an exam will not reveal to students what they do not know in a timely manner, and it will promote mere cramming rather than planned learning. Faculty who plan courses well and disclose them at the detail of a knowledge survey quickly discover that the survey keeps them honest. A student who is following his or her progress will invariably say, “Excuse me, Dr.———, but it seems like we didn’t address that item #——— about———.” Perhaps that class launched into a discussion that carried more value than the item skipped and an expressed item of more value now replaces the original item. On the other hand, perhaps the class launched into a digression and the instructor forgot to address a particularly important point that he or she must yet address. When a class inadvertently strays off track, knowledge surveys reveal whether straying resulted in any important omissions. Surveys also require students to engage material repeatedly. Some of the earliest research on cognition deduced the benefits of time spent in repetition to learning. The use of knowledge surveys ensures at least two additional structured engagements with the entire course material.

6) Good practice communicates high expectations. Students sometimes complain that instructors teach at one level and test at a more challenging level. Knowledge surveys offer the opportunity to detail, in a timely manner, the level of challenge that students should expect. The materials needed to build a rudimentary knowledge survey already reside in most professors’ computers—in the past examinations, quizzes, and review sheets they provided the last time they taught the course. The only work required is to arrange items from these in the order of course presentation. Students appreciate the focus that a knowledge survey brings to the study process, and they will rise to expectations conveyed in a survey, particularly if instructors assert that some of the higher-level items will likely appear on a final exam.

7) Good practice respects diverse talents and ways of learning. One of the best ways to address diverse learners is to be certain to present and engage materials in a variety of ways, in particular ways that make sense in terms of how the brain operates in the learning process (Leamnson, 1999). When one actually has a blueprint of content and levels of thinking that one wants to present, it quickly allows one to ask “What is the best way to present this item, then the following item?” Without such a plan, one can too easily end up lecturing through the entire course, even when the desired outcomes literally scream for alternative methods. A detailed plan of outcomes will obviate a correlative plan for reaching these (see “Pedagogies” Table 4.4).