Web Evaluation in the History Classroom: Reconsidering the Checklist

Julie Kimmel

[email protected]

Philadelphia University

.01 Abstract

This article provides a critical examination of the applicability of common web site evaluation criteria advocated by librarians, such as authority, objectivity, and coverage, when teaching students to use the web in history classes. Based on analysis of student responses to web evaluation assignments, the author argues that common evaluation criteria are too broad and general, and tend to reinforce a simplistic understanding of history as a search for correct information. The article closes with suggestions for developing a more discipline-specific approach to teaching web evaluation.

.02 Introduction

Like many history teachers, I have developed a love-hate relationship with the World Wide Web. I love it for providing my students access to an abundance of wonderful primary source materials. [1] I hate the web when information from questionable web sites appears in student work in inappropriate ways. Often, students turn to the web when they are trying to sidestep the hard work of grappling with course materials. At the same time, many students approach the web with a kind of reverence and an utter lack of judgment. [2] How should history teachers deal with these common problems? Forbidding or limiting student exploration on the web is not feasible at a time when many high school and college students are doing their homework in front of a computer connected to the internet. Nor is this an attractive option given that the web now has so much to offer. Instead, we must teach students how to evaluate the great variety of historical material now available for free on the web. This article examines one common tool for doing so—the generic checklist of evaluation criteria—and makes a case for developing a more nuanced, discipline-specific set of criteria.

At Philadelphia University, where I coordinate first year history courses, we recently added "web evaluation" to our list of learning outcomes. Our university librarians supported us in this mission, pointing us toward a resource page on our library web site with several links to checklists of web evaluation criteria. [3] Within a year of adding this new outcome, almost all instructors had incorporated an assignment requiring students to use these checklists. To test the impact of the checklists on student thinking about history on the web, I began examining student responses to my own web evaluation assignment.

What I discovered after collecting and analyzing student work over two semesters is that the generic checklists generated by librarians are too broad and general to help students make good judgments about historical information on the web. More seriously, generic web evaluation criteria seem to reinforce for students the notion that historical research is best understood as a search for "right answers" rather than a search for primary sources that might yield conflicting interpretations of the past. Loosely defined criteria like "accuracy," "reliability," and "credibility" appeal to students’ common understanding of history as a statement of the truth, and work against efforts to introduce students to an understanding of history as a dynamic product of construction and interpretation. A new set of evaluation criteria must address these common biases directly and provide students with a model for applying habits of historical thinking to the web.

.03 The Assignment

To find out more about how students understood generic criteria like "authority" and "reliability," I designed an assignment that required students to write about how they used the criteria. First, students visited our library’s resource page on "Evaluating Web Sites" and selected one of the available checklists, all of which provided variations on the major criteria advocated by Janet Alexander and Marsh Ann Tate in their influential book, Web Wisdom.  [4] The basic format for these checklists is to present four to six criteria with lists of questions intended to help students make sense of the criteria. So, for example, the Cornell University Library guide asks students to asses the "authority" of web sites and then guides their thinking with questions that range from "What is the author’s background?" to "Are there spelling errors or incorrect use of grammar?" [5] The guide to "Evaluating Web Sites" on our library web site also includes a common mnemonic, C-A-R-D-S, which stands for "credibility," "accuracy," "relevancy," "dates," and "sources" along with lists of questions elaborating on each of these criteria. [6]

Next, students used their selected checklist to evaluate a list of sites that I provided for them. Students had to rank these sites in order from "most appropriate for use in a college paper" to "least appropriate for use in a college paper." In the first semester, students were presented with a list of sites about the experiences of women and minorities during World War II. In the second semester, they worked with a list of sites with content about women in the 1920s. In both cases, the list included sites that ranged from high quality university-based collections of primary sources to high school student papers. My choice of the word "appropriate" reflected my view that there are many sites with value, for example, personal tribute sites, that are not typically appropriate for college papers.

Tables I and II list the web sites that students were asked to assess and indicate what percentage of students found each site to be "very appropriate," "somewhat appropriate," or "not at all appropriate" for a college paper. The tables show that students arrived at a clear consensus about the best and worst sites in each collection. The majority of students gave high ratings to university-based sites created by history faculty and low ratings to student reports posted online. At the same time, however, students experienced considerable confusion about how to handle museum exhibits, essays from sites like About.com and Military.com, oral histories collected by high school students, and even primary sources posted on sites like Historymatters. To provide some point of comparison between how students and instructors view the web, I have also shaded the box that corresponds to my own rating of each site. When several of my colleagues completed the assignment, their ratings matched mine, suggesting that history instructors share at least a rough consensus about what sites are appropriate for use in a college paper. The tables indicate some significant gaps between instructor and student assessments of the appropriateness of sites for use in college papers. (In particular, see Table I, item 4 and Table II, items 2 and 4.)

.03a Table I: Evaluations of Web Sites for Paper on Race and Gender in WWII

Web Sites  [7]	Student Ratings of Web Sites
	Very Appropriate	Somewhat Appropriate	Not at All Appropriate	Total Number of Responses
1. Rutgers Oral History Archives: World War II, Korea, Vietnam	81.5% (22)	18.5% (5)		100% (27)
2. Library of Congress, "World War II, Segregation Abroad and at Home," African American Odyssey: A Quest for Full Citizenship	48% (13)	33.5% (9)	18.5% (5)	100% (27)
3. What Did You Do in the War Grandma?	48% (13)	30% (8)	22% (6)	100% (27)
4. Melissa T. Miller, "A Tribute to Military Pioneers," on Military.com	63% (17)	26% (7)	11% (3)	100% (27)
5. Rosie the Riveter and Women World War II Heroes (A collection of documented student papers posted on the University of Arizona web site and no longer available)	37% (10)	22% (6)	41% (11)	100% (27)
6. "African Americans in the Service of their Country," Father Ryan High School Web Site	4% (1)	18.5% (5)	77.5% (21)	100% (27)

.03b Table II: Evaluations of Web Sites on Women in the 1920s

Web Sites  [8]	Student Ratings of Sites
	Very Appropriate	Somewhat Appropriate	Not at all Appropriate	No. of Students Rating Site
1. Susan K. Freeman, "The New Woman," Clash of Cultures in the 1910s and 1920s, Ohio State University Department of History Web Site.	95% (54)	5% (3)		57
2. "Bobbed Hair Blues: A Mexican-American Song Laments ‘Las Pelonas,’" Historymatters.	20.5% (6)	34.5% (10)	45% (13)	29
3. Chicago Historical Society, Flappers, Fashion, N’All that Jazz.	65% (31)	27% (13)	8% (4)	48
4. Jennifer Rosenberg, "Flappers and the Roaring Twenties," About.com	74% (42)	26% (15)		57
5. Flaming Youth(This site, no longer available, appeared to be a personal hobby site devoted to female film stars of the 1920s.)	16% (9)	31.5% (18)	52.5% (30)	57
6. Flappers and Fashion(This page, no longer available, was a student paper posted on the Friends School of Baltimore site)			100% (54)	54

On their own, these tables tell us only how many students picked each site as "very appropriate" for a college paper and not why they made their choices. However, the assignment also asked students to write a complete paragraph about each web site, explaining how they applied the criteria to make a decision about how to rate the site. These explanations contained a wealth of information not only about how students interpreted and applied evaluation criteria, but also about their assumptions about history and what constituted valuable historical information.

.04 The Trouble with Generic Checklists

The trouble with checklists is that markers of quality and value vary from field to field. Unfortunately, generic checklists are not able to take this into account and, as a result, the guiding questions that accompany most checklists are often vague and difficult to apply. How is a student in an introductory course supposed to answer this common query: "Is the information reliable and accurate?"? Research librarian Marc Meola is particularly hard on his fellow librarians when he calls this question "vacuous and unhelpful." [9] Yet, he is right to point out that this type of question is far more appropriate for an expert than a beginner. When confronted with questions like these, not surprisingly, students struggle and often make very superficial judgments. In the sections below, I consider some of the particular problems posed by commonly used web evaluation criteria, namely "authority," "credibility," "accuracy," "objectivity," "reliability," and "coverage."

.05 Checklists and the Issue of Images

Most checklists do not address the issue of visual information in any substantive way despite the fact that web sites typically make greater use of images than traditional texts. In fact, one of the great values of the web for history educators is its ability to easily display visual information. Ironically, however, students tend to approach the web with a preference for text over images. When students wrote about the assigned web sites, they often dismissed images as decorations rather than crucial parts of the argument and exciting bits of primary source evidence. Several students, for example, questioned whether the Library of Congress exhibit, The African American Odyssey: A Quest for Full Citizenship, (Table I, item 2) would be a valuable resource for a college paper. The exhibit is particularly rich in posters and photographs that provide insight into civil rights activism during World War II. Yet one student wrote, "I am not sure if this page is appropriate. . . . there are citations and links for the pictures but there isn’t much information for them. . . ." Students resisted this site because it required them to engage in the art of interpretation to make use of the images from the exhibit. In a similar vein, a web site for an exhibition on flappers at the Chicago Historical Society web site (Table II, item 3) disappointed another student because "it didn’t contain that much text about flappers." Another student also questioned the usefulness of the site and then commented cheerfully that "it has some good aspects though. There are pictures used that go along with the content of the site." This failure to recognize that images are part of the content of the site and a valuable source of historical information was common.

What these results tell us is that students bring to the web many of the same biases and assumptions about history that they bring into our classrooms, namely that history is about dates, facts, and finding "all true information." In the classroom, we have developed approaches for persuading students to let go of these assumptions and to think like historians. Working with primary sources is so central to history education that the A.P. exam uses the Document Based Question to gauge students’ ability to think historically. Our introductory course at Philadelphia University is typical in its emphasis on primary sources. To help students achieve one of the course’s desired learning outcomes, the "ability to analyze primary source material," faculty assign formal essays that require students to interpret primary sources. In-class workshops provide students with opportunities to practice analyzing primary sources. Yet when students leave the classroom and pursue history on the web, their old attitudes about history return to the forefront. So now we must develop tools for teaching students how to take habits of historical thinking learned in the classroom and apply these habits when they search for information on the World Wide Web.

.06 Toward a New Set of Criteria for Evaluating History on the Web

We are still in an age of experimentation when it comes to using the Internet for teaching and learning. Beginning in the mid-1990s, university librarians responded to students’ growing reliance on the Internet in part by creating and promoting checklists for evaluating information on the web. Today, many within the field of academic librarianship are calling for a more nuanced and contextual approach. Meaningful criteria, they argue, must be discipline-specific, and faculty must take on a leadership role in creating such criteria. [17] What would an effective set of criteria for history look like? Based on my analysis of student work, I suggest we keep four goals in mind when designing discipline-specific criteria.

First, we must develop criteria that prompt students to recognize and think about the different genres and purposes of web sites. A student searching for information about flappers online would find material from museum exhibits, digitized primary sources including archival collections, digitized secondary sources including encyclopedia-style sites and electronic textbooks, blended educational sites with interpretive essays and a selection of primary sources, sites supporting historical documentaries, heritage and tribute sites, personal hobbyist sites, student papers, and papers posted on cheat sites. Like the primary sources we study in our classes, these different sites are created for different reasons and are directed toward different audiences. Therefore, they are best used in different ways (except for papers from cheat sites, which have no legitimate uses). Familiarizing students with genres and purposes of history web sites would be a valuable starting point for evaluation.

Second, we must acquaint students with key institutions within the discipline. Many students see The History Channel or textbook writers as the primary creators of history, as bridges to the "truths" of the past. Their teachers, however, are more likely to see universities, historical societies, museums, and national institutions like the Smithsonian and the Library of Congress as institutions that play a key role in supporting historians in their work of interpreting primary sources from the past. Introducing students to the types of institutions that support historical work will help them to place ideas of "authority" and "reliability" into a disciplinary context. At the same time, however, we should not dismiss the democratic nature of the web or the possibility that amateurs may produce history sites with valuable historical information. Rather, we should remind students that they must apply a range of criteria as they evaluate information on the web.

Third, we should use evaluation criteria to highlight the importance of primary sources—including text, image, and audio sources—and the role of the Internet in expanding access to these sources. When I asked faculty and students at the annual meeting of the American Association for History and Computing to identify the criteria they used to evaluate web sites, they repeatedly focused on the issue of sources. Working groups at the conference gave high ratings to sites that provided access to primary sources and that included complete citations for primary and secondary sources. [18] When I surveyed my colleagues at Philadelphia University, they articulated similar priorities. They noted that sites with quality primary sources provided visitors with the opportunity to engage in independent historical thinking. This important message about how and why historians use the Internet could easily be included in a more discipline-specific list of evaluation criteria.

Fourth, we should design evaluation criteria that move students toward a more complex understanding of history as a process of interpretation. If we are really going to help students understand how historians think about the vast sea of information on the web, we must introduce them to modes of thinking and writing that characterize the discipline of history. This is tricky in a discipline that, according to John Higham, "stubbornly resists codification." In his elegant meditation on the history of the profession, Higham described the practice of history as one involving "commitment, enactment, and design." "History is common," he wrote, "but it is also complex." [19] How can we help students learn to recognize and value this complexity? Perhaps we can take advantage of the fact that most students already recognize and feel comfortable with encyclopedia-style sites that appears to present "only facts." We can compare these types of sites to sites that make historical arguments or place primary sources in context and then ask students to notice the difference between the two. Finally, a new set of evaluation criteria for history might ask students whether sites treat the past as "simple" or as "complex" and a "matter for interpretation."

If we are successful, our students will begin to turn to the web to look for primary source materials that they can use to "do history." Once students learn to value sites with primary sources, they will also need guidance in evaluating the quality and reliability of those sources. The History Section of the Reference and User Services Association of the American Library Association has developed a guide to evaluating primary sources on the web that serves as a valuable starting point. The guide prompts students to think about some of the particular issues that arise when researchers use online primary sources, urging students to determine the origin of the document and highlighting the difference between a transcribed document and a scanned image of an original document. [20]

The final step in this process, of course, is the analysis of the primary sources themselves. An excellent guide to analyzing primary sources online comes from the editors of Historymatters. This expansive site is not only home to an impressive collection of nearly a thousand primary sources for American history, it also includes a feature, "Making Sense of Evidence," that profiles how historians use certain types of documents, ranging from songs to diaries to advertisements. The feature provides students with specific questions to consider as they approach particular types of sources as well as examples of how scholars might approach these documents. [21] This site is exemplary because it provides students with access to both the raw materials of history—primary sources—and the practical tools to analyze these materials.

The development of effective web evaluation criteria can be a tool for improving the way we teach history in the digital age. Academic librarians argue that when students learn to assess information in electronic environments, their ability to think critically about traditional texts improves as well. [22] Yet before we can reap this benefit of teaching with electronic resources, we need to develop an approach to web evaluation that reflects the particular values of historians. Web evaluation criteria that emphasize these values can be another way to move students from a naive understanding of history to a more complex view of history and how historical knowledge is produced. As history educators make greater use of the web, we must develop techniques for encouraging students to use the web to "do history" and not merely to "find right answers."

Notes

1. For a thorough discussion of how the "abundance" of information on the web can transform the teaching of history, see John McClymer, "The Internet: Coping with an Embarrassment of Riches," Perspectives Online 41, no. 5 (May 2003): par. 2, http://www.historians.org/perspectives/issues/2003/0305/0305for2.cfm.

2. T. Mills Kelly, "For Better or Worse? The Marriage of the Web and Classroom," Journal of the Association for History and Computing III, no. 2, (August 2000): par. 18, http://hdl.handle.net/2027/spo.3310410.0003.204.

3. "Evaluation Information on the World Wide Web and Citing Electronic Information," Web Resource Guide 3," Paul J. Gutman Library, Philadelphia University, http://www.philau.edu/library/Pubs/webresource3.htm.

4. Janet E. Alexander and Marsha Ann Tate, Web Wisdom: How to Evaluate and Create Information Quality on the Web (Mahwah, NJ: Lawrence Erlbaum Associates, 1999), 116-119. See also: Susan Beck, The Good, the Bad & the Ugly or Why It’s a Good Idea to Evaluate Web Resources, 1997, http://lib.nmsu.edu/instruction/eval.html.

5. "Evaluating Web Resources," Cornell University Library, http://www.library.cornell.edu/newhelp/res_strategy/evaluating/evaluate.html.

6. "Evaluation Information on the World Wide Web and Citing Electronic Information," Web Resource Guide 3," Paul J. Gutman Library, Philadelphia University, http://www.philau.edu/library/Pubs/webresource3.htm.

7. The Rutgers Oral History Archives: World War II, Korea, Vietnam, Cold War, http://fas-history.rutgers.edu/oralhistory/orlhom.htm; What Did You Do in the War, Grandma?, http://www.stg.brown.edu/projects/WWII_Women/tocCS.html; Library of Congress, "World War II, Segregation Abroad and at Home," African American Odyssey: A Quest for Full Citizenship, http://memory.loc.gov/ammem/aaohtml/exhibit/aopart8.html#08a; Melissa T. Miller, "A Tribute to Military Pioneers," Military.com, http://www.military.com/Content/MoreContent1/?file=BH_Tuskegee; Rosie the Riveter and Women World War II Heroes, http://www.u.arizona.edu/~kari/rosie.htm; "African Americans in the Service of their Country," Father Ryan High School Web Site, http://www.fatherryan.org/blackmilitary/.

8. Susan K. Freeman, "The New Woman," Clash of Cultures in the 1910s and 1920s, http://history.osu.edu/Projects/Clash/NewWoman/newwomen-page1.htm; "Bobbed Hair Blues: A Mexican-American Song Laments ‘Las Pelonas,’" Historymatters, http://historymatters.gmu.edu/d/5116/; Chicago Historical Society, Flappers, Fashion, N’All that Jazz, http://www.chicagohs.org/exhibitions/flappers/; Jennifer Rosenberg, "Flappers and the Roaring Twenties," About.com, http://history1900s.about.com/library/weekly/aa022201a.htm.

9. Marc Meola, "Chucking the Checklist: A Contextual Approach to Teaching Undergraduates Web-Site Evaluation," Portal: Libraries and the Academy 4, no. 3 (July 2004): 336.

10. Janet E. Alexander and Marsha Ann Tate, "Checklist for an Informational Web Page," Wolfgram Memorial Library, July 1996, http://www2.widener.edu/Wolfgram-Memorial-Library/webevaluation/inform.htm

11. Jim Kapoun. "Teaching undergrads WEB evaluation: A guide for library instruction." C&RL News 59. no. 7 (July/August 1998): table 1, http://www.ala.org/ala/acrl/acrlpubs/crlnews/.../teachingundergrads.htm. Kapoun’s criteria are reproduced on the Cornell University Library site as "Five Criteria for Evaluating Web Pages," Olin and Uris Lbiraries, http://www.library.cornell.edu/olinuris/ref/webcrit.html

12. Faculty Guide for History I. School of General Studies, Philadelphia University, April 2002.

13. Sam Wineberg, Historical Thinking and Other Unnatural Acts (Philadelphia: Temple University Press, 2001), 76.

14. Other studies suggest that many students prefer and seek to recreate the linear process of information gathering that they associate with an authoritative text, like a textbook. See, for example: Deborah Vess, "History in the Digital Age: A Study of the Impact of Interactive Resources on Student Learning," The History Teacher 37, no. 3 (May 2004): par. 22, http://www.historycooperative.org/journals/ht/37.3/vess.html.

15. Susan Beck. "Evaluation Criteria." The Good, The Bad & The Ugly: or, Why It’s a Good Idea to Evaluate Web Sources, 1997: http://lib.nmsu.edu/instruction/evalcrit.html.

16. "Evaluating Web Resources," Cornell University Library, http://www.library.cornell.edu/newhelp/res_strategy/evaluating/evaluate.html.

17. Ann Grafstein, "A discipline-based approach to information literacy," Journal of Academic Librarianship. Vol. 28, Issue 4, (July 2002), 197.

18. Julie Kimmel, Presenter, "Teaching Critical Thinking about Historical Information on the Web: The Problem With Checklists and Some Possible Solutions," Presentation, Annual Meeting, American Association for History and Computing, Roosevelt University, Schaumberg, Illinois, April 15 & 16, 2005.

19. John Higham, History: Professional Scholarship in America (Baltimore: The Johns Hopkins University Press, 1983), ix.

20. Reference & User Services Association—History Section, Using Primary Sources on the Web, http://www.lib.washington.edu/subject/History/RUSA/.

21. Kelly Schrum. "Surfing for the Past: How to Separate the Good from the Bad," AHA Perspectives (May 2003): par. 12, http://www.historians.org/perspectives/issues/2003/0305/0305for3.cfm.

22. Janet R. Cottrell, "Teaching students to evaluate web sources more critically: implications from a faculty workshop," College and Research Libraries News 62, no. 2. (February 2001): par. 5, http://www.ala.org/ala/acrl/acrlpubs/crlnews/.../teachingstudents.htm.