Heuristics for Educational Use and Evaluation of Electronic Information: A Case of Searching for Shaker History on the World Wide Web

Deborah Lines Andersen

Traditional teaching of history at the high school and undergraduate level has focused on primary and secondary source documents in print format. In the last several years that tradition has been challenged by information on the World Wide Web, with professors of various backgrounds taking policy stands on whether or not students should be allowed to use electronic sources in their research and writing. This research proposed an experiment to search the World Wide Web for materials about the Shakers, as if it were an assignment for a high school or undergraduate class. Three Web engines were searched, yielding mediocre, informative, and sometimes humorous results. The insights gained form a case study of issues and policy concerns for instructors, librarians, and students, and a technology skills continuum and exercises that could be implemented at the high school or undergraduate level. Among these issues were those of sequencing the teaching of complicated search strategies on the Web, as well as evaluating and authenticating electronic material that is found. Historians rely upon evaluating the primary source documents they use for their research. They need to acquire the same skills for electronically-based documents and teach those skills to their students. Until students and faculty have the skills to attack adequately the highly undifferentiated masses of information that surface on the World Wide Web, it seems certain that they will not find what they are looking for, only use what they find, or shy away from the whole process and use only traditional print sources.

.01 Introduction

Traditional teaching of history at the high school and undergraduate level has focused on primary and secondary source documents in print format. These sources, the grist of historical research, were available only in paper formats because of the dominance of print as an information-transfer medium, and the sophistication (or paucity) of high quality reproduction and transmission technologies.

In the last several years that tradition has been challenged by information on the World Wide Web. Not only are there secondary source documents available on Web sites, but, with the advent of high quality scanning technologies, primary source documents are also becoming available electronically.

Electronic access changes the very way in which history students go about doing research. Card catalogs and archival records have been augmented by search engines and Boolean logic. Thus, information access services have been expanded rather than replaced, creating more choice for researchers. Traditional, paper-based research of primary and secondary documents required traveling to libraries or archives in order to have access to the documents. The advent of the World Wide Web and scanning technologies has brought about the luxury of being able to sit at one's desk and request that information be delivered via the user's personal computer. Of course, the user no longer holds the book or manuscript in her hand. If the document is a primary source it is not possible to feel the paper, or check for various types of ink, or examine the binding of the pages. Nonetheless, it is possible that the user can make an informed decision about whether or not to visit the document many hours away, if the information seems pertinent to his research–a great improvement in savings of research time, dollars, and frustration.

The use of the term "augment" is critical to this discussion. Not all information critical to any research project will be found on the World Wide Web. There is little incentive for individuals to put esoteric or rare manuscripts through a scanning process if they are not going to have wide appeal. Research institutes and governmental organizations have started to make historically pertinent materials available on the Web, but the volume of that information is small when compared with records available in print formats in repositories (or unmarked, brown paper boxes) all over the world.

The idea for this paper originally surfaced in response to several e-mail messages that circulated among the editors of the Journal of the Association For History & Computing. A junior high school teacher had forbidden the use of online resources for a class assignment. "Books only from the library" were the information sources required of students. Responding e-mail related that the same kind of electronic prohibition had been enforced by professors of undergraduate history as well as by a doctoral dissertation committee. With various academics taking policy stands on whether or not students should be allowed to use electronic sources in their research and writing, it seemed worthwhile to examine these issues in greater detail (Murphy 1999, Varn 1999).

In order to accomplish this goal, this researcher proposed an experiment to search the World Wide Web for materials about the Shakers, as if it were an assignment for a high school or undergraduate class. Using three search engines linked to the University at Albany library home page, the researcher gave herself three hours, perhaps more time than a student would actually use, to find primary and secondary source documents on the World Wide Web that could form the basis for a paper on the Shakers. The results of this experiment were mixed, emphasizing the need for faculty and students to question and evaluate the use of World Wide Web resources at each step of the research process. Central to the concerns of this paper is the conviction that faculty and students cannot decide when to use (or not use) the World Wide Web unless they have a good understanding of how it works. Rejecting the Web as an information source, for a specific educational or research situation, should come from a strong knowledge of its use, strengths, and weaknesses.

One of these potential weaknesses is that not all materials that appear on the World Wide Web have had the same kind of peer review and editing that one expects from traditional print journals and monographs. Added to this electronic experiment was the need to evaluate the information found on the Web. To this end, the researcher also looked at sources on the Web that would provide guidelines as to how to value information found there. The insights gained from looking at both primary and secondary sources, and at evaluation tools, form a case study of skills, issues and policy concerns for instructors, librarians, and students. Additionally, Appendix A of this paper presents a technology skills continuum and exercises that could be used as a specific method for teaching students and faculty members to become more critical users and evaluators of World Wide Web materials.

.02 The Shakers

Eleven years later, Ann Lee, the daughter of a blacksmith, joined the society, and by 1774, with eight other individuals, she sailed to New York to form Shaker communities first in Niskayuna and New Lebanon, New York, and then in Hancock, Massachusetts (Ott, pp. 14-15). Mother Ann, as she became known, held two tenants central to the Shaker religion. The first was that the godhead was both masculine and feminine. The second was that Adam and Eve were lost through their original sin, that God was free from original sin, and thus, that the members of the Shaker community would remain celibate–"Hands to work and hearts to God" (Ott, p. 11).

Although only a small number of Shakers are alive today, there are Shaker museums throughout the eastern part of the United States, all boasting museum collections of tools, furniture, and household goods invented and produced by the Shakers, and archival collections of diaries, journals, and ledgers that were kept by the various Shaker communities. The amount of information available about the Shakers (also known as the United Society of Believers in Christ's Second Coming), their inventions, and lifestyle makes for easily accessible research about a former religious community in the United States. There are numerous monograph and journal articles that have been written about the Shakers. The challenge for this project was to see how much material would be available on the Internet, and how difficult it would be to find. Would it be enough to preclude turning to print, paper-based sources?

.03 The Experiment

Given three hours and a connection to the World Wide Web, this researcher's plan was to find three search engines, look for full-text primary and secondary materials on the Shakers, and see if that information would be sufficient to explore topics and write a high school or college level paper. For the purposes of this experiment, the researcher first went to the University at Albany, State University of New York library home page since it has links to a variety of search engines that a student might use (http://library.albany.edu/). "Electric Monk," "Direct Hit," and "Alta Vista" were all listed on the library Web site. The first two search engines were new to this researcher. "Alta Vista" had been on the library Web site for an extended period of time and was familiar to this researcher. This was a haphazard selection process, but probably no more hit-and-miss that what a student would do when first selecting a search engine.

Electric Monk

Electric Monk, claimed to be "The first search engine that understands what you're looking for," allowing the searcher to type in a sentence, as if asking information of a reference librarian (http://www.electricmonk.com/). This researcher started with the query, "I need any information on Shakers."

The results of this ill-formed search were not overwhelming, but at least humorous. Of the top ten answers, all rated as "probably contains your answer," there were salt and pepper shakers (in several different patterns including Kate Grenaway dolls, fruit, and Bart Simpson), movers and shakers, the Blues Shakers (a music group), hand shakers (percussion instruments), and cocktail shakers. None of the top ten had any relation to Ann Lee and her Shakers.

Refining the query for Electric Monk resulted in asking for "Shakers as a religious community." This search yielded another ten sites, with several of them (but not all) pertaining to the correct Shakers. Nonetheless, remembering that this exercise was to find primary and secondary sources that could be used for a basic paper on the Shakers, the search was not a success. The four sites that were perhaps appropriate were

Of these four sites, the first was rated as "probably contains your answer" while the last three were rated "might not contain your answer."

Three issues emerged from this query of a randomly selected search engine.

1. It is important to be as specific as possible in constructing search terms.
2. Search engine rating schemes do not necessarily surface the best sites.
3. The sites which do surface will not necessarily meet the needs of the researcher in providing useable primary and secondary resources.

Direct Hit

The next search engine results were more promising. Direct Hit (http://www.directhit.com/) listed its top three sites when the search terms "Religions Shakers" were used, with a promise to send "an email when the top 10 list changes" if the researcher would subscribe to Direct Hit. Of the top three sites listed, the first was about Quakers. The next two were:

The senior thesis could be used both for the information that it provided in its text, and for the bibliography that it contained (all print, paper materials). The New York Public Library guide was just that–a guide to what was available, but with no primary or secondary documents available online.

As with the Electric Monk search, several lessons could be learned from searching Direct Hit.

1. More specific search terms yield a higher percentage of results.
2. Results that concern the correct topic do not necessarily contain useable documents.
3. Some sites will lead the researcher to print sources that will have to be examined in a library or archives.
4. Primary source documents appear illusive, or at least are hard to identify given the way that information is listed by the search engines (usually just a repeat of the opening lines of the Web page).

Alta Vista

Having taken an hour of the three hours at this point, with only one document, a senior thesis, "in hand," the researcher looked for a more familiar search engine, Alta Vista, and entered "Shakers Massachusetts" to see if a geographic location would yield information about the Shaker communities in question. Of the ten results that were retrieved (with 2,432,590 Web pages found!), some were just about Massachusetts, others were about cocktail, or salt and pepper shakers, and two seemed to be to the point.

This final site at last contained primary source documents about the Shakers. The title "19^th Century Shaker Meeting" was one of eight "reminiscences" that were listed at this Web site, each with a citation to the original source. These were not scanned documents, but retyped copies of original letters and diaries. One could read the words but not see the original manuscript. Finding and reading these took up an additional 45 minutes.

Alta Vista–Again

Having come up with a good source on Alta Vista, the researcher requeried the same search engine, this time asking for "Shakers Hancock," another geographic refinement of the search. Again, the search engine found thousands of Web pages and displayed the first ten. Scanning these yielded one potentially useful site.

This particular site did create additional search terms–other locations that the researcher could use as search commands. It was also from a site that had come up previously in the Alta Vista search (Shaker Workshops) so that particular site might be worth searching in more detail. It did not, however, produce any primary or secondary source documents that could be used for a high school or college research paper. After working this long on the assignment, several additional lessons, and questions emerged.

1. Sites that do not yield useful documents might suggest additional search terms.
2. It takes a long time to scan a hit list to find relevant materials.
3. How long should one spend searching for materials?
4. How many search engines should one look at? This researcher was up to 3 engines, with little time left for additional searching.
5. How does any researcher evaluate the worth of different search engines? Why select one over another?

Given that there were more than 15 search engines available through the University at Albany Library home page, it would have been possible to spend several more hours trying out search terms on a variety of engines, going back when new terms emerged, and, it would appear, not finding too much information that would be helpful. This was a frustrating process that did not yield particularly good results. Given the amount of time spent, there was little return on its investment. On the other hand, in a time before the World Wide Web, three hours of research time would have been hardly enough to travel to a major research library or archives to check catalogs for information on the Shakers, and then examine that information. The potential of the technology appears to have made this researcher much more impatient for immediate information delivery.

.04 Information for Research and Writing

In a traditional research environment there are primary and secondary source documents that historians would use to create a historical argument. These documents differ in terms of their origin, authenticity, and reliability as sources of information. It is usually assumed that print information found in libraries has undergone some kind of rigorous review before it appears on the shelves. Either peer review or heavy editorial process "validates" the information so that researchers can assume a certain amount of reliability (Brandt 1996). Archival materials, and other primary source documents, do not have that level of review. In fact, the historian's job is to assess the worth of these documents, and to see how they support or add to historical arguments and discussion. It could be argued that part of teaching students history is teaching them to evaluate these primary sources.

The advent of materials on the World Wide Web has created a real dilemma for seasoned historians as well as neophyte students of history. The same level of peer review or editorial process does not necessarily exist on the Web. It is often not clear when an electronic document has undergone serious peer review. Primary source documents often come without adequate provenance, so it is again hard to judge how "good" or reliable they are. The user must be much more cautious and evaluative in an electronic environment. The teacher of history must guard against students' believing that all electronic documents are of the same caliber, or indeed, that all information they need can be found electronically.

.05 Evaluation of Information

Information sources can be evaluated based upon a large number of criteria. As previously mentioned, peer review and editorial practice have been the usual benchmarks for good practice in print publication. Furthermore, various journals and publishing houses have developed reputations for printing high-quality scholarly works. Scholars in any academic field additionally have internal "editors" that are based upon their familiarity with the subject matter, their knowledge of individual scholars, their predisposition about the interpretation of the subject matter, and their basic spelling, grammar, and logic checking that allows them to evaluate information and its presentation based upon their previous experience. These are the skills faculties work to instill in high school and college students who take various academic classes.

The addition of electronic information to this mix means that there are more evaluation skills that need to be passed on to high school and college students. These skills fall in two general classes. First, there is the ability to evaluate the format of a Web page and how well it presents information. This is not the interest of this paper, but a very important skill for those students who have reached the stage of developing their own home pages.

The second skill, and pertinent to this paper, is the ability to evaluate information which appears on a Web page. There are standard criteria for electronic information evaluation including last date of update, and source of the material. Other criteria include examining the reliability, credibility, perspective, and purpose of the information (Brandt 1996). More ephemeral is the "face validity" test in which the reader decides if the information seems reasonable, based upon the readerÕs world view and previous experience with the subject matter (Tyburski 1997).

This last skill takes experience and time. There are, however, a series of Web sites that have been developed recently that attempt to teach students about evaluating electronic material. A query of various search engines on the World Wide Web (4/8/99), using the search terms "information evaluation," led to a variety of sources that would help faculty and students to understand better the evaluation of electronic information. Two of these sources were bibliographies with hypertext links to evaluation literature.

These two sources, one from New Zealand and the other from the United Kingdom, included tutorials for students on how to evaluate Web information. Although there are many examples (and one can go to the sites listed above to find them), one site was organized to be particularly useful for this discussion.

EvalWEB, Evaluating WWW Resources was designed as "a tutorial on evaluating web pages to determine their suitability for use as research sources for middle and high school research" (http://www.hudson.edu/hms.comp/evalweb/).* Although last updated on 11/30/97, the material was such that the concepts have not gone out of date. These concepts included evaluating the (1) address, (2) content, (3) author, (4) revision date, (5) links, and (6) drawing a conclusion. In each case there was a short paragraph describing the issue under discussion, and in many there existed a hypertext link to an example site for evaluation. Finally, there were sample sites for the student to evaluate, with subsequent pages that the student could use to compare her conclusions with those of the "expert." The text of the site ends with information worth repeating.

Other sites for faculty and teaching assistants to learn about Web information abound across the Web. In particular, The University of Washington Computer Training workshop, "Teaching Students to Think Critically about Internet Resources," (http://weber.u.washington.edu/~lib560/NETEVAL/index.html)* and Perdue University's "Evaluating World Wide Web Information" (http://thorplus.lib.purdue.edu/rese... lasses/gs175/3gs175/evaluation.html)* presented Web tutorials for helping students and faculty to learn the critical points of Web site evaluation.

.06 Recommendations

From the previous discussion it seems apparent that faculty and students are in need of extensive training about the uses and misuses of electronic information for their research and teaching. Technological illiteracy can lead to assignments that demand no use of the World Wide Web. The ability to find relevant information and evaluate its usefulness is a necessary skill in today's technologically advanced world. Faculty need help in learning to use the Web, in directing students to good sources, and in teaching evaluation of source documents. This last item is a crucial skill for students so that they guard against accepting anything that they see in print. As stated previously, students need to be able to evaluate primary source documents if they are to be good historians. Information evaluation for traditional sources is taught in history departments. That evaluation now needs to be extended to include electronic information as well as material found in attics and archives.

.07 A Technology Skills Continuum

There is no one-size-fits-all skill required for acquiring and manipulating information on the World Wide Web. It seems apparent that both faculty and students need to make distinctions about the various skills that individuals need to acquire, both to use the Web for research and teaching, and to trust the Web for providing appropriate information. Closely related to skills and trust is the need for careful evaluation of what is provided on the World Wide Web.

A technology skills continuum takes into account a necessary linear progression of abilities as well as several variations on those abilities. Table 1 presents those skills.

The first two items are skills that librarians and instructors are starting to assume of all students. Since various initiatives at the junior and senior high school level have brought computing skills into the general curriculum, students arrive in classes prepared to turn on computers and use them to create text documents. Similarly, it is becoming more and more common for junior and senior high school students to have skills and software that allow them to access the Internet for e-mail. Libraries and home computers have made access potentially available to all students. Additionally, students have become sensitized to the quality and reliability of information on the Internet through realization that chat rooms and e-mail pen pal correspondence do not necessarily yield truthful information about others.

World Wide Web access capabilities (item C in Table 1) are not necessarily a tidy package of skills. It is possible to be able to bring up a Web page, given a specific URL, but to be really inept at searching for additional materials on the Web. First one must be able to locate, and then identify, appropriate search engines for the topic in question (C2). One must then be able to use those search engines to perform a fruitful search, keeping in mind that each search engine has its unique set of search algorithms (C3). A fruitful search, using a reasonable set of keywords in one engine, might not yield equally valuable results in subsequent searches of other engines. Thus, another necessary skill is the ability to adopt and refine a search depending upon a change in search engines (C4). Finally, a researcher must be able to evaluate the information that surfaces through these searchers (C5). Actually creating a Web page (D) obviously goes beyond the scope of this paper. The ability to create a page does not necessarily guarantee the ability to search for and appropriately locate others.

One cannot assume that any high school or college student (or any faculty member in a college or university) carries with her a full complement of the skills just mentioned. Inadequate or partial understanding of electronic information can lead to poor choices not only in terms of using materials on the Web, but also in not using that information for teaching and research.

Table 1: A Technology Skills Continuum for WWW Information Acquisition

• A. Generic knowledge about computers (windows; word processing)
• B. Internet access capabilities (modem or Ethernet technologies; e-mail; Web access skills)
• C. World Wide Web access capabilities

1. Ability to locate a URL on the Web
2. Ability to locate search engines
3. Ability to use a variety of search engines
4. Ability to refine a search so as to find relevant topics
5. Ability to evaluate the information found and select appropriate sources

• D. Web page development expertise (knowledge of creation software for Web pages)

.08 Teaching about the World Wide Web

Based upon the previous discussion, it seems absurd to throw students onto the Web and ask them to find information that meets their research needs. Not only is pertinent information hard to find, but it is a frustrating waste of teaching time. Students need to be "checked out" on a variety of skills, such as those presented in Table 1, before they are cast out into cyberspace. Additionally, such skills could also be taught to faculty members who have not developed a fine set of distinctions among the various skills necessary for using the World Wide Web.

In particular, the following is a set of suggestions for creating research competencies on the Web. These should go hand in hand with teaching information evaluations skills, both for print and electronic documents. (Appendix A presents a set of exercises that makes use of the hierarchy of skills listed below.)

• Locate a known item. The simplest of Web skills is that of being given a site and going to it. This skill includes the ability to move through links on a site, get back to the home page, and print out or electronically download pertinent materials for future reference.
• Visit best practice sites. Students would be given a set of URLs for sites that the instructor feels are truly excellent. Students would then exercise the skill of finding these sites as well as looking for evaluation criteria that make them good.
• Create criteria for evaluation of Web-based materials. At a certain point students should be able to list criteria that lead to good or poor practice in electronic dissemination of information.
• Use criteria to evaluate a given item. Based upon criteria developed by students and faculty, students could move to the next level of evaluating a known item, or set of items, given to them by their instructor.
• Complete a search for a known term. As a first step in searching the World Wide Web, students could be given a search term for which the results are known by the instructor. Ideally, the search would provide lessons in what happens when searching a particular engine or engines, providing useful as well as humorously useless results.
• Compare results from several search engines. Again, using a controlled search, where students are given search terms, they should be able to evaluate the results they get when doing searches on two or more search engines.
• Conduct a free search. Finally, students should be ready to do research on the World Wide Web using topics and search engines of their own choice. This level of search does not preclude the use of paper-based documents. In fact, by the time students have reached this level they should be aware of the inadequacies of information on the World Wide Web, and the need to conduct research in a variety of formats.
• Master sophisticated free searching skills. There is a set of skills that greatly improve a searcher's ability to find useful information on the World Wide Web, although it is possible to search blindly, using whatever previously acquired skills one has. These high level skills start with an adequate understanding of one's subject matter in order to create appropriate search terms. Here is the place for dictionary and thesaurus work before spending time on the World Wide Web. Subsequent high-level skills include Boolean logic, and truncation and restriction algorithms. Perhaps equally important is an understanding of the hidden algorithms that various search engines use to create their results. The rating systems of various sites are not transparent. It is worth having students try to uncover how they work, or, very often, do not work to unearth valuable information. (See http://www.albany.edu/library/internet/#search for a series of tutorials in World Wide Web search strategies.)
• Be sensitive to cost/benefit assessment. In addition to these specific steps in locating information on the World Wide Web, students (and faculty) need to concentrate on issues of Web time management, and information overload. Students might be asked to keep track of how much time they spend doing particular searches, running cost/benefit analysis of this method over traditional library and archival research. They need to be aware of when they have enough information ("satisficing"), rather than increasing research time by doing just one more search before stopping an open-ended research project. Similarly, students should have a set of coping skills that will help them deal with information overload, sifting through masses of information to find what is pertinent and appropriate to their work, without having this task consume valuable time that should be spent on creation of insights, synthesis, and historical interpretation.

.09 Conclusion

It seems quite apparent that the World Wide Web and electronic information are not going to disappear in the foreseeable future. Research and education need to move toward providing skills that will allow individuals to make critical information judgments, learn about a wide variety of information sources, become skillful at efficiently finding those sources, and balance the mix of electronic and paper to best meet the needs of historical analysis. The only way to form solid judgments about using (or not using) the World Wide Web is to have a firm understanding of its strengths and weaknesses. When individuals are as skilled in using World Wide Web resources as they are in using libraries then they will be confident selecting the best information resources for a particular information need. Sometimes paper may be the best or only information source available. The rest of the time faculty and students need to be careful about rejecting the message–the information–just because the new medium has created additional challenges in access and evaluation.

.10 Notes

^* Underlining the ephemeral nature of World Wide Web documents, several of the sites that were accessible in the March 1999 URL check of this document were no longer accessible in the April 1999 URL check of the document. It is possible that some documents are permanently lost, or that others have been changed to different domains.

1. This research dealt with print records. There is an entirely different area of research that deals with non-print media. The World Wide Web allows both audio and video to be transmitted. Thus, the argument about whether or not electronic information access is appropriate could be broadened to ask if a photograph or speech could be experienced adequately enough over the Internet so that it could form the basis for a historical discussion. Further, a second broad question asks if any form of access is better than no access if the item is unavailable to the researcher because of travel distance, or accessibility restrictions.

2. Throughout this discussion a distinction must be made between student and faculty research. It would be expected that most of what a serious historical researcher is looking for (faculty and graduate students) would not be found on the World Wide Web. There would be little incentive to scan and make available the unique documents that support historical arguments for these researchers. One would expect that documents that do appear on the Web are of general interest and thus appropriate for high school or undergraduate researchers.

3. See a Norwegian site (University at Bergen) at: (www.hist.uib.no/bomb/) that includes links to oral history bibliographies and websites.

4. It should be perfectly clear at this point that this research was not designed to duplicate the behaviors of a high school or undergraduate student. The researcher holds a masters degree in library science, teaches information science, and has worked at a Shaker museum. The experiment was designed to uncover the sorts of materials and challenges that a student might face in this endeavor.

5. See Andrews (1953), Mahoney (1993) and Sears (1916) for three such print monographs.

6. See Hinman (http://www.albany.edu/sisp/student.html) under student projects for a discussion of evaluation criteria for Web sites, and Andersen for a discussion of history department sites. Also see John Morkes and Jakob Nielsen (1997) for "Concise, Scannable, and Objective: How to Write for the Web," a summary of four years of Web usability studies. (http://www.useit.com/papers/webwriting/writing.html)

7. Boolean logic requires understanding the terms that search engines "understand" to combine multiple search terms. An "and" usually means that one requires sources that include information on two subjects (Shakers and religion) whereas an "or" requires that only one of these terms be present in a source (Shakers or religion). The default search algorithm for search engines is usually of the "or" variety.

Truncation allows one to enter only part of a word, with some symbol such as "*" after it. The truncation should pick up the root with various suffixes (e.g., relig* for religion, religions, religious).

Restriction allows one to narrow a search by adding items, often dates, that will target a particular topic or time period. There is no standard method of accomplishing these search strategies across various search engines–one must learn the search system for each of them.

8. Herbert A. Simon (1955) used the term "satificing" in the context of the private sector firm, looking at how managers made information choices. A variety of constraints were central to his notion of satisficing–individuals did not look for the optimal solution but a solution that was good enough given various feasible possibilities. Time, funding, or personnel resources could all create constraints on finding an optimal solution. Arguably the same holds true for individuals searching the Web.

.11 Bibliography

Andrews, Edward D. 1953. The People Called Shakers: A Search for a Perfect Society. NY: Oxford University Press; and 1963 NY: Dover Publications.

Brandt, D. Scott. 1996. "Evaluating information on the Internet." Purdue University Libraries, West Lafayette, IN (http://thorplus.lib.purdue.edu/~techman/evaluate.htm)

Davenport, Thomas A. 1997. Information Ecology: Mastering the Information and

Knowledge Environment. New York: Oxford University Press.

Mahoney, Kathleen. 1993. Simple Wisdom: Shaker Sayings, Poems, and Songs. NY: Penguin Press.

Murphy, Jamie. 1999. "Technology Training for Faculty." Converge 2(3)30-31. (http://www.convergemag.com)

Ott, John Harlow. 1976. Hancock Shaker Village: A Guidebook and History. Revised edition. Hancock, MA: Shaker Community, Inc.

Sears, Clara Endicott. 1916. Gleaning from Old Shaker Journals. Boston: Houghton Mifflin Co.

Simon, Herbert A. 1955. A behavioral model of rational choice. Quarterly Journal of Economics 69(1)99-118.

Tyburski, Genie. 1997. "Publishers Wanted, No Experience Necessary: Information Quality on the Web." Law Library Resource Exchange; author is the research librarian for Ballard Spahr Andrews & Ingersoll, Philadelphia, PA

(http://www.llrx.com/columns/quality.htm)

Varn, Richard. 1999. "Paying Off Faculty-Development Deficit in Technology." Converge 2(3)42-43. (http://www.convergemag.com)

.12. Appendix A

Exercises in World Wide Web Use: Shaker Museum Sites

Shaker museums and related sites contain a wealth of information about the Shakers and their history. Often mounted and updated by volunteers, these pages form a good information based for teaching about use and evaluation of Web information.

The following exercises follow the text of the paper presented here, suggesting ways in which high school and college instructors could guide their students in the use of Web resources. The exercises are progressive, allowing individuals the chance to develop one set of skills before moving on to the next. In a high school class it is expected that each skill set would take one or two class periods to acquire if students had access to an electronic lab for their work during class time. In a college setting students might be set out on their own to do several of the assignments at a time. The Web sites listed in these exercises were all active as of April 8, 1999.

Know item: (http://www.shakers.org/history.html)

A first step in acquiring WWW skills is simply going to a known site and then following the links that exist at that site. The URL listed for this exercise gives not only a history of the Shakers, but also links to a variety of other information and sources about the Shakers. In particular, the acknowledgements link gives a list of sources that students could look for in the library. Humorously, a link to "recipes" from the original URL is actually a link to a tour of Canterbury Shaker Village in Canterbury, New Hampshire. That Web site is notable because it gives descriptions and pictures of the buildings at that particular museum.

Along with creating a brief history of the Shakers based upon this secondary source material, students could also create a list of potential search terms for finding additional information about the Shakers, on the World Wide Web, and in local library databases for journal articles and catalogs for monographs.

Best practice: (http://www.passtheword.org/SHAKER-MANUSCRIPTS/)

This site gives links to copies of six original Shaker manuscripts, either excerpted versions or the complete text. In particular, http://www.passtheword.org/SHAKER-MANUSCRIPTS/Shakers-Compendium/compndm.htm gives the complete text of the "Compendium," published in 1859 by the Shakers in order to make others outside their community understand more about them. Included are sections on history, settlement, belief, doctrines, and biographies of the founders.

This particular site lends itself to searching for more information about the Shakers than could be found in the "known item" search above and it is a primary source document. For high school students this could form the basis for a series of document based questions (DBQ) about the Shakers. The distinction between primary and secondary sources is quite apparent after looking at the "known item" and "best practice" sites. This site contains a retyped (digital), not scanned (bit mapped) document.

There are sites that do contain bit mapped documents of the Shakers. Students still cannot touch the paper or examine the ink, but they are one step closer to the original. At http://www.hancockshakervillage.org/brdsidlg.html there is a one page broadside from circa 1880 titled "The American Shakers." There is no provenance given for the broadside, although one could probably contact Hancock Shaker Village for more information. The broadside gives basic information about the Shakers, and was an advertising piece.

Another site contains not only a scanned, primary source Shaker document, but also a transcription of that document. See http://www.logantele.com/~shakmus/journal5a.htm for the scanned document and http://www.logantele.com/~shakmus/journal5.htm for the transcription.

Item evaluation: (http://www.hancockshakervillage.org/)

Having already explored the site for the Canterbury, New Hampshire Shaker Museum, students should have the opportunity to compare the information presentation of a variety of sites, deciding what criteria a researcher should use when looking at Web information. The URL given in this section is to a guided tour of the Hancock Shaker Village in western Massachusetts. Both sites use pictures and descriptions of various buildings to give the searcher a feel for the museum site and structures.

The Hancock Web site also contains a "Shaker Link" to other sources of information about the Shakers and about Shaker museums. Depending upon the sophistication of the students, a variety of assignments could be created to have them evaluate the content and presentation of information at one, two, or a number of the sites that are listed here. Some of these sites are far more commercial than others, listing information about programs, hours, and gift shops, while others have more information on buildings, history, and additional sources of information.

• Canterbury Shaker Village http://www.shakers.org/index.shtml
• Shaker Village of Pleasant Hill, Kentucky http://www.shakervillageky.org/
• Shaker Museum of South Union, Kentucky http://www.logantele.com/~shakmus/index.htm
• http://www.logantele.com/~shakmus/othersites.htm (links to other Shaker sites)
• Shaker Heritage Society, Albany, NY http://www.crisny.org/not-for-profit/shakerwv/
• Sabbathday Lake Shaker Village, Maine http://www.shaker.lib.me.us/
• Enfield Shaker Museum, Connecticut http://www.valley.net/~esm/
  • This site has a link to several pages of bibliographic materials on the Shakers, for adults and children. All the books listed were available at the museum shop.

Evaluation criteria:

Having looked at a large number of sites, students should be able to generate a list of criteria they would use if trying to decide the "goodness" of any individual site. This particular exercise could be done as a homework problem or generated with a lot of class discussion and brainstorming. The museum sites listed in the previous section are more than enough to form examples of best practice and sites that could perhaps use some more adjustment. (See "Evaluation of Information" in the body of the paper for a listing of evaluation sites.)

Known search term:

After having come this far students will be ready to use one or more search engines. There are so many search engines available today that individuals or pairs of individuals could be sent out to search using a specific term, and then come back to the class to report their results. As seen in the body of this paper, "Shakers" as a search terms yields more false hits than not, but the instructor might start there. Using specific sites or names will yield more useable results for students.

Comparison of results:

Having done the "known search term" exercise, students will be ready to compare the results they have had in looking for materials on the Shakers. Again, this would lend itself to in-class discussion.

Free searching:

Finally, having gone this far in the search process, instructors might send students out to do their own searching and evaluation of Web materials. This could be directed toward a more narrow paper or presentation topic that a student wished to pursue on the Shakers.

Sophisticated free search:

As mentioned in the body of this research paper, each search engine has its own peculiarities in terms of doing more advanced searches. Some engines have help links for individuals who want to narrow their topics. Others set up sidebars of related topics, or categorize related areas for the researcher. Still others use particular methods of Boolean logic to limit or restrict searches. A sophisticated class of students could spend time determining the various search algorithms that are apparent, or not so apparent, in various search engine procedures and test to determine which methods give the best results. information. (See http://www.albany.edu/library/internet/#search for a series of tutorials that will improved users' ability to conduct more and more sophisticated searches on the Web.)

Cost/benefit assessment:

Finally, it is no surprise to most people who have used the World Wide Web that it has the potential to be an enormous time sink. Students might be required to keep a log of time in/time out while working on the projects suggested here, at the same time keeping track of how many useful items, or useful pages of information were acquired through electronic means. They might also keep track of how many non-electronic bibliographic citations they found. Undergirding this entire assignment set is the very real possibility that it might be faster, or more beneficial to do unknown item searching in a library, rather than spend enormous amounts of time surfing the Web with little research benefit in return.

Web Page Design:

The discussion of assignments to this point has totally avoided the issue of students' creation of their own Web pages. There is pedagogical value to students in having to create and organize information for hypertext display. Nonetheless, all the exercises proposed in this paper could be accomplished without this last skill.