This paper was refereed by the Journal of Electronic Publishing's peer reviewers.

The way humans and computers interact is changing at a dramatic rate. Advances in the speed of computers and the advent of broadband have opened wide the door to new communication technologies, making it easy to transmit text, graphics, sound, animation, video, and live images. The increased use of multimedia technology is changing the face of the Internet and Web-based media, but its impact on users has not been thoroughly researched. This project helps us understand the effect of these changes by investigating users' proficiency at and enjoyment of information-retrieval tasks on Web sites with and without the latest multimedia features. More specifically, we compared users' speed, efficiency, and accuracy in searching, their ability to recall information, and their enjoyment of the user experience on standard HTML-driven Web sites and animated Flash-enhanced sites.[1]

While a difference in interactivity exists between HTML and Flash-enhanced Web sites at the interface level, the most salient difference is aesthetic. A Flash-enhanced site offers the user synchronized sound, rollover effects, intense graphics, and animation. Flash increases the dialog — the interactivity — between the user and the site, and provides the user with a potentially more engaging, immersive, and entertaining experience. The vast majority of HTML sites, on the other hand, are predominantly static, incorporating few multimedia features.

A survey of the literature showed that researchers have been divided on the comparative advantages of HTML-only or Flash-enhanced Web sites.

Pro-HTML

Kirsch (1997) says that the user interface — the design of the Web site — should help the user decide what to do next on the site by logically presenting paths and links. Those guides to the technical and navigational capabilities of the site should be unambiguous and direct. Norman (1988) recommends that hypertext designers use affordances, visual clues to the functions of objects, to give them the information they need to make site-navigation decisions. A good user interface that uses affordances well helps users form strong conceptual models of the site, which in turn fosters the relationship between the users' intentions, the actions they need to take, and successful results.

The most common affordance for HTML navigation is a mouse pointer that changes to a hand when hovered over links, and the underlined hyperlink, which is typically blue (Spool, Scanlon, Snyder, Schroeder, & DeAngelo, 1998). When links are embedded in a Flash file, the mouse pointer may change to the hand when hovering over a link and may also produce a sound or launch a text box (called a "rollover effect" among developers) but typically the links are not underlined or identified by color. Sometimes links and buttons in Flash-enhanced sites are completely unlabeled, and show up only as geometric shapes. Interactivity, such as rollover effects or sound, may assist the user in determining the architecture of the site by providing some feedback, but the design and navigation of many Flash-enhanced sites are not visible to the user; instead, the intense graphics (potentially) place the user in an environment requiring a novel conceptual map.

One potential pitfall associated with Flash was identified by Rajani and Rosenberg (1999). The researchers found, as expected, that users tended to remain on Web pages if they contained interesting information. However, the researchers also found that the navigational aids on the page played a major part in the user experience; if the navigation was confusing then users spent more time planning how to move, rather than concentrating on the interesting information. Rajani and Rosenberg identified a problem that they called "living in the moment," the inability of users to recall information about or describe function included on sites with novel features or navigation systems. The increased stimuli, especially in a multimedia environment, apparently hinders users' abilities to live in the moment, seemingly because they can concentrate only on certain aspects of the site at one time. Ultimately this cognitive taxation limits users' ability to make a cognitive map, thus impairing their ability to navigate through a Web site.

Other researchers have identified Web animation as a source of visual noise. One study suggests that animation makes it more difficult for users to read or skim a Web site (Spool et al., 1998), thus possibly hindering their ability to retrieve information from the site. Similarly, in an examination of graduate students' cognitive processing during Web-based information retrieval, Hess (1999) found that filtering out irrelevant graphical information was key to successfully acquiring information and forming meaning from it (see also Hu, Ma, & Chau, 1999). Therefore, the interactive features of Flash-enhanced Web sites could make it difficult for the user to avoid information overload and locate the relevant information on a site.

Pro-Flash

Hu, Ma, and Chau (1999) examined the impact of interface designs on the perceived relevance of an object in an information-retrieval system. These researchers found that graphical interfaces were significantly superior to list-based interfaces, especially in communicating concepts of interest to a user. The study suggested that multiple visual properties included in an interface may increase user satisfaction. Similarly, van Oostendorp and van Nimwegen (1998) measured users' speed and accuracy in locating information in online newspapers, finding that it took longer to locate information by scrolling and using hyperlinks between levels within a Web site (i.e., features more typically found on HTML-driven sites) than by directly linking a user to desired information (which is more typical of Flash). Furthermore, Spool et al. (1998) found that the use of graphic-laden interfaces had no correlation — either positive or negative — with users' ability to retrieve information; that is, graphic design was unrelated to the ability to find information on a Web site.

Rajani and Rosenberg (1999) found that users enjoyed novel Web sites; research participants indicated that the new technologies were both interesting and fun to use. They particularly liked the way sound enhanced their experience. These findings indicate that many of the features available through Flash-enhanced sites might actually increase (or at least not hinder) information retrieval.

Research Questions

Given the inconsistent and (potentially) contradictory nature of the findings, we wanted to investigate the influence of Web design (HTML and Flash) on a user's experience of information retrieval. The existing literature points to at least five key dependent variables that might be influenced by the design of a Web site during an information-retrieval task: enjoyment of the experience, efficiency (in terms of navigational steps) in task completion, time it takes to complete the task, accuracy of information retrieval, and ability to recall information about the site upon completion of the task. Accordingly, the following research questions were posed as guides for the present study:

• Research Question 1: What is the relationship between information retrieval using different Web site designs (HTML and Flash) and reported enjoyment of the user experience?

• Research Question 2: What is the relationship between efficiency, speed, accuracy, and recall of information retrieval and Web site design (HTML vs. Flash)?

Methods

Sample: A sample of 71 research participants (62 percent were female, 38 percent were male) were randomly assigned to one of two experimental groups defined by the Web sites that they accessed. Participants completed the study individually. Each participant was asked to estimate his or her level of ability in relation to Web usage; 56.7 percent of the respondent claimed to be novice/intermediate users, while the remaining 43.3 percent claimed to be advanced/wizard users.

Procedure: When each participant entered the research facility, he or she was taken to a cubicle with a desk that had a monitor, a mouse, and a keyboard. A researcher explained the procedure and launched a Web browser pointed to a Web page that outlined three tasks for each of two sites, one an HTML site, the other a Flash-enhanced site. The tasks involved answering questions on paper about each Web site on a piece of paper. After answering the questions for one of the Web sites, participants completed questionnaires that measured immediate recall. They repeated the drill on the other Web site. Finally, they completed a questionnaire that measured their attitudes about their experiences.

The monitor and mouse that the participants used were connected to an unseen computer housed in an adjoining cubicle which had its own screen on which we could watch the mouse move. This configuration allowed the researchers to unobtrusively measure both the efficiency (i.e., number of clicks) and time of each task completion for each participant. The Web sites themselves were cached in the permanent storage of the hidden computer to eliminate the variable of network traffic on the time to locate information and to limit the participants' access to only the Web sites used in the study.

Web Sites

Researchers selected two different Web sites on which the information-retrieval tasks would be completed. Each was the Web presence for a technology company: Altair Technologies [formerly http://www.altairtechnologies.com/] and Zymocreations. None of the participants in the study were familiar with either company or their Web sites. Both companies offered two versions of their site: an HTML version and a Flash-enhanced version. The only differences between the Flash-enhanced and HTML versions of each company's site were the inclusion of the animation (both Flash sites included a brief opening series of moving texts and graphics), sound (both Flash sites were accompanied by a music track), and an iconic navigation system (on both Flash sites, linked pages were accessed by clicking representative icons rather than traditional hypertext links).

Information-Retrieval Tasks

Participants completed the same three tasks for each company, but used the HTML site for one company and the Flash-enhanced site for the other. The tasks involved finding a piece of information on the site and recording the information on the task sheet. For each company, the tasks were identical. The use of two different Web sites allowed roughly one-half of participants (47.9 percent) to complete the tasks on the HTML version of the Altair site and the Flash version of the Zymocreations site; the remaining participants completed the tasks on the HTML version of Zymocreations and the Flash version of Altair. This crossed design prevented the subjects from being sensitized to the content during the first set of tasks, which would have greatly increased a participant's ability to complete (or merely recall the results of) the tasks on the second site because the tasks and information location were identical for both the HTML and Flash version of each company's site.

Between the two company sites, the tasks were of equal challenge; that is, Task 1 (the nature of the company's business) for Altair and Zymocreations was one page deep into each site and was located at approximately the same location on those pages. The same is true for Task 2 (a question about manufacturing, which was found three pages deep into the site) and Task 3 (the location of the company's headquarters, which was two pages deep into the site). Therefore, the task-completion experience was functionally equivalent for each participant.

Dependent Measures

Five main dependent variables were measured for analysis. First, efficiency of task completion was unobtrusively observed by the researcher during the testing session. To calculate the efficiency with which each participant completed the assigned tasks, the number of clicks (or navigational steps) used to retrieve the requested information was recorded for each task. The fewer the number of clicks, the more efficiently the task was completed. The number of navigational steps was summed across all three tasks to generate a single variable.

Second, the amount of time (in seconds) needed to complete each task was also unobtrusively recorded by the researchers. Timing began when the participant clicked the link on the task/home page to begin the retrieval of information, and timing ended when the participant clicked on the button to return to the task/home page. The time to complete each task was summed across the three tasks for the HTML and Flash tasks respectively.

Third, because each task required the participant to retrieve and record a specific piece of information, accuracy in completing the task could easily be determined. If the correct piece of information was gathered, then the participant was said to have accurately completed the task (regardless of efficiency and time), and thus received a score of 1 for the task. If the correct piece of information was not recorded, then the participant inaccurately completed the task and was scored a 0 for the task. The accuracy of task completion was summed across all the three tasks for the HTML and Flash tasks respectively, the highest possible score being 3..

Fourth, recall of the information accessed and navigational steps taken during the retrieval of the information was collected for each site. The researchers used the same coding scheme used with the accuracy measure (correct recall = 1, incorrect recall = 0), and summed the scores for each of the four recall items.

Fifth, the enjoyment of the user experience was measured on four items: layout appeal, navigation ease, desire to revisit site, and willingness to recommend site to a friend. The four enjoyment items for the HTML and Flash sites were subjected to a principal components analysis, which each yielded a single factor (Eigenvalue_HTML = 2.18, α_HTML = .72; Eigenvalue_Flash = 2.76, & _Flash = .84) solution explaining 54.6 percent and 68.9 percent of the variance respectively. No item loaded at less than |.67| in either analysis. As a result, the four items were averaged across each design condition and treated as single factors termed HTML enjoyment and Flash enjoyment respectively.

Results

A repeated-measures (General Linear Model) procedure was used to compare the reported enjoyment of the HTML site to that of the Flash-enhanced site. A statistically significant difference (F1,70 = 102.01, p < .01) was observed between the two design conditions, with the Flash-enhanced site (M = 1.99) enjoyed more than the HTML site (M = 3.46). [Note: Lower mean equals more enjoyment.] A statistically significant difference (F1,65 = 6.31, p < .05) was likewise observed between the enjoyment of the conditions when the reported level of Web ability for each participant was used as a covariate.

To compare the efficiency with which each participant completed the assigned tasks, the number of clicks (or navigational steps) used to retrieve the requested information was recorded for each task. A repeated-measures procedure found no significant difference (p < .05) between the two conditions in terms of navigational efficiency. In other words, the participants accessed the requested information on the HTML and the Flash-enhanced site with equivalent efficiency. Similarly, when the ability of the participant was controlled, no significance was found between the two conditions (p < .05).

A similar procedure was used to analyze the length of time it took the participant to access the information. A repeated-measures procedure found no significant difference (p < .05) between the two conditions in terms of time. In other words, the participants accessed the requested information on the HTML and the Flash-enhanced site with equivalent speed. Again, when the ability of the participant was controlled, no significance was found between the two conditions (p < .05).

Accuracy was measured by whether the participants completed the tasks correctly. As was the case with efficiency and time, no significant difference (p < .05) was observed between the two conditions in terms of accuracy using a repeated-measures procedure. In other words, the participants accessed the requested information on the HTML and the Flash-enhanced site with equivalent accuracy. Again, when the reported ability of the participant was controlled, no significance was found between the two conditions (p < .05).

Upon completion of all three tasks per design condition, participants were evaluated on their ability to recall the information accessed during the tasks. Using the same coding scheme used with the accuracy measure (correct recall = 1, incorrect recall = 0), a repeated-measures procedure found a statistically significant difference (F1,70 = 16.44, p < .01) between the two design conditions, with the Flash-enhanced site (M = 2.85) eliciting greater recall than the HTML site (M = 2.08). A statistically significant difference (F1,65 = 4.22, p < .05) was likewise observed between the information recalled in each conditions when the reported level of Web ability for each participant was used as a covariate.

Conclusion: Flash Works

The purpose of the study was to investigate the influence of multimedia features on the completion of information-retrieval tasks on Web sites. The existing literature, while identifying many key aspects of the information-retrieval experience that might be affected by these features, offered little guidance in determining the nature of those potential effects. As the findings indicate, individuals using HTML-driven and Flash-enhanced Web sites did not significantly differ in terms of how efficiently, quickly, or accurately they were able to find information on those sites. In other words, for the purpose of this study, the design of a Web site had no effect on the actual retrieval of information from a Web site. Apparently the potential distraction-oriented problems associated with animation, sound, and cognitive overload previously identified (Hess, 1999; Hu, Ma, and Chau, 1999; Rajani & Rosenberg, 1999; Spool et al., 1998) were found to be insignificant.

While no differences were found in terms of the actual completion of the tasks, significant differences were found in terms of enjoyment and recall. Specifically, users reported more enjoyment of the Flash sites than the HTML sites. This finding is further substantiated by a single item that asked the respondents to identify which site they preferred more: 85.9 percent identified the Flash site.

Also, participants were able to recall more information from the Flash-enhanced sites than from the HTML sites. This is perhaps the most important finding of the project. Does this mean that Flash itself leads to greater recall in all cases? Perhaps, but not necessarily so. The tasks in this project were both forced on the participants and were simplistic and routine, like a school or work assignment. In other words, the information gathered was not necessarily similar to the more conceptual or product-oriented information sought by consumers of goods and services. While Flash may be helpful in aiding recall for situations similar to those tested, it should not be assumed that it will always lead to increased recall by Web users. Futhermore, advertising research suggests that previous attitudes toward brands and products, as well as previous interest in and knowledge levels about those brands and products, greatly impact recall from exposure to information about a company and its products and services. And finally, because enjoyment was higher on the Flash site, it is quite reasonable to suggest that recall was aided by the increased enjoyment. It may be that enjoyment is a necessary condition for increased recall. Means other than Flash of injecting enjoyment into a site might also lead to increased recall of information from that site.

These findings should prove useful to multimedia designers. Web site developers might be well served to create sites driven by Flash or other multimedia technologies. Logic would seem to dictate that increased enjoyment and recall might increase the likelihood that a user would return to the site. And it is just this return traffic that Web developers (and advertisers) are looking for.

While these findings may provide insight, limitations to this project exist. First, the selection of the sites may have helped facilitate the findings. It is possible that the lack of engagement with the content of the sites (i.e., technology companies) may have led the participants to rely more on peripheral cues (i.e., the multimedia features) to drive enjoyment and recall. Second, Flash technology is new, and it may be that novelty that is attractive to users. Third, this project draws a clear distinction between HTML-driven and Flash-enhanced sites. In truth, Flash can easily (and is becoming more typically) incorporated into an HTML design. One future study may seek to replicate these findings while including a third, HTML-Flash hybrid design condition. Other future research may seek to identify psychological factors involved in predicting enjoyment of sites, as well as further investigate the proposed cognitive overload phenomenon identified by many scholars.

The present study serves as an initial investigation into the influence of multimedia features and information retrieval tasks. Certainly, future research is needed to further understand the role of design as it relates to efficiency, searching time, and accuracy. We hope that this project serves as groundwork for those future explorations.

Arthur A. Raney is an assistant professor in the Department of Communication at Florida State University where he researches and teaches in the area of media effects, entertainment studies, and new media technologies. He completed his Ph.D. at the University of Alabama in mass communication in 1998. Before joining the faculty at Florida State University, Dr. Raney served as the managing director of the Institute for Communication Research at the University of Alabama and as a visiting professor in the Department of Telecommunication at Indiana University-Bloomington. He can be reached at araney@mailer.fsu.edu.

Jeremy R. Jackson is currently an Experienced Analyst with Accenture in St. Petersburg, Florida, working in the Resources and Utilities group. Jackson holds an M.S. in Interactive and New Communication Technologies and a B.S. in Interpersonal Communication from Florida State University. He can be reached at jeremy.r.jackson@accenture.com.

Debbie B. Edwards is currently working as a Project Manager for ZCOM2, an interactive division of The Zimmerman Agency, in Tallahassee, FL. She received her M.A. in Interactive and New Communication Technologies and her B.A. in Multinational Business from Florida State University. Prior to returning to graduate school, Ms. Edwards was a Project Manager for a local marketing research firm where she was responsible for individual market research projects for associations, city, state, and national clients. She can be reached at dabcre@yahoo.com.

Karrie Schaffler is currently a Human Performance Consultant with Accenture in Chicago, IL working in the Resources and Chemicals group. She earned an M.S in Interactive and New Communication Technologies and a B.S. in Interpersonal Communication from Florida State University. She is currently applying her communication and technology skills through the design, build and implementation of a web portal for her current client. In her spare time, Karrie enjoys working out, reading, and traveling. You may reach her by e-mail at klschaffler@yahoo.com.

Jean Blutenthal Arrington is a Multimedia Programmer at The Zimmerman Agency-ZCOM2 in Tallahassee, Florida. Arrington has several years experience programming CD ROMS, PDF brandbooks and websites, and using Macromedia Flash, Macromedia Director and Allaire ColdFusion for hospitality clients. She holds an M.A. in Interactive and New Communication Technologies from Florida State University and a B.A. in Communication Studies from Stetson University. She can be reached at jarrington@zcom2.com.

Melissa Price is currently working as a Special Projects Coordinator for the Counsel's Office of the Republican National Committee in Washington, DC. She received her M.A. in Interactive and New Communication Technologies and her B.A. in Public Relations from Florida State University. She can be reached at mprice@rnchq.org.

Notes

1. Flash is a proprietary standard for video, multimedia, and applications from Macromedia. It is the most popular of several tools used for multimedia Web-site design just after the turn of the century, and the plug-in that enables users to experience Flash applications is free. Other tools and scripting languages for multimedia and video on the Web include Real, Director, ColdFusion, ASP (Microsoft Active Server Pages), and PHP.

References

Hess, B. (1999). Graduate student cognition during information retrieval using the World Wide Web: A pilot study. Computers and Education, 33, 1-13. [doi: 10.1016/S0360-1315(99)00011-1]

Hu, P.J., Ma, P., & Chau, P. (1999). Evaluation of user interface designs for information retrieval systems: A computer-based experiment. Decision Support Systems, 27, 125-143. [doi: 10.1016/S0167-9236(99)00040-8]

Kirsh, D. (1997). Interactivity and multimedia interfaces. Instructional Science, 25, 79-96. [doi: 10.1023/A:1002915430871]

Norman, D. (1988). The Design of Everyday Things. New York: Doubleday.

van Oostendorp, H., & van Nimwegen, C. (1998, September). Locating information in an online newspaper. Journal of Computer-Mediated Communication, 4, 1. http://jcmc.indiana.edu/vol4/issue1/oostendorp.html

Rajani, R., & Rosenberg, D. (1999, January). Usable? Or not?: Factors affecting the usability of Web sites. CMC Magazine. http://www.december.com/cmc/mag/1999/jan/rakros.html

Spool, J.M., Scanlon, T., Snyder, C., Schroeder, W., & DeAngelo, T. (1998). Web site usability: A designer's guide. San Francisco, CA: Morgan Kaufmann.

Links from this article:

Altair Technologies, formerly http://www.altairtechnologies.com

ASP (Microsoft Active Server Pages), http://msdn.microsoft.com/library/default.asp?URL=/library/psdk/iisref/aspguide.htm

ColdFusion, http://www.macromedia.com/software/coldfusion/

Director, http://www.macromedia.com/software/director/

Flash, http://www.macromedia.com/software/flash/productinfo/product_overview/

HyperText Markup Language (HTML) Home Page, World Wide Web Consortium (W3C), http://www.w3.org/MarkUp/

Macromedia, http://www.adobe.com/

PHP, http://www.php.net

Real, http://www.real.com

Zymocreations, http://www.zymocreations.com