Introduction

Prior to the launch of the new Primo user interface in the summer of 2018, the University of Haifa conducted a usability study that aimed to determine whether users' information needs were met in a timely, convenient, and pleasant manner, the problems they encountered, and the changes necessary to meet their needs and improve their overall library experience. In order to remain relevant to its users, the library's discovery tool, as well as its services and physical conditions, must continue to make a significant contribution to the student and faculty learning and research processes.

The University of Haifa, located on top of Mount Carmel in Israel overlooking the Mediterranean Sea, has 18,000 students from a wide variety of ethnic backgrounds, studying in the following faculties: Humanities, Social Sciences, Law, Education, Management, Natural Sciences and Social Welfare & Health. Its library has been using Primo as its main discovery tool since 2009. In the summer of 2016, Ex Libris launched a new Primo user interface. Due to some Hebrew language and right to left script issues, the lack of some functionality, and the wish to examine the new interface more carefully, a decision was made at the University of Haifa to delay the new interface adoption and to launch it as a secondary interface to the Primo Classic user interface.

Since August 2017, the library has been customizing the user interface based on a number of known design principles:

1. Enhancing accessibility (for example, the background color in most of the webpages was changed to white)
2. Keeping visual consistency (calm colors, balanced graphics, helpful icons, short captions, bolder and normal-sized fonts, and more)
3. Organizing the information in a consistent and known manner that takes into consideration the emotional and cognitive needs of users (the order of information in the full display of records, positioning the facets' pane on the opposite side, adding a link to a known Hebrew online dictionary—as it is a multilingual institution)
4. Facilitating navigation and short paths to known functions
5. Maintaining quick responses to user actions, and ensuring that the system is available, up-to-date, and functioning

As some of the issues were beyond the responsibility of the Primo Team, customer support cases were opened, questions were sent to the Primo community list, and changes were made after consultation with library colleagues.

Prior to usability testing, a feedback form was published on the website, which provided mostly vague and irrelevant responses. As human-computer interactions dominate patrons' lives, and as many are very familiar with aspects of the user experience, it was hoped that by conducting a systematic usability test they would be able to provide a different perspective to the new Primo user interface design.

Literature Review

Recent technological developments have raised a new type of learner who experiences the learning and research processes in a very different way from the old type of learner. Thus, the user experience is not only about getting the job done, or doing it efficiently, it is also about satisfying users' expectations of receiving a more interactive experience, giving them more control in the discovery process, and offering them as many options as possible to perform their assignments, while taking into consideration their different needs (Dede, 2005; Melcher, 2007). Another characteristic of the new millennial learner is multitasking (Judd & Kennedy, 2011) and the use of a number of media and parallel applications (Rideout, Foehr, & Roberts, 2010, 11–21; Roberts & Foehr, 2008). Some researchers argue that millennials often use fast linear task switching (also known as 'switch of attention') and rarely achieve full multitasking, but this depends on the type of the activity being performed (Wallis, 2006; Wallis, 2010).

Changing the library's main search interface demands consideration of young users as well as older users (lecturers, senior lecturers, and even students) who face technological barriers due to changes and innovations (Wandke, Sengpiel, & Sönksen, 2012). It is important to think about a wide variety of interpretations of terminology, icons, and other elements in the interface. For example, the icon of three dots means "additional options" in a menu, and a white box usually means “type here.” Additionally, smart devices (like smartphones) are useful tools for learners of all ages (Norris, Hossain, & Soloway, 2011), so it is important to have a new user interface that fully supports these devices in order to ensure that the default design, as well as functionality, took into consideration all types of user experience.

According to Krug (2010, pp. 16–17), usability testing is conducted because by observing users, designers will be better equipped to understand their needs. Library discovery tools, like commercial websites, aim for novice and experienced users to navigate them with ease, and to perceive them as an asset to their study or research. Usability tests are usually conducted during the planning stages of a new website, or before customization is done on a purchased product, such as a library discovery tool. It should be an ongoing process and is largely dependent on the ability and willingness of library systems' analysts to implement changes once problems have been identified.

Academic libraries usually conduct usability tests with the aid of an assessment team that decides on the population groups, sample, recruiting methods, and whether to compensate participants for their time (Krug, 2010, p. 39). It is important to adhere to the User Experience Professionals Association’s Code of Conduct (2005). According to Nielsen (2000), it is sufficient to conduct a usability test with five users as almost as many problems are gleaned with five as with double that number. He recommends using the time and resources saved by testing fewer participants for conducting additional rounds of testing. Krug goes one step further by claiming that three users are enough because "you’re not interested in what it takes to uncover most of the problems; you only care about what it takes to uncover as many problems as you can fix" (Krug, 2010, p. 43). He states that the first three users are very likely to encounter the most significant problems, it is less labor-intensive to test three people, and testing more than three often leads to discovering many minor issues, making it harder to see the most serious problems—the ‘can’t see the forest for the trees' effect. Rubin and Chisnell (2008, p. 54) recommend using a Verbal Protocol or a Think Aloud approach as "asking participants to think aloud during their sessions also reveals important clues about how they are thinking about the product or system they are using and whether the way it works matches up with the way it was designed." However, as talking slows down performance, this method should only be used when there is a more flexible time frame for task completion.

Although several usability studies have been conducted on Primo since its inception in 2012 (Brett, Lierman, & Turner, 2016; Kliewer, Monroe-Gulick, Gamble, & Radio, 2016; Nichols, Billey, Spitzform, Stokes, & Tran, 2014; Perrin, Clark, De Leon, & Edgar, 2014; Valentine & West, 2016; Warren, 2017), very few have been carried out on the new user interface. In addition, none have been carried out on a multilingual or bilingual interface (such as English and Hebrew). The current study was performed on the Hebrew version of the interface, which was its default language. Although the mother tongue of several participants—who come from different ethnic backgrounds—was not Hebrew (e.g. English and Russian), English is the language of most of their academic requirements. Furthermore, unlike other libraries, the University of Haifa does not use a blended search for the library catalog and Primo Central Index (Arguello, Wu, Kelly, & Edwards, 2012). This was primarily because a major Israeli database, the Index to Hebrew Periodicals (consisting of mostly Hebrew articles), could not be incorporated into Primo due to its use of a locally-developed thesaurus, rather than Library of Congress Subject Headings. As a result, Haifa's interface has three distinct tabs (in the new user interface they appear in a drop-down menu): (1) Library Catalog; (2) Haifa Index to Hebrew Periodicals; and (3) Primo Central Index—and two interface languages (Hebrew and English), which made the usability test slightly more complicated than with a blended-search, single-language interface.

Methodology

In December 2017, an email invitation and online registration form were sent to a representative sample of randomly selected students and faculty at the University of Haifa. Notwithstanding Nielsen's and Krug's recommendations to test no more than five users, a total of ten users were invited to participate in one of the three sessions, primarily because they were from three different population groups: one was a professor from the Faculty of Management and two were from the Department of Nursing, three were B.A. students from the Departments of Business Administration, Biology, and English Language & Literature, and four were M.A. students from the Departments of Hebrew Language, Hebrew & Comparative Literature, Archaeology, and Philosophy. An additional reason for testing ten users was that it enabled more questions to be asked: five participants were asked to perform one set of tasks for a maximum of one hour, and five others were asked to perform a different set of tasks. Two participants performed both sets of tasks (see appendix).

For internal reasons, the questions consisted of some very specific issues that were known problems, and not just general, easily solvable ones, as recommended by Krug. Specifically, the Media Department requested a question regarding the video recordings collection. The participants' verbalized thoughts and actions were recorded using Camtasia screen recorder, while a member of the assessment team took notes and answered questions (but did not provide solutions to the tasks, unless the participant was at a dead end for more than a minute). This method was more interactive than recommended by Nielsen and Krug, and it allowed participants to state their frustrations, expectations, and recommendations at every stage, which better suits Israeli users. After they had completed each task, the facilitators asked them whether the experience had been positive or negative, and gave them an opportunity to provide general comments on the visual display and layout. After all the sessions were completed, the team members viewed the recordings and their notes and analyzed the findings.

Results

The prevailing view was that the interface was more user-friendly than the classic Primo interface, and was especially suited to undergrads and younger users who are at ease with visual and interactive interfaces. Comments such as "it is intuitive" and "I found it easy to use" were typical of this group. Even when users in this group encountered difficulties, they quickly overcame them. Among older users, comments such as, "It is not simple," "I need to learn," "I need to receive instruction," "Next time, I will perform better," "I don't understand all the symbols," "It is different from the classic Primo," and "It is designed for undergrads" were typical. All participants, however, gave positive feedback on the graphical design of the interface with comments such as, "It has clean lines," "It is simple and modern," and "It is pleasant to the eye."

Most participants easily located specific and general information and performed a variety of tasks. They liked the display of all details on one screen and the save pin and facets, which were considered useful and convenient features. The functions they performed successfully were: finding specific items and accessing their full details, locating physical items and viewing electronic items, navigating within a list of results and sorting them, narrowing results using facets, using the advanced search, saving items and queries, creating alerts, and signing into their account.

The main problems we observed were connected to unclear terminology and layout, however, others were due to lack of familiarity with the interface, lack of technological literacy, and inexperience with library systems in general. The problems are divided into six categories below.

Summary and Conclusions

Although usability testing is a time-consuming pursuit, it proved to be very worthwhile at the University of Haifa as it highlighted various known and new issues, enabling staff to prioritize and take care of them. The usability report was approved by the department heads and solutions were ranked in order of importance. Several issues were implemented immediately by the Primo Team, while others required more planning, or were considered bugs or enhancements and were submitted through various Ex Libris channels. The results and recommendations, as well as the decisions and actions, can be implemented by other institutions based on local and institutional considerations.

Appendix

References

• Arguello, J., Wu, W., Kelly, D., & Edwards, A. (2012). Task complexity, vertical display and user interaction in aggregated search. Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval (pp. 435–444). New York: ACM. Retrieved from https://ils.unc.edu/~jarguell/ArguelloSIGIR12.pdf
• Brett, K., Lierman, A., & Turner, C. (2016). Lessons learned: A Primo usability study. Information Technology and Libraries, 35(1), 7–25. doi:10.6017/ital.v35i1.8965
• Dede, C. (2005). Planning for neomillennial learning styles. Educause Quarterly, 28(1), 7–12. Retrieved from http://www.wou.edu/~girodm/670/Dede_on_neo-millenials.pdf
• Judd, T., & Kennedy, G. (2011). Measurement and evidence of computer-based task switching and multitasking by ‘Net Generation’ students. Computers & Education, 56(3), 625–631. doi:10.1016/j.compedu.2010.10.004
• Kliewer, G., Monroe-Gulick, A., Gamble, S., & Radio, E. (2016). Using Primo for undergraduate research: A usability study. Library Hi Tech, 34(4), 566–584. doi:10.1108/LHT-05-2016-0052
• Krug, S. (2010). Rocket surgery made easy: The do-it-yourself guide to finding and fixing usability problems. Berkeley, CA: New Riders. Retrieved from https://books.google.co.il/books?id=9Q3OQVyX_-QC&pg=PA2&source=gbs_toc_r&cad=4#v=onepage&q&f=false
• Melcher, M. (2007). Usability 2.0 again. Retrieved from https://x28newblog.wordpress.com/2007/05/26/usability-20-again
• Nichols, A., Billey, A., Spitzform, P., Stokes, A., & Tran, C. (2014). Kicking the tires: A usability study of the Primo discovery tool. Journal of Web Librarianship, 8(2), 172–195. doi:10.1080/19322909.2014.903133
• Nielsen, J. (2000). Why you only need to test with 5 users. Retrieved from https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/
• Norris, C., Hossain, A., & Soloway, E. (2011). Using smartphones as essential tools for learning: A call to place schools on the right side of the 21st century. Educational Technology, 51(3), 18–25.
• Perrin, J. M., Clark, M., De Leon, E., & Edgar, L. (2014). Usability testing for greater impact: A Primo case study. Information Technology and Libraries, 33(4), 57–66. doi:10.6017/ital.v33i4.5174
• Rideout, V. J., Foehr, U. G., & Roberts, D. F. (2010). Generation M²: Media in the Lives of 8- to 18-Year-Olds. Menlo, CA.: Henry J.Kaiser Family Foundation. Retrieved from http://www.kff.org/entmedia/upload/8010.pdf
• Roberts, D. F., & Foehr, U. G. (2008). Trends in media use. The Future of Children, 18(1), 11–37. Retrieved from https://www.jstor.org/stable/pdf/20053118.pdf
• Rubin, J., Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley Publishing. Retrieved from http://ccftp.scu.edu.cn:8090/Download/efa2417b-08ba-438a-b814-92db3dde0eb6.pdf
• User Experience Professionals Association. (2005). UXPA Code of Professional Conduct. Retrieved from https://uxpa.org/resources/uxpa-code-professional-conduct
• Valentine, B., & West, B. (2016). Improving Primo usability and teachability with help from the users. Journal of Web Librarianship, 10(3), 176–196. doi:10.1080/19322909.2016.1190678
• Wallis, C. (2006). The multitasking generation. Time Magazine, 167(13), 48–55. Retrieved from http://www.balcells.com/blog/images/articles/entry558_2465_multitasking.pdf
• Wallis, C. (2010). The impacts of media multitasking on children’s learning and development: Report from a research seminar. New York: The Joan Ganz Cooney Center at Sesame Workshop.
• Wandke, H., Sengpiel, M., & Sönksen, M. (2012). Myths about older people’s use of information and communication technology. Gerontology, 58(6), 564–570. doi:10.1159/000339104
• Warren, R. (2017). Usability study identifies vocabulary, facets, and education as primary Primo discovery system interface problems. Evidence Based Library and Information Practice, 12(3), 177–179. doi:10.18438/B89M14