Abstract

Interactive Scientific Publishing (ISP) has been developed by the Optical Society of America with support from the National Library of Medicine at NIH. It allows authors to electronically publish papers which are linked to the referenced 2D and 3D original image datasets. These image datasets can then be viewed and analyzed interactively by the reader. ISP provides the software for authors to assemble and link their source data to their publication. But more important is that it provides readers with image viewing and analysis tools. The goal of ISP is to improve learning and understanding of the presented information. This paper describes ISP and its effect on learning and understanding. ISP was shown to have enough educational value that readers were willing to invest in the required set–up and learning phases. The social aspects of data sharing and the enlarged review process may be the hardest obstacles to overcome.

Introduction

The past twenty years has been called the time of the information super highway. Now we have moved into the time of social media. The web is no longer just a means for requesting and distributing information. It is now a means for social interaction.

Several years ago, the U.S. National Library of Medicine (NLM) became interested in what we considered a new form of publication made possible by the interactive nature of the World Wide Web, interactive publication. At the time, journal articles were beginning to be distributed through the web in digital form, most commonly in PDF format, a use of the web for information distribution, not information interaction. We reasoned that digital distribution of journal articles could allow far more reader interaction than print media. Why couldn’t the reader interact directly with the data on which the publication was based, rather than just looking at the published pictures and graphs? We reasoned that this interaction might give the reader a greater understanding and retention of the publication and an opportunity to try additional analysis on the published data. But would the reader make use of this added opportunity? It was decided to do a pilot project to test this hypothesis.

In order to do this project as we envisioned, one needs free software to provide the means for reader interaction and a series of journal articles, each with its underlying data to provide the information for that interaction. The NLM found a partner in the Optical Society of America (OSA). The OSA, a membership society and publisher, had a vision of the Web as a medium for reader interaction with scholarly publications, a vision which was similar to that of the NLM.

In 2007 the NLM joined with the OSA to do a study under the title of “Interactive Science Publishing” (ISP). The OSA agreed to publish a series of issues of their Medline indexed journals, Optics Express and Applied Optics, on topics dealing with image related biomedical research. Simultaneously with the release of each of the print published issues, a free on–line version would be released. The on–line version would contain the original printed article as well as links to the source image data sets on which the article was based. Each on–line article would be freely downloadable and viewable with standard PDF reader software. The NLM and ISP would provide free, downloadable software which functioned as a “PDF plug–in” as well as independently. This software would allow interaction with the source images associated with the PDF formatted publication being read.

The NLM and the OSA also agreed on another aspect of this means of publication. The articles would be open access when published and the linked data sets would be available as open data. Each of the associated data sets would be stored externally from its published article so that it could be individually indexed, would be searchable through a variety of search engines, and would be individually downloadable[1]. This arrangement allows an individual data set to be associated, assuming author credit is given, with a new publication which may not have been written by the original author (Figure 1). In effect a new “tenure” commodity is being demonstrated and proposed—how many datasets have you published and how many articles have colleagues published based on them?

Figure 1—A page from the “MIDAS” database from which each of a paper’s associated data sets can be accessed externally from its published article. Through this mechanism each dataset can be individually indexed, is searchable through a variety of search engines, and can be individually downloaded.

Freely available computer software that would permit readers to download and interact with the datasets referenced in an ISP digital document was developed by KitWare, a commercial software developer. The software was designed to be easily downloadable by users for installation on their own Microsoft or Apple based computers. Researchers were invited to download and install the software[2] and then download and look at the ISP on–line publications and interact with them[3]. In the initial stage of this study three journal issues were published containing 29 papers and 320 associated data sets (Figure 2).

This was followed by a series of web–based evaluations intended to measure and assess the performance of the software and the learning effect of using ISP on the reader. This initial study indicated that interactive data sets offer potential benefits as a complement to published papers. However, it was also found that the software interface, instructions, and/or software download presented a barrier to the effective use of the interactive functionality. This needed to be improved before a more in–depth evaluation could be undertaken[4].

With these preliminary findings, the reader software was modified to make the installation easier and less prone to installation error. Help files and pop–up labels, associated with each of the reader’s function icons were added. Access to the toll free telephone Help Desk was improved. The upgraded software was released in time for the publication of the fourth issue. This was followed by two additional issues which added a total of 9 papers and 44 additional associated data sets (Figure 3).

Methods

In order to gain a deeper understanding of users' perspectives and to answer the driving question behind this project–does interactive publishing lead to improved understanding and learning?–the OSA and the NLM contracted the CFI Group, a worldwide customer feedback organization, to conduct one–on–one telephone interviews with ISP users after the publication of the sixth issue. An interview guide was developed collaboratively with the OSA and the NLM. Management interviews with Kitware, NLM and OSA were conducted first in order to inform the interview protocol. A list of respondents from the initial phase survey, conducted by ForeSee Results, was provided to CFI Group. Twenty interviewees were recruited from this list to participate in 30–minute interviews. Two of the twenty interviews were used to help further refine the interview guide.

Results

The twenty interviewees who participated in the one–on–one interviews represented a variety of backgrounds in the fields of optics, physics, medicine and engineering. The group consisted of senior research and medical scientists, research assistants, and pre– and post–doctoral students from across the United States. They reported that they accessed journals from daily to monthly. Most had submitted articles to journals and were published authors. All interviewees had some level of experience with the ISP Software as all were selected from a pool of participants who had completed an initial survey about ISP Software. Interviewees’ experience with ISP Software ranged from just a two or three times using the software to “dozens of times” over the past year.

Installation and Start Up

For most respondents the ISP software startup from a downloaded ISP–enabled PDF manuscript was not an issue; it performed as expected (Figure 4). The interface or “look and feel” was important to many of the interviewees as it was the first impression users had of the software. A few did mention that content was much more important and “look and feel” was secondary. Users wanted interfaces that are intuitive, user–friendly, easy to navigate, and did not require much training to get started. For the most part, users thought the ISP Software interface was close to meeting these criteria. While there may have been a few areas within the interface that could be fine–tuned, users felt an overhaul is not necessary.

Figure 4—A PDF page containing an image with which the viewer can interact using the pre–installed ISP software and clicking on “View 1.”

Upon first using the ISP Software, many users did not choose to begin by going through the tutorial. In general, the software tutorial was mostly used in a help mode by those who did use it. This was not because users felt the tutorial was ineffective but rather they mostly felt that the software was intuitive enough to understand it without the tutorial. Often, respondents would use the tutorial just to learn more about a specific feature. The ISP Project should have considered promoting the use of the tutorial. This would likely be a benefit to many users as it was apparent in the interviews that there were quite a few software features of which they had low awareness. Specifically, in their evaluation of ISP Software's tools, many interviewees noted that they had not used the following tools or features, not because they did not need them, but because they were not aware of their availability: computer system hardware check, visualization transfer functions and the lung lesion sizing tool. The few who had familiarity with these tools had positive comments about them. In most instances the computer system hardware check was not needed because the user knew their system fully met the published requirements. However, the visualization transfer functions and lung lesion sizing tools had lower usage due to lack of awareness. Promoting the awareness of these features by strongly encouraging the use of the tutorial might have created a more valuable user experience.

Tools for Interactivity

As for the tools (Figure 5) that were widely used, rotating was the one most mentioned. Not surprisingly, it was also viewed as one of, if not the most useful tool. Along with panning, rotating images was mentioned as being critical to good interactivity. Rotating data sets in 3–D not only had a “wow factor” for many users but in many cases the user thought he or she had a better understanding of the data through manipulating it in this manner. Return to original state was also mentioned as a widely used and highly valuable tool. Users could freely experiment with images knowing that they could easily undo the changes they made. Data rendering was of high quality and highly interactive in the opinion of most users. Most were able to easily manipulate images and the software seemed responsive to the commands they gave. In very few instances did users feel that there was a lag in response to the commands when rendering data. For some users, the responsiveness in rotating images was the one area where the software was a bit slow. Some of these users reported that images seemed “jumpy” when they were being rotated. This may have been caused by the marginal nature of the user’s computer system in meeting the published required system specification, a risk which would have been known to the user if the Computer System Hardware Tool had been used during setup. Since most of the interviewees were using the ISP Software with a high–speed connection speed of downloading files was not an issue.

Figure 5—ISP interactive screen produced by clicking on “View 1” in image of reproduced PDF page shown in figure 4.

Apart from specific tools and features, users saw the overall value in ISP Software conceptually as serving as a “uniform protocol” format for researchers to share their data. Users thought the value derived from this protocol would increase as the number of users and amount of content grew. A few noted that the biggest weakness of the ISP Software was not a specific feature or lack thereof, but rather it being relatively unknown and not being used widely across journals. Therefore, a focus for ISP may not be on the tool itself but in promoting its use in articles and other journals.

Discussion

Our goal was to discover whether or not truly interactive publishing would add educational value for the reader. The simple answer is yes, but circumstances are never simple.

After the publication of our preliminary findings[5] the Radiological Society of North American (RSNA) did a similar experiment[6] but focused on the problems and use of this mode of publishing. The experiment consisted of two articles published in the journal RadioGraphics, both in print and on the web. The articles referred to 11 data sets. Reader software, provided by TeraRecon, was included with each data set download and prohibited the data set from being saved locally. The RSNA tracked the problems inherent in producing and providing this kind of interactivity and user utilization. What they did not track was the educational impact, both long term and short term, on readers.

In our study the original research data was made freely available to the reader as was a free and independent set of tools to re–analyze and visualize that data. Other interactive publications provide appropriate videos to help the reader with better understanding. The reader can start, stop, fast forward and fast rewind the videos at will but cannot deviate from their set content. The reader is never allowed to directly interact with the data or given the ability to ask the perennial “What if . . .” question. This of course puts greater burden on the reader but from the results of this study it would appear that the extra reader effort is worth the investment.

The image data sets involved in this study were all quite large and had to be downloaded by the reader in order for any interaction to be possible. Our readers were all academic or industry based and it would appear that they either did their reading at their work site or they had high–speed internet connections at home as none of them complained about the speed of the data download which would certainly interfere with learning. As time goes on, the download speed problem will be minimized, but I believe we were fortunate in the choice of our reader participants.

In the same vein, the effective speed of the computer being used by the reader is also of critical importance. This became clear in the preliminary study where many readers complained that the downloaded software would not run smoothly, interfering with their interactive experience. A module was added to the software which checked the adequacy of the computer to smoothly run the software package each time it was initiated. The module warned the user if a slow response was predicted and indicated the reason for the slow response, i.e., computer clock speed, memory size, video card, etc. Computer adequacy became a non–issue.

The issues that would not go away centered on the data sets themselves. A scientific publication is made possible by the willingness of investigators to share their findings in a public forum and the willingness and expertise of colleagues to review these findings before publication. In order for a paper to be accepted for ISP, the authors had to be willing to share their data in the public domain. Although this seems to be a tradition in many of the physical sciences and astronomy, and in the genomics community, it was a problem for authors in the medical imaging community. The RSNA avoided this problem by keeping the associated data integral to its associated paper.

All ISP papers were independently reviewed before being accepted for publication. The reviewer’s job was not only to review the paper but also to review the data sets associated with the paper. How does one review the data sets? We have norms and expectations for the proper review of a paper, but what are the norms and expectations for the proper review of data, especially in this situation where the provided software will allow infinite views of that data? The RSNA relates a similar experience.

When this study was conceived it was designed to give us insight into a future reality. Perhaps that is why we needed continuing software and reader support. This will change as the technology of truly interactive publishing becomes more common. The problems of finding authors and recruiting reviewers will require a culture change in parts of our biomedical community.

Perhaps our vision was too far into the future. We set up a situation that created a new tenureable commodity, the shareable open data set. We received no comments about this. The free interactive software reader was not only an ISP PDF add–on, but a complete ISP stand–alone image analysis and visualization program. As best as we were able to tell, none of the readers realized this fact and always only used it in conjunction with their PDF reader.

Conclusion

Truly Interactive Science Publishing was shown to have enough educational value that readers were willing to invest in the needed set–up and learning phases. Problems encountered in network and computer speed can now be minimized by running the ISP software in a cloud computing environment which will minimize the dependence on local computer and network speeds. The social aspects of data sharing and the enlarged review process may be the hardest obstacles to overcome.

Acknowledgments

The author would like to thank his Optical Society of America colleagues Dr. John Childs, Scott Dineen and Grace Klonoski, and Kristen McMurphy of ForSee Results and his National Library of Medicine Colleagues Dr. Elliot Siegel and Dr. Fred Wood for their contributions to this project.

As an Assistant Director for High Performance Computing and Communications at the National Library of Medicine, Dr. Michael Ackerman has been working on the cutting edge of electronic media and open source software and data for over 25 years. His Visible Human Project was an experiment into the utility and usability of 3–dimensional electronic image databases. The Insight Tool Kit (ITK) Project was generally an experiment in the creation, maintenance and usability of Open Source software, specifically in the area of 3–dimensional electronic imaging. His project reported here, Interactive Scientific Publishing (ISP), is an effort to use the full capability of electronic media, not just for the distribution of publications, but to freely interact with the data cited in those publications, specifically 3–dimensional image data. ackerman@nlm.nih.gov

Notes

1. “MIDAS” accessed October 10, 2014, http://midas.osa.org/midaspub/browse

2. “ISP Software” accessed February 5, 2014, http://www.opticsinfobase.org/osaisp/isphowto.cfm

3. “ISP” accessed February 5, 2014, http://www.opticsinfobase.org/isp.cfm#articles

4. Ackerman, M.J., E. Siegel, F. Wood. “Interactive Science Publishing: A joint OSA-NLM project.” Information Services & Use, 30:39–50, 2010.

5. Ibid.

6. Morgan, T.A., E. Flanders, W.W. Olmstead, S.D. Steenburg, E.L. Siegel. “Challenges Encountered and Lessons Learned in Initial Experience with the Next Generation of Interactive Radiology Literature in RadioGraphics.” RadioGraphics 32:929–934, 2012.