The Educational Value of Truly Interactive Science Publishing
Skip other details (including permanent urls, DOI, citation information)
This work is licensed under a Creative Commons Attribution 3.0 License. Please contact firstname.lastname@example.org to use this work in a way not covered by the license. :
For more information, read Michigan Publishing's access and usage policy.
Interactive Scientific Publishing (ISP) has been developed by the Optical Society of America with support from the National Library of Medicine at NIH. It allows authors to electronically publish papers which are linked to the referenced 2D and 3D original image datasets. These image datasets can then be viewed and analyzed interactively by the reader. ISP provides the software for authors to assemble and link their source data to their publication. But more important is that it provides readers with image viewing and analysis tools. The goal of ISP is to improve learning and understanding of the presented information. This paper describes ISP and its effect on learning and understanding. ISP was shown to have enough educational value that readers were willing to invest in the required set–up and learning phases. The social aspects of data sharing and the enlarged review process may be the hardest obstacles to overcome.
The past twenty years has been called the time of the information super highway. Now we have moved into the time of social media. The web is no longer just a means for requesting and distributing information. It is now a means for social interaction.
Several years ago, the U.S. National Library of Medicine (NLM) became interested in what we considered a new form of publication made possible by the interactive nature of the World Wide Web, interactive publication. At the time, journal articles were beginning to be distributed through the web in digital form, most commonly in PDF format, a use of the web for information distribution, not information interaction. We reasoned that digital distribution of journal articles could allow far more reader interaction than print media. Why couldn’t the reader interact directly with the data on which the publication was based, rather than just looking at the published pictures and graphs? We reasoned that this interaction might give the reader a greater understanding and retention of the publication and an opportunity to try additional analysis on the published data. But would the reader make use of this added opportunity? It was decided to do a pilot project to test this hypothesis.
In order to do this project as we envisioned, one needs free software to provide the means for reader interaction and a series of journal articles, each with its underlying data to provide the information for that interaction. The NLM found a partner in the Optical Society of America (OSA). The OSA, a membership society and publisher, had a vision of the Web as a medium for reader interaction with scholarly publications, a vision which was similar to that of the NLM.
In 2007 the NLM joined with the OSA to do a study under the title of “Interactive Science Publishing” (ISP). The OSA agreed to publish a series of issues of their Medline indexed journals, Optics Express and Applied Optics, on topics dealing with image related biomedical research. Simultaneously with the release of each of the print published issues, a free on–line version would be released. The on–line version would contain the original printed article as well as links to the source image data sets on which the article was based. Each on–line article would be freely downloadable and viewable with standard PDF reader software. The NLM and ISP would provide free, downloadable software which functioned as a “PDF plug–in” as well as independently. This software would allow interaction with the source images associated with the PDF formatted publication being read.
The NLM and the OSA also agreed on another aspect of this means of publication. The articles would be open access when published and the linked data sets would be available as open data. Each of the associated data sets would be stored externally from its published article so that it could be individually indexed, would be searchable through a variety of search engines, and would be individually downloadable. This arrangement allows an individual data set to be associated, assuming author credit is given, with a new publication which may not have been written by the original author (Figure 1). In effect a new “tenure” commodity is being demonstrated and proposed—how many datasets have you published and how many articles have colleagues published based on them?
Freely available computer software that would permit readers to download and interact with the datasets referenced in an ISP digital document was developed by KitWare, a commercial software developer. The software was designed to be easily downloadable by users for installation on their own Microsoft or Apple based computers. Researchers were invited to download and install the software and then download and look at the ISP on–line publications and interact with them. In the initial stage of this study three journal issues were published containing 29 papers and 320 associated data sets (Figure 2).
|October 2008||7||45||“Interactive Science Publishing Introductory Issue”|
|March 2009||17||242||“Optical Coherence Tomography in Ophthalmology”|
|October 2009||5||33||“Digital Holography”|
This was followed by a series of web–based evaluations intended to measure and assess the performance of the software and the learning effect of using ISP on the reader. This initial study indicated that interactive data sets offer potential benefits as a complement to published papers. However, it was also found that the software interface, instructions, and/or software download presented a barrier to the effective use of the interactive functionality. This needed to be improved before a more in–depth evaluation could be undertaken.
With these preliminary findings, the reader software was modified to make the installation easier and less prone to installation error. Help files and pop–up labels, associated with each of the reader’s function icons were added. Access to the toll free telephone Help Desk was improved. The upgraded software was released in time for the publication of the fourth issue. This was followed by two additional issues which added a total of 9 papers and 44 additional associated data sets (Figure 3).
|July 2010||4||36||"Imaging in Diagnosis and Treatment of Lung Cancer"|
|October 2011||1||4||"Applications of Nano-optics"|
|December 2011||4||21||"Digital Holography"|
In order to gain a deeper understanding of users' perspectives and to answer the driving question behind this project–does interactive publishing lead to improved understanding and learning?–the OSA and the NLM contracted the CFI Group, a worldwide customer feedback organization, to conduct one–on–one telephone interviews with ISP users after the publication of the sixth issue. An interview guide was developed collaboratively with the OSA and the NLM. Management interviews with Kitware, NLM and OSA were conducted first in order to inform the interview protocol. A list of respondents from the initial phase survey, conducted by ForeSee Results, was provided to CFI Group. Twenty interviewees were recruited from this list to participate in 30–minute interviews. Two of the twenty interviews were used to help further refine the interview guide.
The twenty interviewees who participated in the one–on–one interviews represented a variety of backgrounds in the fields of optics, physics, medicine and engineering. The group consisted of senior research and medical scientists, research assistants, and pre– and post–doctoral students from across the United States. They reported that they accessed journals from daily to monthly. Most had submitted articles to journals and were published authors. All interviewees had some level of experience with the ISP Software as all were selected from a pool of participants who had completed an initial survey about ISP Software. Interviewees’ experience with ISP Software ranged from just a two or three times using the software to “dozens of times” over the past year.
Installation and Start Up
For most respondents the ISP software startup from a downloaded ISP–enabled PDF manuscript was not an issue; it performed as expected (Figure 4). The interface or “look and feel” was important to many of the interviewees as it was the first impression users had of the software. A few did mention that content was much more important and “look and feel” was secondary. Users wanted interfaces that are intuitive, user–friendly, easy to navigate, and did not require much training to get started. For the most part, users thought the ISP Software interface was close to meeting these criteria. While there may have been a few areas within the interface that could be fine–tuned, users felt an overhaul is not necessary.
Upon first using the ISP Software, many users did not choose to begin by going through the tutorial. In general, the software tutorial was mostly used in a help mode by those who did use it. This was not because users felt the tutorial was ineffective but rather they mostly felt that the software was intuitive enough to understand it without the tutorial. Often, respondents would use the tutorial just to learn more about a specific feature. The ISP Project should have considered promoting the use of the tutorial. This would likely be a benefit to many users as it was apparent in the interviews that there were quite a few software features of which they had low awareness. Specifically, in their evaluation of ISP Software's tools, many interviewees noted that they had not used the following tools or features, not because they did not need them, but because they were not aware of their availability: computer system hardware check, visualization transfer functions and the lung lesion sizing tool. The few who had familiarity with these tools had positive comments about them. In most instances the computer system hardware check was not needed because the user knew their system fully met the published requirements. However, the visualization transfer functions and lung lesion sizing tools had lower usage due to lack of awareness. Promoting the awareness of these features by strongly encouraging the use of the tutorial might have created a more valuable user experience.
Tools for Interactivity
As for the tools (Figure 5) that were widely used, rotating was the one most mentioned. Not surprisingly, it was also viewed as one of, if not the most useful tool. Along with panning, rotating images was mentioned as being critical to good interactivity. Rotating data sets in 3–D not only had a “wow factor” for many users but in many cases the user thought he or she had a better understanding of the data through manipulating it in this manner. Return to original state was also mentioned as a widely used and highly valuable tool. Users could freely experiment with images knowing that they could easily undo the changes they made. Data rendering was of high quality and highly interactive in the opinion of most users. Most were able to easily manipulate images and the software seemed responsive to the commands they gave. In very few instances did users feel that there was a lag in response to the commands when rendering data. For some users, the responsiveness in rotating images was the one area where the software was a bit slow. Some of these users reported that images seemed “jumpy” when they were being rotated. This may have been caused by the marginal nature of the user’s computer system in meeting the published required system specification, a risk which would have been known to the user if the Computer System Hardware Tool had been used during setup. Since most of the interviewees were using the ISP Software with a high–speed connection speed of downloading files was not an issue.
Apart from specific tools and features, users saw the overall value in ISP Software conceptually as serving as a “uniform protocol” format for researchers to share their data. Users thought the value derived from this protocol would increase as the number of users and amount of content grew. A few noted that the biggest weakness of the ISP Software was not a specific feature or lack thereof, but rather it being relatively unknown and not being used widely across journals. Therefore, a focus for ISP may not be on the tool itself but in promoting its use in articles and other journals.
Role in Learning
Not all, but certainly many users found value in ISP Software helping them in understanding the material presented by the author. Viewing and manipulating data sets in 3–D allowed the user to access and see data beyond what a PDF article or other conventional methods of displaying 3–D data (such as CAD software) could convey. To that end, it may have helped some to more quickly understand the published information. How the author presented the material and how they chose to display the image data sets also impacted the value the reader could derive. For some readers having the ability to manipulate the data set allowed him or her to “look inside” the data, perhaps for insights beyond what was presented by the author. Conversely, other respondents felt that the 3–D imagery was more of a novelty, which did not necessarily provide them with greater insight. For these users the imagery in ISP Software was more of a “qualitative tool”; they would still need to look at the numbers and perform calculations to gain insight. Regardless of the amount of value the user derived, nearly all interviewees thought that the investment to start using ISP Software was so minimal in terms of time and effort that any benefit they reaped was worth it.
For many respondents, ISP did play some role in helping them understand the material, but the degree to which it helped varied. For some it was not a “fundamental” role, but ISP did augment their understanding of the data. In particular, one respondent felt that “visual learners” would be able to benefit more from having the capability to interact with the data in the manner presented by ISP.
Having the ability to interact with the data in 3–D was not only novel but allowed the user to better see the data “as a whole.” Having another dimension (3–D instead of 2–D) conveyed more information to the user. Seeing and manipulating a 3–D image allows the reader to see complex data sets in a view that may not have been presented by the author and to look further “inside” the data. It also allows the reader to do a check of the author's results. How the author chooses to use the 3–D images is critical to how much value ISP providers the reader.
Respondents were asked about the speed of understanding the material with ISP. ISP Software allows the user to better see the whole data set. This allows the user to save time by seeing all of the information they need at once and answer questions that they would potentially have had to have contacted the author or some other source. Other users were not as sure about the ISP Software improving the actual speed of comprehending material, but some thought it would help them understand more thoroughly. ISP helped the user better customize the information they choose to view and so more information can be accessed by the user. Some used the images from ISP as a teaching tool or to illustrate points they are making in discussions with others.
Some thought the software made the data and the main points that the author was presenting more accessible. Having a 3–D image to support the points allowed the information to become more tangible to the reader. Seeing the data cut down on the time needed to synthesize information and gain a better understanding of what the author was presenting.
Analysis of Data
For some, the ISP Software did not affect how they access or analyze data. For them ISP is more of a “qualitative” tool that provides additional color. If they wanted to do analysis they still needed to put the data set into software that they are more comfortable using, such as Excel or MATLAB. ISP was viewed by some as being able to speed up the analysis or understanding process by allowing the user to quickly manipulate the data to understand connections between the data and the author's conclusions.
Some respondents mentioned that the software would not affect how they approach data sets. They felt the ISP Software and imaging was nice but not going to change their approach to data—which for some came down to the numbers themselves. Software such as MATLAB was mentioned as the main tools for analysis.
Value to Work
Finding content in their areas of interest was a barrier for some to find direct value in the ISP Software. Still while some have yet to find articles in their area of interest, the method of delivering information was found to be more efficient and providing more information than just reading the text. ISP allowed users to explore the fundamental concepts of an article and to experiment so they can better understand the topic presented.
Value Compared to Investment
Respondents were mostly unanimous on the investment of time being worthwhile in terms of benefits received from using ISP. Even for those respondents who did not find much material in their field of interest, the up–front time to install and learn the software was so minimal that it was still worth it to them as well. The intuitive nature of the software helped to lessen the up–front time needed to begin using ISP for most users.
Respondents were very likely to use the ISP Software again in the future. They anticipate that if there is an article in their area of interest that uses ISP they will access it using the software. However, many commented that they would not specifically seek out an article just because it uses ISP if it was not in their area of interest.
Our goal was to discover whether or not truly interactive publishing would add educational value for the reader. The simple answer is yes, but circumstances are never simple.
After the publication of our preliminary findings the Radiological Society of North American (RSNA) did a similar experiment but focused on the problems and use of this mode of publishing. The experiment consisted of two articles published in the journal RadioGraphics, both in print and on the web. The articles referred to 11 data sets. Reader software, provided by TeraRecon, was included with each data set download and prohibited the data set from being saved locally. The RSNA tracked the problems inherent in producing and providing this kind of interactivity and user utilization. What they did not track was the educational impact, both long term and short term, on readers.
In our study the original research data was made freely available to the reader as was a free and independent set of tools to re–analyze and visualize that data. Other interactive publications provide appropriate videos to help the reader with better understanding. The reader can start, stop, fast forward and fast rewind the videos at will but cannot deviate from their set content. The reader is never allowed to directly interact with the data or given the ability to ask the perennial “What if . . .” question. This of course puts greater burden on the reader but from the results of this study it would appear that the extra reader effort is worth the investment.
The image data sets involved in this study were all quite large and had to be downloaded by the reader in order for any interaction to be possible. Our readers were all academic or industry based and it would appear that they either did their reading at their work site or they had high–speed internet connections at home as none of them complained about the speed of the data download which would certainly interfere with learning. As time goes on, the download speed problem will be minimized, but I believe we were fortunate in the choice of our reader participants.
In the same vein, the effective speed of the computer being used by the reader is also of critical importance. This became clear in the preliminary study where many readers complained that the downloaded software would not run smoothly, interfering with their interactive experience. A module was added to the software which checked the adequacy of the computer to smoothly run the software package each time it was initiated. The module warned the user if a slow response was predicted and indicated the reason for the slow response, i.e., computer clock speed, memory size, video card, etc. Computer adequacy became a non–issue.
The issues that would not go away centered on the data sets themselves. A scientific publication is made possible by the willingness of investigators to share their findings in a public forum and the willingness and expertise of colleagues to review these findings before publication. In order for a paper to be accepted for ISP, the authors had to be willing to share their data in the public domain. Although this seems to be a tradition in many of the physical sciences and astronomy, and in the genomics community, it was a problem for authors in the medical imaging community. The RSNA avoided this problem by keeping the associated data integral to its associated paper.
All ISP papers were independently reviewed before being accepted for publication. The reviewer’s job was not only to review the paper but also to review the data sets associated with the paper. How does one review the data sets? We have norms and expectations for the proper review of a paper, but what are the norms and expectations for the proper review of data, especially in this situation where the provided software will allow infinite views of that data? The RSNA relates a similar experience.
When this study was conceived it was designed to give us insight into a future reality. Perhaps that is why we needed continuing software and reader support. This will change as the technology of truly interactive publishing becomes more common. The problems of finding authors and recruiting reviewers will require a culture change in parts of our biomedical community.
Perhaps our vision was too far into the future. We set up a situation that created a new tenureable commodity, the shareable open data set. We received no comments about this. The free interactive software reader was not only an ISP PDF add–on, but a complete ISP stand–alone image analysis and visualization program. As best as we were able to tell, none of the readers realized this fact and always only used it in conjunction with their PDF reader.
Truly Interactive Science Publishing was shown to have enough educational value that readers were willing to invest in the needed set–up and learning phases. Problems encountered in network and computer speed can now be minimized by running the ISP software in a cloud computing environment which will minimize the dependence on local computer and network speeds. The social aspects of data sharing and the enlarged review process may be the hardest obstacles to overcome.
The author would like to thank his Optical Society of America colleagues Dr. John Childs, Scott Dineen and Grace Klonoski, and Kristen McMurphy of ForSee Results and his National Library of Medicine Colleagues Dr. Elliot Siegel and Dr. Fred Wood for their contributions to this project.
As an Assistant Director for High Performance Computing and Communications at the National Library of Medicine, Dr. Michael Ackerman has been working on the cutting edge of electronic media and open source software and data for over 25 years. His Visible Human Project was an experiment into the utility and usability of 3–dimensional electronic image databases. The Insight Tool Kit (ITK) Project was generally an experiment in the creation, maintenance and usability of Open Source software, specifically in the area of 3–dimensional electronic imaging. His project reported here, Interactive Scientific Publishing (ISP), is an effort to use the full capability of electronic media, not just for the distribution of publications, but to freely interact with the data cited in those publications, specifically 3–dimensional image data. email@example.com
“MIDAS” accessed October 10, 2014, http://midas.osa.org/midaspub/browse
“ISP Software” accessed February 5, 2014, http://www.opticsinfobase.org/osaisp/isphowto.cfm
“ISP” accessed February 5, 2014, http://www.opticsinfobase.org/isp.cfm#articles
Morgan, T.A., E. Flanders, W.W. Olmstead, S.D. Steenburg, E.L. Siegel. “Challenges Encountered and Lessons Learned in Initial Experience with the Next Generation of Interactive Radiology Literature in RadioGraphics.” RadioGraphics 32:929–934, 2012.