Digits: Two Reports on New Units of Scholarly PublicationSkip other details (including permanent urls, DOI, citation information)
This work is licensed under a Creative Commons Attribution 4.0 International License. Please contact email@example.com to use this work in a way not covered by the license. :
For more information, read Michigan Publishing's access and usage policy.
New Scholarship in the Digital Age
Making, Publishing, Maintaining, and Preserving Non-Traditional Scholarly Objects
Today’s academic ecosystem is growing beyond the culture of print that once circumscribed it. As Padmini Ray Murray and Claire Squires argue, the “publishing value chain,” from the invention of movable type through the twentieth century, remained surprisingly stable. “The human experience of how we produce, disseminate and perceive text is now, however, being irrevocably transformed by digital technologies.”
Murray and Squires’ observations also hold true for scholarly publishing. Although print-based scholarship remains the gold standard in the humanities, scholars are increasingly producing digital-first objects as part of their research, artistic endeavors, teaching, or other documentary forms. In this report, we refer to such digital artifacts as non-traditional scholarly objects (NTSOs).
Despite their increasing popularity, NTSOs present challenges to publishing and must be reshaped or distorted to fit the social and technical structures of traditional scholarly publishing. Institutions generally have a limited range of supported infrastructures, as well as varying degrees of technical expertise and capacity to adapt. Practitioners still disagree about how credit and prestige are allocated, how collaboration should function, and who ought to maintain responsibility and ownership for projects that are no longer under active development.
Lack of stable standards has led to digital scholarship taking diverse forms, in parallel to the heterogeneity of early printed books. This variety enriches the scholarly landscape, but it comes with a price. Whereas printable scholarship has a clear place in academia, NTSOs struggle to thrive.
This A.W. Mellon-funded report describes the myriad ways digital scholarship is being conceived, produced, distributed, and preserved in the digital humanities. With its long history of digital-first publications, digital humanities practitioners participate in every stage of scholarly production. We interviewed 75 of these practitioners to learn about their processes, what drives them, what holds them back, and how their work fits into a changing academic world.
Through anonymized and aggregated responses, we report on the digital scholarly workflow broken into four categories: (1) making, (2) publishing, (3) maintaining, and (4) preserving digital scholarship. In each section, we report on challenges surfaced in our interviews, with particular attention to the sociotechnical intricacies of that particular phase of an NTSO’s lifespan. We further identify five key stakeholder roles: (1) catalysts, (2) makers, (3) evaluators, (4) hosts, and (5) audiences. We highlight points of agreement and divergence, of values and practices, of frictions and difficulties common to each role.
There is a substantial range of opinion about the social and technical infrastructure needed to support, maintain, and preserve digital scholarship. A primary tension articulated in this report is between the expressive capacities afforded by the digital medium and the constraints of standardized scholarly production. This tension is exacerbated by limitations in even state-of-the-art technical practices, lack of institutional readiness to support such work, and unclear or opposing values with respect to how digital scholarly objects are treated.
In processing the interviews, an unremarked-upon issue became increasingly apparent. Complex NTSOs pass through many hands for many reasons, with no single stakeholder responsible for their trajectories across these spaces. Each party focuses on their own needs, leading to unpredictable difficulties. A recurring result of this disjoint is the lack of capacity of publishers, libraries, and other institutions to steward NTSOs, often on account of difficulty around the hand-offs of these objects from one party to the next. Transferring ownership or stewardship of NTSOs is one of the most significant social and technical challenges faced by today’s practitioners.
We began with the belief that software containerization offered a path towards decreasing these challenges at minimal cost, with the added advantage of creating a more standardized unit of digital publication which will be easier to collaborate on, distribute, and preserve. After conducting this study, we believe even more strongly in the importance of a single, encapsulated format for digital scholarly objects as a necessary intervention into the problems raised here. Digital encapsulation and standardization could do for NTSOs what PDFs did for static digital documents, and what shipping containers did for the global transportation of goods. On the PDF format, Lisa Gitelman writes:
The format prospers both because of its transmissiveness and because of the ways that it supports structured hierarchies of authors and readers (“workflow”) that depend on documents. One might generalize that pdfs make sense partly according to a logic of attachment and enclosure. That is, like the digital objects we ‘attach’ to and send along with e-mail messages, or the nondigital objects we still enclose in envelopes or boxes and send by snail mail, pdfs are individually bounded and distinct.
As Mark Levinson has pointed out, the encapsulation offered by shipping containers was transformative in its reduction of trade costs in the mid-twentieth century, particularly around the hand-offs of goods. In our study, we identify similar high cost points at the hand-offs between NTSOs. We believe the encapsulation of NTSOs would drastically rebalance the digital scholarly value chain, reducing friction at hand-off points in ways similar to the shipping container. However, as with shipping containers, such a technology has the potential to bring harsher conditions to the already contingent laborers associated with these hand-offs.
While the current study outlines particular points of contention or difficulties that software containers might help address, we do not limit our report to the links in the digital scholarly value chain directly related to such technical infrastructure. Instead, we offer an integrative view of the practices, pitfalls, and promise of digital-first scholarly publication.
Ultimately, this report raises the importance of orchestrated interventions into the various stages of the digital scholarship workflow: making, publishing, maintaining, and preserving. An initial intervention could at once reduce pain points and increase the prestige of NTSOs. Ours is not the first group to recommend positive interventions or paths forward. Out of necessity, however, most have focused on smaller subsets of the categories we report on. Many of these projects have reported that problems in other areas of the digital scholarly workflow limited their efficacy. In response to this obstacle, we propose several integrative approaches toward stabilizing and supporting digital scholarship.
Before drafting this report, the project team spent approximately eight months doing fieldwork. We interviewed professionals associated with the production, dissemination, and preservation of digital scholarship. We began each semi-formal interview with a consistent list of questions (see Appendix A), but allowed for relevant and interesting conversational threads and themes to emerge. Most interviews involved one subject and one interviewer asking questions and taking notes. We used audio recording for some responses, and a few interviews took place with pairs of interviewers. In some cases, we also interviewed project teams as a group. Interviews usually lasted approximately one hour.
We interviewed a range of subjects tied to the production, publication, or preservation of non-traditional scholarly objects. We used convenience sampling to generate an interview pool of 75 people whose roles included researchers, publishers, and librarians. The sample included graduate students, independent scholars and developers, contingent employees, tenured faculty, staff, and other established field leaders. Participants' projects varied in size from solo-practitioners project at small institutions to multi-institutional collaborations.
We strove for extensive coverage of the digital scholarship community. Convenience sampling, however, does not necessarily produce an exhaustive inventory of the field. For example, we interviewed far more researchers and librarians than publishers. Our pool converged on common concerns in the wider community, indicating we achieved some qualitative saturation in our data collection, but there is much still do. Our subjects most likely over-represent large universities and liberal arts colleges. In turn, we under-represent less well-funded institutions and community colleges. Supplemental work in these areas would strengthen our findings. Lastly, this report is our distillation of extensive conversations. As such, it leaves out some nuance and specificity. We also heard important points, stories, and insights that were not ultimately included in this document. Despite these limitations, our findings still apply to a large cross section of the digital scholarship community.
The Problems of Non-Traditional Scholarly Objects
We use the term non-traditional scholarly objects (NTSOs) as a subset of digital scholarship. Both differ from traditional forms of print-first scholarship. In our interviews we provided a variety of examples to help clarify our idea of NTSOs, including blogs, Twitter bots, searchable databases, and interactive data visualization essays.
These examples are all web-based. They provide a range of linear and exploratory experiences. They depend on many digital platforms, some custom and others “off-the-shelf.” Their creators had various levels of expertise and technical proficiency. Many of the projects were posted on a scholar's website and disseminated via social media.
We wish to distinguish NTSOs from the more general term of digital scholarship. Digital scholarship is a broader label that can imply ecosystems, contexts, and infrastructures. The key element, as described by Christine Borgman, is the intersection of digital components and scholarship. With the idea of NTSOs, we want to focus attention on objects. We do not seek to bracket the broader social, institutional, and cultural contexts of digital scholarship. Rather, we foreground objects and processes, especially making, publishing, maintaining, and preserving.
These four distinct themes emerged from our interviews with digital humanities practitioners. By making, we refer the practices related to the creation, conceptualization, and construction of NTSOs. Publishing refers to both their publication and dissemination. Maintaining refers to the practice of ensuring that NTSOs remain accessible and operational. Preservation, for the purposes of this report, refers to “all the activities necessary to ensure the long-term accessibility of a resource.” In the context of digital scholarship, this includes making an artifact suitable for inclusion in the long-term scholarly record.
Our report uses section headings to separate content by theme. In each of these four areas, our subjects discussed their biggest challenges. Our report uses subsection headings to identify these challenges. At the end of each subsection, we describe potential interventions and recommendations.
Recommendations for Future Intervention
Recommendations are aimed at practitioners working in one or more of five roles: catalysts, makers, evaluators, hosts, and audiences. These categories are defined in relationship to NTSOs themselves:
Those who facilitate the conception and development of NTSO. This includes funders, digital scholarship centers, university departments, scholarly organizations, etc.
Those who make or directly shape NTSOs. This includes researchers, programmers, and other digital makers, but also in some circumstances peer reviewers, editors, etc.
Those who evaluate NTSOs. This includes editors, peer reviewers, publishers, tenure committees, etc.
Those who host, serve, maintain, or preserve NTSOs. This includes libraries, archives, publishers, hosting companies/platformers, etc. Occasionally, this includes digital makers themselves.
Those who access NTSOs. This includes readers, those who access NTSOs via API, etc.
Breaking the tasks of NTSOs into discrete categories (Making / Publishing / Maintaining / Preserving) or roles (Catalysts / Makers / Evaluators / Hosts / Audience) often occurs implicitly and without intent. Among our key findings is that this natural fracturing is itself an impediment to the long-term success of NTSOs. Many tasks fall through the cracks between categories, and no single body can work to orchestrate success across all links in the scholarly value chain.
With this in mind, we offer an additional category of recommendation, disconnected from any one role:
These recommendations are for the future of NTSOs themselves; how they might act and interact, the shape they may take, and how they may evolve.
Recommendations aimed at NTSOs are those we believe the entire community ought to consider, and cannot be mapped to specific roles.
We asked our interview subjects to comment specifically on pain points and areas of dispute. The proposals draw both from suggestions raised during interviews and from a synthesis of secondary literature. They are intended to be illustrative rather than exhaustive. On account of the variety of perspectives available, some are contradictory. Some recommendations are actionable in the short term, and most point to the need for a coordinated effort from all stakeholders. Many recommendations are actively being tested by A.W. Mellon-funded initiatives like our own. We hope that these challenges and recommendations will guide further discussions, investigations, and interventions.
Report on Interviews and Recommendations
Challenges Making Non-Traditional Scholarly Objects
NTSOs typically require some combination of expertise in research content, proprietary software, and dev-ops. When team members do not have the requisite expertise, the team will often grow. In other cases, projects begin with a technically proficient maker in search of subject matter. Large scale projects may include a range of traditional and hybrid roles. Our interviews were consistent with the truism that bespoke digital projects entail heavy technical labor at the start. Pre-packaged software or templates, in contrast, demand less labor upfront but afford less flexibility.
We did not interview scholars focused on traditional publication tracks. Previous work, however, suggests that digital humanities projects tend to involve more collaborators than print-based humanities publications. These named and unnamed participants come from a wide range of institutional and commercial settings. In this section, we explore who creates NTSOs and how, focusing on common challenges.
The label “digital makers” has been used in many different contexts to describe people who create non-traditional objects, scholarly or otherwise. For example, in 2015 the U.K. foundation Nesta published a report titled Young Digital Makers, which argued, “For most young people digital technology is an everyday part of life. Many are avid consumers of digital media. However they often don’t understand how to manipulate the underlying technology, let alone how to create it for themselves.” For the report’s author, Oliver Quinlan, “digital maker” was a broad but useful construct because it referred to a range of activities that were “distinct from simply using digital devices.” In the context of digital humanities, some practitioners have embraced these labels. For the purposes of this report, our use of the term maker is analogous.
Digital makers reported working along a collaborative spectrum. In many humanities disciplines, researchers follow the sole author / individual scholar model. In this model, a scholar conducts research and claims sole credit for the outputs, which tend to be peer-reviewed articles and monographs.
When others' contributions support this work, an author's acknowledgements sometimes make note of it. Our subjects named librarian consultations, collections access, help with archival materials, research assistantships, and informal peer review as important supportive labor. Yet the individual scholar model does not consider this work coequal with authorship. Some of our subjects, who operate as solo practitioners, showed a preference for this model, perhaps in part due to existing tenure and promotion systems. Other digital makers embraced division of labor to expand their projects' scope. Project partners had expertise they lacked, or helped expand output capacity. A minority of subjects rejected the credit models of traditional scholarship altogether.
When collaboration does occur, some predictable problems arise. The sociotechnical aspects of collaboration, including collaborative credit models and project orchestration environments, seem particularly difficult to navigate. Most humanities disciplines lack strong models for collaboration, and teamwork skills vary greatly. Our research suggests that lack of training deters collaboration. Administrative skills are rare, and many in the humanities consider this labor ignoble. Some interview subjects reported difficult transitions between exerting control over their research process and performing the role of a Principal Investigator. For PIs, the ability to coordinate large project teams is especially important. Secondary communications skills are also crucial. Humanistic knowledge (“subject matter expertise”) and technical proficiency make up two other important axes of skill reported as essential to the development of NTSOs. Some NTSOs are developed by the rare solo digital maker who excels in multiple skillsets, resulting in particularly coherent projects that are often limited in scope.
Well-functioning collaborations, in contrast, can enable digital makers to work beyond what an individual could create by themselves. Such successes seem to give rise to more hybrid project roles. The rewards of such partnerships often included skill development and increased project momentum, but may also show signs of fracture or discontinuity where collaborators meet.
Every NTSO—like every piece of scholarship—must go through many hands. Digital making, however, seems to bring the tension points of collaboration to the fore, and increase the likelihood of conflict. This difference between scholarship in general and NTSOs in particular may relate to the speed at which roles change, or the number of changes a role is likely to undergo before stabilizing. Further, a large digital project could depend on dozens or even hundreds of contributors. Any web-facing digital object, at its core, rests on a complex stack of manufacturers, software developers, internet engineers, and system administrators. Countless members of an NTSO’s audience may also contribute to or co-construct the object, as in crowdsourcing initiatives or annotated editions.
In recognition of these complexities, our report does not attempt to define or contain the idea of a “digital maker.” Instead, we focus on the labor that typifies digital making. Many scholars who would not claim the label "digital maker" take part in digital making. Many other participants remain anonymous or uncredited. Our rhetorical shift calls attention to the innumerable contributions to almost any project.
We found that remaining unnamed often relates to one of three norms:
- The labor was a service offered in exchange for some reciprocal benefit, including funding or university credit.
- The labor was part of some standardized infrastructural apparatus.
- So many participants performed the labor that naming all contributors was impractical.
Reciprocated labor might include work by paid programmers, interns, or students. These kinds of collaborators more often appear by name in website acknowledgements than in books or articles published about digital projects.
Standardized infrastructural labor includes editors, peer reviewers, librarians, and archivists in one category, and technical experts such as in-house developers, digital humanities center personnel, and system administrators in another. Many such professionals receive credit for their work through internal activity reports, though they remain publicly unacknowledged. Such hidden labor is typical in both digital and print mediums. According to some subjects, the erasure from mention in externally-facing publications has been a source of consternation.
Subjects also mentioned cases where too many people had contributed components for acknowledgement to be feasible. When digital projects use outside products, software, and digital infrastructure, the labor of these entities also often remains unacknowledged. In turn, the labor of building and maintaining open source software goes uncredited. Likewise, an NTSO based on audience participation (eg. social annotation projects) or HIT/micro-tasking labor (e.g. Mechanical Turk), may not credit its participants by name.
Collaboration is hard. Some participants in the NTSO ecosystem would prefer to avoid it. Others want to collaborate but find it hazardous and a professional risk. Changing models for credit and compensation can create more conflicts. As the labor of digital making becomes more institutionalized, some types of collaboration seem more appealing. At the same time, technical labor is becoming less visible. We discuss this ostensible contradiction in the next section.
Based directly on interviews, our interpretation of them, or on our interpretation of surveyed literature.
- [Makers] Embrace hybrid roles, flexible teams, and more diffuse definitions of collaboration and making.
- [Makers] Acknowledge labor that is often hidden, such as editors, system administrators, etc. One possible mechanism is by using standard collaborator contribution statements.
- [Evaluators] Value non-traditional contributions as much as recognizable writing work.
- [Catalysts] Create space particularly for generalists or collaborators with shifting roles. Though they may not hyper-specialize in any one area, these contributors often act as translators without whom an NTSO cannot exist.
- [Makers] Learn and be able to describe in some detail the contribution of every member of a project team.
- [Catalysts] Facilitate workspaces and programs where a culture of every team member knowing each other’s contributions is the norm. This may be accomplished, in part, via standardized communication infrastructures and practices.
- [Makers] Seek training in effective project management and collaboration.
- [Catalysts] Create incentives and programs for makers to get trained in project management and collaboration, through funding initiatives, workplace events, etc. Such programs need adequate scaffolding, with clear pathways to gain the expertise necessary in these areas.
Sociotechnical Challenges and Limitations
NTSO production leads to pain points around technology, expertise, and gaps between the two. The variety of projects in our study makes generalization a challenge. This section attempts to identify and categorize the most pressing sociotechnical challenges and limitations we encountered.
Foremost, our interview subjects described tension between experimentation and maintenance. Projects using off-the-shelf platforms and solutions such as Omeka, Scalar, and WordPress were described as less experimental. Other projects were bespoke creations, and required significant technical skills to build. Some proponents of custom projects said that off-the-shelf platforms would not meet their needs. Others told us that their projects exceeded the hardware or systems capacity of their home institutions. In general, more experimental approaches were seen as less maintainable, and more maintainable platforms were seen as less fit for experimentation.
These obstacles appear sociotechnical. Institutions select digital architecture based on complex criteria. Project teams pursuing institutional partnerships for the sake of technical support must often use off-the-shelf platforms. Such compromises constrain projects and deter experimentation, and yet the range of hosted platforms may have been selected with other priorities in mind. Even when technical needs are met, bureaucratic barriers or university policies may slow development, or make implementation more difficult than anticipated. Our interview subjects expressed frustration at this tension.
Our interview subjects were eager to talk about web-hosting decisions and stressed their importance. They discussed four broad categories of hosting solutions:
- “Under-the-desk” servers (any server where the scholar acts as system administrator)
- External services (DigitalOcean, GitHub, WordPress.com, Reclaim Hosting, etc.)
- Institutional partnerships (a university library hosts an Omeka instance, a custom web application, etc.)
- Publisher partnerships (a publisher supports hosting and/or building an NTSO).
In categories 2, 3, and 4, project members tend to cede control of system administration. The work of system administration is often hidden or misunderstood.
A parallel may be drawn between these models and traditional book or journal publication. Authors play a substantial role in creating and publishing books and journal articles, but they tend to cede control over the afterlife of their work, including distribution, access, and preservation. Even during the production process, authors are accustomed to publishers controlling things like page design and printing. The notion of a structured hand-off (such as page proofs) is well understood.
In contrast to print publications, less structured hand-offs occur more frequently in NTSOs. In some cases, the work is parallel and simultaneous. Points of friction can arise between the original team and the production team almost any time in the process. Our interview subjects expressed frustration that security problems, hosting changes, or other sociotechnical issues can force a team to work on a project long after their collaboration has ended or their funding has run out.
Such concerns affect discussions of whose job it is to build the “under-the-hood” pieces of NTSOs. Factors such as funding, institutional personnel, team member expertise, and hardware & software affect these decisions. Negotiations can be complicated, resource-intensive, and difficult to understand. PIs may feel they lack the expertise to conduct such negotiations. These considerations have a strong impact on whether a project is “off-the-shelf” or custom built.
Custom built NTSOs tend to be more complex and idiosyncratic than "off-the-shelf" products. Hand-offs, as a result, are more difficult. Documentation is often absent or out of date. Even with clear records, moving such NTSOs between servers may be expensive and time-consuming.
The implications for digital objects in the scholarly ecosystem are dire. Scholars struggle to hand their digital work to peer reviewers, libraries, publishers, and other maintainers. Maintainers struggle to transfer them to institutions tasked with preserving digital scholarship. The loss of a single key team member may lead to a project’s failure because that team member possesses skills or key knowledge that no other team members has. Under these conditions, libraries cannot hope to host copies of important NTSOs the way many libraries keep copies of important books.
A majority of NTSOs rely entirely on a few off-the-shelf digital tools or platforms. Fewer require some form of customization, coding, or other expert technical labor, and fewer still are entirely bespoke, requiring as much nuance, care, and skill as the humanities work involved. However, the amount of technical effort needed to fit NTSOs into the scholarly publication chain is the inverse of this. The complexity and diversity of bespoke projects means they are often the most difficult to individually accommodate. Though they are in the minority, our subjects reported that bespoke NTSOs often comprise the most interesting and impactful projects within their communities.
Based directly on interviews, our interpretation of them, or on our interpretation of surveyed literature.
- [NTSO] NTSOs must become more easily portable to ease the burden of transferring work between makers, evaluators, and hosts. This would become significantly easier were NTSOs encapsulated in or organized by a single file, particularly one with relevant scholarly metadata.
- [Makers] [Evaluators] [Hosts] Specify and minimize the amount of hand-offs that take place; plan the moments of hand-off carefully.
- [Makers] [Evaluators] [Hosts] Become more comfortable with containerization (or other encapsulation) standards.
- [Makers] [Evaluators] [Hosts] Use encapsulated NTSOs during hand-offs.
- [Catalysts] Incentivize encapsulation, through grant requirements, institutional encapsulation standards, and investing in consortial models to construct standards of scholarly encapsulation.
- [Makers] [Hosts] Use clearly articulated web hosting agreements to reduce the sense of uncertainty around, e.g., whether a university continues to host a faculty project after the faculty has switched institutions.
- [Hosts] Widen range of hosting options to accommodate short-term, low cost sandboxing and prototyping.
- [Makers] Articulate project charters that specify a project’s longevity. Depending on the choice, develop with that longevity in mind, or state clear project end dates.
- [Makers] Learn the hardware and software stack supporting the NTSO enough to be able to describe it.
- [Catalysts] Incentivize makers learning hardware and software stacks by offering trainings and decreasing the disconnect between makers and system administrators.
- [NTSOs] Given that the majority of NTSOs exist on one of a few platforms, the community needs to come together to agree on standard hand-off solutions for these types of objects.
- [NTSOs] Although bespoke NTSOs are in the minority, given that many described them as the most interesting and impactful within their scholarly communities, it is essential that these bespoke objects are not ignored in favor of focusing on off-the-shelf solutions. As much or more effort must be expended on standardizing the bespoke NTSO scholarly value chain, which in turn will be useful for standardizing off-the-shelf solutions.
Our research suggests that NTSO production does not fit traditional humanities funding models. The models focus on paying for books, research trips, conferences, and other initiatives with clear end-dates or end-products. Instead, digital scholarship tends to either be self-funded or dependent upon grants. This introduces a host of problems from the grant-funding economy into the humanities.
All our subjects expressed gratitude for the support of various funding institutions. Many said they were nervous about the consequences if a major grant funder, such as the A.W. Mellon Foundation or the National Endowment for the Humanities, were to stop offering grants for digital work. Some well-funded institutions would be able to support digital scholarship using internal funding, but the vast majority of digital makers do not have access to such resources. As a result, their projects would become limited to what they could self-fund. A change like this would fundamentally alter the production landscape of NTSOs.
The grant-funded economy also shapes the way digital makers choose to run their projects. Universities typically charge overhead rates from external grants to cover indirect costs. These rates are often set with the assumption of scientific research. As a result, overheads in the humanities tend to assume costs that typical digital humanities projects don’t have. In turn, such projects are left with less funding to cover costs that scientific grants typically don’t have.
In more than one interview, we were told that overhead payments take a significant part of a project’s funding. The fact that sciences have more grant funding and more funding sources than the humanities seems to aggravate these concerns, especially since overhead costs contribute more directly to scientific necessities such as lab space. This structure can also lead PIs to make hard decisions based on cost models rather than project needs. For example, teams may outsource development work to avoid paying benefits and make the budget stretch farther. One subject noted that grant-funded project team members, as soon as they were trained, were often “stolen” by other projects and university libraries that could pay them higher salaries.
Hard funding comes with several advantages. Our interview subjects described career stability as one of its most compelling features. Many subjects reported university policies that make soft-funded faculty or staff ineligible to apply for grants in their own right. Several faculty or staff with soft funding reported searching for a grant-eligible faculty members to serve as in-name-only PIs to circumvent such policies. Interview subjects saw hard funding as a way to avoid the negative experience of a project coming to an awkward or even ruinous conclusion. At the end of a grant-funded project, developers are often laid off, and no one remains to address inevitable bugs and security issues. One subject described being responsible for a legacy project that “falls over” at irregular intervals. This particular person, however, did not have the knowledge or expertise to do more than restart the server.
To address problems of soft funding, some archives, libraries, and museums have attempted to make more permanent technical and personnel resources available to project partners. Such organizational partners face a different set of challenges from lone scholars or one-off projects. Most often, they are constrained by technical debt from past projects, as well as limited capacity of personnel and technical breadth. In response, many reported limiting the types of objects they are willing to work with in order to accommodate as many projects as possible. A minority of our subjects said they take on bespoke production with only one or two projects at a time. Based on our research, few organizations have secured long-term funding for bespoke production. Exemplary projects produced in these cases seem, to others, impossible to imitate. Even when funding is stable, interview subjects reported an absence of protocols for or commitments to ongoing maintenance and preservation.
Some of our interview subjects reported funding some or all of their projects out of pocket. Web servers operating at home or under someone's desk were more common than we expected. Self-funded services such as Reclaim Hosting, Digital Ocean, or Amazon Web Services were even more common. The costs of these solutions varied greatly. The most common reasons for such setups were as follows:
- The technical needs of the maker exceeded the capacities of their home institution.
- The project's preferred software or hardware was not permitted by the institution.
- Bureaucratic hoops proved too complex or too onerous for the maker or project team.
Self-funding offered the advantage of making projects easier to move between institutions. Likewise, they offer complete freedom to experiment and develop projects using various approaches. They also shift the burden of tech support onto the individual or team working on the project. No subject we interviewed who went this route received compensation for the monetary costs or time debt produced by their self-hosted systems.
As evidenced in our interviews, several publishers, libraries, and others are thinking about these issues and have collectively offered some solutions. Our subjects, however, reported being under-resourced, understaffed, and missing the crucial expertise to execute these ideas.
Based directly on interviews, our interpretation of them, or on our interpretation of surveyed literature.
- [Catalysts] Support research into the start-up costs of creating digital projects, and the long-term costs of maintaining and preserving them. Include in these cost evaluations everything from a maker’s time to equipment costs and sysadmin labor.
- [Hosts] Ensure hosting, maintenance, and preservation costs are transparent, and include the full stack of development and technical requirements.
- [Makers] Clearly document all project activities, time spent, and costs incurred.
- [Makers] Write librarians, technical staff, and others into grant applications to ensure these and related costs are accounted for.
- [Hosts] Clarify costs of technical infrastructure and staff to ensure they can be accounted for in grants.
- [Catalysts] Encourage grant applicants to include more complete sociotechnical costs in grant applications, and accept applications with such details as integral to the projects being undertaken.
- [Evaluators] Press for more clarity in method with respect to personnel time, contribution statements, and technical infrastructure.
- [Hosts] Particularly libraries, lean into the analogy between the laboratory in the sciences and the library in the humanities. Use this analogy to demand a larger cut of indirect costs levied by the university, and to help project teams secure grant funding for project infrastructure costs.
- [Hosts] Fill the role of “laboratory for the humanities” by providing makers with more flexible web hosting and cloud storage, particularly for preliminary work.
Copyright and sensitive data
Our subjects raised logistical and ethical concerns about copyright and sensitive data. Those working with post-1924 United States materials reported more problems with copyright. The new affordances of text-mining complicate questions of access, as many academic distributors only permit browsing access to copyrighted materials. Others charge fees for large-scale, computational use cases (e.g., text-mining). In the United States, analysis of copyrighted materials for scholarly purposes qualifies as fair use, but the re-distribution of copyrighted materials as part of a dataset does not. Some vendor licensing agreements bar any redistribution of data, even if copyright is not a factor. Such complexities have led to creative solutions like the HathiTrust Research Center’s original “walled garden,” which enabled pre-defined text analysis algorithms to run on copyrighted materials. HTRC has also provided “non-consumptive” versions of texts in the form of term frequency tables.
In our interviews, several other issues related to copyright came up. Scenarios were complex and broad ranging. There was no consensus among our subjects around fairly common questions, such as the conditions necessary to call a work transformative. In cases where licensing was more of a concern than copyright, our subjects voiced similar frustration and uncertainty.
Many of our subjects discussed concerns related to sensitive data. Humanists are often unfamiliar with Institutional Review Board (IRB) requirements and procedures. IRB is often described as a baseline ethical standard and not a cure-all. As our interview subjects pointed out, ethical considerations beyond IRB are often crucial. Further, IRB-exempt projects can encounter ethical quandaries when attempting to create digital datasets.
One common example of this is Twitter data, which is publicly available and thus—theoretically—free for scholars to analyze yet which can include sensitive materials belonging to or associated with marginalized groups who did not consent to be collected and studied this way.
Another useful example is the digitization of the On Our Backs lesbian porn magazine archive. While the digitizers believe that they were operating within the limits of current copyright, most of the contributors to the magazine did so when it was a limited print-run magazine—many before the modern internet existed—and some later contributors explicitly withheld consent to having their images posted online for anyone with a browser to find. Given the potentially catastrophic personal and professional harm that could occur to the contributors through this digitization, this may be considered an unethical digital project regardless of its legality. Makers must therefore exercise caution when creating digital projects, even if they believe they have the legal right to do so.
- [Catalysts] Foster and offer legal and ethical training for digital scholarship, similar to training required in the sciences around IRB and similar issues.
- [Makers] Take time to learn about the legal and ethical issues surrounding digital scholarship before embarking on projects. If an instructor, teach students about these issues.
- [Evaluators] Keep legality and ethics in mind when evaluating data-rich research. Be particularly mindful of situations that are legal but not ethical.
- [Catalysts] [Hosts] With respect to legally ambiguous but ethically clear situations, cultivate a risk-tolerant atmosphere that encourages experimentation.
- [Catalysts] [Hosts] [Evaluators] [Audiences] Encourage and incentivize open data practices.
- [Makers] When ethical, adopt radically open data practices. Avoid using copyright claims as a means of staving off criticism and discouraging engagement with “under-the-hood” elements of a project.
Of the social aspects of creating a digital project, credit models were consistently mentioned as the most likely to generate friction for a project team. As discussed above, the humanities lack strong models for collaboration. Existing norms obfuscate a great deal of labor and may foster resentment among team members.
The structural aspects of credit influence how some would-be digital makers approach collaboration, particularly with developers and librarians. Some interview subjects reported a common attitude of de facto authority and control. They reported visitors to digital scholarship centers or libraries arriving with all project details predetermined, expecting staff to construct the project without providing any feedback or guidance. The vast majority of our subjects from digital scholarship centers and libraries objected to this model. They said it reinforced problematic hierarchical structures within their institutions, as well as divisions between the digital and the humanities in the community at large.
Some felt that categorizing technical labor as “service” devalues those contributions. In many cases, the technical aspects of a project are foundational to the project's scholarly intervention or argument. This service model may also inhibit or close off career pathways for developers within academia. One subject argued that these paths should be analogous to computing industry career pathways, with the goal of ensuring that talented developers find intellectual fulfillment within their positions. Many must fight a double battle to receive both external and internal recognition for their work. Such attitudes may reinforce the idea that digital scholarship is not “real” scholarship. In turn, they may undercut broader efforts to legitimize digital humanities in the wider humanities community.
In contrast, two of our subjects who began on the academic track and ended up as digital humanities developers in permanently-funded positions expressed relief and excitement over the very aspects that others found problematic. Both said they had some agency to shape projects and sometimes lead their own, but spend most of their time on what their employers assign. Authorship credit is possible in their positions, but not inevitable. Both expressed distaste for the prestige-driven academic tenure process and said they were happy to have so-called “alt-ac” positions.
The majority of our subjects agreed on the need for credit models that acknowledged the work of project team members; they also expressed that this was easier said than done. Who, for example, should be included as a co-author on an article written about a digital project? Tarkang et. al. suggests that authorship denotes “those who deserve credit and can take responsibility for the work.” In the sciences, “the work” includes lab research and the “writing, submission, and editing required for a paper.” In some humanities disciplines, where lab research does not take place, authorship consists solely of writing, submitting, and revising. An article based on a digital project might follow either of the two models. Friction can arise when contributors to a project are not listed as authors because they did not take part in the composition of an article or book.
Many interview subjects discussed the difference between team credit and individual credit. They reported creating an “About Us” page that lists everyone associated with a project. In one interview, we were told that this system is analogous to films where credits roll on for ages at the end. Such credits, they said, only last as long as the project remains online and often do not provide readers with an adequate understanding of an individual team member’s contribution to the project.
Even when digital labor is made visible, it is often misunderstood. The amount of time, effort, and skill that goes into activities such as data wrangling are especially opaque. Many of the people we spoke with expressed uncertainty about how to claim credit for the project work on their own CVs and portfolios. Where does one put pedagogical materials, datasets, software, contract work, consulting work, funded Kickstarters, or Patreon donations? These problems go beyond the CV itself; they speak to uncertainty about how to describe and categorize their work. One subject expressed a desire to see these questions lead into a deeper analysis of what people are getting credit for and why. What, in other words, does tenure mean? The uncertainty of the present moment, they said, creates an opportunity to raise these questions for digital scholarship, as well as academia more broadly. Such a reassessment, they said, raises the question of how prestige drives scholarly publication. Meanwhile, most of our interview subjects focused on the value of their own work in the current credit system.
- [Catalysts] [Hosts] Revise requirements for how credit is articulated to ensure labor does not go unrewarded. Arguably, taking implicit credit for others' work is a form of plagiarism, yet the stigma of a plagiarism accusation is currently much greater than failing to share credit. Apply social pressure to balance these credit norms.
- [Evaluators] Require clear contribution statements when evaluating digital projects.
- [Makers] Via scholarly organizations, coordinate norms for claiming credit in NTSOs on CVs and portfolios. Define contribution types and roles, but non-dogmatically. Some subjects preferred a “total collaboration” model, since many particularly vibrant NTSOs included contributors who were involved in every aspect of its creation.
- [Evaluators] Accept non-traditional contributions as credit in cases of hiring, tenure, and promotion.
- [Catalysts] [Makers] Standardize the use of project charters and other formal agreements to reduce the friction around credit statements early on in projects.
- [NTSOs] NTSO metadata standards must evolve to accommodate flexible, ambiguous, or expansive contribution statements.
Challenges Publishing Non-Traditional Scholarly Objects
In this study, we have separated publication and maintenance under different section headings. Though they often overlap, they present distinct challenges to digital scholarship. We employ the term publishing to collapse a range of activities:
- Peer review.
- Manuscript preparation, including editing, proofreading, quote checking, and production design.
- Distribution, marketing, publicity, and indexing.
The prestige value of a specific imprint, as well as the financial models that support these activities, are also included in this section. Not all publishers take on these roles, but they distinguish publishing from our other categories. Publishing serves as a convenient umbrella term for this report.
Avenues for the publication of print materials are well established, as is the division between published and unpublished. In contrast, the line for digital scholarship between published and not published remains amorphous, without a clear distinction between “made”, “online”, and “published.” In the current academic ecosystem, the labor of making a finalized public-facing, digital object often falls on its creators, while typesetting and finalization of “print-ready” versions of print scholarship falls on publishers or editors. Traditional print scholarship benefits from a level of codification that NTSOs presently lack.
Some have attempted to codify NTSO production by creating avenues to move NTSOs into the traditional credit pipeline, but the most prevalent options for such transfer struggle to accommodate digital scholarship on its own terms. For example, NTSOs are often connected to traditional scholarly publications under the following circumstances:
- Digital or print publications that stamp peer-review approval on pre-made digital objects.
- Journals that publicly review digital objects (as with a book review).
- A print publication (i.e., companion piece) authored by members of the digital project team.
- a print publication with a digital supplement or appendix, where the publication is seen as the primary, peer reviewed object.
In three out of four of these circumstances, a journal or publisher partially accommodates an NTSO, but the final product does not sit alongside its print counterpart. In these cases, the NTSO is not peer reviewed and does not earn prestige or credit equal to the print publication to which it is attached.
In choosing among these “shoehorn avenues,” interview subjects said they must consider:
- How the eventual NTSO will be cited.
- Where and how it will appear in their CV.
- The stated requirements of their chosen career path.
- How to justify it to a dissertation director or a to hiring, tenure/promotion committee (where applicable).
Academic prestige was an abiding concern, and pessimism was abundant in our interviews. They described options as scarce, lacking in prestige, and often ill-fitted to their work. Most described the labor of transforming their work for the print ecosystem as difficult, with dubious benefits. Others, especially public humanists and digital artists, reported that their audience and peers were more concerned with public impact or critical reception than academic prestige, and thus did not need to “shoehorn” NTSOs into pre-existing academic publication ecosystems. Many of these interviews subjects, however, said they work outside the research tenure stream and are less tied to traditional scholarly metrics than most of their colleagues. The perception that experimental work of making NTSOs was best suited for post-tenure faculty or alt-ac jobs was widespread.
Our interview subjects identified peer review as an especially important aspect of traditional publishing, but adapting peer review for NTSOs is daunting. Peer reviewers must have expertise in the NTSO’s technical form and its content. As a result, qualified peer reviewers can be hard to find. When a project's content and technical form can be compartmentalized, reviewers with one expertise or the other can be enlisted. NTSOs built with off-the-shelf tools may help with such separation. On rare occasions, however, an NTSO’s content and technical form are completely intertwined. Given the relatively small intersection between digital humanities scholars and scholars of a particular subfield, all qualified reviewers might already be attached to the project under review either as active team members or as advisors.
Our subjects expressed concern about any peer review system too sophisticated for an average scholar. For example, in digital exhibits a reviewer must be able to assess whether the technology choices make sense for the NTSO. For digital objects with complex computational elements, a reviewer needs to be able to determine whether the source material, data, and methods work together to support the object's central argument. Access to a digital object’s “front end” and source code is often necessary. If an NTSO requires the installation of software or dependencies to run locally for evaluation, many potential peer reviewers would be excluded.
Another difficulty inherent in adopting the traditional peer-review process for NTSOs comes from requested revisions. Peer reviewers might consider all aspects of the NTSO equally revisable but projects that rely on standard content management systems and other off-the-shelf solutions such as Omeka, Scalar, or Wordpress can only make changes that the platform allows and affords. A relatively simple suggestion for revision could be, in this context, difficult or impossible to accommodate from a technical perspective.
Bespoke projects can be similarly difficult to revise, and may cause even more difficulties. An ad hoc solution to a particular problem could be easy to program but, depending on the scaffolding of project elements, even seemingly minor revisions might require rebuilding the project from the ground up. As with other challenges, the issue here is sociotechnical. Often, large projects enlist outside consultants who are available on a term-limited basis. Technical services might be funded through grants, provided as part of a course, or extended as grant-in-aid from a digital scholarship center or library. In such cases, revisions are possible from a purely technical standpoint. Social barriers, instead, make revision impractical and unlikely to occur.
- [Catalysts] [Makers] [Hosts] Start the peer review process early, perhaps during the initial design and development phases, so that NTSO revisions can be incorporated before a technical point of no return.
- [Catalysts] [Makers] [Evaluators] Foster the education of peer review standards for NTSOs. This can perhaps be combined with other teaching goals, such as technical literacy. Utilize scholarly organizations such as MLA, ALA, and AHA to host seminars and pre-conference workshops on best practices for reviewing NTSOs.
- [Catalysts] [Makers] [Hosts] For moments when early integrated peer review is not possible, foster and adopt open, post-publication peer review models.
- [NTSO] NTSOs must be portable between collaborators and peer reviewers, and executable such that non-technical peer reviewers are still able run and review the object.
Despite the difficulties of publishing NTSOs, many of our interview subjects remained committed to adapting core elements of monograph publishing to their work. Academic prestige was at the heart of this support, as was loyalty to peer review as a process. Some said they were concerned with how NTSOs appear on their CVs, and many said they preferred Omeka and Scalar because of their “monograph whiff.” A booklike object, they said, would be easier for hiring, promotion, and tenure committees to understand.
There was no consensus as to whether monographs, articles, or conference papers were most analogous to NTSOs. Some suggested that such a judgment depended on the size and scope of the project. Almost all our subjects, however, sought to draw comparisons to traditional modes of publication. A few also related experiences of having NTSOs evaluated as service instead of scholarship. The subjects who recalled such experiences found them objectionable. One attributed the misjudgment to an overly narrow definition scholarship, i.e., that only a prose-like intervention articulating and defending a critical argument should count. Some subjects voiced the idea that “the model is the argument,” but they also conceded that such scholarship would be less recognizable to some reviewers. In other words, our interview subjects saw the norms of print as crucial to traditional prestige.
Many of our subjects were especially concerned about the stigma of self-publication. This label, they felt, would lead reviewers to dismiss the work and disqualify it from being “serious scholarship.” There was some disagreement (and even tension) about whether publishers were a healthy part of the scholarly ecosystem. Some called for prestigious journals in their field to create more space for NTSOs. Their core idea was to extend the prestige of traditional, print-based scholarship to contexts where digital scholarship could appear.
Others called for the reform of (or even an end to) prestige-based scholarship. A press’ reputation, one interview subject argued, is too often relied upon to determine the importance and prestige of a scholarly work. Outsourcing scholarly gatekeeping to publishers, they said, prevented scholars from reading and judging scholarship on its own merits. As stated previously, many of our subjects who chose “alt-ac” career paths expressed great relief at “being freed” from the structures of prestige, credit, and promotion. Many of these subjects directly associated these structures with a publisher-centric system.
- [Catalysts] [Makers] Articulate criteria for NTSOs to be treated as equal to monographs, journal articles, and conference presentations via scholarly organizations, tenure guidelines, grant programs, and other initiatives. Reduce the knowledge gap regarding the amount and kinds of labor that go into producing NTSOs.
- [Makers] [Evaluators] Continue experimenting with radically alternative credit models.
- [Hosts] Put resources into and create space for NTSOs. Treat NTSOs as equally valuable and valued as print-based scholarship.
- [Evaluators] Adopt capacious notions of scholarship that include NTSOs and self-published work. Work to divorce the means of distribution from the granting of prestige.
In our interviews, issues of prestige were directly linked to intended audience. Many digital makers have traditional academic audiences in mind for their scholarship, including disciplinary scholars in the humanities and STEM, which can be further divided by level of specialization. Imagined classroom use was also a common intended audience for interviewees NTSOs. Several of the people we interviewed identify strongly with the public humanities and see their audiences as various segments of the public. They expressed interest in matters of social justice, cultural heritage communities, and policy-making. The great range in size of perceived audiences appears to have a strong impact on the software, tools, and platforms used to create and publish NTSOs. In cases where audience response (approval, engagement, etc.) is a priority, the norms of prestige are perhaps less salient.
Those who said they prioritized public humanities showed little interest in attempting to emulate traditional publishing models. Instead, they argued that effectively self-publishing—that is, paying for their own project hosting and taking responsibility for the full lifecycle of the project—enabled them to reach their intended audiences. The scholars we spoke with who were engaged in public-facing scholarship tended to work on smaller or solo projects, and either worked in contexts where these projects were sufficient to keep them employed, or worked on enough more traditional-looking projects that they rested on these when it came time for assessment, tenure, promotion, or re-appointment.
- [Makers] Build NTSOs with user-centric design, particularly but not exclusively when building for general audiences.
- [NTSOs] NTSO standards should align with modern user-focused web standards, including for mobile compatibility, accessibility, and minimalism. Adopting these standards, conveniently, will also help NTSOs become more easily accessible and preservable, and decay more gracefully.
- [Catalysts] [Hosts] Build networks, indices, databases, and other aggregators to collect NTSOs and make them more discoverable by non-traditional audiences. Such networks would do well to include secondary educators, community organizers, and other stakeholders.
- [Makers] Proactively seek inclusion of NTSOs in aggregators depending on the intended audience. Reaching out to non-governmental organizations and for-profit entities may be relevant, depending on the NTSOs focus.
- [Audience] Normalize paying for NTSOs or NTSO consortia out of respect for the labor involved.
- [Makers] Directly justify decisions to self-publish if going up for tenure or promotion, describing how the work was received by scholars and broader audiences.
- [Evaluators] Accept non-traditional audiences and venues as legitimate markers for success.
Discoverability and access
Most of our interview subjects agreed that discovery and access are two of the largest barriers to success for NTSOs. Making self-published materials visible is difficult. As we have suggested, many NTSOs are published on scholars’ personally-maintained websites.
NTSOs hosted by digital centers or other groups with institutional websites tend to be more visible. Some interviewees noted they now pay increased attention to dissemination. Putting something online “is not enough anymore,” one subject said. They suggested further that scholars must do more than ever before to market even traditionally published scholarship.
Our subjects noted that aggregators, distributors, and library catalogues do not prioritize NTSOs, even those hosted by traditionally prestigious publishers including Stanford University Press. Scholars generally know that they can find monographs, journals, and articles using library-integrated services like the MLA bibliography, JSTOR, EBSCO, ProQuest, and Worldcat. None of the subjects we spoke with were aware of any such systems that indexed digital projects. NTSOs, as a result, are less visible and discoverable than print scholarship. According to one subject, many well established aggregators, in fact, see NTSOs as a potential threat to their market share. Their focus has been embedding digital assets into proprietary distribution models. In some cases, vendors have focused on adding metadata to open access or public domain materials. Improved search or browse functionality ostensibly justifies re-distributing free materials for profit.
Some of our interview subjects had negative reactions to these attempts. They wanted to know why for-profit companies were in control of academic scholarship that they felt should be under the control of their makers or at least available without charge. (None offered solutions for the increased burden this would place on makers.) Other subjects were more focused on outcomes. One such subject said they had begun using Google Scholar because it indexes recent scholarship better than their university library catalog. Newer aggregators generally do not provide easy ways to search for digital projects, datasets, libraries, bots, or other digital project “detritus” that might be of interest to a digital scholar.
One interview subject had a particularly negative reaction to Google Scholar. The root of this particular critique was Google's opaque metadata and indexing standards, but it suggested a broader concern. New indexes, like the aggregators they seek to supplant, do not account for NTSOs.
- [NTSOs] Standardization must form around NTSO formats and metadata before indices/aggregators can pick them up.
- [Catalysts] Encourage standardization around local or funded NTSOs.
- [Catalysts] Fund, create, or join consortia tasked with creating generalized indices for NTSOs of particular forms.
- [Makers] Learn, adopt, or help create standards for NTSOs, particularly with respect to standard locations and structures for metadata.
- [Catalysts] Fund studies and support projects into how to make repositories more NTSO-compatible.
Financial models and licensing
The publishers we interviewed spoke of financing and maintaining NTSOs as two main stumbling blocks to integrating them into existing business practices. This was true for traditional publishers, higher education institutions publishing on their own platforms, and individual creators who self-published. Currently, there is no business model established for NTSOs that is as clear as the models for print publishing. In a print context, publishers can reliably calculate cost-per-book upfront, and have developed industry-standard methods for cost recovery.
Our interview subjects making NTSOs expressed reluctance to work with for-profit publishers. These publishers might have the resources to experiment, one subject said, but their goals conflict with academia’s. University presses were seen as a more ethical option. Many such presses have expressed interest in publishing NTSOs, but their business models often prevent them from experimenting freely. Outside grants allow some experimentation, but this does not constitute a sustainable model.
Offering NTSOs alongside a monetizable print component can sometimes offset these costs. However, this structure can also create tension between the print and digital component, since the financial model implies that the print component is the main product. Several of the press representatives we spoke to said they were hesitant to try a digital-first model, for fear of cannibalizing sales. The few interview subjects who had taken this approach said they used embargo periods—during which time only the print publication would be available for sale—to assuage this fear. More often, access to a digital object would come gratis, on its own or with purchase of a print edition.
Publishers we spoke to said that lack of technical uniformity and clear production pipelines are the root drivers of costs. Each digital object requires different labor, which creates a larger financial burden for publishers. Hiring expertise in various technologies, they said, would be especially cost prohibitive. Combined with a lack of consensus around cost-recovery models, these forces push production labor to NTSO project teams and their host institutions.
Based on our interviews, this financial model is relatively common. However, it makes NTSO publication less appealing for project teams, as they are not able to offset their labor as much as they would like. Many NTSO makers are not able to pursue this model, as they lack the necessary funding, expertise, or institutional support. Such factors keep NTSOs marginal, which further inhibits their legitimacy and potential prestige.
Presses may also avoid NTSO publication because unclear lines of ownership can sometimes develop with large scale digital projects. In our interviews, this issue came up several times. If a project has been handed off from one lead investigator to another, it may be unclear who has the ability to offer control and ownership of the component pieces. The concern is also partially a financial one because licensing concerns are seen as a risk to cost recovery. If licensed material is part of project, that license is a continuing cost. The licensing party could discontinue services or abruptly change their price.
In contrast, some makers cited the control they gained by self-hosting NTSOs as an upshot. Many may use self-hosting to sidestep issues of intellectual property, since they are not accountable to publishers’ legal teams to produce signed intellectual property agreements. Self-hosting can also allow an NTSO to continue to function after official financial support has been exhausted. Although self-hosting can sometimes increase the lifespan of a project, interview subjects who said they preferred to maintain control of their materials seemed more comfortable with the idea of digital ephemerality. Some NTSOs, they said, do not need to be as durable as printed books.
- [NTSOs] Financial models appear to be held back by a collective action problem. Publishers seem to be risk-averse because it is difficult to assess costs upfront. Outside forces can easily disrupt an NTSOs ability to pay for itself long-term. Normalizing NTSO production could alleviate both problems. Meanwhile, many scholars are dissuaded from taking on NTSO production because of their marginal status. One solution would be to produce a critical mass of non-traditional scholarly output to try and make it less marginal (supply side). The other would favor targeting NTSO demand by appealing to new audiences (consumer side).
Challenges Maintaining Non-Traditional Scholarly Objects
Project teams, digital scholarship centers, and host institutions are responsible for most NTSO maintenance. Likewise, they are the primary distributors of these objects. With print publication, scholarly presses handle production, distribution, and preparing materials for long-term access (typesetting, printing on acid-free paper, indexing, etc.). Libraries are then primarily responsible for the care and preservation of these print publications. With NTSOs, maintenance burdens have shifted to those who have not traditionally been responsible for such tasks, and who are often ill-prepared for their requirements.
Our subjects pointed to many issues that project maintainers need to address. Such labor includes scoping resource requirements, preparing projects for preservation, and retroactively dealing with “legacy projects” built on outdated platforms.
There was no consensus among our subjects about who should be responsible for project maintenance. Several said that their labor in making an NTSO should end at the point of publication. As with a print resource, the resulting object would become someone else’s responsibility. One of our subjects said that long-term maintenance concerns prohibited faculty from considering NTSO production. If a project might only be maintained for three to five years, they felt their time and effort would be better spent writing a traditional book. Others said the shifted maintenance burden made it easier to take their projects with them when they left institutions. Still others viewed direct control over how long their scholarship remained available online as a benefit.
Disparate notions of maintenance
Many of the people we interviewed questioned the way we were using the term preservation and wondered whether the scholarly community as a whole was using it appropriately with respect to NTSOs. They asked if maintenance of digital project could be considered preservation. If not, where was the boundary between these two activities? Interview subjects asked “what is the difference between the live web site and the archived site? We can’t get rid of preservation, but can we get rid of maintenance?” Collectively they raised questions about how separate these activities are.
This report adopts the position that maintenance and preservation must be viewed as separate categories. Conflating the two obscures much of the labor of NTSO production. For the purposes of this report, we have tried to use maintain and preserve consistently. A project that is being maintained continues to be accessible via the same or similar means as originally designed. A project that is being preserved may not be rapidly or easily accessible in its original context but may continue to be accessible in the long term. Web archiving provides an illustrative example of how we distinguish between maintenance and preservation. A web project that is hosted on its original domain on the same platform or perhaps flattened into a static site is being maintained. A web archive of that site that is accessed via the Wayback Machine or is being stored without public access is being preserved. The modality and availability of access is how we, for the purposes of this report, separate the two practices.
- [Makers] [Hosts] [Catalysts] Pursue education and outreach strategies to close gaps in understanding about maintenance.
- [Hosts] [Catalysts] Create professional development opportunities targeting maintenance task competencies.
- [Makers] Clearly define maintenance tasks and document work.
- [Makers] [Hosts] Use project charters and other documents to help set clear expectations for maintenance at a project’s outset.
- [Catalysts] Direct grant funding towards normalizing NTSO maintenance activities.
Our interviews returned often to the so-called “legacy projects” problem. Teams, centers, departments, or institutions have existing obligations to maintain certain NTSOs. Some digital content, likewise, is seen as too important to lose. Since many of these projects originated twenty or more years ago, many use non-standard or now-out-of-date technologies. They may be dependent legacy database engines, webservers, and operating systems. These projects can constitute major security risks for hosting institutions, especially when code with known exploits is running on public-facing servers. One common technique involves hijacking a server’s resources to create a zombie or bot. A project may continue to function while its resources are used to send spam emails, mine cryptocurrencies, or spread malware. An NTSO's functionality may degrade or disappear due to changing browser or platform standards, security patches, or changes to external resources.
- [NTSOs] It would greatly reduce costs and complexities were many actively maintained legacy NTSOs transferred to a static or encapsulated preservation state. This would require significant initial investment by [Catalysts] and [Hosts], but would reduce overall costs of long-term maintenance and preservation.
- [Makers] [Hosts] Help establish clear pathways between maintenance and preservation to ease hand-offs.
- [Makers] [Hosts] Avoid models where continuous updating and ongoing maintenance is the norm. Many NTSOs could adopt a model more like a scholarly monograph, with various editions in stable preservation states.
- [NTSOs] Software containers, though not a panacea for maintenance challenges, can reduce maintenance overhead and mitigate the effects of “dependency hell.” Containers are often perceived as difficult to adopt but are often praised as time-savers once adopted.
- [Makers] Consider simplifying NTSOs using approaches like minimal computing.
- [Makers] Build graceful degradation into original objects or their metadata.
Technical and personnel resources
Maintaining digital objects requires significant personnel and technical resources. Both of these raise costs. Some interview subjects, especially solo practitioners, said they took on maintenance and stewardship responsibilities themselves, including personally paying for hosting, renewing domains, updating software, responding to copyright issues, moderating user-comments, and updating metadata for search engine indexing. With few exceptions, our subjects reported letting digital objects remain inaccessible when their projects became too difficult to maintain. They rarely put effort into long-term access or preservation beyond a reliance on, for example, GitHub or an institutional repository (if the IR could accept the project in the first place).
One solo practitioner said their projects might remain offline for months at a time before someone notices and informs them. Fixing the underlying issue could take longer still. More than anything, maintenance requires people tasked with the labor of maintenance. One interview subject said that lack of personnel was “the ultimate hurdle.” All the computers in the world, they said, aren’t enough to maintain projects if there aren’t enough people with knowledge and expertise to work on them. Those with outside support (institution, publisher, etc.) sometimes shared short-term maintenance labor. More often, solo practitioners or teams reported making hard choices about how to best use limited resources.
Maintenance is a 24/7 enterprise. It combines ensuring an NTSO maintains functionality and remains reasonably secure. As one subject reminded us, it could even include walking into a server room and moving a computer because of a leak in the roof. In our section on making NTSOs, we described the trade-offs between standardization and experimentation. These decisions affect a project's eventual maintenance demands. Institutions committed to maintaining multiple software stacks (each perhaps requiring its own virtual machine) must have the expertise on hand to deal with each of these stacks. Hosts with standardized system stacks can maintain more projects at once, but at the expense of expressive capacity. Many content management systems, though, are perpetual targets of attacks. As a result, using platforms such as Omeka or Scalar incur additional maintenance costs in dealing with the security risks of standardization. In our interviews, the precise cost of accommodating multiple software stacks vs. standardized systems remained unclear.
- [Makers] [Hosts] Create documentation that addresses an eventual project hand-off directly.
- [Evaluators] Help establish and defend the norm among NTSOs and NTSO production teams that documentation will directly address expectations for any future project hand-off.
- [Makers] [Hosts] Develop consistent editorial policies, agreements, etc. before hand-offs take place. This recommendation may make preservation easier, either by freeing hand-off teams from making curatorial decisions or providing clear guidance for those decisions.
- [Makers] [Hosts] Allocate the necessary time and resources for maintenance teams to keep sites functional and secure. New maintenance needs, such as recently identified security risks, can arise at a moment's notice.
As noted in our interviews, maintenance costs are difficult to estimate ahead of time and can grow quickly. Costs can remain low for longer periods of time, with sudden increases for upgrading or short-term troubleshooting. One interview subject compared digital scholarship to venture capitalism. They said this paradigm focuses on starting projects rather than maintaining them in the long term. Based on our interviews, although funders mandate maintenance and sustainability, current funding systems do not accommodate these requirements. Sustainable NTSOs need permanent infrastructure and personnel, which are hard to maintain with project-based expenditures.
Our interview subjects were practical about the resource requirements of maintaining projects. Many (in small and large institutions) pointed out the need to be realistic about what individuals, centers, or libraries can do. Determining whether a project should be maintained or preserved, they said, required balancing several factors:
- Audience or community demand for the resource.
- The level of access needed to meet current demand.
- Commitments made to grant-funders, partners, etc.
- The value or significance of the project.
Several interview subjects spoke of both intellectual and monetary value. Some suggested that digital project maintainers should not be afraid to justify and even monetize their projects to support maintenance costs. One said they wanted to “take the hubris” out of digital projects, arguing: if you let it go and it’s valuable, people will step in to ensure its survival.
- [Makers] [Hosts] Work to make maintenance costs more transparent, and incorporate these costs into budgets.
- [NTSOs] Broadly speaking, the digital scholarship community should be open to cost models that involve monetizing to support maintenance costs (e.g., following the model of Omeka.net, MLA member resources, etc.).
- [Hosts] Develop models to pass maintenance burdens from a project's creators to the people or communities who are most invested in its survival. This is how libraries maintain access to certain valued physical books or journals. Even institutions that want to act as maintainers of access to NTSOs, however, often lack the resources and infrastructure to do so.
- [Catalysts] [Hosts] Normalize budget lines related to institutional infrastructure and budget for continuing costs, rather than project-based expenditures. This will require a shift in institutional culture, and may require new types of institutions built around maintaining and making accessible NTSOs. Increased funding will be necessary, and might come from lines such as university indirect rates.
Challenges Preserving Non-Traditional Scholarly Objects
As we've noted, the people we spoke with tended to conflate maintenance and preservation. The soft edge between the two speaks to the need for more clarity about the differences between these two practices. It may also suggest that project teams are keeping NTSOs in states of active maintenance while trying to preserve them. The digital scholarship community has debated how to ethically and practically use the Internet as a publication space and a repository space. What’s more, previous scholarship has pointed out that NTSOs blur the lines between these two activities.
In our interviews, we focused on questions of what to preserve, why to preserve it, how to enact effective preservation strategies, and how long a preserved digital object should last. We also asked about who should perform the labor and who should pay for it. Others have pointed out that effective NTSO preservation begins before an object is built. Keeping this idea in mind, we have made efforts to frame preservation in relation to making, publishing, and maintaining.
Interview subjects had strong opinions about what were their most essential outputs. Some argued that their data, ideally in rawest form, were the fundamental building blocks of their work. Others said metadata was an essential output that must be preserved as well. Many pointed out that metadata was necessary for automated indexing systems and would enable citation. Some felt that code, or to a lesser extent software-dependencies, were important digital outputs to preserve.
Less often mentioned digital outputs included project documentation and user interfaces. Many tasked with preserving digital projects expressed frustration at the lack of explanatory documentation. This was true for experts working on custodial legacy projects and non-custodial preservation. We heard often that production was prioritized over documentation and metadata.
Tension between ephemerality and permanence, combined with existing incentives, may reinforce such priorities.
Many of our subjects felt that digital interfaces were the least important aspects of their projects to preserve. This feeling may be because they did not consider the design and user experience of their NTSOs to be scholarly outputs worthy of study in their own right.
Others said they thought preserving video recordings of user interactions was sufficient for preserving an interactive user interface. However, they recognized a loss of context when trying to preserve digital projects in this way. One interview subject in particular used the extended metaphor of preserving old Nintendo games: “You can have a Nintendo and you can have a Mega Man II cartridge, but you can’t have the sleepovers where you played Mega Man II until four in the morning.”
- [Makers] [Hosts] [Catalysts] Continue education and outreach efforts. As with maintenance, the perceived burden of preservation is shaped by gaps in understanding.
- [Makers] Develop better explanatory documentation to address how projects will be preserved, especially in post-custodial preservation contexts. This is particularly important for points of hand-off.
- [Catalysts] Pursue grant programs to increase education, outreach, and documentation initiatives, aimed specifically at increasingly NTSO preservability.
Purpose of Preservation
Our interview subjects had vastly different views on the purpose and usefulness of preservation. Where Ithaka’s “The State of Digital Preservation in 2018” frames digital preservation as a necessity, our subjects questioned the preservation of NTSOs. Lack of consensus on this subject might reflect a digital scholarship value system in flux. Our report, unlike Ithaka’s, focuses specifically on digital scholarship, so many of our subjects responded to the idea of preserving argument-driven objects.
One group of subjects expressed little interest in preserving NTSOs. Some considered their projects as a form of digital ephemera. Such NTSOs may be important for a time but also significant in that they are not designed to last. Others articulated a tension between the freedom to explore and the demands of preservability. Their experimental “flights of fancy,” one said, simply weren’t worth long-term care. The idea that not everything can or should be preserved came up in many of our interviews.
Most of our interview subjects did not share this view. NTSOs, they said, should be preserved due to scholarly standards of evidence, especially to ensure the veracity of future arguments built on prior work. The comparison we heard most related to citing sources. As long as an NTSO is making a contribution to the scholarly conversation, they said, it should be available for scholars to reference. Unlike in the sciences and social sciences, where arguments and experiments might become “stale” after a few years, humanities arguments and evidence may be referenced for decades. Our subjects said this time scale makes preservation more important in the humanities.
NTSOs that no longer contribute to contemporary debates often have value to other audiences. Some of these may be cultural heritage objects in themselves. Others may inform future intellectual histories. One of our interview subjects argued that self-analysis and reflection requires access to the historical record. Studying the historiography or legacy of digital fields requires access to the NTSOs created in those fields. They asked, how much of the history of humanities computing and digital humanities has been lost due to the challenges of preserving early digital objects?
Finally, and most pragmatically, our interviewees pointed to issues revolving around contractual requirements. Many grant-funded digital projects have data management plans or other commitments to keeping project outputs accessible. Institutions and scholars are often bound (by law or contract) to preserve the outputs of sponsored research. Others said preserving their project outputs was a moral imperative. We heard this position most with public-facing resources used by a particular community, or in the classroom.
The decisions around what to preserve are challenging in the face of scarce resources. Meanwhile, NTSOs are becoming increasingly complicated. The people we spoke with said they felt no obligation to spend time and money preserving projects of interest to “just one professor.” They did note, however, that they felt a stronger obligation to preserve social justice projects, regardless of usage statistics. This raises the issue of when and what to preserve, which we address in the next section.
- [Makers] [Audiences] [Evaluators] Participate in conversations with communities of interest in determining what is preserved and why.
- [Makers] Reach out to potential stakeholders early in the design of an NTSO. Expectations for such community building is likely to increase over time.
- [Makers] [Audiences] [Evaluators] Clearly articulate motives and needs. Different reasons for preservation can inform different preservation strategies.
- [Makers] [Hosts] Continue to develop levels of preservation in concert with users or audiences' needs.
When to preserve
In the previous section, we raised the issue of how to determine an NTSO’s value. Appraisal, especially in libraries and archives, is a well-established process used to determine the value of records. These determinations inform decisions about preservation and deaccessioning. The labor of preservation is often invisible from the outside and taken for granted, as noted by several subjects. In traditional scholarly publishing, preservation takes place after-the-fact and the author is not involved. The scholars we interviewed, however, had a deeper understanding of appraisal and preservation than we expected. Such understanding perhaps arose from their need to play a greater role in these processes than would be necessary with printed materials.
In the context of NTSOs, a crucial and related concept is graceful degradation, the gradual loss of functionality over time. Digital objects do not automatically degrade in this way. If neglected, they may pass from functionality to a complete inaccessibility with little or no transition time. Some of our subjects lamented their inability to preserve NTSOs in their entirety. Most, however, said they were comfortable with graceful degradation, as long as core functionality, or the content, remained. As we noted before, ephemerality was also acceptable on a case-by-case basis.
Some of those we spoke to said that wanted the original versions of their work to be available for as long as possible. One specifically argued for the 10-year “life of a laptop” to become standard. Another suggested 5-10 years with a 20-year lifespan in special cases, followed by an explicit shut-down process. However, as we noted when discussing maintenance, most projects do not have an explicit shut-down procedure or predetermined end-of-life. This absence can give rise to what several of our interview subjects called “zombie projects,” with no clear hand-off between maintenance and preservation. It may also increase the number of projects that are still online and functional but no longer maintained, and at risk of system failure.
Our interview subjects were particularly emphatic about the need, sometimes, to embrace ephemerality. This was almost always expressed as a condition of digital experimentation. Knowing that their work does not—and indeed should not—exist in the long-term can encourage scholars to “hack” or “play.” One person said they’d found it liberating to think of a website existing only when visited by a user’s browser.
Some of our interview subjects felt that preservation standards for born-digital content exceed norms for print. Books and paper, one person said, are only considered preservable because of the infrastructure that we have built to protect them. A book outside on the pavement would only last days. Books go out of print and journals fold, so why, with digital projects, should we have an idea of “we paid for it, it should exist forever”? Some subjects said they focused on more personal short-to-medium timelines such as a semester, a graduate career, a job search, or tenure and promotion.
We detected a strong sense in many of our interviews that projects in need of long-term preservation would emerge organically from the field. Institutions and agencies willing to invest the time and resources necessary to maintain, reinvent, and generally preserve notable projects would presumably step forward. This perspective may be naïve or callous, but we believe it is important to note. A significant divide may exist between the digital preservation community and people who tend to create NTSOs.
As we noted earlier, some of our subjects did argue for the need to preserve as much as possible. One person argued that digital scholarship should last as long as non-acidic paper. Another wanted active projects to be maintained for decades. Preservation of content and outputs, one said, should be preserved “forever,” or the lifespan of the institution stewarding the digital object. Other subjects said they thought that only the most ground-breaking or frequently used digital projects needed long-term preservation, but we did not speak extensively about the selection criteria.
- [NTSOs] Although much work on this subject has been and continues to be undertaken, NTSOs must coalesce around proven models of graceful degradation.
- [Makers]Embrace graceful degradation methods in the development of NTSOs
- [Hosts] If an NTSO is not built to gracefully decay, consider building in that functionality during the hand-off between institutions, or between maintenance and preservation.
- [Makers] Have open conversations about the lifecycle of your project with your team members, and to articulate your project’s goals as they pertain to ephemerality or preservation.
- [Hosts] [Audiences] [Makers] Consider graceful degradation models that include video surrogates to capture the spirit of the original work in modified form, narrated by the author to maintain the project’s original argumentative structure for the academic record.
- [Hosts] [Audiences] [Makers] Consider, alternatively, shut-down processes that include “flattening” the digital object into a standardized or easier to preserve format, such as static HTML. A version like this would enable a digital project to remain accessible without excessive resource costs.
Continuing to Institutionalize Preservation
When asked how to establish effective digital preservation for their NTSO, many of our subjects said they hadn’t thought about it. We found that almost anyone in the development chain for a project can disavow responsibility for preservation. (One might say “it isn’t my job,” or “let’s wait and see what happens.”) Many reported taking one of these approaches in the past. Some said they stored NTSOs on a server or relied upon free, commercial services such as GitHub despite acknowledging that these weren’t preservation solutions. Anything more, some said, would exceed their available resources.
We saw strong consensus that, in an ideal world, libraries and archives would maintain and preserve NTSOs. Many said they already ask archivists, special collections librarians, and scholarly communications librarians for guidance on digital preservation. Some added that NTSOs developed with library involvement have a better chance of long term sustainability. They said they considered well established roles and responsibilities crucial for digital preservation.
Several of our subjects stressed that universities and colleges must preserve NTSOs themselves. They warned that higher education’s failure to preserve NTSOs would ensure a for-profit takeover of the labor. There was particularly strong resistance to the idea of corporate, for-profit, or vendor ownership of NTSOs. Some subjects worried the different incentives, time-scales, and values of for-profit vendors made them ill-suited for the role of cultural steward. Even some who held this opinion felt vendors were the only option, however, given a lack of local institutional expertise, resources, or technical support for dealing with digital preservation.
We saw overall agreement that the current social, technical, and financial realities are obstacles. Some stressed that smaller or less-funded institutional libraries had particular challenges. Most of our subjects, however, said that such concerns inhibit all libraries from taking on such stewardship roles.
- [Makers] [Hosts] Use "Memorandums of Understanding" between project teams and libraries to establish clear roles and responsibilities for digital preservation. These might be comparable to the practice of donation agreements associated with material and print collections.
- [Catalysts] [Hosts] Pursue consortium models for preservation to enable smaller or less-well funded institutional libraries to pool resources and take advantage of economies of scale.
- [Catalysts] [Hosts] Ensure considerable resources are available at cultural heritage institutions to deal with digital maintenance and preservation. Lack of sufficient resources appears to be the most significant reason this task gets outsourced to other organizations in ways that practitioners feel are inappropriate or inadequate.
Preservation requires ongoing technical resources, even if the project is not accessible. Potential costs include:
- Server mirroring and storing backups.
- Maintaining links (e.g., between datasets and code).
- Format migration.
- Re-deploying to new platforms.
This work, especially platform migration, requires increasing technical proficiency. The digital preservation community is exploring new technologies to streamline preservation, but some of these have a steep learning curve. Some newer preservation technologies, such as emulation or containerization, they said, are overkill or unproven for preservation. One subject said, “we don’t need [to emulate] a whole desktop environment” to provide access to a single digital project.
Only a few of those we interviewed had experience with emulation and containerization. Many held the view that either would be difficult to learn. Even those with direct experience questioned how a workforce with the necessary expertise would be developed.
Learning any new skill requires training and time commitment. New competencies can also broaden job descriptions and increase expectations. Already overextended project team members are understandably wary of proposed changes with such potential.
Attitudes about whose job it is to preserve NTSOs may constitute, in itself, a sociotechnical obstacle to effective preservation. Our subjects were sensitive to shifting responsibilities for digital preservation. As we've discussed, many we interviewed said they don't want to be responsible for long-term preservation. Several raised the prospect of a scenario where a digital maker leaves the institution where they deposited a project. This contingency seemed to provoke particular concern. Preservation, they said, should work like books. This is to say that preservation would remain the responsibility of libraries, archives, and other memory institutions, even after a maker has left.
Deep technical expertise, preservation personnel, and technological infrastructure come at significant financial cost. We heard several times that these costs need to be "baked into " project budgets. Others said they felt that grant funding should not support preservation costs. Most agreed that preservation was best accounted for as part of the overhead of a university or other institutional home.
- [Hosts] Investigate the potential of emulation or containerization as mechanisms to make active preservation easier.
- [Makers] [Catalysts] Consider using software containers for maintenance, but not preservation. However, note that making an NTSO more maintainable will often make it easier to preserve.
- [Hosts] [Audiences] [Catalysts] Explore use cases for containers where users requests access to a preserved NTSO. With a software container, a project could be recovered from its preserved state and brought temporarily into some active, temporarily maintained state for consultation. This mode of access would require technically skilled personnel who can manage the technical stack, “revive” a preserved project to make it accessible, and shut it down after the access needs have been met.
- [Hosts] Create infrastructure for NTSO creators to self-deposit their work. Such a system would need to include metadata and documentation. Many institutional repositories, at present, are unable to accommodate complex NTSOs in a way that would allow them to be easily revived.
Conclusion & Looking Forward
We envision a future in which scholarship that embraces its digital affordances and materiality are placed on equal footing with typeset, print-ready scholarly publications.
Currently, non-traditional scholarly objects (NTSOs) fit poorly in the academic world. They are less prestigious, more difficult to find, and more likely to suffer neglect than their printable counterparts. The stages of and roles involved in an NTSO’s life are ill-defined and contentious. The rich variety of NTSOs is both a blessing and a curse, resulting in an explosion of creative, transformative scholarship that by its nature defies academic norms.
Challenges faced by NTSOs will not disappear soon. Their inevitable growth is a function of the changing environments in which scholars work. A sound fitting of NTSOs within their academic world will require a series of informed, orchestrated interventions that take into account every aspect of their complex lives.
Our perspective on how to intervene emerged organically from the work of this report. Based on hundreds of hours of interviews and a survey of secondary literature, we relay common pain points associated with digital scholarship. The NTSO workflow is broken into stages (1. Making, 2. Publishing, 3. Maintaining, and 4. Preserving), with subsections organized by topic. We further identify five roles (1. Catalysts, 2. Makers, 3. Evaluators, 4. Hosts, and 5. Audiences).
Though the report’s structure suggests stages and roles are easily separated, the opposite is true. Stages and roles blur together, rarely following a neat trajectory. This blurriness betrays a lack of standard protocol engendered by the uncertainty of the new. We do not anticipate the stages and roles organized here will be those eventually settled upon, but they offer a useful starting point to articulate challenges surrounding NTSOs.
Across all stages, several broad and overlapping problem areas suggesting intervention strategies became apparent, including:
- Sociotechnical challenges being treated as technical challenges.
- Gaps in expectations and communication between roles leading to poorly aligned practices.
- Friction around hand-off points and periods of transition.
- Nonexistent or competing standards preventing NTSOs from thriving.
(1) Sociotechnical challenges are all-too-easily mistaken for purely technical hurdles. When dealing with NTSOs, individuals catalysts, makers, and hosts often try to address sociotechnical challenges with technical solutions. Even this study fell into this trap, initially positing software container technology alone as a possible cure for challenges faced by NTSOs. In interviews, we heard of decisions between digital platforms being driven entirely by technical capacities and limitations. The capabilities of such platforms are important, but so are factors like what the institution will support, what previous team members have used in the past, and what is the norm for a particular field. As these are sociotechnical challenges, separating the technical from the social, cultural, or institutional challenges is impossible; each dimension must be considered in tandem. Interventions, similarly, must address this spectrum. Clear communication of these factors at every stage is essential.
(2) Gaps in expectations and communication between roles lead to poorly aligned practices. Despite the fact that development, publication, and preservation of digital scholarship depends on a shared ecosystem, conversations in that ecosystem often fall out of sync. For example, NTSO makers seem more likely to blur distinctions between maintenance and preservation, which can obscure labor and shortchange resource needs. Likewise, motives and incentives shape expectations. A scholar including an NTSO in their tenure portfolio has specific needs for preservation and evaluation. Yet, the scholarly infrastructure at their institution may be unable or unwilling to accommodate a one-off, bespoke digital project. The motivations and incentives of the individual scholar are not aligned with the institution. Many problems we observed appeared related to a lack of communication or a failure to understand motivational contexts. Interventions must be built with members of different roles in close conversation with one another.
(3) Periods of transition and points of hand-off are the most crippling moments in the life of an NTSO, and perhaps the best starting point for an orchestrated intervention. NTSOs are often distributed across many files, systems, hand-driven modes of stewardship. Packaging and transferring NTSOs between collaborators, roles, or stages can be intensely difficult. Often, without the same group of people, technologies, and resources brought to bear on its creation, an NTSO will be impossible to move from one party to another. Even when it can be moved, as with any fragile object, an NTSO’s transfer can require significant costs and expertise. And because of the collaborative nature of NTSOs, they might need to change hands frequently. We heard frustrations like "how do I submit a digital object without self-publishing it?" or "how do I know what technical stack would work best for a particular scholarly journal?" We argue these frictions suggest a deeper question: How might the digital scholarship ecosystem normalize transfers of ownership or stewardship for NTSOs? One essential steps in this process will be the encapsulation of NTSOs, clearly demarcating an object and its context of functionality.
(4) Nonexistent or competing standards prevent NTSOs from thriving. The lack of agreed upon NTSO standards contributes significantly to the friction around hand-offs, and impedes the normalization of digital scholarship. For example, we learned about an NTSO developed within a particular library context which was technically incompatible with a journal seeking different sorts of digital content. Not only could the subject not get their NTSO published—they were unable to deposit it in the same library’s digital preservation system. Although standards have arisen around NTSOs, they often compete, or look quite different across stages. Getting past these difficulties will require more orchestrated interventions.
One question that underlies these challenges is: “Who can take action to effect change?” We avoid suggestions of how the world ought to be, disconnected from specific actors who can bring about the change. In recognition of how easy (or arguably cheap) it is to recommend that other parties make broad, sweeping changes, we pointed our recommendations toward practitioners in each of the five identified roles. Our last category of recommendation pertains to NTSOs as objects, and how we might collectively shape them to suit scholarly needs.
How and where to act first is a difficult subject. A well-established way to overcome collective action problems is to disrupt the balance with an orchestrated intervention. It would not be enough to make it easier for journals to accept bespoke websites. Such an effort would need a community of practitioners and peer reviewers to test and use the new tool. Likewise, establishing a new approach to hosting digital scholarship without bringing in stakeholders from publishing and preservation will surely fail. Some well-known failures in digital scholarship have espoused the philosophy, "if we build it they will come." Orchestrated interventions, in contrast, make building coalitions and lining up beta testers co-equal priorities to prototyping. Any attempt to create smoother hand-offs, in particular, must keep this in mind. Solutions not building toward each other are building away from each other.
After extensive research, we identified no panaceas or silver bullets. The path forward seems clearer than when we began, thanks to the continued efforts of many stakeholders. But the future we envision, in which NTSOs are as prestigious, discoverable, and easy to hand-off as their print-first counterparts, is still far off. Standards need better articulation and coordination. More shared tools and platforms must be developed. Crucially, incentives must shift to encourage and reward experimentation.
Perhaps most importantly, there is an urgent need for a single format (or set of formats) for NTSOs that encapsulates both the content and the structure of these complex digital objects. Such a format must balance its ability to enable maximally expressive scholarship alongside its need to constrain NTSOs to standard shapes. The design must support a broad range of digital projects while also reducing friction at the various points of hand-off. Standardized formats could offer clear targets tied to existing reward structures and publication systems, thus addressing the challenges around incentives.
Like with the codex book or the PDF, we anticipate a paradigm of encapsulation for NTSOs would eventually curtail the expressive diversity of scholarly objects. In the meantime, we are excited to watch the tension at the heart of this balance foster works of profound creativity and value. And we hope the approaches suggested here will help bring these works the legitimacy, durability, and wide audience they deserve.
- Agate, Nicky, Rebecca Kennison, Stacy Konkiel, Christopher Long, Jason Rhody, and Simone Sacchi. “About HuMetricsHSS.” HuMetricsHSS, September 2019. https://humetricshss.org/about/.
- AIMS Work Group. “AIMS Born-Digital Collections: An Inter-Institutional Model for Stewardship.” White Paper, 2012. http://www2.lib.virginia.edu/aims/whitepaper/AIMS_final.pdf.
- Altman, Micah, Jeffrey Bailey, Karen Cariani, Jim Corridan, Jonathan Crabtree, Michelle Gallinger, Andrea Goethals, et al. “2015 National Agenda for Digital Stewardship.” Report. NDSA Coordinating Committee, September 2014. http://www.digitalpreservation.gov/documents/2015NationalAgendaExecSummary.pdf.
- Arp, Laurie Gemill, and Megan Forbes. It Takes a Village: Open Source Software Sustainability. A Guidebook for Programs Serving Cultural and Scientific Heritage. LYRASIS, 2018.
- Bailey, Charles W. Digital Curation Bibliography: Preservation and Stewardship of Scholarly Works. USA: CreateSpace Independent Publishing Platform, 2012.
- Belcher, Wendy Laura. Writing Your Journal Article in Twelve Weeks: A Guide to Academic Publishing Success. SAGE, 2009.
- Borgman, Christine L. Scholarship in the Digital Age: Information, Infrastructure, and the Internet. Cambridge, MA: MIT Press, 2008.
- ———. “The Digital Future Is Now: A Call to Action for the Humanities.” Digital Humanities Quarterly 3, no. 4 (January 2, 2010). https://escholarship.org/uc/item/0fp9n05s.
- Bryson, Tim, Miriam Posner, Alain St. Pierre, and Stewart Varner. SPEC Kit 326: Digital Humanities (November 2011). SPEC Kit. Association of Research Libraries, 2011. https://doi.org/10.29242/spec.326.
- Burkert, Mattie. “London Stage Database.” Mattie Burkert (blog), October 15, 2015. https://mattieburkert.com/london-stage-project/.
- ———. “Recovering the London Stage Information Bank: Lessons from an Early Humanities Computing Project.” Digital Humanities Quarterly 011, no. 3 (August 28, 2017).
- Butler, Brandon, Amanda Visconti, and Ammon Shepherd. “Archiving Dh Part 3: The Long View.” Scholars’ Lab (blog), April 30, 2019. https://scholarslab.lib.virginia.edu/blog/archiving-dh-part-3-the-long-view/.
- Carlin, Claire, Ewa Czaykowska-Higgins, Janelle Jenstad, Elizabeth Grove-White, Corey Davis, John Durno, Lisa Goddard, et al. “The Endings Project,” 2019. https://projectendings.github.io.
- Daigle, Bradley, Lorrie Chisholm, Brantley Craig, Elizabeth Gushee, and Matthew Stephens. “Valley of the Shadow.” University of Virginia Library Digital Production Group, 2003. https://dcs.library.virginia.edu/sustaining-digital-scholarship/valley-of-the-shadow/.
- Davis, Robin Camille. “Taking Care of Digital Efforts: A Multiplanar View of Project Afterlives.” In Proceedings of the Modern Languages Association 2015. Vancouver, 2015. https://robincamille.com/presentations/mla2015/.
- Elliott, Michael A. “The Future of The Monograph in the Digital Era: A Report to the Andrew W. Mellon Foundation.” Journal of Electronic Publishing 18, no. 4 (December 17, 2015). https://doi.org/10.3998/3336451.0018.407.
- Eve, Martin Paul. “Open Access Publishing Models and How OA Can Work in the Humanities.” Bulletin of the Association for Information Science and Technology 43, no. 5 (2017): 16–20. https://doi.org/10.1002/bul2.2017.1720430505.
- Fenlon, Katrina, Megan Senseney, Maria Bonn, and Janet Swatscheno. “Humanities Scholars and Library-Based Digital Publishing: New Forms of Publication, New Audiences, New Publishing Roles.” Journal of Scholarly Publishing, April 12, 2019. https://doi.org/10.3138/jsp.50.3.01.
- Galey, Alan, and Stan Ruecker. “How a Prototype Argues.” Literary and Linguistic Computing 25, no. 4 (December 1, 2010): 405–24. https://doi.org/10.1093/llc/fqq021.
- Germano, William. Getting It Published, 2nd Edition: A Guide for Scholars and Anyone Else Serious about Serious Books. University of Chicago Press, 2009.
- Gitelman, Lisa. Paper Knowledge: Toward a Media History of Documents. Durham ; London: Duke University Press Books, 2014.
- Griffin, Gabriele, and Matt Steven Hayler. “Collaboration in Digital Humanities Research – Persisting Silences.” Digital Humanities Quarterly 012, no. 1 (April 4, 2018).
- Groeneveld, Elizabeth. “Remediating Pornography: The On Our Backs Digitization Debate.” Continuum 32, no. 1 (January 2, 2018): 73–83. https://doi.org/10.1080/10304312.2018.1404677.
- Guiliano, Jennifer, and Roopika Risam. “Reviews in Digital Humanities.” Reviews in Digital Humanities, August 30, 2019. https://reviewsindh.pubpub.org/.
- Gunsalus, C. K., Edward M. Bruner, Nicholas C. Burbules, Leon Dash, Matthew Finkin, Joseph P. Goldberg, William T. Greenough, Gregory A. Miller, and Michael G. Pratt. “Mission Creep in the IRB World.” Science 312, no. 5779 (June 9, 2006): 1441–1441. https://doi.org/10.1126/science.1121479.
- Haynes, Anthony. Writing Successful Academic Books. Cambridge University Press, 2010.
- Humphreys, Alex, Christina Spencer, Laura Brown, Matthew Loy, and Ronald Snyder. “Reimagining the Digital Monograph: Design Thinking to Build New Tools for Researchers.” Journal of Electronic Publishing 21, no. 1 (2018). http://dx.doi.org/10.3998/3336451.0021.102.
- Ithaka S+R. “Life Cycle of a Digital Resource.” Ithaka S+R (blog), 2016. https://sr.ithaka.org/life-cycle-of-a-digital-resource/.
- Jules, Bergis, Ed Summers, and Vernon Mitchell, Jr. “Documenting the Now: Ethical Considerations for Archiving Social Media Content Generated by Contemporary Social Movements: Challenges, Opportunities, and Recommendations.” White Paper, April 2018. https://www.docnow.io/docs/docnow-whitepaper-2018.pdf.
- Keener, Alix. “The Arrival Fallacy: Collaborative Research Relationships in the Digital Humanities.” Digital Humanities Quarterly 009, no. 2 (August 7, 2015).
- Kemp, Angie, Lee Skallerup, and Kris Shaffer. “What Do You Do with 11,000 Blogs? Preserving, Archiving, and Maintaining UMW Blogs - A Case Study.” Journal of Interactive Technology and Pedagogy, no. 15 (2019). https://jitp.commons.gc.cuny.edu/what-do-you-do-with-11000-blogs-preserving-archiving-and-maintaining-umw-blogs-a-case-study/.
- Kirby, Jasmine Simone. “How NOT to Create a Digital Media Scholarship Platform: The History of the Sophie 2.0 Project.” IASSIST Quarterly 42, no. 4 (February 22, 2019): 1–16. https://doi.org/10.29173/iq926.
- Klein, Martin. “A Web-Centric Pipeline for Archiving Scholarly Artifacts.” Keynote presented at the TPDL/DCMI, 2018. https://www.slideshare.net/martinklein0815/a-webcentric-pipeline-for-archiving-scholarly-artifacts.
- Kretzschmar, William A., and William Gray Potter. “Library Collaboration with Large Digital Humanities Projects.” Literary and Linguistic Computing 25, no. 4 (December 1, 2010): 439–45. https://doi.org/10.1093/llc/fqq022.
- Langmead, Alison, Tracey Berg-Fulton, Thomas Lombardi, David Newbury, and Christopher Nygren. “A Role-Based Model for Successful Collaboration in Digital Art History.” International Journal for Digital Art History, no. 3 (July 27, 2018). https://doi.org/10.11588/dah.2018.3.34297.
- Lavoie, Brian F, Eric Childress, Ricky Erway, Ixchel M Faniel, Constance Malpas, Jennifer Schaffner, Titia van der Werf, and OCLC Research. The Evolving Scholarly Record. Dublin, Ohio: OCLC Research, 2014. http://www.oclc.org/content/dam/research/publications/library/2014/oclcresearch-evolving-scholarly-record-2014.pdf.
- Lavoie, Brian F, Constance Malpas, and OCLC Research. Stewardship of the Evolving Scholarly Record: From the Invisible Hand to Conscious Coordination, 2015. http://www.oclc.org/content/dam/research/publications/2015/oclcresearch-esr-stewardship-2015.pdf.
- Levinson, Marc. The Box: How the Shipping Container Made the World Smaller and the World Economy Bigger. Princeton, NJ: Princeton University Press, 2008.
- Linclon, Matthew, Zoe LeBlanc, Rebecca Sutton Koeser, and Jamie Fulsom. “The State of Digital Humanities Software Development Roundtable.” Pittsburgh, PA, 2019. https://www.conftool.org/ach2019/index.php?page=browseSessions&form_session=139&presentations=show.
- Lockridge, Timothy, Enrique Paz, and Cynthia Johnson. “The Kairos Preservation Project.” Computers and Composition 46 (December 1, 2017): 72–86. https://doi.org/10.1016/j.compcom.2017.09.002.
- Maron, Nancy L. “The Department of Digital Humanities (DDH) at King’s College London 2011: Cementing It Status as an Academic Department, Case Study Update 2011.” Ithaka Case Studies in Sustainability. Ithaka S+R, 2011. https://sr.ithaka.org/publications/the-department-of-digital-humanities-ddh-at-kings-college-london-2011/.
- Maron, Nancy L, and Sarah Pickle. “Sustainability Implementation Toolkit: Developing an Institutional Strategy for Supporting Digital Humanities Resources.” Ithaka S+R (blog), June 18, 2014. https://sr.ithaka.org/publications/sustainability-implementation-toolkit/.
- ———. “Sustaining the Digital Humanities: Host Institution Support beyond the Start-Up Phase.” Ithaka S+R, June 18, 2014.
- Maron, Nancy L, K. Kirby Smith, and Matthew Loy. “Sustaining Digital Resources: An On-the-Ground View of Projects Today.” New York: Ithaka S+R, August 11, 2015. https://doi.org/10.18665/sr.22408.
- Maron, Nancy, Kimberly Schmelzinger, Christine Mulhern, and Daniel Rossman. “The Costs of Publishing Monographs: Toward a Transparent Methodology.” Journal of Electronic Publishing 19, no. 1 (Summer 2016): 52. https://doi.org/10.3998/3336451.0019.103.
- Maxwell, John W., Alessandra Bordini, and Katie Shamash. “Reassembling Scholarly Communications: An Evaluation of the Andrew W. Mellon Foundation’s Monograph Initiative (Final Report, May 2016).” Journal of Electronic Publishing 20, no. 1 (2017). https://doi.org/10.3998/3336451.0020.101.
- Maxwell, John W., Erik Hanson, Leena Desai, Carmen Tiampo, Kim O’Donnell, Avvai Ketheeswaren, Melody Sun, Emma Walter, and Ellen Michelle. “Mind the Gap.” PubPub: Simon Fraser University / MIT Press, July 2019. https://mindthegap.pubpub.org/.
- Mazanec, CeciIia. “#ThanksForTyping Spotlights Unnamed Women In Literary Acknowledgments.” NPR.org, March 30, 2017. https://www.npr.org/2017/03/30/521931310/-thanksfortyping-spotlights-unnamed-women-in-literary-acknowledgements.
- McCarty, Willard. “Collaborative Research in the Digital Humanities.” In Collaborative Research in the Digital Humanities, 13–22. Routledge, 2016.
- Meneses, Luis, Jonathan Martin, Richard Furuta, and Ray Siemens. “Quantifying the Degree of Planned Obsolesce in Online Digital Humanities Projects.” presented at the ACH, Pittsburgh, PA, July 2019.
- MLA Task Force. “Report of the MLA Task Force on Evaluating Scholarship for Tenure and Promotion.” Profession 2007, no. 1 (November 26, 2007): 9–71. https://doi.org/10.1632/prof.2007.2007.1.9.
- Morgan, Paige C. “The Consequences of Framing Digital Humanities Tools as Easy to Use.” College & Undergraduate Libraries 25, no. 3 (July 3, 2018): 211–31. https://doi.org/10.1080/10691316.2018.1480440.
- Murray, Padmini Ray, and Claire Squires. “The Digital Publishing Communications Circuit.” Book 2.0 3, no. 1 (June 1, 2013): 3–23. https://doi.org/10.1386/btwo.3.1.3_1.
- Newbold, Bryan. “About Fatcat.” fatcat!, 2019. https://fatcat.wiki/about.
- Nowviskie, Bethany. “Evaluating Collaborative Digital Scholarship (or, Where Credit Is Due).” Journal of Digital Humanities 1, no. 4 (Fall 2012).
- Nowviskie, Bethany, and Dot Porter. “The Graceful Degradation Survey: Managing Digital Humanities Projects Through Times of Transition and Decline.” In Proceedings of Digital Humanities 2010. King’s College, London, 2010. http://dh2010.cch.kcl.ac.uk/academic-programme/abstracts/papers/html/ab-722.html.
- Oppegaard, Brett, and Michael Rabby. “The App-Maker Model: An Embodied Expansion of Mobile Cyberinfrastructure.” Digital Humanities Quarterly 010, no. 3 (August 17, 2016).
- O’Sullivan, James. “The Equivalence of Books: Monographs, Prestige, and the Rise of Edge Cases.” Convergence 24, no. 5 (October 1, 2018): 494–503. https://doi.org/10.1177/1354856518780457.
- Owens, Trevor. The Theory and Craft of Digital Preservation. Johns Hopkins University Press, 2018.
- Padilla, Thomas, Laurie Allen, Hannah Frost, Sarah Potvin, Elizabeth Russey Roke, and Stewart Varner. “Final Report — Always Already Computational: Collections as Data,” May 22, 2019. https://doi.org/10.5281/zenodo.3152935.
- Pendergrass, Keith, Walker Sampson, Tim Walsh, and Laura Alagna. “Toward Environmentally Sustainable Digital Preservation.” The American Archivist 82, no. 1 (June 2019): 165–206. https://doi.org/10.17723/0360-9081-82.1.165.
- Pitti, Daniel V. “Designing Sustainable Projects and Publications.” In A Companion to Digital Humanities, edited by Susan Schreibman, Ray Siemens, and John Unsworth, 469–87. Malden, MA, USA: Blackwell Publishing Ltd, 2004. https://doi.org/10.1002/9780470999875.ch31.
- Plantin, Jean-Christophe, Carl Lagoze, and Paul N Edwards. “Re-Integrating Scholarly Infrastructure: The Ambiguous Role of Data Sharing Platforms.” Big Data & Society 5, no. 1 (June 2018): 205395171875668. https://doi.org/10.1177/2053951718756683.
- Posner, Miriam. “Digital Humanities and the Library.” Blog. Miriam Posner’s Blog (blog), April 2013. http://miriamposner.com/blog/digital-humanities-and-the-library/.
- Quinlan, Oliver. “Young Digital Makers: Surveying Attitudes and Opportunities for Digital Creativity across the UK.” Nesta, March 2015. https://media.nesta.org.uk/documents/youngdigmakers.pdf.
- Reed, Ashley. “Managing an Established Digital Humanities Project: Principles and Practices from the Twentieth Year of the William Blake Archive.” Digital Humanities Quarterly 008, no. 1 (April 17, 2014).
- Rieger, Oya. “The State of Digital Preservation in 2018: A Snapshot of Challenges and Gaps.” Ithaka S+R, October 29, 2018. https://doi.org/10.18665/sr.310626.
- Robertson, Tara. “Concerns about Reveal Digital’s Statement about On Our Backs.” Tara Robertson (blog), October 12, 2016. https://tararobertson.ca/2016/oob-part3/.
- ———. “Digitization: Just Because You Can, Doesn’t Mean You Should.” Tara Robertson (blog), March 20, 2016. http://tararobertson.ca/2016/oob/.
- ———. “Update on On Our Backs and Reveal Digital.” Tara Robertson (blog), August 16, 2016. https://tararobertson.ca/2016/oob-update/.
- Rosenthal, David. “Personal Pods and Fatcat.” DSHR’s Blog (blog), April 18, 2019. https://blog.dshr.org/2019/04/personal-pods-and-fatcat.html.
- Russell, Andrew, and Lee Vinsel. “Hail the Maintainers.” Aeon, 2016. https://aeon.co/essays/innovation-is-overvalued-maintenance-often-matters-more.
- Schreibman, Susan, Laura Mandell, and Stephen Olsen. “Introduction.” Profession, 2011, 123–35.
- Shweder, Richard A., and Richard E. Nisbett. “Don’t Let Your Misunderstanding of the Rules Hinder Your Research.” The Chronicle of Higher Education, April 19, 2017. https://www.chronicle.com/article/Don-t-Let-Your/239823.
- ———. “Long-Sought Research Deregulation Is Upon Us. Don’t Squander the Moment.” The Chronicle of Higher Education, March 12, 2017. https://www.chronicle.com/article/Long-Sought-Research/239459.
- Sieczkiewicz, Robert. “On the Diversity of Digital Decay.” In Proceedings of Keystone DH 2016. Pittsburgh, PA, 2016. http://keystonedh.network/2016/abstracts/#submission-40.
- Sikes, Sara B. “A Design Process Model for Inquiry-Driven, Collaboration-First Scholarly Communications – DH2018.” Mexico City, 2018. https://dh2018.adho.org/en/a-design-process-model-for-inquiry-driven-collaboration-first-scholarly-communications/.
- Sikes, Sara, Maria Bonn, and Elli Mylonas. “Building Capacity for Digital Scholarship & Publishing: Three Approaches from Mellon’s 2014-2015 Scholarly Communications Initiative,” 3. Montreal, Canada, 2017. https://dh2017.adho.org/abstracts/135/135.pdf.
- Silvia, Paul J. How to Write a Lot: A Practical Guide to Productive Academic Writing. American Psychological Association, 2018.
- Smart, Scott, Charles Watkinson, Gary Dunham, and Nicholas Fitzgerald. “Determining the Financial Cost of Scholarly Book Publishing.” Journal of Electronic Publishing 19, no. 1 (Summer 2016). https://doi.org/10.3998/3336451.0019.102.
- Smithies, James, Carina Westling, Anna-Maria Sichani, Pam Mellen, and Arianna Ciula. “Managing 100 Digital Humanities Projects: Digital Scholarship & Archiving in King’s Digital Lab.” Digital Humanities Quarterly 013, no. 1 (April 26, 2019).
- Sula, Chris Alen. “Digital Humanities and Libraries: A Conceptual Model.” Journal of Library Administration 53, no. 1 (January 1, 2013): 10–26. https://doi.org/10.1080/01930826.2013.756680.
- “Supporting the Digital Humanities: Report of a CNI Executive Roundtable.” Coalition for Networked Information, 2016. https://www.cni.org/wp-content/uploads/2016/05/CNI-SupportDH-exec-rndtbl.report.F14.pdf.
- Tarkang, Elvis E., Margaret Kweku, and Francis B. Zotor. “Publication Practices and Responsible Authorship: A Review Article.” Journal of Public Health in Africa 8, no. 1 (June 27, 2017). https://doi.org/10.4081/jphia.2017.723.
- Terras, Melissa, James Baker, James Hetherington, David Beavan, Martin Zaltz Austwick, Anne Welsh, Helen O’Neill, Will Finley, Oliver Duke-Williams, and Adam Farquhar. “Enabling Complex Analysis of Large-Scale Digital Collections: Humanities Research, High-Performance Computing, and Transforming Access to British Library Digital Collections.” Digital Scholarship in the Humanities, May 2, 2017. https://doi.org/10.1093/llc/fqx020.
- The Information Maintainers, D. Olson, J. Meyerson, M. A. Parsons, J. Castro, M. Lassere, D. J. Wright, et al. “Information Maintenance as a Practice of Care.” White Paper, June 17, 2019. https://doi.org/10.5281/zenodo.3251131.
- The Prototyping Team of the Los Alamos National Laboratory, and The Web Science and Digital Library Research Group at Old Dominion University. “About.” My Research Institute, 2018. https://myresearch.institute/about/.
- Thomas, William G., and Patrick D. Jones. “History Harvest.” University of Nebraska-Lincoln, 2019. https://historyharvest.unl.edu/.
- Trettien, Whitney. Cut/Copy/Paste: Fragments of History. University Of Minnesota Press, Forthcoming. https://manifold.umn.edu/projects/cut-copy-paste.
- Vinopal, Jennifer, and Monica McCormick. “Supporting Digital Scholarship in Research Libraries: Scalability and Sustainability.” Journal of Library Administration 53, no. 1 (January 1, 2013): 27–42. https://doi.org/10.1080/01930826.2013.756689.
- Visual Media Workshop at the University of Pittsburgh. “The Socio-Technical Sustainability Roadmap.” The Socio-Technical Sustainability Roadmap, October 2018. http://sustainingdh.net.
- Waters, Donald J. “Monograph Publishing in the Digital Age.” The Andrew W. Mellon Foundation. Shared Experiences Blog (blog), July 22, 2016. https://mellon.org/resources/shared-experiences-blog/monograph-publishing-digital-age/.
- Wilkin, John, Ronald W. Bailey, Antoinette Burton, Allen Renear, Harriett Green, Megan Senseney, Marilyn Thomas-Houston, et al. Publishing Without Walls, 2016. https://publishingwithoutwalls.illinois.edu/.
- Zundert, Joris van. “On Not Writing a Review about Mirador: Mirador, IIIF, and the Epistemological Gains of Distributed Digital Scholarly Resources.” Digital Medievalist 11, no. 1 (August 3, 2018): 5. https://doi.org/10.16995/dm.78.