Digital Archives and Data Science: Building Programs and Partnerships for Health Sciences Research

Introduction

When the UCSF Industry Documents Library first launched in 2002, it was one of the earliest digital libraries to offer access to its collections on the web. From the beginning, librarians, archivists, technologists, and the public health research community at the University of California, San Francisco (UCSF) grappled with how best to manage, index, search, and analyze millions of digitized documents which had been released from historic litigation against the tobacco industry. As the interrelated fields of information science, digital archives, and data science evolved, the Industry Documents Library (IDL) developed its own software tools and approaches to meet researchers’ “big data” needs, at a time when few out-of-the-box solutions existed and none fit the unique requirements of a project which combined elements of digital archives, law libraries, and corporate accountability. For two decades the IDL focused on stewarding its data, building custom open-source tools for search and access, and supporting new public health research methods for studying and responding to the spread of tobacco-related diseases. During this period the IDL collection grew from a few thousand pages to a few million pages, and now contains more than 94 million pages in over 15 million documents. As the volume of available material continues to increase – the IDL anticipates adding more than 5 million new documents in the next year – the need to build partnerships and design collaborative strategies for managing data at scale is more imperative than ever. The expanding data science landscape now provides significant opportunities for the IDL to move beyond custom isolated solutions, and to embrace an approach which engages with an advancing library and data science knowledge base, innovative new ideas developed by other libraries and archives, and an expanding network of potential collaborators and partners.

This chapter begins with an overview of the UCSF Industry Documents Library, and highlights examples of how IDL scholars have applied data science methods to investigate specific questions about the tobacco industry’s impact on public health. It describes the application of emerging computational tools for digital archival appraisal and description, and the potential for these tools to dramatically improve the IDL’s capacity for curating its data. The chapter then describes the IDL’s current and potential collaboration opportunities with the UCSF Library’s Data Science Initiative (DSI), which serves as a campus hub for education and support in data science. This is followed by a description of another UCSF archival initiative which serves as a model for the IDL’s data science engagement efforts: the UCSF Archives and Special Collections project “No More Silence: Opening the Data of the HIV/AIDS Epidemic using Natural Language Processing Techniques.” This project offers rich instruction in how to prepare data for computational analysis, build a data science research community, develop training workshops, and connect with other campus partners. The chapter concludes with recommendations identified by the IDL, UCSF partners, and others in the library and archives profession, on how to encourage staff development and skill building, build cross-campus partnerships, provide opportunities for mutual learning through data science student internships, integrate data science tools into existing workflows, and plan for the long-term sustainability of data science activities.