The Scholarly Impact of Books Acquired via Approval Plan Selection, Librarian Orders, and Patron-Driven Acquisitions as Measured by Citation Counts

David C. Tyler, Brianna D. Hitt, Francis A. Nterful, and McKenna R. Mettling*

Patron-driven acquisition has been an important, if contentious, topic for decades, with numerous programs having been piloted, adopted, and reported on, largely favorably, in the library literature. Still, questions and doubts persist for academic libraries, especially where the composition of vendor plans and packages and the judgment of patrons are concerned. Past literature has approached the assessment of patron-driven acquisition by analyzing circulation/usage, comparing peer-library holdings, seeking patrons’ or librarians’ judgments of utility and suitability, looking for evidence of collection imbalances, and testing for overlap in patrons’ and librarians’ purchases. To contribute to this literature, this study addresses scholarly impact and examines whose selections—approval plans’, librarians’, or patrons’—have been most heavily cited. For the social sciences, the sciences, and the humanities, the authors gathered topic-matched random samples of books acquired via approval plans and librarian orders during the first five years of operation of their institutions’ interlibrary loan purchase-on-demand patron-driven acquisition program and compared their citation counts to the counts of books acquired via the program. Google Scholar was employed to tally citations.

Introduction

Academic librarians with collection development responsibilities and library administrators who have followed the library literature on acquisitions and collection development since the 1999 publication of Perdue and Van Fleet’s study of the successful program at Bucknell University1 could hardly be unaware of the explosion of interest in patron-driven acquisition (henceforth, PDA)—or, alternately, demand-driven (DDA), user-driven, and/or evidence-based acquisition. In 2010 and in 2012, respectively, the ACRL Research Planning and Review Committee identified collection growth driven by patron demand and patron-driven e-book acquisition as top ten trends in academic libraries.2 As well, several authors contemporaneously named PDA one of the most widely discussed acquisition models in the library literature.3 In fact, Walker declared US research libraries to be on a PDA “tipping point” in 2011.4

The PDA approach would certainly seem to have been quickly and widely adopted. In a national survey reported on in 2010, Lenares and Delquie found just 32 respondents with a current program, 42 with plans to implement a program within a year, and 90 institutions indicating that they would like to implement a program within the next three years.5 The following year, Osorio presented a survey of librarians in Illinois that found that a mere 12.6 percent and 23.1 percent of respondents had a PDA or DDA program for print books or e-books, respectively.6 In a 2013 article adapted from a 2012 report to The Andrew W. Mellon Foundation, Esposito, Walker, and Ehling estimated that there were just 400 to 600 programs in existence.7 More recently published survey results, however, have shown that between 85 and 95 percent of respondents report that their libraries have PDA/DDA programs in place.8

The field’s high level of interest and the increasing number of programs piloted and adopted by academic libraries have produced a veritable avalanche of literature. Caminita, in a chapter published in 2014, reported that searches of Google Scholar and of EBSCO’s Library, Information Science and Technology Abstracts (LISTA) employing a selection of terms associated with college libraries and PDA found more than 2,000 and more than 400 items, respectively, had been indexed since 2009.9 A search of Google Scholar employing Caminita’s terms performed by the lead author of this study during the fall of 2017 for items indexed since the beginning of 2013 found nearly 12,000 items.10 Thus, it would seem that the field’s interest in PDA has been and continues to be quite keen, and adoption of the model has been widespread. In fact, several in the field have argued that this “new” patron-driven approach has or will soon become standard library practice,11 although perhaps it should be noted that several authors, both PDA critics and supporters, have taken pains to declare that patron-driven collecting is actually nothing new and is part of what libraries have always done.12 One can only speculate, but perhaps this reflects a desire on the part of the model’s critics to call into question the boosterish enthusiasm of some of PDA’s promoters and a desire on the part of the model’s promoters to reassure that a seemingly new and potentially risky acquisition practice is not actually new and is, therefore, not risky.

Regardless of whether one enthusiastically embraces or unenthusiastically laments the development and adoption of PDA, the authors of this study would caution that it is early yet to draw firm conclusions concerning the model and its effects. Thus far, the published literature has cast PDA as a new and exciting disruptive technology/practice that moves academic libraries away from a wasteful and moribund “just-in-case” model and toward a lean and responsive “just-in-time” model;13 as a subversion of library hegemonies, hierarchies, and authority/power relations;14 as a potential revolution in library work;15 as a major strategic change;16 as a “powerful shift in how libraries acquire materials”;17 and as a continuation of or a return to past practice in a new guise …something both new and not.18 From the midst of this promotional whirl, several authors have thoughtfully called into question the PDA model and argued that it has several failings that will produce numerous unfavorable outcomes. Early studies, though, regardless of the metrics employed or the issues addressed and assessed, have largely produced findings and conclusions favorable to the PDA model.19 Many of these findings have not, though, been entirely generalizable or particularly robust. As Walker has noted, the published research on, as well as the criticism of, PDA “has consisted, in the main, of opinion pieces debating the merits of PDA and case studies of libraries’ experiences implementing PDA.”20 Likewise, Dewland and See have noted that a shortcoming of the model and its literature has been the absence of consistent, objective, and standardized metrics for success.21

Remedying this entirely would be well beyond the current authors’ abilities or the scope of a single article, yet the authors do hope that they may here introduce an approach that will address one of the more frequently raised criticisms of PDA: its potential for acquiring books of low scholarly merit. Unfortunately, “scholarly merit” is something of an amorphous and multifaceted measure with a very high potential for subjectivity. Unsurprisingly, the library literature thus far has tended to approach the idea obliquely and fractionally rather than directly and comprehensively, most commonly through assessments of PDA-purchased books’ appropriateness for academic library collections. One facet of scholarly merit that has yet to be addressed is scholarly impact. So far as the authors have been able to ascertain, no one has yet examined whether the books supplied/ordered via the various selection/acquisition modes—approval plan selection, librarian firm order, or PDA—tend to be more heavily cited. To begin to remedy this, the authors propose to examine the citation counts of the books of the three disciplines—the social sciences, the sciences, and the humanities—acquired via the three main acquisition methods (henceforth, “Order Type[s]”) employed by the University of Nebraska-Lincoln Libraries (UNL Libraries). The books in question were added to the circulating collection during the first five years (2003/04 to 2007/08) of the UNL Libraries’ interlibrary loan-driven purchase-on-demand (ILL-PoD) program for print books. For those unfamiliar with the terminology, Carrico, Leonard, and Gallagher have defined ILL-PoD as follows: “ILL-PoD is a patron-driven purchase model whereby patrons request books not owned by their local library, and based on various criteria…, the book is purchased rather than borrowed from another library.”22 For the criteria employed by the UNL Libraries, excepting the list of barred publishers, please refer to appendix A. The citation counts for the books were collected via Google Scholar from January to April of 2016, roughly 8 to 13 years after they had been added to the collection and 12 years, on average, after they had been published. In this manner, the authors hope to discover which Order Type, if any, attracted the most citations, which the authors hope will serve as an at least partial measure of the Order Types’ scholarly impact and merit.

Review of Literature

As was noted above, a great deal has been written about the PDA model and about PDA/DDA programs. Much of the published research literature that has assessed and evaluated the outcomes of PDA programs has supported the approach,23 but some of the library literature has been cautionary, if not outright critical. The criticisms of PDA pertinent to this study can be distilled to two issues: first, the difference in the objectives of the traditional, collection-building, “just-in-case” model and the more patron-oriented, “just-in-time,” PDA model; and second, the different motivations and aims of traditionalist academic librarians, of vendors and publishers, and of library patrons. When developing collections in accord with the traditional model, academic librarians act as experts, agents, and gatekeepers, in which roles they identify, collect, manage, and preserve the best and most significant scholarship of their day and of the past, as identified by themselves and other authorities, such as the teaching faculty and reviewers of academic books. The intent behind the model is to build a collection to meet institutions’ educational missions, to preserve the scholarly record, and to meet not just the needs of the day’s patrons but the needs of future scholars working 10, 20, 50, and even 100 years hence.24

By way of contrast, the PDA model was derived at least in part from an approach to manufacturing and warehousing developed by Toyota that was designed to meet customers’ needs as they arose, rather than anticipating them.25 As Costello explains in the preface to her recent book on the subject, “Demand-driven acquisitions (DDA) describes any acquisitions process that is driven by the desires of patrons or their usage of materials rather than predictive processes like package plans and librarian selection.”26 Thus, as one of its most vigorous proponents, Rick Anderson of the University of Utah, has suggested, the PDA model is not intended for building collections in the traditional sense at all.27

For PDA’s critics, the fairly recent, widespread, and enthusiastic adoption of PDA/DDA approaches by academic libraries marks an abdication of one of academic librarians’ main professional responsibilities,28 the “ultimate source of pride of many collection specialists,”29 through a turning away from collection-building as an end in itself, what Anderson has termed the “monument to Western Civilization” collections mindset.30 For PDA’s advocates, it instead marks a positive turn toward the adoption of what several authors have termed a facilitated-collection or collections-as-service mindset centered on present patrons’ expressed, rather than anticipated, needs.31 Not incidentally, it also marks something of a reduction in academic librarians’ control over the composition of their libraries’ collections and a corresponding empowerment of patrons.32 As such, it is not difficult to suppose why traditionalist academic librarians who understand collection building to be one of their core duties find PDA worrisome and upsetting, especially when some of its more radical champions have suggested it ought to become the primary, if not the sole, method for adding books to academic libraries’ collections.33

In addition to being concerned with the lessening of collection building as academic librarians’ raison d’être, PDA’s critics have also called into question vendors’/publishers’ and patrons’ motives, expertise, and knowledge. With respect to vendors/publishers, they have warned that book jobbers and publishers are primarily in the business of selling books, so academic libraries ought to be wary of their PDA plans and e-book DDA packages lest their library catalogs come to resemble online shopping malls crowded with inappropriate and low-quality materials.34 Similarly, with respect to PDA patrons, several authors have warned against, or have reported hearing colleagues opine against, allowing patrons too much freedom to add books to academic libraries’ collections. Critics have cautioned that patrons tend to be too narrowly focused on their immediate needs, that patrons request/access books whose utility and scholarly worth is unknown to them, that patrons will unbalance collections in their pursuit of idiosyncratic or trivial popular topics, and so forth.35

The criticisms of vendors strike the authors of this study as somewhat problematic. It has been a generally accepted professional truism that good collection development is a product of good partnerships and the sharing of expertise between librarians on the one hand and vendors and publishers on the other, and one can certainly see this attitude expressed in the library literature from both sides.36 Yet, in the critics’ support, there has been some recent, empirically supported counter-literature that has found that approval plan selections produce fewer circulations than do librarians and/or patrons’ purchases37 and offer worse value in terms of cost-per-circulation,38 although it should also be noted that a number of studies have found that approval plan selections slightly outcirculated books ordered by individual selectors.39 Approval plans also have been found to do a poor job of providing libraries with books from small presses and from small societies, with highly specialized books, and with books whose subject matter or approach fall outside traditional disciplinary boundaries.40 There also does seem to be some evidence that DDA e-books packages warrant scrutiny. Two recent studies have both reported their authors having discovered nonacademic materials in their e-book packages despite both pairs of authors having worked closely with vendors to exclude such materials.41 An internal study of e-book subscription packages conducted at the UNL Libraries similarly found questionable materials in the vendors’ packages, as well as books from suspect publishers. Thus, cautions against giving over collection development too much to vendors/publishers would certainly seem worth considering.

The library literature’s response to criticisms of patrons as selectors has been varied, and the authors would be hard-pressed to do justice here to the voluminous scholarship on this subject or to summarize it succinctly. For a lengthy overview of the approaches taken in the literature, the authors recommend Costello’s recent Evaluating Demand-Driven Acquisitions.42 For the authors’ current purposes, it should suffice to note that the attitudes apparent behind much of the criticisms of patrons have been chided for exhibiting a snobbish paternalism43 and that much of the early research on PDA programs would seem to support the model’s ability to deliver collection-appropriate books of acceptable scholarly merit. As Tyler et al. have noted:

The primary worry about inappropriate books has largely been answered in the existing body of literature. Authors have reported small numbers or percentages of their programs’ requested or purchased items as being inappropriate to their libraries, with subject, Dewey call number or Library of Congress (LC) subclass, genre, readership-/content-level, publisher, or material type variously employed as criteria. However, librarians who have directly reviewed the requested or purchased items or who have reviewed requests by subject criteria, publisher, material type, readership-level, reviewers’ recommendations, or against peer institutions’ holdings, have found PDA items to largely have been worthy and appropriate purchases for their collections.44

Still, although critics’ dire predictions against patrons as selectors have yet to find much solid support in the research literature, they should not be dismissed entirely. There does seem to be some scattered evidence in the critics’ favor. For example, with respect to the problem of “hot topics,” there is, of course, the infamous “banana books” incident, recounted by Polanka at the Charleston Conference, wherein a library’s PDA budget was quickly spent in response to a class assignment.45 Additionally, concerning patrons’ much-feared fondness for less-scholarly materials, Goedeken and Lawson have discovered that their patrons appear to have an affection for books in Wiley’s Dummies series.46 Of interest to critics questioning patrons’ motivations for accessing/requesting books would be the finding of Reynolds et al. that patrons’ second-most-frequent reason given for using the “Suggest a Purchase” service at Texas A&M was for recreational reading.47

A few critics, rather than call into question all patrons, have limited themselves to suggesting that perhaps only undergraduates ought to be barred from adding books to academic libraries collections and that only faculty and, perhaps, graduate students should be allowed. With respect to the issue of undergraduates’ potential for being problematic selectors, one recent study of a PDA program for undergraduates did find that the books undergraduates requested tended not to be widely held by peer institutions,48 whereas studies of programs that were less restricted by patrons’ status have found PDA/DDA books to be held widely by peer institutions or held widely in general.49 The same study on undergraduates as selectors also reported that selections made by undergraduates appeared not to obtain as great a circulation advantage as has been reported elsewhere for less restricted programs.50 In yet another study, Purdue University’s librarians found that books requested by undergraduates tended to be subsequently checked out most frequently by other undergraduates.51 These studies raise the strong possibility that undergraduates’ PDA/DDA purchases may be qualitatively different from purchases made by faculty, graduate students, and other types of patrons, which has understandably made placing the patrons on an equal footing to selectors an issue of concern for some PDA critics.52

In light of the uncertainties recounted above, and owing to the fact that no one has yet looked into the scholarly impact of PDA books, it would seem worthwhile to examine whether books provided by vendors or publishers via approval plans or e-book packages, purchased by librarians via firm ordering, or acquired via PDA/DDA programs produce a greater scholarly impact. As well, given the uncertainty surrounding the various types of academic library patrons as purchasers of scholarly works, it would also seem worthwhile to examine whether PDA books acquired by the various types of patrons are of commensurate interest to scholars.

Background, Data Collection, and Method of Analysis

Toward the end of the 2002/03 school year, the manager of the UNL Libraries’ ILL department brought Perdue and Van Fleet’s article on Bucknell’s successful PDA program to the attention the Collection Development Committee and suggested that it might consider a pilot project involving ILL PoD. As a result, since the start of the 2003/04 school year, the UNL Libraries has operated an ILL PoD PDA program for recently published nonreference academic books. After five years’ operation, in response to a call for papers for a special issue of the journal Collection Management, data were drawn for the PDA-acquired books and for all print books added to the circulating collection during the interval via traditional acquisition modes. The resulting data set produced several articles concerning the circulation performance of this set of books, the cost-per-circulation, whether the books were being monopolized, and the selection patterns of librarians and patrons at the level of Library of Congress (LC) subclass.

For the current study, the authors gathered records for PDA-acquired books in the data set with LC subclasses associated with the social sciences, the sciences, and the humanities. From the contemporaneous approval plans’ selections and librarian firm-ordered titles, the authors then drew equal numbers of books for each LC subclass via simple random selection without replacement. Subclasses for which not all three Order Types had purchases were not included in the draw. Thus, the authors produced three “matched” samples for each discipline’s purchases (for example: if PDA patrons had purchased 10 books in subclass QA—Mathematics, then 10 books for that subclass would have been randomly drawn from the approval plans’ and from the librarians’ books, resulting in a pool of 30 books for purposes of analysis and comparison). Very small numbers of books were purchased by the PDA patrons in some LC subclasses, so the authors, to facilitate analysis, found it necessary to merge several of the LC subclasses into topical groups (in other words, “Topic”). For the composition of the Topic groups of each discipline, please see table 1.

In addition to being curious in general about the performance of the selection modes that comprise the three Order Types, the authors’ attention, as was noted above, was also drawn to the performance of the library’s patrons by the literature critical of PDA/DDA programs for failing to exclude certain sorts of patrons. Therefore, the authors further grouped the books in the UNL Libraries’ PDA data sets into books ordered by one of five “Patron Types” by status. As can be seen from table 2 below, PDA at UNL has mostly been a means for graduate students and for members of the faculty to make additions to the circulating collection, which is consistent with results reported elsewhere in the library literature for PDA programs for print materials,53 although not necessarily for e-book DDA programs.54 Unfortunately, the UNL Libraries’ ILL department’s data collection form allowed patrons to self-report their status, which reporting was not mandatory, so it would appear from the table that roughly 11.4 percent of PDA patrons did not sufficiently identify themselves and had to be classed as “Unknown.”

To gather data on the apparent scholarly interest in/impact of these books, as indicated by cumulative citation over the study’s interval, the lead author searched Google Scholar from January through April of 2016 for the books’ index entries and citation counts. In a number of instances, there were multiple index entries for books due to variations in how they had been cited, so each book was searched for with three different combinations of search terms drawn from the author, title, and publisher fields of their catalog records. Upon discovering the books’ index entries, the books’ citations were recorded and, if necessary, summed. In a few instances, books in the data set proved to be earlier or later editions or to be books from numbered series. In such instances, only citation counts for the appropriate editions or series numbers were included in the tallies. In a very few instances, index entries for the books were not discoverable. In those instances, the lead author conducted a fourth search that employed the books’ first authors’ last names and a few words from the books’ titles as a phrase. The lead author then manually tallied the number of times the books appeared in the first 10 pages of search results. The author found, when forced to employ this approach, that none of these books appeared beyond the fifth or sixth page of search results.

The collected data produced highly right-skewed distributions of citation counts (see appendix B). As an initial, informal testing of the waters, the authors attempted within each discipline to test for differences in performance by Order Type, by Order Type within Topic groups, by Patron Type within the PDA subset, and by Patron Type within Topic groups using Poisson regression, but this initial test run produced results that were unlikely, with nearly every comparison proving to be significant, and the Poisson models were poor fits. Subsequent model fitting suggested that the negative binomial distribution was a much better fit for the data, so the authors reanalyzed the performance of the Order Types and Patron Types using negative binomial regression analysis with p value adjustments for multiple comparisons.

Research Questions

As should be evident from the foregoing, the foci of this study are the citation count performances of the three Order Types and, within the PDA data sets, the performances of the five Patron Types. For each of the three disciplines, the authors first asked a pair of questions: If the three Order Types had been given an opportunity to select/purchase equal numbers of books on the same Topics for the UNL Libraries, would there be differences in the books’ performance as generators of citation activity, and, if so, which Order Type’s books would have been cited more? As a follow-up research question, the authors asked: Would there be differences in Order Types’ citation count performances within Topic groups? If there were no significant differences between Order Types’ performances in general, the authors reasoned, perhaps there would be performance differences concealed within particular subject areas. Alternately, if there were differences between the Order Types’ performances in general, these differences might be present across all Topics or might be limited to outsized differences within just a few. For the study’s final research question, the authors repeated Research Question #1 but turned their attention to the PDA data sets and to the five Patron Types: Within the PDA data sets of each discipline, did any of the Patron Types outperform any of the others?

Results

Research Question #1: For each discipline, were there significant differences in performance by Order Type? If so, which Order Types were cited more?

For this first question, the authors tested within disciplines the three Order Types’ citation counts for statistically significant differences and then performed post hoc tests with adjustments for multiple comparisons to determine which Order Types, if any, outperformed one or more of the others. As the F values for the three tests of fixed effects in table 3 show, there would fairly definitely seem to be at least one significant difference in performance present in each discipline. To determine the nature and direction of the differences between Order Types, the authors performed the aforementioned post hoc comparisons, whose results are to be found in the “Differences of Order Type Least Squares Means” portions of the table.

For the social sciences, as one can see from the reported t values and adjusted p values presented in the top third of the table, the librarians significantly outperformed both the approval plans and the patrons, and the patrons significantly outperformed the approval plans. (Note: In reading this and the tables to follow, positive t values indicate that the Order Type or Patron Type in the left-hand column outperformed the one in the right-hand column; a negative value indicates the reverse.) How great were the differences in citation-count performance? To answer this question, one will need to subtract the mean values found in the table. For example, the difference in citation counts between the approval plans and the librarians in the social sciences was 161.03–362.00 = –200.97. That is to say that the book jobbers’ selectors underperformed the UNL social sciences librarians by roughly 201 citations per book. Similarly, the librarians outperformed the PDA patrons by roughly 86 citations per books, and the patrons outperformed the approval plan selectors by roughly 115 citations per book.

For the sciences, the post hoc comparisons show that there were two statistically significant performance differences to be found among the Order Types. Both the librarians and the patrons handily outperformed the approval plans. The librarians outperformed the approval plans by approximately 301.46 citations per book, and the patrons outperformed the approval plans by roughly 227.28 citations per book. The librarians did outperform the patrons by approximately 74 citations per book. Were one to multiply this difference by the number of books in question, one would find that the librarians’ books attracted more than 25,000 more citations than did the patrons’ books; that certainly seems like a large amount, but, when one consults the table, one can see that this difference was not statistically significant. In fact, it was not significant even prior to the adjustment for multiple comparisons (p = 0.1046). Still, it is difficult not to see it as substantial.

Finally, for the humanities, as was the case with the social sciences, all three post hoc comparisons reveal statistically significant differences. The librarians again outperformed the approval plans’ selectors. Unlike with the social sciences, the PDA patrons outperformed both of the other Order Types. If one consults the mean values displayed and/or consults appendix B, one will notice that the parameters and counts for the humanities tended to be quite a bit smaller than those of the social sciences and of the sciences, although the shape of the citation counts’ distributions tended to be quite similar. Despite these differences between the disciplines, statistically significant differences were still very much present in the humanities. The humanities librarians attracted almost 10,000 more citations than did the approval plans, and the patrons garnered roughly 20,000 more than the former and almost 30,000 more than the latter.

Research Question #2: For each discipline, were there significant differences in performance between Order Types within Topics? If so, which Order Types were cited more?

As a follow-up to the first research question, the authors parsed the data sets a bit more finely and again tested the performance of the Order Types, this time within the Topic categories presented in table 1. The authors tested the Order Types’ performances within the 10 Topic categories of the social sciences; the 13 categories of the sciences, although only the results of 9 will be presented; and the 11 Topic categories of the humanities. The four missing sciences categories—Arts & Crafts, Building Construction, Dentistry, and Therapeutics & Pharmacy—were included in the modeling and testing of the data, but they comprised so very few books that the authors felt that it was not worthwhile—and would perhaps be misleading—to report their results.

As was the case above, the authors looked for the presence of statistically significant differences in the three disciplines’ data sets and then followed with a battery of post hoc comparisons to determine where and in what direction the significant differences in performance within Topics, if any, occurred. Because of the sizable number of comparisons being made, the authors unfortunately found it necessary to split the table employed above into three parts, one for each discipline. In the upper portion of each discipline’s subtable, the authors once again will report the results of the tests of fixed effects, this time looking for Order Type and Topic effects and, most important, for Order Type*Topic interaction effects. In the remainder of each subtable, the authors will again report on the post hoc comparisons performed, as in table 3 above.

As one can see from a quick review of tables 4a-c below, there were statistically significant Order Type*Topic interaction effects for all three disciplines, with the social sciences and the humanities producing roughly equal F values; but, with the sciences, despite its enormous F value displayed in table 3 above, producing a value slightly less than one half of either. With Topic taken into account, perhaps the sciences exhibited fewer or less extreme differences than the other disciplines. To obtain a clearer picture of what occurred in the data sets, one will need to turn to the post hoc comparisons.

First, for the social sciences, the post hoc testing determined that performance differences between Order Types were not present across all Topic groups. As the t values and adjusted p values in table 4a show, there were statistically significant differences in performance in just 8 of the 30 comparisons made, after the adjustment for multiple comparisons. For example, in the Anthropology & Recreation Topic group, there was no statistically significant difference in performance between the approval plans and the librarians, despite the latter having attracted roughly 48 more citations per book. The PDA patrons, however, outperformed both in a statistically significant manner, garnering roughly 219 more citations per book than did the approval plans and roughly 171 more per book than did the librarians. The close reader of the tables may notice that some of the differences in means appear to be quite large, yet the adjusted p values show the differences not to be significant at all, such as was the case with the Topic Social Sciences (General) & Statistics. In such cases, an explanation will likely be found by referring back to the composition of the samples presented in table 1 and referring to the standard error of the means column (“S.E. Mean”) in the table under review. If one were to do so with this example case, one would see that the number of books being tested was small and that the standard errors, partially as a result, were quite large. Thus, one could speculate that had more books been purchased in the LC subclasses H and HA, then this Topic would also have produced a batch of statistically significant results. Unfortunately, as things stand, this can only be a speculation.

With respect to the sciences, if one were to peruse the mean values reported for each Topic, one would see that the librarians’ books attracted more citations in eight of the nine Topic categories reported on than did the approval plans’citations, with the sole exception being Agriculture. Likewise, the patrons’ mean citations were higher than the approval plans’ citations in all nine Topics. Both results would certainly be in line with the more general results reported on in table 3. If one were to review the t and adjusted p values for the approval plans’ post hoc comparisons, one would see that the approval plans were statistically significantly outperformed in 9 of the 18 comparisons involving them, and they were outperformed by nearly statistically significant amounts in an additional three comparisons—versus librarians in Medicine (General) and versus patrons in Chemical Technology & Manufacturing and Engineering & Technology (General). In reviewing the Topic means for librarians and for patrons, one can see that the performance results were more mixed. Librarians outperformed patrons by substantial amounts in four Topics, patrons substantially outperformed librarians in three topics, and the performance of both was roughly equal for Medicine (Clinical & Internal) and just slightly favorable for patrons in Medicine (General, Public Health, & Pathology). Although some of the performance differences appear substantial to the eye, however, none of the post hoc comparisons for librarians and patrons produced differences that were statistically significant. In fact, just one, the comparison for Physical Sciences, was even close to being significant.

Finally, let us turn to the humanities. A review of the reported means shows the performance of the approval plans to be quite mixed vis-à-vis the librarians, with the approval plans having higher means in six Topics and with the librarians having higher means in five. The approval plans’ performance against the PDA patrons was not as good, with the latter having higher means in 11 of the 12 Topic categories. Against the PDA patrons, the librarians’ performance was similarly poor, with the patrons having higher mean values for nine Topics. A review of the post hoc comparisons supports this sense of the Order Types’ performances within Topics. Of the 33 post hoc comparisons, 15 proved to be statistically significant, with the patrons outperforming the approval plans or the librarians in 12 of them. Additionally, the patrons outperformed the approval plans by a nearly significant amount in the Topic Fine Arts and the librarians in the Topic Non-English Languages & Literatures. Of the three remaining comparisons whose results were statistically significant, the approval plans outperformed both the librarians and the patrons in Photography, and the librarians outperformed the approval plans in Philology, Linguistics, & Literature (General). The librarians also nearly outperformed the approval plans by a statistically significant amount in English Language & Literature. Thus, the results reported above for the general samples presented in table 3, which showed the patrons strongly outperforming both the approval plans and the librarians and which showed the performance difference between the approval plans and the librarians to have been slightly less pronounced, would seem to have been well supported here.

Research Question #3: For each discipline, were there significant differences in performance by Patron Types? If so, which types were cited more?

The study’s third research question is, essentially, the application of Research Question #1 to the citation counts of just the PDA-acquired books, with the five Patron Type categories assuming the place of the three Order Types. As such, tables 5a–c below should each be read in much the same manner as were the disciplines’ sections of table 3 above. In the upper portion of each discipline’s section of the table, one will again find the tests of fixed effects, which indicate whether or not at least one statistically significant difference was present in the data sets being analyzed. As one can see via a quick review of the three pertinent sections of the tables, there was strong evidence of at least one statistically significant difference in the social sciences data set, considerably weaker evidence of at least one difference in the sciences data set, and strong evidence for at least one difference in the humanities data set.

For the social sciences, as the t and adjusted p values in the table show, there were several statistically significant differences in performance between Patron Types, and every Patron Type was involved in at least one such pairing. The surprise in the social sciences PDA data set for the authors was the performance of Staff, which outperformed all of the other Patron Types and outperformed three of the other four by statistically significant amounts. Its outperformance of the Unknown patrons was also nearly significant prior to the adjustment of p values. The next-best-performing Patron Type among the categories whose members were identifiable was Graduates, who significantly outperformed Faculty and Undergraduates. Surprisingly, Faculty did not significantly outperform Undergraduates, although it should be noted that the Faculty mean was higher.

Before proceeding to the next discipline, the authors thought it would be worthwhile to note the magnitude of some of the differences in performance in the social sciences. The top-performing Patron Type, Staff, outperformed Faculty by roughly 565 citations per book, Graduates by roughly 452 citations per book, and Undergraduates by roughly 617 citations per book. Graduates, by way of comparison, slightly outperformed Faculty and Undergraduates by roughly 113 and 165 citations per book, respectively, and Faculty outperformed Undergraduates by just 52 citations per book. The performance of social sciences’ Staff was quite remarkable.

In the PDA data set for the sciences, the Patron Type performances were more in keeping with what one would expect from having read the cautionary PDA literature. Faculty and Graduates had roughly equal means and handily outperformed the other three Patron Types by more than one hundred citations per book. Surprisingly, however, none of the post hoc comparisons proved to be statistically significant after the adjustment of p values. Faculty and Graduates did significantly outperform the Staff and Unknown types prior to adjustment, but not the Undergraduates. A closer inspection of the data set revealed that Undergraduates books had the greatest performance variability, so this may in part account for the lack of a detectable difference. The bulk of Undergraduates books performed no better than did Staff and Unknown books, but the best-performing of these books performed as well as the best-performing Graduates books and nearly as well as the best-performing Faculty books. From this, one would expect that, were Undergraduates acquisitions to be increased, eventually a statistically significant difference in performance would develop.

The performances of the Patron Types in the humanities’ PDA data set were more problematic than they were in the social sciences or sciences. The tests of fixed effects showed the presence of at least one likely real difference in the humanities’ PDA data set; but, after the adjustment for multiple comparisons, the only significant differences that remained were those where the Unknown type outperformed the Faculty, Graduates, and Staff Patron Types, which information is, clearly, not useful. If one were to review the preadjustment p values, one would see that Graduates nearly significantly outperformed Faculty and Staff and that Undergraduates nearly outperformed Staff. Though Graduates and Undergraduates had nearly equal means, the latter’s performance did not as closely approach significance, likely because of the small number of books purchased by Undergraduates and because of the comparatively high variability in their citation performance, as was the case with the sciences’ Undergraduates above.

Analysis

For the social sciences, the results displayed in table 3 suggest that some of PDA’s naysayers’ and critics’ concerns about potential differences in the quality of vendors’, librarians’, and patrons’ book selections may be justified and that their arguments in favor of librarian title-by-title selection were warranted. Librarians’ selections during the interval had more of an impact, as measured by citation counts, than did book vendors’ and patrons’ selections by statistically significant amounts. Patrons’ selections, of course, also had more impact than did book vendors’ selections for the approval plans, also by a statistically significant amount. The librarians’ sample of 491 books garnered 42,293 more citations than did those of the patrons, for a difference of roughly 86 citations per book. When looking at these numbers, it would be difficult to argue that librarian selection did not add value or that librarian selection and patron selection were equivalent where scholarly impact is concerned.

That said, these results would suggest to the authors that the more important story was the dismal performance of the approval plans’ book selectors. Over the study interval, the sample of titles that the book vendors sent to the UNL Libraries garnered almost 100,000 fewer citations than did the titles in the librarians’ sample and slightly more than 56,000 fewer citations than did the patrons’ purchases. However correct one may count PDA’s critics concerning patrons’ failings as book selectors for the ages, the citation counts would suggest that the critics would be substantially more justified in concluding that book vendors deserve close, constant, and careful scrutiny.

As was noted above, to produce a more nuanced picture of the three selectors’/purchasers’ performance, the authors elected also to compare the performance of the Order Types within Topics; but, where the approval plans were concerned, there was very little nuance to be found. In table 4a, the librarians’ purchases had higher mean citation counts in 7 of the 10 Topic groups, and most of these differences showed the librarians’ books to have collected two, three, or four times as many citations. Likewise, the patrons outperformed the book vendors in 6 of the 10 Topic groups, with an additional two near ties. The sole Topic group where the approval plan selectors outperformed the other Order Types was “Military & Naval Sciences,” and even there the differences in performance were not statistically significant.

There may be some room for nuance where Librarians and PDA Patrons are concerned. The librarians had higher mean counts in half of the Topic groups, the patrons had higher mean counts in 4 of the 10 Topic groups, and the two had roughly equal performances in one group. Of the performance differences, each Order Type had just one that was statistically significantly different. Thus, of the three Order Types, the authors would suggest that firm-ordering librarians and ILL-PoD-requesting patrons were much more nearly comparable in their performances than were the book vendors and the librarians or the book vendors and the patrons, although, again, the librarians were clearly the superior selectors when the samples were looked at as a whole.

For both PDA’s critics and its advocates, the results presented by table 5a must have produced the most bafflement, in that they both confirmed and countered expectations. Of the five Patron Types, for example, Undergraduates performed the worst, just as PDA’s critics have contended they would. However, of the two Patron Types that both critics and advocates esteem, only one, Graduates, outperformed Undergraduates by a statistically significant amount. The Faculty decidedly did not. A review of the results for a Patron Types within Topics analysis performed by the authors but not reported on did suggest that Faculty books pretty consistently produced higher mean citation counts than did Undergraduates books in Topic groups where purchasing overlapped, but, again, the differences in Faculty and Undergraduates performance were largely statistically indistinguishable. Librarians critical of PDA have suggested to the lead author that this unexpected finding should be read as the result of undergraduates and faculty having purchased books with different qualities that both attract low citation counts. On the one hand, the former, in accordance with the anti-undergraduate and anti-PDA argument, obviously must have purchased books with low scholarly merit. On the other hand, the latter, in accordance with librarians’ generally pro-faculty attitude, must have purchased books of high scholarly merit, but these obviously were so advanced, abstruse, and/or esoteric as to attract less widespread attention. This may well be the case, but, without empirical support, this argument seems to the authors to be tainted by motivated reasoning.

And what should PDA’s critics and advocates make of the results for the university staff? This is a group of library patrons that has largely been ignored in the PDA/DDA literature, yet here they clearly outperformed the other three groups with identifiable patrons. The social sciences’ Staff purchases attracted roughly 5,200 more total citations than did Faculty’s, despite their having made 30 purchases to Faculty’s 103. In essence, the Staff books attracted 4.5 citations for every citation that the Faculty books attracted. Perhaps one ought to expand Walter’s argument against letting undergraduates add to the collection and make a point of including university staff in PDA/DDA purchasing.

How ought one to interpret this surprising result? Unfortunately, because of policies intended to protect patron privacy, the authors were unable to delve into just what was behind the outsized performance of the university staff, but we are able to offer some plausible speculation based upon our knowledge of local culture that may suggest that the staff’s performance may be read as a local effect. First, some university administrators at UNL may be listed in the patron database as members of the university staff rather than as members of the faculty. Second, it is not entirely uncommon at UNL for recently graduated graduate students, especially students in education and in a handful of the social sciences, to take various sorts of staff positions on campus. Once in these positions, some of them spend a few years conducting research and building their résumés prior to applying for faculty and administrative positions at UNL or elsewhere. Thus, it might be that some sizable percentage of the Staff performance could more accurately be characterized as having been produced by administrators with active research agendas and by lingering, highly motivated graduate students intent on improving their chances in a tough career market. Thus, if one’s institution has a culture similar to that of UNL, one may see a similar performance from one’s staff; if not, then likely not. Regardless, this study’s findings for Research Question #3 suggest that, in the social sciences at least, some of PDA’s critics’ assumptions concerning the intersection between academe’s natural hierarchy and the quality of what academic PDA patrons purchase may be inaccurate and misleading.

Where the sciences were concerned, the statistics presented in appendix B and the results of the first batch of statistical tests reported on in table 3 suggest that PDA’s critics’ opinions in favor of librarian title-by-title selection were, again, justified. Their concerns with PDA, however, do not appear to have been supported by the data analyzed here. Far and away, the concerns that were most clearly and substantially supported by the UNL Libraries’ sciences data were, once again, critics’ concerns over the motivations and/or performances of vendors. Both the librarians’ and the patrons’ science books had higher citation counts by statistically significant amounts than did the book jobbers’ approval plan selections. Librarians’ books garnered just over 103,000 more citations than did the plans’ books, and Patrons’ books garnered almost 78,000 more. Librarians’ books did outperform Patrons’ by more than 25,000 citations, but this difference, again, proved not to be statistically significant.

Thus, in reviewing the Order Types in general, it would also be difficult to argue that librarian selection in the sciences does not add value, and it would certainly be appropriate to see the results of this study as supporting the contention of several of the more cautionary voices in the PDA debate that librarian selection should not be abandoned in favor of universal PDA just yet.55 The results of this study do not, so far, support the contention of reactionary voices in the debate that have suggested PDA/DDA ought not to be employed in academic libraries. The citation counts of the PDA Patrons books were nearly as good as were Librarians, so there would seem to be little justification for excluding academic library patrons in the sciences from collection development or for closely monitoring and mediating their collecting, especially if the distribution of one’s library’s PDA requests tends to resemble the UNL Libraries’ skew in favor of Faculty and Graduates. With minimal and reasonable restrictions in place (again, see appendix A), the UNL Libraries’ patrons performed quite well.

The same, of course, cannot be said for the book vendors’ contributions to the collection via approval plans. The results of the table 3 tests for the sciences, as they did for the social sciences, suggest that critics of vendors may be more right than they know. Approval plan selectors for the UNL Libraries have not only proven themselves to be poor at selecting books that meet local needs,56 but they have, so far, proven themselves to be poor at identifying and selecting books that meet the interests of researchers generally, as well.

Again, to produce a more nuanced picture of the three Order Types’ performance, with Research Question #2 the authors also compared the sciences’ books’ performance by Order Type within Topic groups. Of the 27 head-to-head, within-Topic comparisons reported on, the 9 involving Librarians versus PDA Patrons showed no statistically significant differences in performance, even prior to the adjustments made to their p values to account for multiple comparisons. In fact, the preadjustment p values showed only a single difference in performance being nearly significant. Thus, these results support the conclusion that there is little to choose from where Librarian and PDA Patron selection in the sciences are concerned.

The same, once again, cannot be said for the approval plan selections. Half of the 18 post hoc comparisons showed Approval Plans as having been outperformed by either the Librarians or the Patrons by statistically significant amounts, even after adjustment for multiple comparisons. An additional three comparisons—against Patrons in Chemical Technology & Manufacturing and in Engineering & Technology (General) and against Librarians in Medicine (General, Public Health, & Pathology)—were also nearly significant. Thus, Approval Plans’ performance proved to be pretty consistently poor across the several Topics.

With respect to the criticisms and cautions raised and aired concerning which sorts of patrons ought or ought not to be allowed to add books to academic libraries’ collections, the results for the final research question would appear to offer some support for the contention that members of the faculty and graduate students can be relied upon as worthy selectors, but that undergraduate students and others might not be as reliable. In the set of tests reported on in table 5b, Faculty and Graduates citation counts were statistically indistinguishable, and there was also little to distinguish one from the other in additional Patron Types within-Topics post hoc tests performed later by the authors but not reported on. In the general tests of performance, both well outperformed Undergraduates, Staff, and the Unknown Patron Types, although none of the post hoc head-to-head comparisons’ results proved to be statistically significant. Still, Faculty and Graduates would seem a safer and more consistent bet in the sciences.

The humanities data—including the statistics in appendix B, the general tests of the Order Types presented in table 3, and the Order Types within-Topics tests presented in table 4c—offer very strong support for the PDA approach. For advocates of librarian title-by-title selection, this data set also offers some welcome, if not entirely unproblematic, support as well. Consistent with the results presented for the social sciences and for the sciences, the humanities’ results for the general testing of the three Order Types support both PDA and librarian title-by-title selection approaches to some extent and call into question the value of approval plans for meeting researchers’ needs. The humanities PDA books attracted tens of thousands more citations than did the librarians’ firm-ordered books and the approval plans’ selections, and they, unsurprisingly, strongly and statistically significantly outperformed both in table 3. As well, in the 22 Order Types within-Topics comparisons reported on in table 4c, the PDA books pretty consistently outperformed the other two Order Types, with 18 of the 22 PDA mean citation counts being higher than either the Librarians’ or Approval Plans’ mean counts and with 12 of the 22 post hoc comparisons in table 4c favoring PDA by statistically significant amounts. Thus, it would seem worthwhile to allow humanities patrons a fair amount of freedom to add books to an academic library collection.

Librarians, as was the case with the other disciplines, also outperformed Approval Plans by a statistically significant amount in table 3, although the humanities’ difference in performance was not as outsized as were those of the social sciences or the sciences. In table 4c, which presented the Order Types within-Topics comparisons, however, the Librarians versus Approval Plans relationship was shown to be much less clear. The within-Topics comparisons actually would appear to have been something of a draw. The two Order Types had higher means in roughly equal numbers of Topic categories, and the majority of the post hoc comparisons strongly support failing to reject the null hypothesis. In essence, table 4c suggests that the liaison librarians for languages and literatures were able to outperform the approval plans, but none of the other humanities librarians were able to distinguish themselves in any meaningful way.

Finally, for those critics who have taken a position against allowing undergraduates to add books to academic library collections via PDA, the UNL Libraries’ humanities data would offer little to no support. None of the known Patron Types performed better or worse than any of the others by statistically significant amounts, although there was some slight performance variation between types of patrons. Thus, the results reported here would support letting humanities students and faculty, and perhaps even university staff, to order what they like within the guidelines and constraints of the program.

Limitations to the Study

The first limitation to this study of which to be cognizant is one that affects much of the library literature generally and, as Walker noted, the PDA literature particularly: this was a single study conducted with data from a single site. This, of course, limits its generalizability, as it is impossible to determine whether or not it was merely characteristics of the setting that produced the effects presented. Thus, the results reported and conclusions drawn may not be universal, and studies of this sort should be conducted at other sites to reduce or eliminate the impact of local factors.

The issue of the study’s generalizability naturally leads to the second important limitation to the study: the issue of its robustness. Recently, the robustness of scientific findings in a myriad of fields has been called into question,57 and this has led to something of a crisis in validity and to a renewed interest in replication. Recently, researchers have had difficulty reproducing the results of highly cited papers in clinical research,58 the findings of classic studies in psychological science,59 the conclusions medical studies and drug trials,60 findings concerning structural brain-behavior correlations,61 and so forth. Reproducibility is a defining feature of science, the final arbiter of findings’ validity, and the cornerstone of cumulative science,62 and this should be no less the case in library science. The current study, as far as the authors are aware, has been the first of its kind. Without additional studies of this sort, it will be difficult, if not impossible, to determine how much the results of this study actually explain rather than merely describe, to determine whether the authors have observed systematic regularities, idiosyncrasies in the data, or mere products of chance.63 Thus, it would be incorrect to conclude that the issues raised by PDA’s critics and advocates have been firmly and entirely settled here.

The third important limitation to this study centers upon the type of PDA program studied. As has been noted, previous research on PDA has shown that Order Type has been a good predictor of circulation.64 At the UNL Libraries, ILL PoD purchasing has been associated with greater circulation than librarians’ orders and approval plans’ selections.65 In essence, that an academic library patron has a need, perceived a book to have the potential to meet that need, was willing to go through the UNL Libraries’ ILL process to request it, and was then willing to wait for a few days to receive the requested item has predicted that the requested item will circulate significantly more in the short term. Thus, ILL-PoD–derived findings may not hold for other sorts of PDA programs that have lower barriers to patron satisfaction, such as catalog-integrated DDA programs for e-books that allow one to merely click on a link to access a book.66 The ILL-PoD findings also may not hold for programs that merely encourage patrons to recommend books for purchase, such as mediated “suggest-a-book” programs that direct patron input to selector librarians for vetting.67

A fourth limitation to the study involves the research questions and the metric employed to answer them. As was noted above, this study only addresses scholarly merit via scholarly impact as measured by citation counts. As such, it offers the conclusion that Order Type may be a useful predictor of scholarly attention, at least in the short term. This study has nothing to say about the quality of the scholarship in the books under examination, about how lasting their influence will be in the long run, about how well received they were by scholars, or about any of the other, more qualitative conclusions one could come to about the books and their impact.

A fifth limitation to the study that the authors would like to note involves the samples employed. For the first research question, depending upon the discipline, the study had between 300 and 500 books per Order Type to analyze. These samples should be sufficiently large to draw and support some meaningful conclusions. As the study progressed to its second question, however, the authors had to “slice” the samples more finely. For the third question, the authors employed just the books that comprised the three PDA samples. Thus, the results for these latter questions should be treated a bit more cautiously, and the conclusions drawn should be treated as being less certain.

Finally, the reader should keep in mind that this study did not control for the disciplinary affiliations of the librarians and PDA patrons. The study’s samples were drawn and divided solely on the basis of Order Type and Topic. As a result, one cannot conclude that librarians or patrons with various disciplinary affiliations were or were not particularly good at selecting books of high interest in their areas of specialization. For example, the samples for the “Education” Topic contained books purchased by librarians with liaison assignments to departments in the College of Education and Human Sciences, but the samples also contained books purchased by librarians with assignments to other departments and programs, such as anthropology, sociology, ethnic studies, and so forth. In most Topics, pertinent disciplinary affiliations did account for the majority of purchases in most Topics, but one should still not conclude too much from this study about the value of narrow and specific disciplinary knowledge.

Conclusions

The conclusions to be drawn from this study would seem to be several. For critics who have warned against a too eager, uncritical, and comprehensive adoption of PDA as a collections panacea and for advocates of the value of librarians as selectors, this study has provided some much-needed empirical support. Librarians performed very well in the social sciences and in the sciences, as well as or better than did the PDA patrons and much better than did the approval plans’ selectors. In the humanities, of course, the librarians were substantially outperformed by the patrons, but they did still significantly outperform the approval plans in their turn. Thus, it would seem far too soon to worry over whether selection is dead, as a panel discussion at a recent Charleston Conference provocatively inquired. The results of this study would support the myriad panelists’ arguments that librarian selection is still valuable and vital.68

For PDA critics who have loudly opined against allowing patrons in general or against undergraduates in particular to add titles to academic libraries’ collections, this study provides some much-needed evidential discouragement. The PDA patrons performed well in all three disciplines, and the cases for the faculty and against the undergraduates were nowhere near as clear or strong as PDA’s critics would apparently have them. Graduate students were consistently good performers across all disciplines and within many Topics, and Faculty performed very well in the sciences, but the results of this study certainly do not support a blanket disparagement or lionization of any one type of patron. Certainly, concerned academic librarians could more closely monitor undergraduate students’ requests if doing so felt necessary, but if their institutions’ undergraduate students perform as well and make up as small a percentage of PDA purchases as do UNL’s, then the authors would be inclined to suggest that such heightened gatekeeping would largely be a waste of time.

Which of the final two conclusions to this study will prove the most pertinent will depend upon the nature of the reader’s library and the principles guiding its collecting, as well as where one’s library falls along what Nardini, in an early article on the history of the adoption of approval plans, posited as the depth and breadth versus precision divide.69 Some number of this journal’s readers are employed by large research libraries with budgets and mandates oriented more toward comprehensiveness. Many more, no doubt, are employed by academic libraries with constrained budgets whose collection-building focus is directed more toward meeting patrons’ current needs.

For the large research library whose intent is to collect more comprehensively and to build, as Anderson would have it, a cultural monument, this study should provide some reassurance that operating a PDA program will not precipitate the arrival of a “bibliopocalypse” that destroys the library, regardless of what some PDA critics appear to have been prophesying. The UNL PDA books in this study attracted good amounts of scholarly attention during the study interval, and nothing in this study would suggest that PDA polluted or debased the UNL Libraries’ collection with low-quality dross, beneath the interest of scholars. Past surveys of PDA programs have found that early adopters were devoting 1 to 5 percent of their book budgets to PDA.70 Given the results of this study and of past studies of PDA books’ circulation and collection suitability, it would be difficult to imagine that a large research library, however monumental its collection, could not profitably devote at least such a small percentage of its book budget to meeting its current patrons’ expressed needs.

For the more typical sort of academic library, the library hoping to direct limited collection dollars toward impactful titles, the results of this study suggest that the PDA critics’ cautions against vendors’ and publishers’ plans and packages could profitably be heeded. Nardini, in a recently published chapter, positioned approval plans as an important part of academic libraries’ selection infrastructure, one that provides libraries with numerous benefits and advantages, especially in the areas of efficiency and collection balance.71 From the results of this study, however, it would be difficult not to conclude that, as prices continue to increase and as collection dollars grow ever more dear, approval plans’ advantages may be outweighed by their failings. The results of this and past studies at the UNL Libraries incline the authors to wonder whether, in the librarian-vendor relationship, the shadowy outlines of a classic principal-agent problem may not be discerned, the sort of problem where one party believes the other to be acting to advance its interests when the second party is in fact acting to advance its own.72 Contrary to the rosy picture advanced by Nardini in the aforementioned chapter, much of the early criticisms of the approval plan model warned of just such potential conflicts of interest.73 Approval plans and e-book packages may well deliver the efficiencies and savings promised, but at UNL the approval plans appear to have been doing so with books that comparatively few researchers want. The not infrequent response in the field to this dilemma is to suggest that vendors’ products require monitoring and modification to function perfectly,74 but employing this approach at UNL actually produced worse performance on several occasions. If academic libraries with similarly constrained book budgets find themselves experiencing similarly unsatisfactory results with vendor plans or packages, they may wish to investigate the benefits of divesting.

To close: as was noted above, the library literature on PDA has so far been largely positive and supportive of the model, and a number of studies have substantiated the claims of its advocates, especially where circulation/usage and collection suitability have been concerned. The counter literature skeptical of or cautioning against PDA, to this point, has warned of the dangers and shortcomings of the model almost entirely speculatively. The authors hope that this study has contributed to a continued, evidence-driven, and empirically supported conversation on the PDA model and furthered the examination of its benefits and pitfalls for academic libraries. Effective and efficient materials selection is a difficult process, and academic librarians engaged in collection building often find themselves striving to satisfy multiple and sometimes contradictory masters. Concerning the struggle to purchase books that meet current patrons’ needs and the simultaneous struggle to purchase books of high and sustained scholarly interest, this study suggests that PDA’s advocates may be right: Giving the patrons what they want today may very well be supplying the collection with what it needs for tomorrow.

APPENDIX A. UNL University Libraries’ ILL Purchasing Project Criteria (2010–2011)

All books requested via ILLiad, published between 2007 and 2010 and not owned by UNL Libraries must be considered for purchase (books in IRIS that are counted as missing, lost, or on search cannot be purchased). Selected books will be judged by the following criteria. Books must be available from Barnes and Noble online and must arrive in a timely manner. We cannot order any book that takes longer than 2–3 days to ship. We also cannot order books that are marked as “Pre-orders.” The price limit on purchasing project books is $175; we may not order books that cost more than this amount. Books purchased have academic merit. The following type of books may not be purchased:

• Textbooks (any book designated by OCLC or Barnes & Noble as a textbook)
• Foreign language books
• Fiction, poetry, plays
• Music scores
• Lab manuals, workbooks, field guides
• Solutions manuals
• Popular interest (nonfiction, best sellers, self-help, and the like)
• Popular biographies (biographies can be decided on a case-by-case basis)
• Journal volumes/serials
• Computer books
• Anthologies
• Older editions of a book that still fall within the 3-year period

Most books not marked as textbooks published by university presses are appropriate for the ILL purchasing program. Another way to judge a book that may be published by an unknown publisher but is not disqualified by any of the above criteria is to look in the OCLC holdings to see what other libraries own this item. If it is owned by many GWLA and ARL libraries, it should be purchased. If it is owned by mostly public libraries, it should be obtained via interlibrary loan. If it is a new book not owned by many libraries or you think it would be a good addition to the collection, decide which subject area the book falls under and consult the library liaison. Ask if they feel the book is a worthwhile purchase. If they feel that it is, you may buy it. If the liaison is not available or the book falls under the prohibited categories but still seems like a good addition to the library collection, ask [the Chair of Technical Services] for permission to purchase it.

APPENDIX B. Sample Distributions and Parameters

Notes

1. Jennifer Perdue and James A. Van Fleet, “Borrow or Buy? Cost-effective Delivery of Monographs,” Journal of Interlibrary Loan, Document Delivery & Information Supply 9, no. 4 (1999): 19–28.

2. Association of College and Research Libraries, Research, Planning and Review Committee, “2010 Top Ten Trends in Academic Libraries: A Review of the Current Literature,” College & Research Libraries News 71, no. 6 (2010): 286–92; Association of College and Research Libraries, Research, Planning and Review Committee, “2012 Top Ten Trends in Academic Libraries: A Review of the Trends and Issues Affecting Academic Libraries in Higher Education,” College & Research Libraries News 73, no. 6 (2012): 311–20.

3. Cristina Caminita, “E-Books and Patron-Driven Acquisitions in Academic Libraries,” in Customer-based Collection Development: An Overview, ed. Karl Bridges (Chicago: ALA Editions, 2014), 1–12; Karin J. Fulton, “The Rise of Patron-driven Acquisitions: A Literature Review,” Georgia Library Quarterly 51, no. 3 (2014): Article 10, available online at http://digitalcommons.kennesaw.edu/glq/vol51/iss3/10 [accessed 28 November 2017]; Denise A. Garofalo, “Tips from the Trenches,” Journal of Electronic Resources Librarianship 23, no. 3 (2011): 274–76; Rick Lugg, “Collecting for the Moment: Patron-driven Acquisitions as a Disruptive Technology,” in Patron-driven Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 7–22; Norm Medeiros, “King of the Hill,” OCLC Systems & Services 28, no. 2 (2012), 64–66.

4. Kizer Walker, “Patron-Driven Acquisition in US Academic Research Libraries: At the Tipping Point in 2011?” Bibliothek Forschung und Praxis 36, no. 1 (2012): 126–30.

5. Deborah Lenares and Emilie Delquié, “Give the People What They Want: Patron Driven Acquisition: Results and Reflections on a Survey Completed by Publishers Communication Group” (presentation, STM Spring Conference, Le Meridien Cambridge-MIT, Cambridge, MA, April 21–29, 2010).

6. Nestor L. Osorio, “A User’s Participatory Selecting Model: Librarians Point of Views” (presentation, Illinois Library Association Annual Meeting, Rosemont, IL, October 19, 2011), available online at http://hdl.handle.net/10760/16507 [accessed 26 January 2018].

7. Joseph Esposito, Kizer Walker, and Terry Ehling, “PDA and the University Press: A Report Prepared for The Andrew W. Mellon Foundation,” Journal of Scholarly Publishing 44, no. 3 (2013): s1–s62.

8. Steven Carrico, Michelle Leonard, and Erin Gallagher, Implementing and Assessing Use-Driven Acquisitions (Lanham, MD: Rowman & Littlefield, 2016); Tony Horava and Michael Levine-Clark, “Current Trends in Collection Development Practices and Policies,” Collection Building 35, no. 4 (2016): 97–102.

9. Caminita, “E-Books and Patron-Driven Acquisitions in Academic Libraries,” 1–12.

10. Google, Google Scholar, n.d., s.v. “college, libraries, patron, driven, acquisitions,” available online at http://scholar.google.com/ [accessed 31 October 2017].

11. Dee Ann K. Allison, The Patron-driven Library: A Practical Guide for Managing Collections and Services in the Digital Age (Oxford: Chandos Publishing, 2013); Rick Anderson, “Collections 2021: The Future of the Library Collection Is Not a Collection,” Serials 24, no. 3 (2011): 211–15; Faye A. Chadwell, “What’s Next for Collection Management and Managers? User-centered Collection Management,” Collection Management 34, no. 2 (2009): 69–78; Joseph Esposito, Kizer Walker, and Terry Ehling. “The New Supply Chain and Its Implications for Books in Libraries,” Educause Review 47, no. 5 (2012): 58–59; Esposito, Walker, and Ehling, “PDA and the University Press,” s1–s62; Fulton, “The Rise of Patron-driven Acquisitions,” Article 10; Edward A. Goedeken and Karen Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” College & Research Libraries 76, no. 2 (2015): 205–21; Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 126–30.

12. John Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions: A Summary,” in Customer-based Collection Development: An Overview, ed. Karl Bridges (Chicago: ALA Editions, 2014): 159–76; Barbara Fister, “Problematizing Patron-driven Acquisitions,” Library Journal: Peer to Peer Review (blog, November 11, 2010); Barbara Fister, “Puzzled by Patron-driven Acquisitions,” Library Babel Fish (blog, November 11, 2010), available online at https://www.insidehighered.com/blogs/library-babel-fish/puzzled-patron-driven-acquisitions [accessed 12 April 2019]; Mary Gilbertson, Elizabeth Chadbourn McKee, and Lutishoor Salisbury, “Just in Case or Just in Time? Outcomes of a 15-month Patron-driven Acquisition of e-Books at the University of Arkansas Libraries,” Library Collections, Acquisitions & Technical Services 38, no. 1/2 (2014): 10–20; William Miller, “Patron-driven Acquisitions (PDA): The New Wave in Book Acquisitions Is Coming,” Library Issues: Briefings for Faculty and Administrators 31, no. 5 (2011): 1–4; Scott A. Smith, “PDA: Driving Off the Cliff or, New Wine in Old Bottles,” Against the Grain 23, no. 3 (2011): 40; Suzanne M. Ward, Guide to Implementing and Managing Patron-Driven Acquisitions (Chicago: American Library Association, 2012).

13. Esposito et al., “The New Supply Chain and Its Implications for Books in Libraries,” 58–59; Esposito et al., “PDA and the University Press,” s1–s62; Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21; Lugg, “Collecting for the Moment,” 7–22.

14. Kathleen A. Lehman, “Collection Development and Management,” Library Resources & Technical Services 58, no. 3 (2014): 169–77; Miller, “Patron-driven Acquisitions (PDA),” 1–4; Jane Schmidt, “Demand-driven Acquisitions: The Hegemony of the Canon Interrupted,” in Creating Sustainable Community: The Proceedings of the ACRL 2015 Conference (Portland, Oregon, March 25–28, 2015), 168-73, available online at www.ala.org/acrl/sites/ala.org.acrl/files/content/conferences/confsandpreconfs/2015/Schmidt.pdf [accessed15 November 2017]; Elena S. Smith, “Power and Practice in Academic Library Materials Selection Paradigms,” SLIS Student Research Journal 1, no. 2 (2011): Article 6, available online at http://scholarworks.sjsu.edu/slissrj/vol1/iss2/6/ [accessed 1 May 2012]; William H. Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” Library Resources & Technical Services 56, no. 3 (2012): 199–213.

15. Medeiros, “King of the Hill,” 64–66.

16. Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 126–30.

17. Carrico et al., Implementing and Assessing Use-Driven Acquisitions, xiii.

18. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Esposito et al., “PDA and the University Press,” s1–s62; Gilbertson et al., “Just in Case or Just in Time?” 10–20; Smith, “PDA: Driving Off the Cliff or, New Wine in Old Bottles,” 40; Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 126–30.

19. David C. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials: A Chronological Review and Summary of Findings,” Library Philosophy and Practice (2011): Article 635, available online at http://digitalcommons.unl.edu/libphilprac/635/ [accessed 26 January 2018]; David C. Tyler, Joyce C. Melvin, MaryLou Epp, and Anita M. Kreps, “Patron-driven Acquisition and Monopolistic Use: Are Patrons at Academic Libraries Using Library Funds to Effectively Build Private Collections?” Library Philosophy and Practice (2014) 11, Article 1,149, available online at http://digitalcommons.unl.edu/libphilprac/1149/ [accessed 26 January 2018].

20. Walker, “Patron-Driven Acquisition in US Academic Research Libraries,” 129.

21. Jason C. Dewland and Andrew See, “Notes on Operations: Patron Driven Acquisitions: Determining the Metrics for Success,” Library Resources & Technical Services 59, no. 1 (2015): 13–23.

22. Carrico et al., Implementing and Assessing Use-Driven Acquisitions, 95.

23. Laura Costello, Evaluating Demand-driven Acquisitions (Cambridge, UK: Chandos Publishing, 2016); Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 635; Tyler et al. “Patron-driven Acquisition and Monopolistic Use,” Article 1,149.

24. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213; Aaron Wood, “Mainstream Patron-Driven Acquisition: Topicality Over the Scholarly Record… and the Cello Suites,” Against the Grain 25, no. 5 (2013): 22.

25. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21.

26. Costello, Evaluating Demand-driven Acquisitions, ix.

27. Joseph Esposito, “A Dialogue on Patron-Driven Acquisitions,” The Scholarly Kitchen (blog, January 3, 2012), available online at https://scholarlykitchen.sspnet.org/2012/01/03/a-dialogue-on-patron-driven-acquisitions/ [accessed 15 November 2017].

28. Sheila Corrall, “Community Engagement in Collection Development: Social Responsibility or Professional Abdication?” (presentation, Community Engagement & Social Responsibility: ALISE’17 Annual Conference, Atlanta, GA, January 17–20, 2017).

29. James A. Cogswell, “The Organization of Collection Management Functions in Academic Research Libraries,” Journal of Academic Librarianship 13, no. 5 (1987): 270.

30. Rick Anderson, “What Patron-Driven Acquisition (PDA) Does and Doesn’t Mean: An FAQ,” The Scholarly Kitchen (blog, May 31, 2011), available online at https://scholarlykitchen.sspnet.org/2011/05/31/what-patron-driven-acquisition-pda-does-and-doesnt-mean-an-faq/ [accessed 15 November 2017]; Anderson, “Collections 2021,” 211–15.

31. Anderson, “Collections 2021,” 211–15; Lorcan Dempsey, “Library Collections in the Life of the User: Two Directions,” Liber Quarterly 26, no. 4 (2017): 338–59; Julie Linden, Sarah Tudesco, and Daniel Dollar, “Collections as a Service: A Research Library’s Perspective,” College & Research Libraries 79, no. 1 (2018): 86–99; Doug Way, “Transforming Monograph Collections with a Model of Collections as a Service,” portal: Libraries and the Academy 17, no. 2 (2017): 283–94.

32. Miller, “Patron-driven Acquisitions (PDA),” 1–4; Schmidt, “Demand-driven Acquisitions: The Hegemony of the Canon Interrupted,” 168–73; Smith, “Power and Practice in Academic Library Materials Selection Paradigms,” Article 6; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213.

33. Douglas Jones, “On-demand Information Delivery: Integration of Patron-driven Acquisition into a Comprehensive Information Delivery System,” Journal of Library Administration 51, no. 7/8 (2011): 764–76; Michael Levine-Clark, “Developing a Multiformat Demand-driven Acquisition Model,” Collection Management 35, no. 3/4 (2010): 201–07; Michael Levine-Clark, “Developing a Model for Long-term Management of Demand-driven Acquisitions,” Against the Grain 23, no. 3 (2011): 24–26; Michael Levine-Clark, “Building a Demand-Driven Collection: The University of Denver Experience,” in Patron-driven Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 45–60; Peter Spitzform, “Patron-driven Acquisition: Collecting as if Money and Space Mean Something,” Against the Grain 23, no. 3 (2014): 20–24; Peter Spitzform and Pongracz Sennyey, “A Vision for the Future of Academic Library Collections,” International Journal of the Book 4, no. 4 (2007): 185–89.

34. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Fister, “Problematizing Patron-driven Acquisitions” (blog); Fister, “Puzzled by Patron-driven Acquisitions” (blog); Jean-Mark Sens and Anthony J. Fonseca, “A Skeptic’s View of Patron-driven Acquisitions: Is It Time to Ask the Tough Questions?” Technical Services Quarterly 30, no. 4 (2013): 359–71.

35. Buschman, “Seven Reasons to Be Skeptical about Patron-Driven Acquisitions,” 159–76; Chadwell, “What’s Next for Collection Management and Managers?” 69–78; Candice Dahl, “Primed for Patron-driven Acquisition: A Look at the Big Picture,” Journal of Electronic Resources Librarianship 24, no. 2 (2012): 119–26; Lindsey Reno, “Out of the Shadows: A Public Face for Acquisitions in Academic Libraries,” Against the Grain 24, no. 6 (2012): 22–26; F.K. Rottmann, “To Buy or to Borrow: Studies of the Impact of Interlibrary Loan on Collection Development in the Academic Library,” Journal of Interlibrary Loan & Information Supply 1, no. 3 (1991), 17–27; Sens and Fonseca, “A Skeptic’s View of Patron-driven Acquisitions,” 359–71; Smith, “PDA: Driving off the Cliff or, New Wine in Old Bottles,” 40; Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213; Ward, Guide to Implementing and Managing Patron-Driven Acquisitions; Aaron Wood, “Mainstream Patron-Driven Acquisition,” 22.

36. Audrey Fenner, “The Approval Plan: Selection Aid, Selection Substitute,” The Acquisitions Librarian 16, no. 31/32 (2004): 227–40; Sherry E. Gelbwasser, “Vendor-Library Relationships in ARL Libraries,” The Acquisitions Librarian 8, no. 16 (1996): 109–17; Alison Griffin and Sarah Forzetting, “Behind the Scenes: Approval Profiles from a Vendor’s Perspective,” in Library Collection Development for Professional Programs: Trends and Best Practices, ed. Sarah Holder (Hershey, PA: IGI Global, 2012), 16–31; Bob Nardini, “Approval Plans and Patron Selection: Two Infrastructures,” in Patron-driven Acquisitions: History and Best Practices, ed. David A. Swords (Berlin and Boston: Walter de Gruyter, 2011), 23–43; Robert F. Nardini, “Approval Plans: Politics and Performance,” College & Research Libraries 54, no. 5 (1993): 417–26.

37. Wenli Gao, Irene Ke, and Cherie Turner, “Programming Plus Subject Expertise: A Combined Approach for Approval Profile Modification,” Collection Management (forthcoming); Irene Ke, Wenli Gao, and Jackie Bronicki, “Does Title-By-Title Selection Make a Difference? A Usage Analysis on Print Monograph Purchasing,” Collection Management 42, no. 1 (2017): 34–47; David C. Tyler, Christina Falci, Joyce C. Melvin, MaryLou Epp, and Anita M. Kreps, “Patron-driven Acquisition and Circulation at an Academic Library: Interaction Effects and Circulation Performance of Print Books Acquired via Librarians’ Orders, Approval Plans, and Patrons’ Interlibrary Loan Requests,” Collection Management 38, no. 1 (2013): 3–32; David C. Tyler, Yang Xu, Joyce C. Melvin, Marylou Epp, and Anita M. Kreps, “Just How Right Are the Customers? An Analysis of the Relative Performance of Patron-Initiated Interlibrary Loan Monograph Purchases,” Collection Management 35, no. 3/4 (2010): 162–79.

38. David C. Tyler, Joyce C. Melvin, Yang Xu, Marylou Epp, and Anita M. Kreps, “Effective Selectors? Interlibrary Loan Patrons as Monograph Purchasers: A Comparative Examination of Price and Circulation-Related Performance,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 21, no. 1/2 (2011): 57–90.

39. Denise Brush, “Circulation Analysis of an Engineering Monograph Approval Plan,” Collection Building 26, no. 2 (2007): 59–62; Erin L. Ellis, Nikhat J. Ghouse, Monica Claassen-Wilson, John M. Stratton, and Susanne K. Clement, “Comparing Approval and Librarian-Selected Monographs: An Analysis of Use,” in Library Data: Empowering Practice and Persuasion, ed. Darby Orcutt (Santa Barbara, CA: Libraries Unlimited, 2010), 53–68; Marcie Kingsley, “Circulation Statistics for Measuring Approval Plan Effectiveness,” Against the Grain 8, no. 4 (1996): 1, 16–17; James Cory Tucker, “Collection Assessment of Monograph Purchases at the University of Nevada, Las Vegas Libraries,” Collection Management 34, no. 3 (2009): 157–81; Michele Wolff, “Evaluating the Art Approval Plan: A Case Study,” Art Documentation 18, no. 1 (1999): 5–9.

40. John S. Brantley, “Approval Plans, Discipline Change, and the Importance of Human Mediated Book Selection,” Library Collections, Acquisitions, and Technical Services 34, no. 1 (2010): 11–24; Mary Eldredge, “Major Issues in Approval Plans: The Case for Active Management,” The Acquisitions Librarian 8, no. 16 (1996): 51–59.

41. Kady Ferris and Tina Herman Buck, “An Ethos of Access: How a Small Academic Library Transformed Its Collection-building Processes,” Collection Management 39, no. 2/3 (2014): 127–44; Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21.

42. Costello, Evaluating Demand-driven Acquisitions.

43. Esposito, Walker, and Ehling, “PDA and the University Press,” s1–s62.

44. Tyler et al., “Patron-driven Acquisition and Monopolistic Use,” 11.

45. Jason Price and John McDonald, “Beguiled by Bananas: A Retrospective Study of the Usage & Breadth of Patron vs. Librarian Acquired eBook Collections,” in Proceedings of the Charleston Library Conference (Charleston, South Carolina, 2009), doi:10.5703/1288284314741; Gabrielle Wiersma and Yem Fong, “Patron-driven e-Book Solutions: Moving beyond the Banana Books Incident,” in Proceedings of the Charleston Library Conference (Charleston, South Carolina, 2011), doi:10.5703/1288284314973.

46. Goedeken and Lawson, “The Past, Present, and Future of Demand-driven Acquisitions in Academic Libraries,” 205–21.

47. Leslie J. Reynolds, Carmelita Pickett, Wyoma Vanduinkerken, Jane Smith, Jeanne Harrell, and Sandra Tucker, “User-driven Acquisitions: Allowing Patron Requests to Drive Collection Development in an Academic Library,” Collection Management 35, no. 3/4 (2010): 244–54.

48. Jeffrey H. Waller, “Undergrads as Selectors: Assessing Patron-driven Acquisition at a Liberal Arts College,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 23, no. 3 (2013): 127–48.

49. Gilbertson et al., “Just in Case or Just in Time?” 10–20; Doug Way, “The Assessment of Patron-initiated Collection Development via Interlibrary Loan at a Comprehensive University,” Journal of Interlibrary Loan, Document Delivery & Electronic Reserve 19, no. 4 (2009): 299–308.

50. Waller, “Undergrads as Selectors,” 127–48.

51.. Judith M. Nixon and E. Stewart Saunders, “A Study of Circulation Statistics of Books on Demand: A Decade of Patron-Driven Collection Development, Part 3,” Collection Management 35, no. 3/4 (2010): 151–61.

52. Walters, “Patron-driven Acquisition and the Educational Mission of the Academic Library,” 199–213.

53. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 635.

54. Emily K. Chan and Susan L. Kendall, “Identifying Users of Demand-driven E-book Programs: Applications for Collection Development,” in Imagine, Innovate, Inspire : Proceedings of the ACRL 2013 Conference (Chicago: Association of College and Research Libraries, 2013), 301–08.

55. Rick Anderson, “Is Selection Dead? The Rise of Collection Management and the Twilight of Selection,” Against the Grain 23, no. 2 (2011): 40–44.

56. Tyler et al., “Patron-driven Acquisition and Circulation at an Academic Library,” 3–32; Tyler et al., “Just How Right Are the Customers?” 162–79.

57. John P.A. Ioannidis, “Why Most Published Research Findings Are False,” PLoS Medicine 2, no. 8 (2005): e124, available online at http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124 [accessed 23 April 2018].

58. John P.A. Ioannidis, “Contradicted and Initially Stronger Effects in Highly Cited Clinical Research,” JAMA 294, no. 2 (2005): 218–28.

59. Open Science Collaboration, “Estimating the Reproducibility of Psychological Science,” Science 349, no. 6251 (2015): aac4716; Harold Pashler and Eric-Jan Wagenmakers, “Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?” Perspectives on Psychological Science 7, no. 6 (2012): 528–30.

60. C. Glenn Begley and Lee M. Ellis, “Drug Development: Raise Standards for Preclinical Cancer Research.” Nature 483, no. 7391 (2012): 531–33; Florian Prinz, Thomas Schlange, and Khusru Asadullah, “Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets?” Nature Reviews Drug Discovery 10, no. 9 (2011): 712.

61. Wouter Boekel, Eric-Jan Wagenmakers, Luam Belay, Josine Verhagen, Scott Brown, and Birte U. Forstmann, “A Purely Confirmatory Replication Study of Structural Brain-Behavior Correlations,” Cortex 66 (2015): 115–33.

62. Jennifer Crocker and M. Lynne Cooper, “Addressing Scientific Fraud,” Science 334, no. 6060 (2011): 1182; Jeremy Freese, “Replication Standards for Quantitative Social Science: Why Not Sociology?” Sociological Methods & Research 36, no. 2 (2007): 153–72; Gary King, “Replication, Replication,” PS: Political Science & Politics 28, no. 3 (1995): 444–52; Barbara R. Jasny, Gilbert Chin, Lisa Chong, and Sacha Vignieri, “Again, and Again, and Again…,” Science 334, no. 6060 (2011): 1225; Matthew C. Makel and Jonathan A. Plucker, “Facts Are More Important Than Novelty: Replication in the Education Sciences,” Educational Researcher 43, no. 6 (2014): 304–16; Stefan Schmidt, “Shall We Really Do It Again? The Powerful Concept of Replication Is Neglected in the Social Sciences,” Review of General Psychology 13, no. 2 (2009): 90–100.

63. Nassim Nicholas Taleb, Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (New York: Random House, 2005).

64. Tyler, “Patron-driven Purchase on Demand Programs for Printed Books and Similar Materials,” Article 635.

65. Tyler et al., “Patron-driven Acquisition and Circulation at an Academic Library,” 3–32; Tyler et al., “Just How Right Are the Customers?” 162–79.

66. Costello, Evaluating Demand-driven Acquisitions.

67. Rebecca Schroeder, “When Patrons Call the Shots: Patron-driven Acquisition at Brigham Young University,” Collection Building 31, no. 1 (2012): 11–14.

68. Anderson, “Is Selection Dead?” 40–44.

69. Nardini, “Approval Plans: Politics and Performance,” 420.

70. Lenares and Delquié, “Give the People What They Want.”

71. Nardini, “Approval Plans and Patron Selection,” 23–43.

72. Michael C. Jensen and William H. Meckling, “Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure,” Journal of Financial Economics 3, no. 4 (1976): 305–60.

73. Association of Research Libraries, Approval Plans in ARL Libraries, SPEC Kit, no. 83 (Washington, DC: ARL, 1982), 79–84; Margaret Dobbyn, “Approval Plan Purchasing in Perspective,” in Background Readings in Building Library Collections, 2nd ed., eds. Phyllis Van Orden and Edith B. Phillips (Metuchen, NJ: Scarecrow Press, 1979): 194; Jane Maddox, “Approval Plans—Viable?” Journal of Academic Librarianship 1 (1976): 22; Rose Mary Magrill and Mona East, “Collection Development in Large University Libraries,” in Advances in Librarianship, vol. 8, ed. Michael H. Harris (San Diego, CA: Academic Press, 1978): 6; Norman D. Stevens, “The Resources and Technical Services Division of ALA,” Library Resources & Technical Services 13 (1969): 334.

74. Leslie Czechowski, “Edging toward Perfection: Analysis of a New Approval Plan in a Health Sciences Library,” Library Collections, Acquisitions, & Technical Services 32 (2008): 107–11; Eldredge, “Major Issues in Approval Plans,” 51–59.