The Impact of Academic Library Resources on Undergraduates’ Degree Completion

Krista M. Soria, Jan Fransen, and Shane Nackerud*

The purpose of this study was to examine the impact of first-year undergraduates’ (n = 5,368) use of academic library resources in their first year on their degree completion or continued enrollment after four years of study. Propensity score matching techniques were used to construct treatment (library users) and control (library nonusers) groups with similar background characteristics and college experiences. The results suggest that using the library at least one time in the first year of enrollment significantly increased the odds that students would graduate in four years or remain enrolled after four years as opposed to withdrawing from the university. First-year students who used electronic resources and books also had significantly improved odds of graduation over withdrawing, while students who used electronic books and took a library instruction course had significantly improved odds of remaining enrolled over withdrawing.

Over the last decade, calls to explore the value of academic libraries in supporting college students’ success have yielded several promising research studies that have investigated relationships between students’ use of academic libraries, retention,1 and academic achievement.2 Given the prioritization of retention and graduation rates among colleges and universities,3 in addition to mounting public pressures for institutions to encourage students’ success, such library impact research studies continue to have advancing relevance in the field of academic libraries research and assessment and in higher education as a whole.

While researchers exploring the associations between academic library use and students’ success have used increasingly more robust data sets with large sample sizes, improved their research methodologies and data analyses, and developed progressively nuanced research designs, there remains a series of unanswered questions yet to be addressed. For instance, while graduation rates are important indicators of the success of higher education institutions, there are, at present, few research studies that have explored whether academic libraries and their various components may contribute to students’ long-term success over the course of their enrollment. Stone and Ramsden4 examined library data across multiple institutions and found significant, positive relationships between library resource use and students’ level of academic degree attainment; however, as the authors cautioned, the results are limited because the researchers did not measure students’ use of library resources at an individual level and only measured students’ level of degree attainment (that is to say, first-class or ordinary/third-class degrees). As a consequence of those limitations, Stone and Ramsden were not able to demonstrate a direct causal relationship between students’ use of academic libraries and their graduation rates.

Previous studies have hinted at the potential for libraries to impact students’ longitudinal educational trajectories all the way to the point of graduation. In some studies, researchers have not directly explored students’ use of academic library resources but have instead investigated relationships between students’ retention and broader factors, such as library expenses or expenditures;5 book expenditures, acquisitions, collection sizes, and circulation;6 ratio of library professional staff to students;7 and professional library staff salaries and number of professional library staff.8

In other studies, researchers have examined libraries data at the individual student use level. For instance, the research of Soria et al.9 suggested that students who used academic libraries in their first semester of enrollment at least once were significantly more likely than their peers who did not use the library to persist to their second semester of enrollment. Continuing this line of research, the newest study of Soria et al.10 found that first-year students who use the library at least once in their first year of enrollment were significantly more likely to be retained to their second year of enrollment than their peers who did not use the library. The research designs of Soria et al. are more robust than other studies examining bivariate correlations between students’ use of libraries and success outcomes because they take into consideration factors that might also bear weight on student outcomes (such as demographic variables). Yet, according to Chiteng Kot and Jones,11 a “consistent shortcoming” in those studies is that they fail to “take into consideration the fact that a variety of factors may contribute to students’ decisions to use library resources and that these factors may be, in turn, related to the student outcomes of interest.”

To address the limitation described by Chiteng Kot and Jones,12 the purpose of this study is to explore the impact of library resources on college students’ longitudinal outcomes as related to their graduation or continued enrollment after four years of study. We used quasi-experimental procedures (propensity score matching techniques) to construct control (nonlibrary users) and treatment (library users) groups similar to those found in randomized experiments. These steps were used to reduce the potential bias found within students’ self-selection as they decide to use specific library resources. In this study, we sought to explore the average treatment effect (ATE) to examine differences in graduation and continued enrollment between students who used academic library resources (the “treatment” group) to those who did not use academic library resources (the “control” group).

Conceptual Framework

Like previous studies examining the effects of students’ use of academic libraries on academic outcomes, in this study we used the input-environment-output model of Astin13 as our conceptual framework. The inputs or covariates examined within this model include students’ precollege characteristics, experiences, and demographics. The environment includes experiences during higher education (in the present study, using the library, among other indicators). The outputs in the present study include college students’ graduation or continued enrollment after four years of study. Inputs can have an effect on both environmental experiences and outcomes, which is why researchers commonly take inputs into consideration when building their statistical models. Indeed, to test the true impacts of environmental experiences, it is important to consider the direct effects of input variables on outcomes while also examining the potential effects those input variables have on the environmental variables. In the case of students’ use of academic libraries, for instance, self-selection bias may contribute to systematic differences between students who decide to use library resources and those who do not use library resources.

To attempt, in part, to reduce some of those self-selection biases, educational researchers are more frequently using quasi-experimental designs in their analyses. In most experimental studies, researchers randomly assign participants to a control group or a treatment group to better test the effects of the treatment. While these randomized controlled trials are considered the “gold standard approach” for estimating the effects of treatments,14 such randomization is not often possible in educational settings; it would be unethical, for instance, to deny the use of academic library resources to students who may be randomly assigned to a control condition. The rationale behind quasi-experimental techniques is to simulate the characteristics of experimental designs by matching groups of students based upon demographic variables or other observable characteristics such that the only differences between these students is the type of “treatment” received (such as using academic libraries).15 Using quasi-experimental design methods, researchers match students based upon pretreatment (such as precollege) characteristics that approximate randomization by balancing the observable characteristics between the treatment and control groups.16 The resultant effects of such steps can help researchers to better estimate the effects of “treatments” on outcomes with a greater degree of accuracy. Therefore, in the present study, we used propensity score matching techniques to estimate the effects of first-year students’ use of academic libraries in their first year of enrollment with their four-year graduation and continued enrollment rates.

Methodology

Sample

The sample used in this study is drawn from the 2011 entering class of first-year undergraduates at a large, public university located in the Midwest (N = 5,368). The sample included slightly more females (n = 2,803, 52.2%) than males (n = 2,565, 47.8%). Within the sample, 5.8 percent were international students (n = 310); 2.6 percent, Hispanic (n = 169); 1.1 percent, American Indian or Native American (n = 61); 10.7 percent, Asian (n = 574); 3.8 percent, Black (n = 204); 0.3 percent, Hawaiian (n = 18); 74.8 percent, White (n = 4,013); and 0.4 percent (n = 19) had a nonspecified race/ethnicity.

Our use of first-year students only in this study was intentional because we sought to take steps to eliminate potential confounding biases that may impact students’ success to better isolate the effects of library use. For example, if we had studied juniors’ or seniors’ use of academic libraries, students’ graduation could be attributed to their ongoing persistence instead of their use of libraries—those students would only have one or two years to finish their degrees, so they might decide to stay and complete their education as opposed to transferring or withdrawing given that they had completed so many credits already. Furthermore, these students might have already engaged in any number of collegiate experiences associated with higher graduation rates, so narrowing our sample to first-year college students reduced the chance that confounding variables would interfere with our ability to more accurately judge the impact of academic libraries use.

Measures

Covariate measures. The covariate or input measures used for propensity score matching analyses were intentionally selected because of their known relationships to students’ use of academic library resources. These measures included students’ race/ethnicity,17 sex,18 on-campus residency,19 first-generation status,20 participation in freshman seminar-like programs,21 students’ college of enrollment,22 AP credits earned,23 socioeconomic status as measured by Pell grant,24 and incoming ACT/SAT scores.25

All indicators were derived from institutional records. When ACT scores were missing, SAT scores were converted to ACT scores according to ACT concordance tables. We included two measures related to students’ participation in specialized seminars: Access to Success, a small advising community aimed to increase the retention of students from underrepresented backgrounds,26 and elective freshman seminars taught by faculty in a wide variety of disciplines. Students’ college of enrollment was dummy-coded with the liberal arts college (which admits the greatest proportion of first-year students) serving as a common referent. Frequencies and mean values for the entire sample (save for the variables described in the sample section) are shown in table 1.

Environmental measures: Students’ use of academic libraries. The environmental or “treatment” variables included students’ use of academic libraries at least once during the academic year, in addition to at least one use in five primary areas: loaning books (including interlibrary loans and electronic books), using electronic resources (including academic journals accessed, website used, and databases searched), logging into library computer workstations, enrolling in library instruction courses (introduction courses, workshops, or instruction integrated into courses), and reference resources used (such as meeting with a peer research consultant or chatting with a reference librarian). We collected all of these data points from existing records on students’ use of a variety of library services. We coded students’ use of libraries in these areas to reflect whether they had ever once used the library in this area (0 = never used, 1 = used at least one time). The results suggest that 82.2 percent of students used the libraries at least once, students were most likely to have used electronic resources (78.3%), and students were least likely to have met with a peer research consultant or chatted online with a reference librarian (5.5%).

We extracted collection loans—primarily book checkouts and renewals—from the University’s Ex Libris Aleph27 catalog transaction records, and no distinctions were made between initial checkout and renewal. E-book views, database logins, and electronic journal logins used a “click-through” script to capture usage of these resources. Computer workstation use at the libraries required that users log in through a shared computer management software service called Cybrarian™.28 Login data included Internet ID and date of transaction, which we extracted from the Cybrarian database. We gathered reference transactions that occurred via live Internet chat from QuestionPoint™,29 (“QuestionPoint [Computer Software]” 2013) and we parsed the data into a list of Internet IDs. Students scheduled appointments with peer research consultants, which made it possible to harvest Internet IDs from appointment lists. Note that we could not include other types of reference transactions, such as asking a question at a service desk or making an appointment with a liaison, because Internet IDs are not collected for those transactions.

Outcomes measures: Students’ graduation and continued enrollment. The dependent variables of interest in this study included first-year students’ rate of graduation in four years and their rate of continued enrollment in four years (if not yet graduated or withdrawn). In the sample, 17.6 percent (n = 946) of the original class of 5,368 had withdrawn without graduating, while 63.1 percent (n = 3,388) of students graduated and 19.3 percent (n = 1,034) were still enrolled in courses after their fourth year. While policymakers and administrators would likely be most concerned about students’ rate of graduation in four years, we found value in also including students’ continued enrollment after four years in our analyses, as such a measure points toward students’ progress toward degree completion.

Data Analyses

We used propensity score matching techniques in SPSS 23.0 using the procedures outlined by Thoemmes.30 We began by using binary logistic regression to compute propensity scores for individual students. We used the five outcomes (using one of the five library resources at least one time) as dependent variables and the independent covariates listed above in the regressions to calculate the probabilities of using a library resource at least once in each area during students’ first year of enrollment. To be most effective, propensity scores should be based upon selected covariates that have significant relationships with both the “treatment” condition (that is to say, use of a library resource in an area) and the outcome (graduation or continued enrollment in four years).31 We removed only one covariate that was not significantly (p < .05) associated with either the treatment conditions or the outcomes—the nonspecified race/ethnicity variable.

Next, we used 1:1 nearest neighbor matching, meaning that each student in the treatment condition is matched to a student in the untreated condition who has the most similar estimated propensity score. We matched with replacement, meaning that some students in the control groups were “reused” in matches with students in the treatment group, a process that can reduce bias.32 We adjusted for those frequency weights in subsequent analyses, as some of the control units were used more than once in matching. We discarded all units that fell outside of the area of common support to avoid extrapolation to units that were so dissimilar that no comparisons could be made to other units.33

Next, it was important to check whether the matching procedures balanced the distribution of variables in both the treatment and control groups. We examined the standardized mean differences (the mean differences between the two groups divided by the standard deviation of the control group) in the treatment and control groups before and after matching. We detected no large imbalances above .25 after matching in each analysis, meeting the threshold suggested by Rosenbaum and Rubin.34 These results suggest that, before matching procedures were implemented, the covariates within the treatment and control groups differed significantly. These results also suggest that the propensity score matching decreased bias by making the observed and treatment groups more similar with regard to their covariates.

Finally, to test whether there are significant differences between library resource users’ and nonusers’ graduation rates in four years or continued enrollment in four years, we used multinomial logistic regression analyses. Multinomial logistic regression techniques are appropriate when comparing several outcomes against a referent variable. In this instance, we compared the odds of graduating in four years and of continued enrollment after four years against withdrawal from the university. We included the propensity scores as controls to remove the component of their correlation that is due to the assignment process.35 To evaluate the accuracy of the models, we computed the proportional by chance accuracy rate and maximum by chance accuracy rates and found that both exceed the criterion that models’ accuracy should be 25 percent better than chance.36 The final models were significantly different from the intercept-only models, suggesting that the independent variables contribute significantly to the prediction of the outcome and an adequate model fit.37 Goodness-of-fit statistics were nonsignificant (p > .05), and the pseudo R2 values suggested that the models had weak effect sizes. The results of these tests suggest that the models were adequate; however, the variables entered into the models did not explain very much of the variance in students’ graduation or continued enrollment in four years, meaning that other variables not entered into the models may help to better explain student graduation outcomes.

Results

All of the multinomial regressions predicted students’ four-year graduation or continued enrollment in four years over their withdrawal from the university. The first model assessed whether first-year students who used the library at least once during their first year of enrollment had significantly greater odds of graduation or continued enrollment (over withdrawal). The results suggest that first-year students who used any library resource at least once had significantly greater odds of continued enrollment (eβ = 1.389, p < .01) and graduation in four years (eβ = 1.441, p < .001) over their peers who did not use any library resources (see table 2). The results ultimately suggest that students who used any library resource at least one time were 1.389 times more likely to be enrolled in four years or 1.441 times more likely to have graduated in four years than peers who did not use any library resources.

The results of the second model suggest that first-year students who used electronic resources at least once were significantly more likely than their peers to both graduate in four years (over withdrawing) and were also more likely to have continued enrollment over withdrawing (see table 2). The odds of continually enrolling were nearly one-and-a-half times greater for first-year students who used electronic resources (eβ = 1.450, p < .001). The odds of graduating in four years were nearly doubled for students who used electronic resources (eβ = 1.924, p < .001).

The results of the third model suggest that students who used books at least once in their first year of enrollment had significantly improved odds of graduating in four years (eβ = 1.337, p < .01), although no effect was observed for continued enrollment. The fourth model suggested that using workstations had no effect on the odds of continued enrollment or graduation in four years over withdrawal from the university. The fifth model suggested that students who registered for at least one library instruction class or who had library instruction embedded in classes had significantly improved odds of continued enrollment in four years over withdrawal (eβ = 1.402, p < .01). Finally, the results of the last model suggested meeting with a peer reference consultant or chatting with a reference librarian had no effect on the odds of continued enrollment or graduation in four years. Given that these interactions with libraries staff or peer research consultants may have been relatively brief, it may be unsurprising to see such results.

Discussion and Recommendations

The results of this study suggest that using the library at least one time in the first year of enrollment significantly increased the odds that students would graduate in four years or remain enrolled after four years as opposed to withdrawing from the university. First-year students who used electronic resources and books also had significantly improved odds of graduation in four years (over withdrawing), while students who used electronic books and took a library instruction course had significantly improved odds of remaining enrolled after four years (over withdrawing). These results lend continued support to the growing bodies of research on students’ use of academic libraries and the attendant benefits of using libraries as associated with student success outcomes. As noted by Chiteng Kot and Jones,38 “investing in the university library is investing in student success.” As signified by this article, encouraging first-year students to make use of academic libraries in their first year of enrollment may yield dividends on students’ long-term success at their institutions.

However, it is not enough that students simply make use of their libraries—students should receive instruction and guidance on how to make the most of their academic libraries experiences. Along those lines, we also recommend that students receive opportunities to learn effective library research skills as soon as they enter their college or university within new student orientation programs or first-year seminar courses.39 Course-based instruction and working with faculty to create library-based assignments may also be effective ways to reach first-year undergraduate students. While face-to-face peer reference and reference librarian online chats were not significantly associated with student success outcomes in this study, we argue that these interpersonal interactions also remain critical components of students’ acquisition of vital research skills.40

Previous research also reveals that the majority of incoming undergraduate students consider libraries to be an important, very important, or essential part of their education experience and that library use is positively associated with both a student’s engagement in scholarship and his or her overall academic involvement.41 In other words, students value libraries and associate them with successful university or college attendance. Providing opportunities for students to make use of the library may not only help students achieve better grades or stay in school, it may also give students a sense of belonging. Library instruction and early undergraduate student use of databases, e-journals, or print collections may appear to primarily have short-term benefits, but library use also has a longitudinal impact, in that it gives students the confidence that they are scholars and belong at the university.

Limitations and Future Research Recommendations

The data used in this study were drawn from a large, public, university, a factor that may limit generalization to other institutional types.42 Furthermore, even though propensity score matching analyses were used to achieve a degree of balance in the covariates, it is indeed possible that important unobserved covariates were not included in analyses that may contribute to students’ use of academic library resources or their success in terms of graduation or continued enrollment. Chiteng Kot and Jones43 suggested, for example, that students’ academic motivation may be a factor associated with their decisions to use library resources. Without inclusion of variables such as academic motivation, estimated treatment effects of library use may therefore remain biased and should be interpreted with caution.44

We were not able to capture additional variables related to students’ use of libraries that may be important markers of their libraries experiences; for instance, Chiteng Kot and Jones45 found that students’ use of study rooms housed in the library was significantly associated with their academic performance. Stone and Ramsden46 discovered that students identified the library as an important physical space to meet others and engage in collaborative academic group work. The nearest variable we had to capture the physical benefits of academic libraries in our study was students’ use of workstations, although it would be far more preferable to use variables such as library entrance data or Wi-Fi location data to more effectively measure the impact of the physical attributes of libraries on student outcomes. We also advocate additional research on the effectiveness of peer research consultations or chats with reference librarians to better understand whether these services may have additional impacts on student outcomes if modified or moderated by factors such as duration or perceived helpfulness of the interaction.

Conclusion

Given the evidence presented here, we recommend continued research on the potential impacts of academic libraries on students’ outcomes. We recommend that academic libraries develop projects of their own to determine which of their interventions or services are most successful in reaching undergraduate students during their first year. Furthermore, it would be helpful to expand research in this field to discover which library services lead to more engagement with the library’s resources throughout students’ undergraduate careers. With such information, libraries can make decisions about those mediations and measure results on an ongoing basis. We encourage all academic libraries to begin or continue gathering usage data and mediation data at whatever scale is possible—and then to analyze those data in ways that allow library staff to provide the best possible experience for all students.

Notes

1. Gaby Haddow, “Academic Library Use and Student Retention: A Quantitative Analysis,” Library & Information Science Research 35 (2013): 127–36; Gaby Haddow and Jayanthi Joseph, “Loans, Logins, and Lasting the Course: Academic Library Use and Student Retention,” Australian Academic & Research Libraries 41, no. 4 (2010): 233–44; Elizabeth M. Mezick, “Return on Investment: Libraries and Student Retention,” Journal of Academic Librarianship 33, no. 5 (2007): 561–66, doi:10.1016/j.acalib.2007.05.002; Krista M. Soria, Jan Fransen, and Shane Nackerud, “Library Use and Undergraduate Student Outcomes: New Evidence for Students Retention and Academci Success,” portal: Libraries and the Academy 13, no. 2 (2013): 147–64, doi:10.1353/pla.2013.0010; Krista M. Soria, Jan Fransen, and Shane Nackerud, “Stacks, Serials, Search Engines, and Students’ Success: First-Year Undergraduate Students’ Library Use, Academic Achievement, and Retention,” Journal of Academic Librarianship 40, no. 1 (2014): 84–91.

2. Felly Chiteng Kot and Jennifer L. Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance: A Propensity Score Matching Design,” College & Research Libraries 76, no. 5 (2015): 566–86; Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes”; Soria, Fransen, and Nackerud, “Stacks, Serials, Search Engines, and Students’ Success”; Shun Han Rebekah Wong and T.D. Webb, “Uncovering Meaningful Correlation between Student Academic Performance and Library Material Usage,” College & Research Libraries 72, no. 4 (2011): 361–70.

3. Gregory A. Crawford, “The Academic Library and Student Retention and Graduation: An Exploratory Study,” portal: Libraries and the Academy 15, no. 1 (2015): 41–57; Megan Oakleaf, The Value of Academic Libraries (Chicago, Ill.: Association of College & Research Libraries, 2010).

4. Graham Stone and Bryony Ramsden, “Library Impact Data Project: Looking for the Link between Library Usage and Student Attainment,” College & Research Libraries 74, no. 6 (2013): 546–59.

5. Crawford, “The Academic Library and Student Retention and Graduation”; Sidney Eng and Derek Stadler, “Linking Library to Student Retention: A Statistical Analysis,” Evidence Based Library and Information Practice 10, no. 3 (2015): 50–63; Elizabeth M. Mezick, “Return on Investment: Libraries and Student Retention,” Journal of Academic Librarianship 33, no. 5 (2007): 561–66, doi:10.1016/j.acalib.2007.05.002.

6. Boris Teske, Michael DiCarlo, and Dexter Cahoy, “Libraries and Student Persistence in Southern Colleges and Universities,” Reference Services Review 41, no. 2 (2013): 1–13.

7. Mark Emmons and Frances C. Wilkinson, “The Academic Library Impact on Student Persistence,” College & Research Libraries 72, no. 2 (2011): 128–50.

8. Eng and Stadler, “Linking Library to Student Retention.”

9. Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes.”

10. Soria, Fransen, and Nackerud, “Stacks, Serials, Search Engines, and Students’ Success.”

11. Chiteng Kot and Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance,” 568–69.

12. Ibid.

13. Alexander W. Astin, What Matters in College: Four Critical Years Revisited (San Francisco, Calif.: Jossey-Bass, 1993).

14. Peter C. Austin, “An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies,” Multivariate Behavioral Research 46, no. 2 (2011): 399–424.

15. Tatiana Melguizo, Gregory S. Kienzl, and Mariana Alfonso, “Comparing the Educational Attainment of Community College Transfer Students and Four-Year College Rising Juniors Using Propensity Score Matching Methods,” Journal of Higher Education 82, no. 3 (2011): 265–91.

16. Sascha O. Becker and Andrea Ichino, “Estimation of Average Treatment Effects Based on Propensity Scores,” Stata Journal 2, no. 4 (2002): 358–77.

17. Paul W. Grimes and Marybeth F. Charters, “Library Use and the Undergraduate Economics Student,” College Student Journal 34, no. 4 (2000): 557–71; Krista M. Soria, Shane Nackerud, and Kate Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries,” Journal of Academic Librarianship 41, no. 5 (2015): 636–43.

18. Soria, Nackerud, and Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries”; Ethelene Whitmire, “Academic Library Performance Measures and Undergraduates’ Library Use and Educational Outcomes,” Library & Information Science Research 24, no. 2 (2002): 107–28, doi:10.1016/S0740-8188(02)00108-1.

19. Soria, Nackerud, and Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries”; Ethelene Whitmire, “The Relationship between Undergraduates’ Background Characteristics and College Experiences and Their Academic Library Use,” College & Research Libraries 62, no. 6 (2001): 528–40.

20. Soria, Nackerud, and Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries.”

21. Chiteng Kot and Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance,” 568–69; Soria, Nackerud, and Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries.”

22. Chiteng Kot and Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance,” 568–69; Soria, Nackerud, and Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries.”

23. Chiteng Kot and Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance,” 568–69.

24. Chiteng Kot and Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance,” 568–69; Soria, Nackerud, and Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries.”

25. Soria, Nackerud, and Peterson, “Socioeconomic Indicators Associated with First-Year College Students’ Use of Academic Libraries.”

26. Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes.”

27. “Aleph Integrated Library System [Computer Software],” Ex Libris Group (2012).

28. “Cybrarian [Computer Software],” Cybrarian Corporation (2013), available online at www.cybrarian.com [accessed 2 August 2017].

29. Ibid.

30. Felix J. Thoemmes, “Propensity Score Matching in SPSS” (2012), available online at http://arxiv.org/ftp/arxiv/papers/1201/1201.6385.pdf [accessed 2 August 2017].

31. M.H. Clark and Nicole L. Cundiff, “Assessing the Effectiveness of a College Freshman Seminar Using Propensity Score Adjustments,” Research in Higher Education 52 (2011): 616–39.

32. Elizabeth A. Stuart, “Matching Methods for Causal Inference: A Review and a Look Forward,” Statistical Science: A Review Journal of the Institute of Mathematical Statistics 25, no. 1 (2010): 1–21, doi:10.1214/09-STS313.

33. Thoemmes, “Propensity Score Matching in SPSS.”

34. Paul R. Rosenbaum and Donald B. Rubin, “Constructing a Control Group Using Multivariate Matched Sampling Methods That Incorporate the Propensity Score,” American Statistician 39, no. 1 (1985): 33–38.

35. Melguizo, Kienzl, and Alfonso, “Comparing the Educational Attainment of Community College Transfer Students and Four-Year College Rising Juniors”; Christopher Winship and Stephen L. Morgan, 1999. “The Estimation of Causal Effects from Observational Data,” Annual Review of Sociology 25 (1999): 659–707.

36. Carrie J. Petrucci, “A Primer for Social Worker Researchers on How to Conduct a Multinomial Logistic Regression,” Journal of Social Service Research 35, no. 2 (2009): 193–205.

37. Ibid.

38. Chiteng Kot and Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance,” 583.

39. Margit Misangyi Watts, “The Place of the Library versus the Library as Place,” in Challenging and Supporting the First-Year Student: A Handbook for Improving the First Year of College, eds. M.L. Upraft, J.N. Gardner, and B.O. Barefoot (San Francisco, Calif.: Jossey-Bass, 2005), 339–55.

40. Elizabeth B. Fitzpatrick, Anne C. Moore, and Beth W. Lang, “Reference Librarians at the Reference Desk in a Learning Commons: A Mixed Methods Evaluation,” Journal of Academic Librarianship 34, no. 3 (2008): 231–38; Marie L. Radford, “Encountering Users, Encountering Images: Communication Theory and the Library Context,” Journal of Education for Library and Information Science 42, no. 1 (2001): 27–41; Lauren Reiter and J.P. Huffman, “Yes, Virginia, It Will Scale: Using Data to Personalize High-Volume Reference Interactions,” Journal of Academic Librarianship 42, no. 1 (2016): 21–26.

41. Krista M. Soria, “Factors Predicting the Importance of Libraries and Research Activities for Undergraduates,” Journal of Academic Librarianship 39, no. 6 (2014): 464–70; Krista M. Soria, Jan Fransen, and Shane Nackerud, “Beyond Books: The Extended Academic Benefits of Library Use for First-Year College Students,” College & Research Libraries 77, no. 1 (2016): 8–22.

42. Soria, Fransen, and Nackerud, “Library Use and Undergraduate Student Outcomes.”

43. Chiteng Kot and Jones, “The Impact of Library Resource Utilization on Undergraduate Students’ Academic Performance.”

44. Ibid.

45. Ibid.

46. Stone and Bryony Ramsden, “Library Impact Data Project.”