Relationships between Journal Publication, Citation, and Usage Metrics within a Carnegie R1 University Collection: A Correlation Analysis

William H. Mischo, Mary C. Schlembach, and Elisandro Cabada*

This study examines the correlational relationships between local journal authorship, local and external citation counts, full-text downloads, link-resolver clicks, and four global journal impact factor indices within an all-disciplines journal collection of 12,200 titles and six subject subsets at the University of Illinois at Urbana-Champaign (UIUC) Library. While earlier investigations of the relationships between usage (downloads) and citation metrics have been inconclusive, this study shows strong correlations in the all-disciplines set and most subject subsets. The normalized Eigenfactor was the only global impact factor index that correlated highly with local journal metrics. Some of the identified disciplinary variances among the six subject subsets may be explained by the journal publication aspirations of UIUC researchers. The correlations between authorship and local citations in the six specific subject subsets closely match national department or program rankings.

Introduction

There has been a great deal of interest in attempting to determine correlational relationships between various individual journal title metrics within a collection, including local publication, citation, and usage (now download) measures and journal global impact factor measures. Gathering the raw measurement numbers associated with specific journals and establishing relationships between these variables can serve to inform a library’s collection development and management decisions, including journal subscription, cancellation, and retention decisions. From a public service and subject liaison perspective, this data can be used to construct a knowledgebase identifying departmental and faculty research concentrations and areas of focus.

The data can also be used to indicate if the collection is adequately supporting the research and instructional needs of faculty and students. It can be used in the generation of a library’s core journal list, which can provide an evidence-based listing of journals necessary to meet the instructional and research needs of the library’s primary constituents.1

The raw publication, citation, and usage data and correlation measures can assist in developing mechanisms to calculate a journal’s overall local composite value and can contribute to providing more data-driven assessments of a library’s journal collections. Collecting and correlating authorship, citation, and usage data can also allow patterns of journal use to emerge, resulting in a more accurate picture of journal value than cost-per-use calculations or other value gathering methods. Data gathered in this process can also be used to defend any local administrative tax that could be applied to academic departments or colleges to fund the library.

Libraries are also interested in determining the degree to which any one of the local journal metrics, particularly full-text downloads or local citations, can be used as a proxy for predicting any of the other values. This might allow, for example, predictive statements about publication numbers or citations to be made from usage numbers, or vice versa, and for one measure to serve as a predictive proxy for another measure. If this were uniformly the case, libraries could focus on collecting one or two types of measurements and be certain that the other local metrics would be proportionate.

In the same way, libraries are also interested in determining to what extent the journal global impact factor indices, such as the Institute for Scientific Information (ISI) Journal Citation Reports (JCR) Journal Impact Factor (JIF), (abbreviated as ISI JCR) correlate with local citation, publication, and usage metrics. A high correlation would in theory allow impact factor index values to be used in journal collection decisions or serve as a proxy for the local data variables.

Libraries have local usage data available in the form of the Counting Online Usage of Networked Electronic Resources (COUNTER) full-text download usage reports.2 COUNTER full-text download data is provided by commercial and professional society publishers in the form of spreadsheets for specific journals, giving monthly and yearly download data. Local publication and citation information is commonly available via several tools, among them the Local Journal Utilization Report (LJUR) from ISI, Scopus API extracted data, the SciVal PURE current researcher profile information system, the Symplectics Elements platform, and other research management resources.

This paper examines the relationships between specific journal title publication, citation, usage, and impact metrics from 17,934 journals in all disciplines, including 12,200 active titles, gathered from scholarly activities involving researchers at the University of Illinois at Urbana-Champaign, a Carnegie R1 university. The study analyzes and calculates the correlations over five years of local journal authorship numbers; five years of local and external citation counts; two years of full-text downloads, two years of link-resolver data, and the values from four global journal impact factor indices. It also examines correlation pair values for the journals in six subject subset areas: engineering; life sciences; social sciences; chemistry; history and philosophy; and literature. Several of these are monographic focused literatures and were included to present a more comprehensive scholarly communication model.

Literature Review

There is a long and rich literature on journal publication, citation, and usage metrics, particularly in the area of citation analysis, which is defined as the examination of citations from journal articles, dissertations, or other publications to determine trends and patterns of use.3 Ashman reviewed and categorized 88 studies on citation analysis culled from the library literature published from 1995 to 2008, categorizing the articles into nine types of literature profiles in the areas of public service, assessment, and collection-related areas.4 Hoffmann and Doucette reviewed 34 articles on citation analysis methodologies published from 2005 to 2010 and found that the articles typically did not provide enough information to make them adaptable for practical collection management decisions.5

Many studies have examined the relationships between citation and usage, in both local and global settings, first with print collections, then later with e-journal collections. Research relating to this topic appears in both the library and informetrics literature.

Surveying the studies on the relationship between citations and usage, McGillivray and Astell note that “these (studies) have not produced a definitive answer.”6 Pastva et al. examined the literature on citation and usage analysis and stated that, “some studies found a significant correlation between citation and usage data, while others found no significant correlation, highlighting the importance of methodology and local citation behaviors.”7

Several studies have looked at the relationship between usage and citation at the article or paper level and sometimes at both the article and journal title levels. Brody, Harnad, and Carr examined download and citation patterns at the paper level within the ArXiv.org e-print archive and found that the number of times an article was read was related both to the number of times it was cited and the age of the article. The authors also determined that short-term Web usage predicted medium-term citation impact.8 Kurtz et al. examined citation rates and readership rates with respect to publication date within the NASA Astrophysics Data System and developed a model for the relationship between reads and cites which incorporates obsolescence and derives a citation function that is based on several components of a usage function.9 In a review of usage and citation studies, Kurtz and Bollen assert that the relationship between usage and citation is complex and state: “with the accurate description of use being so complex, it is perhaps not surprising that the relation between use and citation has not been convincingly established.”10 They describe the difficulty in comparing usage information at the article level with citation histories and show that the interpretation of usage frequency as a function of publication date is quite complex.11 Schlögl and Gorraiz found that there were differences between downloads and citations in terms of obsolescence characteristics, where the half-life of the articles that are downloaded and the median cited half-life are significantly different. They found that the average cited half-life was 5.6 years and the mean usage half-life was 1.7 years, complicating the correlation relationship.12

Many of the studies examining the relationships between citation, usage, publication, and impact factor metrics have been carried out in specific subject disciplines and have typically covered a small subset of a discipline’s journals.13 In addition, several studies have found that the correlation between citation and usage data is dependent on subject discipline.14

Several previous studies have examined broader correlations between publication, usage, citation, and impact factor metrics within a library environment. Duy and Vaughan found a significant correlation between electronic journal usage and both LJUR local citation counts and library shelving counts for 112 chemistry and biochemistry journals, but found no significant correlation between the ISI JCR impact factor data and local electronic journal usage.15 McDonald, using subsets from 1,521 journal titles from the California Institute of Technology Library, found that print journal usage and, later, online journal usage was a valid predictor of local citation rates in journals.16

De Groote, Blecic, and Martin examined 2,619 health science journals and found high correlation values between download data, link-resolver data, and local citation rates.17 They concluded that link-resolver data were a good predictor of usage statistics in this environment. However, Gallagher, Bauer, and Dollar found that the usage data captured by link resolvers represented less than 10% of the total e-journal usage as identified by vendor download data.18

Chew et al. collected metrics data from 700 e-journals within 12 disciplinary subject areas at the University of Minnesota and analyzed correlation values.19 The study found marked disciplinary variation in the resulting correlations and also significant discrepancies between Scopus and ISI Web of Science calculated values. Some of the sample sizes were quite small.

Pastva et al. conducted a citation and usage analysis of over 33,000 articles (from an indeterminate number of journal titles) published between 2007 and 2016 by researchers in the Feinberg College of Medicine at Northwestern University.20 The correlations they derived between journal title usage and citing data were fairly weak and open to interpretation.

In a study of the University of Houston School of Communication faculty publications, Gao analyzed correlations among citation count, journal impact factor values, and journal usage.21 The journal sample sizes for the studied factors ranged from 147 journal titles to 108 titles. Gao found significant correlations between journal impact factor values and journal usage but no correlation between citation count and impact factor.

Several studies have been performed in research or vendor settings. In a comprehensive scientific impact analysis, Bollen et al. studied 7,675 journals and compared a number of journal citation and usage measures derived from usage log data from the 2008 Los Alamos Metrics from Scholarly Uses of Resources (MESUR) Project with several external impact factor measures.22 The authors performed a principal component analysis over a 39x39 factor correlation matrix and found 10 usage-based measures that appear to be stronger indicators of scientific prestige than the ISI JCR and other citation-based impact factor measurement systems. Bollen et al. comments that: “these results should give pause to those who consider the JIF (ISI JCR) the gold standard of scientific impact.”23 An earlier study by Bollen et al. (2005) also questioned the validity of the ISI JCR as a valid assessment of journal impact and suggested that usage-based measures were more accurate on a local level.24

Gorraiz, Gumpenberger, and Schlögl looked at the use of citation and download global data from 362 ScienceDirect journals over 10 years covering four subject disciplines: arts and humanities, computer science, economics and finance, and oncology.25 They found that the disciplines with the highest citation rates are not those with the highest download rates and the proportion of downloaded documents is dramatically higher than the proportion of cited documents. The authors claim that citations are often insufficient to assess the impact of the research output in many disciplines and downloads do not necessarily measure actual usage but must be considered as a complement to the “bibliometric citation-restricted horizon.”26

Elkins et al. examined the correlation between four journal impact factor indices, including the ISI JCR JIF, Eigenfactor’s article influence index, SCImago’s journal rank index, and Scopus’ trend line index.27 Paired values for the four all showed strong correlations between the four impact factor indexes.

Moed and Halevi carried out a detailed analysis of the relationship between downloads and citations by examining 62 journals from the Elsevier ScienceDirect repository, finding large differences in the degree of correlation between downloads and citations across various subject fields.28 They examined the correlations at both the journal and article levels finding that downloads were a good predictor of citations but that citations were a less valid predictor of downloads.

In addition, tools have been developed for library managers that are designed to aid in evaluating journal collections using journal title metrics. The California Digital Library developed the Weighted Value Algorithm (CDL-WVA) dashboard for collection selectors. Knowlton, Sales, and Merriman found that faculty selection differed significantly with the bibliometric values provided by the CDL-WVA tool.29 The Canadian Research Knowledge Network (CRKN) also utilized the CDL-WVA assessment tool in examining consortial packages of publisher journals.30

Methodology

The UIUC is a Carnegie R1 university with over 34,000 undergraduates, over 17,000 graduate students, and 1,900 tenure-system faculty, offering degrees in over 150 programs. In 2019, UIUC awarded almost 14,000 degrees, including 874 PhDs. The UIUC Library supports this wide variety of instructional and research programs with comprehensive journal subscriptions from all major commercial and professional society publishers. In 2021, the UIUC Library supported 108 subject or central collection funds on a $19.5 million materials budget.

The goal of this study was to examine the relationships between a number of local journal title publication, citation, link resolver clickthroughs, and usage measures within a large research university setting and calculate the correlations of these local metrics with four global impact factor indices. Examining these local journal metrics along with the global impact factor data assists the library to better determine the scholarly activities of UIUC researchers and to accurately characterize journal value measures for collection development and retention purposes. To obtain the local publication and citation data for this analysis and correlation of journal research activity metrics, the University Library purchased the 2017 Local Journal Utilization Report (LJUR) for UIUC from ISI, now owned by Clarivate.

The LJUR data provides summary information on UIUC researcher journal title authorship and citation numbers for the database of journals covered by the ISI platform. The LJUR data covers all academic disciplines and provides local publication and citation data at the journal title level for the journals covered by the extended ISI source list that is comprised of the journals covered by the former Science Citation Index, the Social Sciences Citation Index, and the Arts and Humanities Citation Index. The LJUR database covers the years 1981 through 2017 and includes UIUC publication and citation data from 17,934 journals.

The LJUR data is packaged as a Microsoft Access relational database containing five tables: (1) a journal list of 17,934 titles with columns for standard title abbreviation, ISSN, active or inactive status, and publisher; (2) a source publication table of 8,587 journal titles that UIUC authors have published in from 1981 through 2017 with columns for the total number of articles published and published articles for individual years from 1981 through 2017; (3) several tables of UIUC author local citation numbers for 15,785 journals with total times cited and citations by year; (4) a table of 21,423 journal titles (with 14,140 unique titles) including a title and an ISI subject descriptor; and (5) several tables of 14,338 journal titles showing the number of times outside authors have cited articles written by UIUC authors.

For this study, the data from the LJUR was used as a base to construct a journal title master table within a relational database that contained the 17,934 LJUR journal titles list, including the ISSN and EISSN numbers, the publisher information, and the active/inactive designation. Several scripts were written that extracted publication and citation information from the other LJUR database tables and added this data as additional columns into the master table. The columns that were added contained the total number of locally authored articles in each journal for a five-year period from 2013 to 2017, the total number of local citations from UIUC authors for each title from 2013 to 2017, and the total of external citations to UIUC authored articles for each journal from 2013 to 2017. There were 12,200 journals in the master table that the LJUR designated as active titles.

In order to better process and add the data for the additional usage, impact factor, and SFX (the Ex Libris local link resolver used by libraries) clickthrough numbers to the table of 17,934 journal records, additional ISSN and EISSN numbers for individual titles were added as columns using data from the Australian Research Council Excellence in Research for Australia (ERA) 2018 Journal List and ISSN/EISSN data from the Scopus source journal list. This provided a more comprehensive list of ISSNs and EISSNs that could be used as linking keys for matching journal titles and extracting download data from COUNTER tables and the impact factor data from the various services.

Subject descriptors were added as a column to the journal title master table to more accurately identify and extract subsets of journal titles by subject disciplines. A script was written to extract the ISI subject descriptors from the appropriate LJUR table which contained subject terms drawn from five research areas described by 251 subject categories in the ISI Web of Science descriptor scheme. To augment these LJUR Web of Science descriptors, subject terms from Scopus were added as a separate column to the journal title master table. Scopus journal titles are classified under four broad subject clusters which are further divided into 27 major subject areas and 300+ minor subject areas. The Scopus subjects were added using ISSN and EISSN numbers as the linking key. The two subject columns were used in SQL statements on the journal title master table to retrieve relevant journals in six disciplinary categories covering engineering, chemistry, social sciences, biosciences, history/philosophy, and literature subsets.

In addition, within the master title list, a column was added for SFX link resolver requests and clickthroughs for the years 2017 and 2018.

Four global impact indices were also used in the analysis. All the impact factor indices use global citation statistics to assign a value to individual journal titles typically calculated by taking the number of cited articles in a journal over a specific period and dividing that number by the number of articles published in that same period. The impact factor indices used in this analysis were: the ISI JCR JIF five-year impact factors from 2018; the SCIMago Journal Rank (SJR) values which use average citations within a subject field, the quality of citing journal, and a page rank algorithm on top of the usual measurement of citations divided by articles; the Eigenfactor scores from 2018 in which citations from highly ranked journals are weighted to generate a higher citation score than those from poorly ranked journals and normalized the journal scores by rescaling the total number of journals in the ISI JCR; and the Elsevier 2018 SNIP (Source Normalized Impact per Paper) values, which weights citations based on the total number of citations in a subject field over three years. All of these additional impact factor values have been added as separate columns in the journal title master entries using ISSN and EISSN numbers as the linking key.

For the usage data, the study utilized the publisher-provided COUNTER full-text download usage reports from 35 commercial and professional publishers and four aggregators—EBSCO, ProQuest, Ovid, and JSTOR. The publisher list includes all the major commercial publishers, e.g., Elsevier, Wiley, Springer-Nature, and Taylor and Francis, and many professional society publishers. The COUNTER data, in the form of spreadsheets of specific journal monthly and yearly full-text download data, was uploaded as individual tables in a separate companion publisher information relational database. Each publisher table contained COUNTER JR1 full-text usage report statistics from either 2015 or 2018. Over 44,000 journal titles are represented in the 39 COUNTER supplied publisher tables in the 2015 data and over 45,000 titles were in the 2018 data with 31,918 unique journal titles represented over both years. The duplicates are often titles appearing in both the publisher and aggregator tables. From there, scripting programs were written to move the 2015 and 2018 COUNTER downloaded data into an aggregated column for each specific matching journal in the journal title database master table. If a journal title appeared in more than one COUNTER table (for example a publisher and an aggregator), the numbers were added together to obtain a total number of downloads for that journal title. The journal titles were matched in the journal title master table using ISSN and EISSN numbers as linking keys.

There were 10,604 of the 17,934 journals in the LJUR all-disciplines corpus that had COUNTER download numbers available and 9,190 of the 12,300 active journal titles with available download data, so the COUNTER statistics covered a high majority of the journal titles in this study.

All the raw data used in this analysis, in the form of relational database tables with multiple columns, is being made available in the UIUC Library’s Illinois Data Bank dataset repository under https://doi.org/10.13012/B2IDB-6810203_V1. In addition, the processing scripts and Pearson correlation code is available at https://doi.org/10.13012/B2IDB-0931140_V1.

Processing

The correlation processing was set up to examine nine journal metric indicators: the LJUR supplied local authorship, LJUR local citation, LJUR external citation, full-text COUNTER download usage, link resolver clickthrough results, and four global journal impact service indexes. The numeric values for these nine indicators are all stored as columns in the records in the journal title master table. Note that this correlation analysis was carried out at the journal title level and does not include any analysis at the journal article or paper level.

A web interface over the database master journal title table was created with a search function that allowed retrieval by journal title, publisher, and subject and sorting capabilities by each of the journal title metrics. The web site tool serves an administrative and search function. It displays data records on single or groups of journal titles with the journal title metrics and can inform subscription, retention, and cancellation decisions, assist liaison librarians in understanding department and group research fronts, and contribute to the identification of core journal lists.

There were 12,200 designated active journal titles in the list of 17,934 titles from the LJUR database at the time of the analysis. The correlations over the metric data elements were carried out on both the 12,200 active titles and the entire corpus of 17,934 journal titles. There was essentially no difference in the correlation values of the active titles analysis and the values on the entire corpus. Because the study was using a two-year total of download data and five-year totals of local authorship and local citation, it was determined that the 12,200 active journal set would serve as a more accurate base sample for correlation calculations. The correlation analyses were carried out over the complete all-disciplines set of 12,200 journal titles and over six subsets comprised of engineering journals; chemistry journals; social science journals; biosciences journals; history and philosophy journals; and literature journal titles.

A server-side correlation generator script was written that produced a web site dashboard that allowed the authors to select the desired journal title metric indicators from the nine options and select either the all-disciplines journal corpus or one of the six disciplinary subset areas. The correlation generator produces a series of two-way correlations on the selected journal title indicators, producing a maximum of 26 (nine values taken two indicators at a time) pairwise correlation values (or fewer if less of the nine indicator factors are chosen). The correlations can be run over the 17,934 journal title corpus and also the 12,200 active journal title subsets.

The correlation generator calculates Pearson’s R values for the two pairwise data points. Pearson’s R gives values from –1 to 1 where –1 is a perfect negative correlation; 0 is no correlation; and 1 is a perfect positive correlation. Pearson’s is intended to be used in situations where the raw numeric data is available, as in this case. Several previous studies used the Spearman’s rho correlation statistic in cases where ranked data, not numeric data, was available. All the Pearson’s values in this analysis are significant at the p < .001 or lower value.

Global Impact Factor Measures

Table 1 shows the pairwise correlation analysis over the journal title values from the four global impact factor indices, using the 12,200 LJUR 2017 active journal titles as the base. Three of the global impact factor indices journal title values showed a high correlation to each other: the ISI JCR and SNIP (N=8,121, R= 0.7674); the ISI JCR and SJR (N=8,136, R=0.877); and the SNIP and SJR (N=11,669, R=0.7438). While the SNIP, SJR, and the ISI JCR correlate highly with each other, the normalized Eigenfactor stands out as not correlating highly with any of the other three impact factor indices: ISI JCR and Eigenfactor (N=8,348, R=0.4346); SNIP and Eigenfactor (N=8,127, R=0.2392); SJR and Eigenfactor (N=8,142, R=0.398).

Table 2 presents the correlations between local publications, citations, and downloads with the three highly correlated global impact factor indices: ISI JCR, SNIP, and SJR. None of the three impact factor indices exhibited a significant R value with the publication, citation, or usage measures for the cohort of journals in the study. This reinforces results obtained in numerous studies that show that the ISI JCR is often not a useful measure for local citation and publication activities and typically cannot serve as a proxy for local scholarly communication measures.31

Based on these results, only the normalized Eigenfactor Score values were used in the remaining analysis, along with the five local publication, citation, link resolver, and usage indicators. Chew et al. found that the Eigenfactor and SNIP (but not the ISI JCR) values provided significant correlations in certain disciplines with local publication and citation data.32 They found local authorship and impact factor values correlated strongest with Eigenfactor but also with SNIP in several disciplines.

Correlations Over the Journal Title Metrics

Table 3 shows the correlation results from the 12,200 active journal titles and journal value indicators for the all-disciplines analysis. Of the 15 pairwise value combinations (six items taken two at a time), only three Pearson R values are below .5 (shown in red text): the pair SFX clickthroughs and outside citations (N=11,709, R=0.4351); SFX clickthroughs and articles written (N=11,709, R=0.4941); and downloads and outside citations (N=9,190, R=0.4959). All other R values are above .5 (shown in blue text) with the next lowest being the pair downloads and articles written (N=9,190, R=0.5282), SFX clickthroughs and locally cited (N=11,709, R=0.5863), and normalized Eigenfactor and articles written (N=8,408, R=0.5937). All the other values are R=.64 or higher.

Tables 4 through 9 show the correlations over the six journal value indicators in each of the six subject discipline journal subsets included in the study. These are biosciences (Table 4), social sciences (Table 5), engineering (Table 6), literature (Table 7), chemistry (Table 8), and history and philosophy (Table 9).

Several of the correlation relationships, both in the all-disciplines set and the subject subsets, bear further examination. The specific disciplinary correlations exhibit some interesting differences with the all-disciplines analysis, particularly in three pairwise relationships: downloads and locally cited; articles written and locally cited; and downloads and articles written.

Downloads and Locally Cited

Numerous studies have shown that the relationship between usage and citation is very complex, particularly at the article or paper level.33 Issues involving usage and citation obsolescence characteristics, where the half-life of the articles that are downloaded and the median cited half-life are significantly different, play a key role in the relationship between usage and citation.34 At the article level, the articles appearing at the top of a citation ranking are not necessarily the most frequently downloaded articles, and vice versa.35 Several researchers have noted that different disciplines have different citation practices and protocols concerning citation behavior. Citation habits differ from one scientific area to another and there are several reasons for both citing an article and for downloading an article.36 Importantly, it has been established that correlations between downloads and citations are higher when calculated at the journal title level than at the article level.37

This study uses journal title level metrics, with two combined years of download data and a combined five years of both local citation and publication authorship data. Within this more simplified approach, in the all-disciplines overarching set, the often-studied correlation between the indicator pair downloads and locally cited is highly significant at N=9,190, R=0.7843. In the subject discipline analyses, the correlation is also high in the biosciences (N=922, R=0.7760), engineering (N=817, R=0.7240), chemistry (N=575, R=0.8667), and literature (N=272, R=0.6859), but lower in the social sciences subset (N=938, R=0.5416) and the history and philosophy (N=96, R=0.4107) journals subset. Even given the complexity, in this study using two years of download data and five years of local citations at the journal title level, the correlations were high overall and high in most of the six subject subsets.

Vaughan, Tang, and Yang analyzed 150 journals in 69 fields and found higher correlations between downloads and citations in the social sciences and humanities fields than in science, engineering, and medicine fields.38 In this study, the sciences and engineering fields yielded the highest correlations and the social sciences and humanities (except for literature) were lower.

Within the scholarly communications system, researchers are, for the most part, citing the most relevant and important articles in their field and faculty and students are downloading the most relevant articles for their research and instruction. Moed and Halevi suggested that for downloads and citations, there was a high correlation between the two in specialized fields in which the readers tend to be the active researchers but in fields where the reader population is wider and more diverse than the research community, the correlations are lower.39 That observation is generally supported by the results of this analysis. It is also the case that some less academic articles in more general journals are being downloaded for classroom use and not research.

Articles Written and Locally Cited

While the overarching all-disciplines correlation between the values in the articles written and locally cited pair is significantly high at N=12,200 and R=0.7698, there are clear differences in the values derived in the six subject discipline subsets. Looking at Tables 4 through 9, the biosciences journals (N=1,204, R=0.5401) in Table 4, the social sciences journals (N=1,123, R=0.5059) in Table 5, the literature journal subset (N=366, R=0.4173) in Table 7, and the history and philosophy journals (N=160, R=0.5213) in Table 9 are all below the all-disciplines value. The engineering publications subset shown in Table 6 (N=1,065, R=0.8392) and chemistry in Table 8 (N=747, R=0.9291) exhibit higher correlations than the other disciplines.

From the scholarly communications standpoint, the faculty are citing the most important articles in the most prestigious journals in the bibliographies of their research and, at the same time, are trying to have their research published in those most prestigious journals. Research faculty aspire to be published in the same journals that are publishing the most highly cited articles. The all-disciplines correlation between the indicators articles written and locally cited is very high at R=0.7698 and that shows a significant university-wide ability to publish in the same journals that are being cited. But the correlation varies across the six subsets. Interestingly, the articles written and locally cited correlations in the six subject discipline subsets examined in this study match very closely with the associated program rankings in the U.S. News and World Report 2022 Graduate School Rankings where the UIUC chemistry program is ranked at #6, engineering is #10 (with computer science at #5), biological sciences is #27, sociology is #49 and social work is #22 (but psychology is #7), English is #20 with literary criticism at #18, and history is ranked #21. In fact, one could argue that perhaps a factor in the determination of the prestige of a department or program could be the strength of the correlation between a faculty group’s articles written and their locally cited indicator pair. This might be incorporated into the suite of algorithms of the ranking services. Looking at the six subset areas in this case, UIUC researchers in some of the higher ranked science and engineering programs look to be better able to publish in the journals that they are citing most frequently.

Downloads and Articles Written

From this study, it appears that downloads are not strongly predictive of local authorship, given the all-disciplines correlation of the indicator pair downloads and articles written at N=9,190, R=0.5282. In the subject discipline subsets, there are three higher correlation values in the history and philosophy journals (N=96, R=0.7735), chemistry journals (N=575, R=0.8871), and engineering journals (N=817, R=0.6002). The other three subject discipline analyses show lower correlations with biosciences (N=922, R=0.4025) and social science (N=938, R=0.4605) lower, and literature (N=272, R=0.5201) somewhat lower. An examination of the five correlations involving the downloads values shows that the correlations for the pair downloads and articles written are typically lower in the all-disciplines journals and the subset disciplines than the other correlations involving downloads. These lower relationships may be due to the same issue contributing to the other lower correlations involving the articles written measures. They are related to the aspirational aspects of UIUC authorship, where the researchers may not be able to consistently publish, because of low acceptance rates or focus of their research, in the journals that, in this case, contain those articles that faculty and students are downloading to support their research. It is also possible that in the broader or more popular fields, there may be numerous downloads of articles by non-researchers in the field. However, in that case the history and philosophy correlation values would be expected to be low as they were for downloads and locally cited pair but here, they are in fact the highest in the disciplinary set.

Overall, the highest download numbers and local citation numbers come from many of the prestigious journals where the faculty aspire to publish.

SFX Local Link Resolver

De Groote, Blecic, and Martin defined the term SFTARs (successful full-text article requests) to indicate how many times articles in a journal are retrieved from a local link resolver full-text link appearing in an abstracting and indexing (A&I) service.40 Their study of medical journals found significant correlations between local link resolver requests and local citations. In this study, the correlation between SFX clickthroughs and locally cited pair in the all-disciplines journal set was moderate (N=11,709, R=0.5863). Interestingly, the SFX clickthroughs and locally cited pair values are highest in the social science journals (N=1,100, R=0.6436), engineering journals (N=1,022, R=0.7046), and chemistry journals (N=722, R=0.7755).

The two highest correlations involving the SFX local link resolver clickthroughs in the all-disciplines journals are SFX clickthroughs and downloads (N=9,041, R=0.7297) and SFX clickthroughs and normalized Eigenfactor (N=8,075, R=0.7295). The other values, including the correlations with articles written and local citations were lower.

In the UIUC environment, the use of local link resolver links is reduced by several factors. Some users go directly to a publisher or journal site and bypass any link resolver usage. In addition, some full-text links appear on aggregator sites and in discovery systems offering users direct to full-text links or direct to publisher site links. Direct publisher site links appear in major A&I services such as Scopus, PubMed, or Web of Science and aggregators and discovery systems offer direct full-text links or DOI links to publisher sites, both of which bypass the local link resolver.

It may be that UIUC SFX usage was not uniform across all A&I services and publisher sites and that it was consistently used only in certain subject A&I services and not in others or that users were clicking on the direct to PDF links in some A&I services rather than the SFX links. This has gotten more complicated in the UIUC Library where the discovery service pulls out direct PDF links from EBSCOHOST and ProQuest services. In the current environment, the SFX local link resolver has been replaced by the Alma link resolver.

External Citation Values

The LJUR data provides external citation values for the journal title articles authored by UIUC researchers that are cited by outside researchers. In the all-disciplines set of journals, the correlation for the pair outside citations and locally cited (N=12,200, R=0.7613) is high and it is significantly high for all the disciplinary subsets except for history and philosophy (N=160, R=0.4363).

The correlations for the outside citations and downloads pair (N=9,190, R=0.4959) were low in the all-disciplines set but higher in the biosciences, engineering, literature, and chemistry subsets. The correlation for outside citations and SFX clickthroughs (N=11,709, R=0.4351) was lower in the all-disciplines journals than it was in all the subset journal collections except for history and philosophy.

The correlation value for the outside citations and articles written pair (N=12,200, R=0.7907) in the overarching all-disciplines collection is significantly high but the outside citations and articles written pair exhibits the exact same differences in correlation values within the six subject subsets that were present in the local citations and articles written values.

Overall, the external citation correlations do not appear to contribute to a better understanding of the relationships between publication, citation, and usage metrics. The fact that the outside citations and articles written pair exhibited the same subject subset differences as the local citations and articles written pair in the six journal sets again implies that UIUC faculty in some departments or programs are not always writing in the same highly regarded journals that they are citing or that outside researchers are citing.

Normalized Eigenfactor

The normalized Eigenfactor Score values are the only global impact factor measures that exhibit significant correlation values with local publication, citation, and download data. This is particularly true at the all-disciplines level. The normalized Eigenfactor and locally cited pair (N=8,408, R=0.7858) and normalized Eigenfactor and downloads pair (N=6,189, R=0.8165) are the highest all-disciplines correlations. Several of the Eigenfactor and articles written correlations in the subject subsets are low, with the biosciences (N=1,162, R=0.3956), social sciences (N=201, R=0.3595), and history and philosophy (N=15, R=0.3774) subsets exhibiting low pairwise correlation values.

Limitations

The study used publication and citation data from 2013 to 2017 and download data from 2015 and 2018 in order to accommodate the projected half-life and obsolescence issues connected with the complex relationships between usage, citations, and publications. This placed the study in a period before open access became as prevalent as it is currently. The implications of open access full-text downloads and authoring are not known but should be investigated using later years for the study.

The authorship, citation, and download data numbers are all raw numbers and are not log normalized or weighted. There is no weight given to first or last authors listed on the articles and all cited articles are treated the same. It is not clear if weighting would influence the correlations in any way.

The study looked at only six subject subset areas. There is a clear need to examine the metric correlations within additional disciplines—some of the other locally highly ranked subject areas and some of the lesser ranked programs—to see if the conclusions regarding program strength and the relationship between the articles written and locally cited parameters and several other pairs hold true. It would be possible to automate the process to introduce a script that would present the appropriate SQL commands to derive the subject discipline subsets to calculate the R values and summarize and collect the results.

The COUNTER only publisher full-text download data encompassed 31,918 journal titles but the ISI LJUR coverage extended to 17,934 total journals including 12,200 active journals. The Scopus API journal coverage includes almost 25,000 current journals and would be more extensive than the coverage provided by the LJUR data. Repeating this study using UIUC authored journal articles and processed using the Scopus API would provide more extensive journal title coverage and allow additional journal metric pair correlations to be performed.

Conclusion

The goal of this study was to investigate the correlational relationships between journal title metrics from the UIUC multi-disciplinary research journal collection and over six subject subset journals in biosciences; chemistry; social sciences; history and philosophy; literature; and engineering. The particular metric indicators making up this analysis were local publication and citation data; COUNTER supplied full-text downloads; local link resolver clickthroughs; and four global impact factor index values. This analysis was carried out over a large sample of 12,200 active journals in all subject disciplines with publication and citation data supplied through the ISI LJUR service. Full-text download numbers from COUNTER were available for 9,190 journal titles in the active journal title set.

The exercise of assembling the raw journal title publication, citation, and download values over a collection-scale set of journals was useful in itself. A web interface over the database table was created with a search function that allowed retrieval by journal title, publisher, and subject with sorting capabilities by each of the journal title metrics. The web site tool provides data on individual journal titles which can inform subscription, retention, and cancellation decisions, assist liaison librarians in understanding department and group research concentrations, and could contribute to the generation of core journal lists. The pairwise correlation values over the journal title metrics provide insight into scholarly communication patterns, the relationships between the various journal metrics, and the bibliometric interactions in operation at UIUC. These correlation values can be compared to the values at other R1 university libraries. They can also provide evidence of the ability of one or more of the metrics to be used as a proxy for the others. The process methodology and protocols for this study can serve as a model or blueprint for other academic libraries looking to investigate these relationships in other institutional settings.

An analysis of the four global impact factor indices showed that the ISI JCR, the SNIP, and the SCIMago JCR indices did not exhibit significant positive correlations with the publication, citation, or download indicators. Only the normalized Eigenfactor values showed significant correlation with the local data.

The relationship between local download usage and local citation has been the subject of many previous investigations. Earlier studies have shown that the relationships involving downloads and citations, particularly when they are examined at the article level rather than the journal title level, are quite complex. The data in this study were comprised of two combined years of download data from 2015 and 2018 and a combined five years (from 2013 to 2017) of both local and external citation and publication authorship data. The correlations between the important downloads and locally cited values were calculated at the journal title level, where it has been shown to be higher than at the article level. The analysis found an overall strong positive correlation between journal usage, in the form of full-text downloads, and locally cited journal titles. In the all-disciplines overarching journal set, the correlation between downloads and locally cited pair was high (N=9,190, R=0.7843) and the R values were also high (from 0.5416 to 0.8667) in five of the six subject subset journal collections examined in the study. The history and philosophy subset R value was 0.4107. While earlier investigations have proven inconclusive, this study shows strong correlations in the all-disciplines set and most subject subsets between full-text downloads and local citations.

One explanation offered in the literature for the subject discipline differences may lie in the observation that there are higher correlations between the two metrics in specialized fields in which the readers tend to be the active researchers but lower correlations in fields where the reader population is wider and more diverse than the research community. Within the all-disciplines 12,200 active journals, and in most subject disciplines, this study’s correlation results do imply that download measurements can predict local citations and vice versa.

Researchers are citing the most important articles in the most prestigious journals in their field. At the same time, they are attempting to publish their research in the most prestigious journals, which are typically the journals that they and other researchers are predominantly citing. Research faculty aspire to be published in the same journals that are publishing the most highly cited articles. The all-disciplines correlation of the articles written and locally cited is very high at R=0.7698, demonstrating a significant ability of UIUC faculty to publish in the same journals that they are citing. This study found, however, that, for the six subject journal title subsets that are identified from the overarching collection, the articles written and locally cited correlation matches very closely with their associated department or program ranking in the U.S. News and World Report 2022 Graduate Schools Rankings. One criterion for ranking a subject department or program might be to calculate the strength of the correlation between the group’s articles written and locally cited journal title metrics. The aspirational publishing aspect may also affect the correlations between the indicators downloads and articles written where many of the articles are downloaded from highly cited journals where researchers aspire to publish.

The study found that SFX link resolver correlations were high when matched with the downloads indicator and the normalized Eigenfactor measures. The link resolver correlation values with articles written and local citations were lower. The link resolver and external citation indicators were not regarded as very useful measures for understanding publication, citation, and usage behaviors or activities.

With the addition of journal title subscription information to the metric data assembled in this study, it is fairly easy to calculate a journal composite value using the weighted set of local publication, citation, and download number values to derive a journal composite value which can then be divided by the subscription price to obtain an overall value score. The UIUC Library has produced this assigned value journal table, although there is some difficulty in assigning an individual journal subscription price to journals purchased as part of an overarching “big deal” package.

The study revealed some interesting interactions and relationships between the journal metrics. There are limitations and subtleties with each of the journal title measure correlations. Chew et al. noted that “it is generally conceded that the metrics, when taken in aggregate, provide a more complete picture on journal value and importance.”41 A number of studies show that the various journal metrics need to be applied and combined in a strategic manner in order to obtain meaningful results.42 De Groote, Blecic, and Martin noted that citation data describes research activity but that vendor, publisher, and link-resolver statistics also reflect educational and clinical usage.43 Given these complex and interrelated factors and the analysis presented in these study results, it would appear that multiple metrics may need to be employed to make definitive statements about journal publication, citation, and usage relationships and interactions. The study also demonstrated that there are some significant disciplinary differences in the local indicator correlation values across the six subject subsets. It is also clear from the study that a more nuanced profile of user publication, citation, and usage activity than some other measures, such as the commonly quoted cost per use metric, are possible and desirable.

Notes

1. Philip M. Davis, “Where to Spend our E-journal Money? Defining a University Library’s Core Collection Through Citation Analysis,” portal: Libraries & the Academy 2, no. 1 (2002): 155–166, https://doi.org/10.1353/pla.2002.0009.

2. Gaby Haddow, “Level 1 COUNTER Compliant Vendor Statistics are a Reliable Measure of Journal Usage,” Evidence Based Library & Information Practice 2, no. 2 (2007): 84–86 https://doi.org/10.18438/B83G6S.

3. Kristin Hoffmann and Lise Doucette, “A Review of Citation Analysis Methodologies for Collection Management,” College & Research Libraries 73no. 4 (2012): 321–335. https://doi.org/10.5860/crl-254.

4. Allen B. Ashman, “An Examination of the Research Objectives of Recent Citation Analysis Studies,” Collection Management 34, No. 2 (2009): 112–128, https://doi.org/10.1080/01462670902725885.

5. Hoffmann and Doucette, “A Review of Citation Analysis Methodologies for Collection Management,” 327.

6. Barbara McGillivray and Mathias Astell, “The Relationship Between Usage and Citations in an Open Access Mega-Journal,” Scientometrics 121, no. 2 (2019): 818, https://doi.org/10.1007/s11192-019-03228-3.

7. Joelen Pastva, Jonathan Shank, Karen E. Gutzman, Madhuri Kaul, and Ramune K. Kubilius, “Capturing and Analyzing Publication, Citation, and Usage Data for Contextual Collection Development,” Serials Librarian 74, no. 1–4 (2018): 102–110, https://doi.org/ 10.1080/0361526X.2018.1427996.

8. Tim Brody, Stevan Harnad, and Leslie Carr, “Earlier Web Usage Statistics as Predictors of Later Citation Impact,” Journal of the American Society for Information Science & Technology 57, no. 8 (2006): 1060–72, https://doi.org/10.1002/asi.20373.

9. Michael J. Kurtz, Guenther Eichhorn, Alberto Accomazzi, Carolyn Grant, Markus Demleitner, Stephen S. Murray, Nathalie Martimbeau, and Barbara Elweli, “The Bibliometric Properties of Article Readership Information,” Journal of the American Society for Information Science and Technology, 56 (2005): 111–128, https://doi.org/10.1002/asi.20096.

10. Michael J. Kurtz and Johan Bollen, “Usage Bibliometrics,” Annual Review of Information Science and Technology 44 (2010): 1–64, https://doi.org/10.1002/aris.2010.1440440108.

11. Ibid., 23.

12. Christian Schlögl and Juan Gorraiz, “Comparison of Citation and Usage Indicators: The Case of Oncology Journals,” Scientometrics 82, no. 3 (2010): 567–80, https://doi.org/10.1007/s11192-010-0172-1; Christian Schlögl and Juan Gorraiz, “Global Usage Versus Global Citation Metrics: The Case of Pharmacology Journals,” Journal of the American Society for Information Science & Technology 62, no. 1 (2011): 161–70, https://doi.org/10.1002/asi.21420.

13. Schlögl and Gorraiz, “Comparison of Citation and Usage Indicators: The Case of Oncology Journals”; Juan Gorraiz, Christian Gumpenberger, and Christian Schlögl, “Usage Versus Citation Behaviours in Four Subject Areas,” Scientometrics 101, no. 2 (November 2014): 1077–95, https://doi.org/10.1007/s11192-014-1271-1; Liwen Vaughan, Juan Tang, and Rongbin Yang, “Investigating Disciplinary Differences in the Relationships Between Citations and Downloads,” Scientometrics 111, no. 3 (2017): 1533–45, https://doi.org/10.1007/s11192-017-2308-z; Stephanie H. Wical and R. Todd Vandenbark, “Combining Citation Studies and Usage Statistics to Build a Stronger Collection,” Library Resources & Technical Services 59, no. 1 (2015): 33–42, https://doi.org/10.5860/lrts.59n1.33; Johan Bollen, Herbert Van De Sompel, Joan A. Smith, and Rick Luce, “Toward Alternative Metrics of Journal Impact: A Comparison of Download and Citation Data,” Information Processing and Management Vol. 41, no. 6 (December 2005): 1419–40. https://doi.org/10.1016/j.ipm.2005.03.024; H. Chu and T. Krichel, “Downloads vs. Citations in Economics: Relationships, Contributing Factors and Beyond,” Proceedings of 11th International Conference of the International Society for Scientometrics and Informetrics (2007): 207–15, https://www.issi-society.org/publications/issi-conference-proceedings/proceedings-of-issi-2007/; Daniel E. O’Leary “The Relationship Between Citations and Number of Downloads in Decision Support Systems,” Decision Support Systems 45, no. 4 (2008): 972–80. https://doi.org/10.1016/j.dss.2008.03.008; Jason C. Dewland, “A Local Citation Analysis of a Business School Faculty: A Comparison of the Who, What, Where, and When of Their Citations,” Journal of Business & Finance Librarianship 16, no. 2 (2011): 145–58, https://doi.org/10.1080/08963568.2011.554740; Irene Ke and Jackie Bronicki, “Using Scopus to Study Researchers’ Citing Behavior for Local Collection Decisions: A Focus on Psychology,” Journal of Library Administration 55, no. 3 (2015): 165–78, https://doi.org/10.1080/01930826.2015.1034035.

14. Henk F. Moed and Gali Halevi, “On Full Text Download and Citation Distributions in Scientific-Scholarly Journals,” Journal of the Association for Information Science & Technology 67, no. 2 (2016): 412–31, https://doi-org.proxy2.library.illinois.edu/10.1002/asi.23405; Katherine Chew, Mary Schoenborn, James Stemper, and Caroline Lilyard, “E-journal Metrics For Collection Management: Exploring Disciplinary Usage Differences in Scopus and Web of Science,” Evidence Based Library and Information Practice 11, no. 2 (2016): 97–120, https://doi.org/10.18438/B85P87; Vincente P. Guerrero-Bote and Felix Moya-Anegón, “Relationship Between Downloads and Citations at Journal and Paper Levels, and the Influence of Language,” Scientometrics 101, no. 2 (2014) 1043–65, https://doi.org/10.1007/s11192-014-1243-5.

15. Joanna Duy and Liwen Vaughan, “Can Electronic Journal Usage Data Replace Citation Data as a Measure of Journal Use? An Empirical Examination” Journal of Academic Librarianship 32, no. 5 (2006): 512–17, https://doi.org/10.1016/j.acalib.2006.05.005.

16. John D. McDonald, “Understanding Journal Usage: A Statistical Analysis of Citation and Use,” Journal of the American Society for Information Science & Technology 58, no. 1 (2007): 39–50, https://doi.org/10.1002/asi.20420.

17. Sandra L. De Groote, Deborah D. Blecic, and Kristin E. Martin, “Measures of Health Sciences Journal Use: A Comparison of Vendor, Link-Resolver, and Local Citation Statistics,” Journal of the Medical Library Association 101, no. 2 (2013): 110–19, https://doi.org/10.3163/1536-5050.101.2.006.

18. John Gallagher, Kathleen Bauer, and Daniel M. Dollar, “Evidence-based Librarianship: Utilizing Data from All Available Sources to Make Judicious Print Cancellation Decisions,” Library Collections, Acquisition and Technical Services 29, no. 2 (June 2005): 169–79, https://doi.org/10.1016/j.lcats.2005.04.004.

19. Chew et al., “E-journal Metrics for Collection Management: Exploring Disciplinary Usage Differences in Scopus and Web of Science.”

20. Pastva et al., “Capturing and Analyzing Publication, Citation, and Usage Data for Contextual Collection Development,” 102–10.

21. Wenli Gao, “Beyond Journal Impact and Usage Statistics: Using Citation Analysis for Collection Development,” Serials Librarian 70, no. 1–4 (2016): 121–27, https://doi.org/10.1080/0361526X.2016.1144161.

22. Johan Bollen, Herbert Van de Sompel, Aric Hagberg, and Ryan Chute, “A Principal Component Analysis of 39 Scientific Impact Measures.” PLoS ONE 4, no. 6 (2009): 1–11, https://doi.org/10.1371/journal.pone.0006022.

23. Ibid., 10.

24. Bollen et al., “Toward Alternative Metrics of Journal Impact: A Comparison of Download and Citation Data.”

25. Gorraiz, Gumpenberger, and Schlögl, “Usage Versus Citation Behaviours in Four Subject Areas.”

26. Ibid., 1093.

27. Mark R. Elkins, Christopher G. Maher, Robert D. Herbert, Anne M. Moseley and Catherine Sherrington, “Correlation Between the Journal Impact Factor and Three Other Journal Citation Indices,” Scientometrics 85, no. 1 (2010): 81–93, https://doi.org/10.1007/s11192-009-0090-2.

28. Moed and Halevi, “On Full Text Download and Citation Distributions in Scientific-Scholarly Journals.”

29. Steven A. Knowlton, Adam C. Sales, and Kevin W. Merriman, “A Comparison of Faculty and Bibliometric Valuation of Serials Subscriptions at an Academic Research Library,” Serials Review 40, no. 1 (2014): 28–39, https://doi.org/10.1080/00987913.2014.897174.

30. Eva Jurczyk and Pamela Jacobs, “What’s the Big Deal? Collection Evaluation at the National Level,” portal: Libraries & the Academy 14, no. 4 (2014): 617–31, https://doi.org/ 10.1353/pla.2014.0029.

31. Bollen et al., “A Principal Component Analysis of 39 Scientific Impact Measures”; Bollen et al., “Toward Alternative Metrics of Journal Impact: A Comparison of Download and Citation Data”; Chew et al., “E-journal Metrics for Collection Management: Exploring Disciplinary Usage Differences in Scopus and Web of Science”; Schlögl and Gorraiz, “Comparison of Citation and Usage Indicators: The Case of Oncology Journals”; Schlögl and Gorraiz, “Global Usage Versus Global Citation Metrics: The Case of Pharmacology Journals”; Klaus G. Altmann and G.E. Gorman, “Can Impact Factors Substitute for the Results of Local Use Studies? Findings From an Australian Case Study,” Collection Building 18, no. 2 (June 1999): 90–94, https://doi.org/10.1108/01604959910265878; Davis, “Where to Spend our E-journal Money? Defining a University Library’s Core Collection Through Citation Analysis”; Janice Kreider, “The Correlation of Local Citation Data with Citation Data from Journal Citation Reports,” Library Resources & Technical Services 43, no. 2 (1999): 67–77, https://doi.org/10.5860/lrts.43n2.67; Stephen P. Harter and Thomas E. Nisonger, “ISI’s Impact Factor as Misnomer: A Proposed New Measure to Assess Journal Impact Factor,” Journal of the American Society for Information Science 48 (Dec. 1997): 1146–48, https://doi.org/10.1002/(SICI)1097-4571(199712)48:12<1146::AID-ASI9>3.0.CO;2-U.

32. Chew et al., “E-journal Metrics for Collection Management: Exploring Disciplinary Usage Differences in Scopus and Web of Science.”

33. Kurtz and Bollen, “Usage Bibliometrics”; Schlögl and Gorraiz, “Comparison of Citation and Usage Indicators: The Case of Oncology Journals”; Schlögl and Gorraiz, “Global Usage Versus Global Citation Metrics: The Case of Pharmacology Journals”; Moed and Halevi, “On Full Text Download and Citation Distributions in Scientific-Scholarly Journals”; Gorraiz, Gumpenberger, and Schlögl, “Usage Versus Citation Behaviours in Four Subject Areas.”

34. Kurtz et al., “The Bibliometric Properties of Article Readership Information”; Kurtz and Bollen, “Usage Bibliometrics”; Schlögl and Gorraiz, “Comparison of Citation And Usage Indicators: The Case of Oncology Journals”; Schlögl and Gorraiz “Global Usage Versus Global Citation Metrics: The Case of Pharmacology Journals.”

35. Moed and Halevi, “On Full Text Download and Citation Distributions in Scientific-Scholarly Journals”; Gorraiz, Gumpenberger, and Schlögl, “Usage Versus Citation Behaviours in Four Subject Areas.”

36. Guerrero-Bote and Moya-Anegón, “Relationship Between Downloads and Citations at Journal and Paper Levels, and the Influence of Language”; A. J. Nederhof, R. A. Zwaan, R. E. De Bruin, and P. J. Dekker, Assessing the Usefulness of Bibliometric Indicators for the Humanities and the Social Sciences,” Scientometrics 15 (1989): 423–435, https://doi.org/10.1007/BF02017063; Gorraiz, Gumpenberger, and Schlögl, “Usage Versus Citation Behaviours in Four Subject Areas”; Kurtz and Bollen, “Usage Bibliometrics.”

37. Moed and Halevi, “On Full Text Download and Citation Distributions in Scientific-Scholarly Journals”; Vaughan, Tang, and Yang, “Investigating Disciplinary Differences in the Relationships Between Citations and Downloads”; Henk F. Moed, “Statistical Relationships Between Downloads and Citations at the Level of Individual Documents within a Single Journal,” Journal of the American Society for Information Science & Technology 56, no. 10 (2005): 1088–97, https://doi.org/10.1002/asi.20200; Guerrero-Bote and Moya-Anegón, “Relationship Between Downloads and Citations at Journal and Paper Levels, and the Influence of Language.”

38. Vaughan, Tang, and Yang, “Investigating Disciplinary Differences in the Relationships Between Citations and Downloads.”

39. Moed and Halevi, “On Full Text Download and Citation Distributions in Scientific-Scholarly Journals.”

40. De Groote, Blecic, and Martin, “Measures of Health Sciences Journal Use: A Comparison of Vendor, Link-Resolver, and Local Citation Statistics.”

41. Chew et al., “E-journal Metrics for Collection Management: Exploring Disciplinary Usage Differences in Scopus and Web of Science,” 99.

42. Chew et al., “E-journal Metrics for Collection Management: Exploring Disciplinary Usage Differences in Scopus and Web of Science”; Wical and Vandenbark, “Combining Citation Studies and Usage Statistics to Build a Stronger Collection.”

43. De Groote, Blecic, and Martin, “Measures of Health Sciences Journal Use: A Comparison of Vendor, Link-Resolver, and Local Citation Statistics.”