07_Fosnacht

Information Literacy’s Influence on Undergraduates’ Learning and Development: Results from a Large Multi-institutional Study

This paper investigated the reliability and validity of the National Survey of Student Engagement’s Experiences with Information Literacy module, an assessment instrument developed in collaboration with a group of instructional librarians. After identifying three information literacy–related factors in the module, it assessed the relationship between the factors and students’ engagement in Higher-Order Learning and Reflective and Integrative Learning activities and students’ perceived gains. The results from these analyses indicated that information literacy activities were positively and significantly correlated with student engagement and students’ perceived gains.

Today undergraduates face a conundrum. They can access more information on their cellphones at a moment’s notice than previous generations could access in a multistory university library. However, the increasing inclusion of terms like “alternative facts” and “fake news” in the popular discourse highlights how much of the information digested by students is of questionable quality.1 This reality makes knowing how to search for, use properly, and evaluate information a critical skill for the 21st century.

Information literacy has been widely recognized as a critical college learning outcome for decades.2 However, assessing information literacy skills has been difficult for librarians and other assessment professionals, as the conventional methods frequently focus on small samples at or within a single institution, testing knowledge of specific skills, are time intensive (rubrics), or use instruments that have not been psychometrically validated.3 Due to these concerns, instructional librarians approached the National Survey of Student Engagement (NSSE) to collaborate with NSSE researchers on expanding the information literacy content in NSSE.4 The collaboration resulted in the creation of the Experiences with Information Literacy module, which institutions can elect to append to the core NSSE instrument. In contrast to other information literacy assessments, the module mainly focuses on student engagement in activities associated with information literacy skill development and the extent to which instructors emphasize the proper use of information, which makes the module’s results actionable. Additionally, most baccalaureate-granting colleges and universities periodically administer NSSE to their students,5 and 208 institutions have chosen to administer the module to their students in the first four years NSSE has offered the module. Therefore, the module is relatively inexpensive and efficient to administer, and its data are already available to a wide variety of institutions.

This study has two main purposes. First, it seeks to establish the reliability and validity of the NSSE Experiences with Information Literacy module. Second, through its investigation of the predictive validity of the module’s factors, it examines how engagement in information literacy activities contributes to students’ learning and development. To accomplish these objectives, the study uses data from nearly 145,000 first-year students and seniors attending 208 colleges and universities. The results have the potential to highlight the usability of the module and provide further evidence of the importance of information literacy in promoting student learning outcomes.

Student Engagement Theory

Both the study and the creation of the NSSE Experiences with Information Literacy module were informed and guided by student engagement theory. The theory combines empirical research and several well-known concepts in the higher education literature.6 Student engagement uses Pace’s quality of effort concept, which postulates that student learning is a product of students’ quality and quantity of effort, as well as Astin’s student involvement theory, which theorizes that student retention is related to involvement in academic and cocurricular activities.7 In addition to focusing on students’ activities, it emphasizes the role of the institution to promote student learning and development in and outside of the classroom. Chickering and Gamson demonstrated a variety of effective educational practices that support student learning.8 Additionally, Kuh and colleagues exhibited how institutions can structure their institutions to improve student learning outside of the classroom.9 When applying the theory to information literacy, the theory views institutions as having the responsibility to emphasize and instruct students on the proper and effective uses of information and providing ample or adequate information resources for students. Furthermore, students are expected to exhort time and effort to search, evaluate, and use information sources properly.

Research Questions

As the validity of a survey instrument requires multifaceted evidence,10 this study investigated the following research questions to establish the validity of the NSSE Experiences with Information Literacy module and examine how engagement in information literacy activities promotes student learning and development.

  1. What information literacy factors exist within the NSSE’s Experiences with Information Literacy module?
  2. Do the factors adequately fit the observed data?
  3. Are the factors reliably measured?
  4. After controlling for other factors, how do the factors correlate to students’ engagement in higher-order learning and reflective and integrative learning skills?
  5. After controlling for other factors, how do the factors relate to students’ perceived gains?

Methods

Data

To answer the aforementioned research questions, I used data from the 2014 to 2018 administrations of the National Survey of Student Engagement (NSSE). NSSE examines students’ engagement in educationally beneficial activities, their perceptions of the campus environment, self-assessments of how their college experience has improved their learning and development, and students’ background characteristics. The 2014 to 2018 timeframe included all NSSE administrations that included the Experiences with Information Literacy module. The module is a set of additional questions that institutions can elect to append to the core NSSE instrument that focuses on how often students use and evaluate information, the extent to which instructors emphasized the proper use of information, and an item asking how much students’ college experience has influenced their ability to use information effectively. NSSE staff and a group of instructional librarians collaboratively developed the module.11 The designers of the module used their academic training as well as their experience teaching information literacy skills, and they consulted existing information literacy standards and rubrics when creating the module.12 In keeping with student engagement theory, the module was not designed to test a specific student’s information literacy capabilities. Rather, it seeks to investigate how often students use information literacy skills and how institutions promote the proper use of information through their curriculum, as students’ time and effort in educationally beneficial activities are presumed to promote student learning and development. The module was pilot-tested in 2013 and subsequently refined and officially added to NSSE in 2014.

Since 2014, 60,631 first-year and 83,762 senior students responded to the Experiences with Information Literacy module. These students attended 208 U.S. bachelor’s-granting institutions. The response rate was 22 percent for first-year students and 24 percent for seniors. Previous research indicates that NSSE data produces accurate and reliable group means at this response rate.13 Table 1 contains the characteristics of the sample. About two-thirds of the respondents were white. Six out of 10 students were female. Three in four students were 25 years old or younger. More than half of the sample had a parent who earned a bachelor’s degree or higher. The most common major was business; however, the sample was dispersed well among a variety of academic disciplines. A plurality of the respondents attended a research university. Two-thirds of the sample attended a public institution. I obtained the data from the Indiana University Center for Postsecondary Research, which operates NSSE. Researchers interested in replicating the analyses or using the data can request a data use license from the Indiana University Center for Postsecondary Research.

The primary data used were 13 of the 14 items on the Experiences with Information Literacy module. The lone item not included in the analyses was the final item asking about how students’ college experiences influenced their ability to use information literacy effectively, which did not comport with the engagement focus of the other items. A list of the items used from the module and their distributions are displayed in table 2. I also used data on two of the NSSE Engagement Indicators: Higher-Order Learning and Reflective & Integrative Learning. The Engagement Indicators are summary measures of various dimensions of student engagement. Information on the reliability and validity of the Engagement Indicators is available from NSSE’s Psychometric Portfolio.14 In addition to the Engagement Indicators, I used NSSE’s perceived gains scale, which is a composite of 10 items inquiring about how students’ college experience influenced their learning and development. The reliability of the perceived gains scales was .91. A list of the items contained in the two Engagement Indicators and perceived gains scale is displayed in the appendix. To estimate effect sizes efficiently, I standardized the Engagement Indicators and perceived gains scale to have a mean of 0 and standard deviation of 1. I also used data on a variety of student characteristics to control for differences in students’ background and academic characteristics. The control variables were sex, race/ethnicity, time spent working, major field, college grades, educational aspirations, parental education, age, and enrollment level. The Engagement Indicator and control variables were captured on the core NSSE instrument or provided by institutions.

Table 1

Descriptive Statistics of the Sample By Class Level

First-year (%)

Senior (%)

Race/ethnicity

Asian/Pacific Islander

6

6

Black/African American

7

7

Hispanic/Latin@

12

11

White

63

67

International

5

4

Other race/ethnicity

6

5

Sex

Female

67

63

Male

33

37

Parental Education

Less than high school

5

5

High school

16

19

Some college

11

11

Associate's

9

10

Bachelor's

29

28

Master's

22

19

Doctoral or professional

8

8

Major field

Arts & humanities

9

10

Biological sciences

12

9

Physical sci., math, & computer sci.

6

5

Social sciences

11

13

Business

15

17

Communications

4

4

Education

7

7

Engineering

7

7

Health professions

16

15

Social service professions

5

5

All other

4

7

Undecided, undeclared

3

<1

Basic 2010 Carnegie Classification (aggregated)

Research universities

42

45

Master's colleges and universities

43

42

Baccalaureate colleges—Arts & Sciences

6

5

Baccalaureate colleges—Diverse Fields

6

5

Other/not classified

3

3

Control

Public

68

68

Private

32

32

Note: Values are percentages and may not sum to 100 due to rounding. Sample is unweighted.

Table 2

Distribution of the Experiences with Information Literacy module items by class level

Variable name

Item

Response option

First-year (%)

Senior (%)

During the current school year, about how often have you done the following?

INL01A

Completed an assignment that used an information source (book, article, Web site, etc.) other than required course readings

Never

2

2

Sometimes

21

15

Often

37

30

Very often

40

53

INL01B

Worked on a paper or project that had multiple smaller assignments such as an outline, annotated bibliography, rough draft, etc.

Never

5

8

Sometimes

26

26

Often

39

32

Very often

31

34

INL01C

Received feedback from an instructor that improved your use of information resources (source selection, proper citation, etc.)

Never

5

9

Sometimes

28

29

Often

40

34

Very often

27

29

INL01D

Completed an assignment that used the library’s electronic collection of articles, books, and journals (JSTOR, EBSCO, LexisNexis, ProQuest, etc.)

Never

18

13

Sometimes

34

26

Often

28

26

Very often

20

35

INL01E

Decided not to use an information source in a course assignment due to its questionable quality

Never

25

27

Sometimes

40

39

Often

24

22

Very often

11

13

INL01F

Changed the focus of a paper or project based on information you found while researching the topic

Never

16

17

Sometimes

43

44

Often

29

26

Very often

12

14

INL01G

Looked for a reference that was cited in something you read

Never

16

13

Sometimes

37

35

Often

31

31

Very often

15

21

INL01H

Identified how a book, article, or creative work has contributed to a field of study

Never

20

17

Sometimes

38

36

Often

28

28

Very often

14

18

During the current school year, how much have your instructors emphasized the following?

INL02A

Not plagiarizing another author’s work

Very little

2

3

Some

7

9

Quite a bit

21

21

Very much

70

66

INL02B

Appropriately citing the sources used in a paper or project

Very little

2

4

Some

9

11

Quite a bit

27

25

Very much

61

61

INL02C

Using scholarly or peer-reviewed sources in your course assignments

Very little

4

6

Some

15

13

Quite a bit

30

25

Very much

51

56

INL02D

Questioning the quality of information sources

Very little

6

9

Some

20

22

Quite a bit

32

28

Very much

43

41

INL02E

Using practices (terminology, methods, writing style, etc.) of a specific major or field of study

Very little

8

6

Some

24

18

Quite a bit

30

29

Very much

37

47

Notes: Percentages may not sum to 100% due to rounding.

Analyses

Following NSSE’s standard practices, I conducted all of the following analyses for first-year and senior students, separately. The initial analyses conducted an exploratory factor analysis (EFA) to identify information-literacy factors in the Experiences with Information Literacy module. I randomly sampled half of the students for use in the EFA (to preserve data for use in subsequent analyses). The Kaiser-Meyer-Olkin measure of sampling adequacy was .90, which Kaiser15 described as “marvelous.” Due to the ordinal nature of the module variables, the EFA used a diagonally weighted least squares estimator, as it does not assume that the variables are continuous.16 I identified the appropriate number of factors through an examination of the scree plot of the extracted factors eigenvalues17 and practical experiences based on the development of the module. The factor loadings were rotated using an oblique, oblimin rotation with a Kaiser normalization.

After identifying an initial set of latent factors in the module, I conducted a confirmatory factor analysis (CFA) to verify the factor structure of the underlying data. CFAs differ from EFAs in that the factor structure is hypothesized by the researcher rather than derived from the data. The CFA was performed using the half of the sample not used in the EFA. Like the EFA, the analysis used a diagonally weighted least squares estimator, as the module variables were ordinal.18 The initial structure of the CFA model was based upon the EFA results. I allowed the latent factors to correlate as the EFA results suggested the factors were significantly correlated. The model was subsequently refined based upon the model fit information and modification indices. I used the following model fit information to assess whether the CFA met generally accepted standards. These criteria were a root mean square error of approximation (RMSEA) not significantly different from .05 or lower, comparative fit index (CFI) of .95 or higher, Tucker‐Lewis Index (TLI) of .95 or higher.19

I assessed the reliability of the factors identified by calculating Cronbach’s α coefficient for each of the factors. Finally, I investigated the predictive validity of the factors by examining their correlation with NSSE’s Higher-Order Learning and Reflective & Integrative Learning Engagement Indicators and the perceived gains scale, holding constant other characteristics. In these analyses, I standardized the factors to have a mean of 0 and a standard deviation of 1. To estimate the predictive validity of the factors, I estimated two regression models for each of the three outcome measures. The first model contained the control variables previously described and school-level fixed effects to account for the institution attended. The second model added the information literacy factors. This two-step approach allowed me to investigate how the inclusion of the information literacy factors improved the prediction of these outcomes.

Limitations

Readers should keep in mind the study’s limitations when interpreting the results. While the sample is large and diverse, it was a convenience sample and may not be generalizable to the larger population of undergraduates in the United States. Institutions choose to administer NSSE and the Experiences with Information Literacy module to their students. Thus, the average institution is most likely more committed to assessment and evaluation and information literacy than the typical four-year institution. For example, research universities were overrepresented and baccalaureate colleges were underrepresented in the sample. Additionally, the data were self-reported by students and maybe subject to social desirability bias. Finally, the module focused on engagement in information literacy activities, but it does not investigate the quality of information experiences. For example, an instructor could frequently instruct their students to use peer-reviewed sources, but this emphasis without instructing students how to access and identify peer-reviewed articles is of low quality.

Results

Exploratory Factor Analysis

I began the analyses by estimating an EFA using a random half of the sample for both first-year and senior students discretely. An analysis of the scree plots for both class levels indicated that three factors should be extracted from the data. The rotated loadings for the three factors are available by class in table 3. After analyzing the results, I titled the factors: information use, information evaluation, and instructors’ emphasis. The first four items in the module loaded onto the information use factor. These items inquired about how often students used various information sources in their coursework or received feedback from an instructor on their information use. The second factor, information evaluation, contained the four remaining items in the first question set. These items focused on how students evaluated and used information sources to conduct a richer literature review or revise their topic of study. The five items in the second question all loaded onto the third factor: instructors’ emphasis. These items all inquired about specific activities that instructors can emphasize to improve students’ use of information. One substantial cross-loading greater than .30 was observed for first-year students, but not seniors—the item asking if the student “Completed an assignment that used the library’s electronic collection of articles, books, and journals” loaded onto both the information use and evaluation factors. However, the loading with the larger magnitude was for information use, which comported with seniors’ results. The correlations between the factors ranged from .32 to .49 for first-year students and .37 to .54 for seniors, indicating that the three factors capture independent, but correlated, phenomena.

Table 3

Exploratory factor analysis factors and rotated loadings by class level

First-year

Senior

Variable

Item

Information Use

Information Evaluation

Instructors’ Emphasis

Information Use

Information Evaluation

Instructors’ Emphasis

INL01A

Completed an assignment that used an information source other than required course readings

.601

.615

INL01B

Worked on a paper or project that had multiple smaller assignments

.805

.806

INL01C

Received feedback from an instructor that improved your use of information resources

.587

.618

INL01D

Completed an assignment that used the library’s electronic collection of articles, books, and journals

.359

.341

.424

INL01E

Decided not to use an information source in a course assignment due to its questionable quality

.617

.660

INL01F

Changed the focus of a paper or project based on information you found while researching the topic

.684

.717

INL01G

Looked for a reference that was cited in something you read

.720

.663

INL01H

Identified how a book, article, or creative work has contributed to a field of study

.692

.601

INL02A

Not plagiarizing another author’s work

.785

.843

INL02B

Appropriately citing the sources used in a paper or project

.855

.901

INL02C

Using scholarly or peer-reviewed sources in your course assignments

.830

.813

INL02D

Questioning the quality of information sources

.848

.817

INL02E

Using practices of a specific major or field of study

.655

.659

Notes: Item labels shortened. Factor loadings were rotated using an oblimin rotation with a Kaiser normalization. Loadings with an absolute value less than .30 are not displayed.

Confirmatory Factor Analysis

With the remaining portion of the sample, I conducted a CFA. Figure 1 contains the initial conceptual model between the three factors. As shown in the conceptual model, the three factors are presumed to be latent variables measured by the specific survey items from the module. Additionally, the three latent factors were allowed to correlate, as the EFA results indicated a significant correlation between the factors. Despite the cross-loading observed for first-year students on the use of electronic collections, I attributed this variable to the information use factor due to the higher loading observed for the item on the information use factor and the results from the senior EFA model. When analyzed, the initial conceptual model did not adequately fit the data according to the model fit information. The modification indices indicated that the model needed to account for residual correlations between some of the items. After allowing some of the items to correlate, an adequate model fit was achieved. The RMSEA was .051, 90% CI [.050, .052], the CFI was .988, and the TLI was .983 for the first-year sample. For seniors, the RMSEA was .051, 90% CI [.050, .052], the CFI was .989, and the TLI was .985. All three criteria met the generally accepted standards for CFAs for both classes.20 Figures 2 and 3 contain the final model and standardized factor loadings between the latent and observed variables for first-year and senior students, respectively.

Figure 1

Confirmatory Factor Analysis
Conceptual Model

Figure 1. Confirmatory Factor Analysis Conceptual Model

Note: Error terms not displayed

Figure 2

Confirmatory factor analysis final model for first-year students

Notes: Standardized loadings. Standard errors in parentheses. Error terms not displayed.

Figure 3

Confirmatory factor analysis final model for senior students

Figure 3. Confirmatory factor analysis final model for senior students

Notes: Standardized loadings. Standard errors in parentheses. Error terms not displayed.

Construct Reliability

After confirming the construct validity of the three factors in the model, I estimated the reliability of the three factors by calculating the Cronbach’s α for each composite variable. For first-year students, the αs for information use, information evaluation, and instructors’ emphasis were .77, .80, and .85, respectively. For seniors, the coefficients were .78, .79, and .86 for information use, information evaluation, and instructors’ emphasis, respectively. All α coefficients exceed the generally accepted threshold of .70 used for social science research and thus considered reliable for use in research contexts.

Predictive Validity

Finally, I assessed the predictive validity of the three factors by estimating regression models predicting the following NSSE composite variables: Higher-Order Learning, Reflective & Integrative Learning, and Perceived Gains. For each outcome, I first estimated a regression model that included the aforementioned control variables and institution-specific fixed effects. The second model added the three information literacy factors. I estimated these models separately for first-year and senior students. As the dependent variables and information literacy factors were standardized, the results represent the expected standard deviation change in the dependent variable for a standard deviation change in an independent variable.

Table 4 contains the regression coefficient estimates of the information literacy factors when predicting the outcome measures, holding constant student and institutional characteristics. For each of the three outcome measures, each of the information literacy factors uniquely predicted a significant proportion of the variation in the Higher-Order Learning, Reflective & Integrative Learning, and Perceived Gains scales. Additionally, these relationships were all positive and not trivial, as all of the coefficient estimates were greater than 0.09. Therefore, a standard deviation change in one of the information literacy factors would be expected to result in roughly a tenth to quarter standard deviation change in Higher-Order Learning, Reflective & Integrative Learning, and Perceived Gains. Despite the significant intercorrelation of the information literacy factors, each of the factors exerted a unique and significant influence on the outcomes. Therefore, the effects of more engagement in different types of information literacy activities appear to be additive. Finally, as demonstrated by the R2 change and final R2 statistics, the information literacy factors accounted for 17 to 26 percent of the variance in the outcome measures. This finding indicates that information literacy activities play an essential role in students’ learning and their perceived gains.

Table 4

Fixed effect estimates of the relationship between information literacy factors and selected outcomes by class level

Higher-Order Learning

Reflective & Integrative Learning

Perceived Gains

 

Est.

Sig.

Est.

Sig.

Est.

Sig.

First-year students

Information Use

0.20

***

0.15

***

0.21

***

Information Evaluation

0.14

***

0.24

***

0.19

***

Instructors’ Emphasis

0.19

***

0.13

***

0.24

***

R2 change

.18

.17

.26

Final R2

.20

 

.21

 

.28

 

Seniors

Information Use

0.23

***

0.19

***

0.25

***

Information Evaluation

0.12

***

0.21

***

0.09

***

Instructors’ Emphasis

0.19

***

0.13

***

0.24

***

R2 change

.19

.21

.21

Final R2

.22

 

.26

 

.26

 

* p < .05, ** p < .01, *** p < .001

Notes: Models hold constant race/ethnicity, time spent working, major field, grades, transfer status, educational aspirations, parental education, age, and enrollment status. Models include institution-specific fixed effects. The information literacy factors and outcome measures were standardized with a mean of 0 and standard deviation of 1. R2 change is the amount of additional explained variance by the information literacy factors after holding accounting for student and institutional factors. Final R2 is the total amount of variation in the dependent variable explained by the model.

Discussion

In this study, I sought to investigate the validity and reliability of the Experiences with Information Literacy module using a large multi-institutional sample of first-year and senior students. To answer the first research question, I identified factors within the module through an EFA. The EFA results indicated that there were three latent factors within the module. I titled the factors Information Use, Information Evaluation, and Instructors’ Emphasis. Information Use focuses on students’ basic use of information in their course assignments. Information Evaluation focuses on a deeper and richer use of information by students through activities like assessing information quality, changing the focus of a paper based upon a literature review, and identifying the uniqueness of an information source. The final factor, Instructors’ Emphasis, examined the extent to which instructors encouraged the proper use of information in their courses. I subsequently estimated a CFA to confirm that the proposed factors adequately fit the data using a CFA to answer the second research question. After accounting for residual correlations between some of the items, the CFA results adequately fit the data and supported the findings of three information literacy factors identified by the EFA. The third research question asked if the three factors could be reliably estimated. The factors were demonstrated to be reliable, as their Cronbach’s α coefficients ranged from .77 to .86. The fourth question focused on the predictive validity of the three factors in relation to engagement in higher-order learning and reflective and integrative learning activities. After controlling for student characteristics and the institution attended, each of the three factors was significantly and positively correlated to student engagement in these domains. The three information literacy factors also increased the R2 by about .20, indicating that the factors accounted for a substantial portion of the variation in student engagement related to higher-order learning and reflective and integrative learning. The final research question similarly inquired about the predictive validity of the factors in relationship to students’ perceived gains. The results were similar to the engagement findings in that the information literacy factors were significantly and positively related to perceived gains and that the factors accounted for a relatively high proportion of the variance explained.

Implications

As validity is a multifaceted concept,21 the results from the study combine to indicate that the Experiences with Information Literacy module is a psychometrically valid instrument that can be used to assess undergraduates’ engagement with information literacy skills. The results supported the continued and expanded use of the module by institutions and instructional librarians in their assessment and evaluation activities. The module is relatively novel in that it focuses on the process of how students develop information literacy skills rather than directly testing their knowledge, thus adding a new method of assessment for instructional librarians. This feature of the module allows for users to identify specific areas of improvement. For example, an institution’s module results may reveal that instructors placed less emphasis on questioning the quality of information sources. In response, instructional librarians may work with faculty members to emphasize information quality in their lessons, grading, and syllabi. NSSE also provides institutions with the raw data to compare students by their characteristics. Such a subgroup analysis may indicate that instructors in a particular discipline placed less emphasis on the use of scholarly sources, and instructional librarians may work with faculty in that discipline to increase the emphasis on this topic.

The validation of the module also leads to multiple lines of future inquiries. First, what student and institutional characteristics are correlated with information use and evaluation and instructors’ emphasis? For example, how does information literacy engagement vary by academic field or parental education? Similarly, are factors like the student-to-librarian ratio or selectivity related to information engagement? Second, how does participation in library instructional sessions influence students’ information engagement? The module data could demonstrate the long-term effectiveness of information literacy programming. Third, does information literacy engagement have compensatory effects on outcomes like retention and GPA? Prior work by NSSE researchers has shown that more general forms of engagement are especially effective in promoting the retention and GPA of underrepresented minority students;22 does this finding hold for information-related engagement? Finally, given the increasing difficulty of determining high-quality news sources, does information-literacy engagement during college assist students to identify and evaluate news sources in their postcollegiate lives?

Conclusion

The study’s results demonstrate that the NSSE Experiences with Information Literacy module is an assessment tool meeting generally accepted standards for validity in social science research. They go on to highlight the significant and positive relationship between first-year and senior students’ engagement in information literacy activities and both Higher-Order Learning and Reflective and Integrative Learning. Additionally, engagement in information literacy activities was positively correlated with students’ self-assessments of their college learning gains, providing additional evidence of the importance of information literacy as a learning outcome. Finally, the study supports the use of the module by both institutions and librarians to assess and improve information literacy outcomes for students.

APPENDIX. Items Comprising the Higher-Order Learning and Reflective & Integrative Learning Engagement Indicators and Perceived Gains Scale

Higher-Order Learning

During the current school year, how much has your coursework emphasized the following:

[Response options: Very little, Some, Quite a bit, Very much]

  • Applying facts, theories, or methods to practical problems or new situations
  • Analyzing an idea, experience, or line of reasoning in depth by examining its parts
  • Evaluating a point of view, decision, or information source
  • Forming a new idea or understanding from various pieces of information

Reflective & Integrative Learning

During the current school year, how often have you:

[Response options: Never, Sometimes, Often, Very Often]

  • Combined ideas from different courses when completing assignments
  • Connected your learning to societal problems or issues
  • Included diverse perspectives (political, religious, racial/ethnic, gender, etc.) in course discussions or assignments
  • Examined the strengths and weaknesses of your own views on a topic or issue
  • Tried to better understand someone else’s views by imagining how an issue looks from his or her perspective
  • Learned something that changed the way you understand an issue or concept
  • Connected ideas from your courses to your prior experiences and knowledge

Perceived Gains

How much has your experience at this institution contributed to your knowledge, skills, and personal development in:

[Response options: Very little, Some, Quite a bit, Very much]

  • Writing clearly and effectively
  • Speaking clearly and effectively
  • Thinking critically and analytically
  • Analyzing numerical and statistical information
  • Acquiring job- or work-related knowledge and skills
  • Working effectively with others
  • Developing or clarifying a personal code of values and ethics
  • Understanding people of other backgrounds (economic, racial/ethnic, political, religious, nationality, etc.)
  • Solving complex real-world problems
  • Being an informed and active citizen

Notes

1. For example: Jessica Taylor and Danielle Kurtzleben, “This Week in Trump’s ‘Alternative Facts,’” National Public Radio (last modified Jan. 29, 2017), available online at www.npr.org/2017/01/29/512068148/this-week-in-trumps-alternative-facts [accessed 5 February 2017]; John Wagner, “Trump Rants about ‘Fake News’ as He Marks Black History Month,” Washington Post (last modified Feb. 1, 2017), available online at https://www.washingtonpost.com/news/post-politics/wp/2017/02/01/trump-rants-about-fake-news-as-he-marks-african-american-history-month/ [accessed 5 February 2017].

2. American Library Association, “Presidential Committee on Information Literacy: Final Report,” American Library Association, available online at www.ala.org/acrl/publications/whitepapers/presidential [accessed 1 February 2017]; Kathleen L. Spitzer, Michael B. Eisenberg, and Carrie A. Lowe, Information Literacy: Essential Skills for the Information Age (Syracuse, NY: ERIC Clearinghouse on Information & Technology, Syracuse University, 1998); Association of College and Research Libraries, “Information Literacy Competency Standards for Higher Education” (Chicago, IL: Association of College and Research Libraries, 2000); Todd J. Wiebe, “The Information Literacy Imperative in Higher Education,” Liberal Education 102, no. 1 (2016).

3. Joseph R. Matthews, The Evaluation and Measurement of Library Services (Westport, CT: Libraries Unlimited Incorporated, 2007); Association of College & Research Libraries, The Value of Academic Libraries: A Comprehensive Research Review and Report. Researched by Megan Oakleaf (Chicago, IL: Association of College & Research Libraries, 2010); María Pinto, “Viewing and Exploring the Subject Area of Information Literacy Assessment in Higher Education (2000–2011),” Scientometrics 102, no. 1 (2015).

4. Kevin Fosnacht, “Information Literacy and NSSE Introducing the Experiences with Information Literacy Module,” College & Research Libraries News 75, no. 9 (2014).

5. Indiana University Center for Postsecondary Research, “The Carnegie Classification of Institutions of Higher Education, 2015 Edition” (Bloomington, IN: Indiana University Center for Postsecondary Research, n.d.); National Survey of Student Engagement, “About NSSE,” National Survey of Student Engagement, available online at http://nsse.indiana.edu/html/about.cfm [accessed 1 February 2017].

6. Alexander C. McCormick, Jillian Kinzie, and Robert M. Gonyea, “Student Engagement: Bridging Research and Practice to Improve the Quality of Undergraduate Education,” in Higher Education: Handbook of Theory and Research, ed. Michael B. Paulsen (Dordrecht, The Netherlands: Springer, 2013).

7. Alexander W. Astin, “Student Involvement: A Developmental Theory for Higher Education,” Journal of College Student Personnel 25, no. 4 (1984); C.R. Pace, Measuring the Quality of College Student Experiences: An Account of the Development and Use of the College Student Experiences Questionnaire (Los Angeles, CA: Higher Education Research Institute, 1984).

8. Arthur W. Chickering and Zelda F Gamson, “Seven Principles for Good Practice in Undergraduate Education,” AAHE Bulletin 39, no. 7 (1987).

9. G.D. Kuh et al., Involving Colleges: Successful Approaches to Fostering Student Learning and Development outside the Classroom (San Francisco, CA: Jossey-Bass, 1991).

10. Samuel Messick, “Validity,” in Educational Measurement, 3rd edition, ed. Robert L. Linn (New York, NY: American Council on Education and Macmillan, 1989), 13–103.

11. Fosnacht, “Information Literacy and NSSE Introducing the Experiences with Information Literacy Module.”

12. Association of American Colleges & Universities, “Information Literacy Value Rubric,” Association of American Colleges & Universities, available online at https://www.aacu.org/value/rubrics/information-literacy [accessed 1 February 2017]; Association of College and Research Libraries, “Information Literacy Competency Standards for Higher Education.”

13. Kevin Fosnacht et al., “How Important Are High Response Rates for College Surveys?” Review of Higher Education 40, no. 2 (2017).

14. National Survey of Student Engagement, “Psychomentric Portfolio,” available online at http://nsse.indiana.edu/html/psychometric_portfolio.cfm [accessed 10 February 2017].

15. Henry F. Kaiser, “An Index of Factorial Simplicity,” Psychometrika 39, no. 1 (1974).

16. Timothy A. Brown, Confirmatory Factor Analysis for Applied Research (New York, NY: Guilford, 2006).

17. Raymond B Cattell, “The Scree Test for the Number of Factors,” Multivariate Behavioral Research 1, no. 2 (1966).

18. Brown, Confirmatory Factor Analysis for Applied Research.

19. Li‐tze Hu and Peter M. Bentler, “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis: Conventional Criteria Versus New Alternatives,” Structural Equation Modeling: A Multidisciplinary Journal 6, no. 1 (1999).

20. Hu and Bentler, “Cutoff Criteria for Fit Indexes in Covariance Structure Analysis.”

21. Messick, “Validity.”

22. George D. Kuh et al., “Connecting the Dots: Multi-Faceted Analyses of the Relationships between Student Engagement Results from the NSSE, and the Institutional Practices and Conditions That Foster Student Success,” (Bloomington, IN: Center for Postsecondary Research, 2007).

* Kevin Fosnacht is Associate Research Scientist at the Indiana University Center for Postsecondary Research; email: kfosnach@indiana.edu. Author’s note: A previous version of this manuscript was presented at the 2017 meeting of the Association of College & Research Libraries in Baltimore, MD. ©2020 Kevin Fosnacht, Attribution-NonCommercial (http://creativecommons.org/licenses/by-nc/4.0/) CC BY-NC.

Copyright Kevin Fosnacht


Creative Commons License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

Article Views (Last 12 Months)

No data available

Contact ACRL for article usage statistics from 2010-April 2017.

Article Views (By Year/Month)

2020
January: 0
February: 1
March: 524
April: 50