Hernon and Dugan (2002) outline three levels of institutional outcomes to which libraries might contribute, a) easy to gather measures, b) lower order outcomes, and c) higher order
outcomes. The easy to gather measures include retention and graduation rates, preparation for the workplace as assessed by employers, and the number of graduates pursuing post-graduate
education. These measures could be correlated with library use but direct connections are difficult to make. Hernon and Dugan define lower order outcomes as evidence of skills learned, such as awareness of resource availability or skills in using information gathering tools. Higher order competencies are defined as critical thinking and problem solving skills. Evidence of connections to library services are yet even more difficult to make at this level. Hernon and Dugan also note that
“higher order competencies, require direct evidence of achievement … [and] their achievement can be neither assumed or inferred, rather the connection or association must be established” (Hernon & Dugan, 2002, p.103).
Writing in 1992, Ronald Powell exhorted researchers to develop methods for exploring and communicating the ways in which the academic library provides value to its users. He suggested several candidate outcomes including grades, critical thinking skills, communication skills, graduation, and retention (1992). Powell identified several methods useful for identifying outcomes including focus groups, interviews, and user panels. Poll recommends the use of both qualitative methods (surveys, interviews, focus groups, user self-assessments of gains) and
86
quantitative methods for “finding correlations between library use and a person’s academic success” including pre and post-tests, performance monitoring and data mining, unobtrusive observation, analysis of citation data over time, and comparison of data about user success (e.g. graduation, grades) with library use data (Poll, 2007, p. 33 – 34). Numerous authors suggest libraries focus on the assessment of information literacy outcomes (e.g. Smith, 2007; Hernon & Dugan, 2002).
In recent decades some progress has been made in identifying the ways undergraduates benefit from interactions with the library. The next sections of this paper review this literature. Outcomes considered include retention, academic performance, gains in information literacy skills, and the degree to which student-library interactions correlate with known ‘best-practices’ in higher education.
2.4.4.1 Retention
There is evidence that library use correlates with retention. Kramer and Kramer (1966) found a positive relationship between library use and persistence, as non-readers dropped out 40% more often than readers. Elizabeth Mezick (2007) used library expenditures, student FTE figures, and library staffing data from ARL statistics 2002-2003 and ACRL Library Trends and Statistics 2003 and retention data from the Integrated Postsecondary Educational Dataset (IPEDS) to explore correlations between library expenditures and student retention. She found moderate relationships between total library expenditures (r2 = 0.205, p < 0.001), total library materials (r2 = 0.237, p < 0.001), and expenditures on serials (r2 = 0.211, p < 0.001) and student retention. A major weakness to this study is that Carnegie Classification was the only control variable used in the analysis and the limited number of variables in the model. Stanley Wilder (1990) and Darla Rushing and Deborah Poole (2002) report a different type of library influence on undergraduate retention: holding a job in the library. Wilder suggested that working in the library “demystifies” library work, possibly reducing anxiety associated with library research and being associated with academic role models could be contributors to retention (1990). Rushing and Poole (2002)
87
reported that efforts to provide students with meaningful work and involving them in the life of the library may be factors contributing to retention. The finding that holding a student job in the library is tied to retention is in alignment with Vincent Tinto’s Theory of Student Departure which suggests that retention is related to the degree to which a student integrates into the academic and social life of the campus (Tinto, 1975, 1993). Holding a campus job, especially in an organization that supports the academic mission, is a form of integration that has been found to be related with “higher levels of effort and involvement” including “use of the library,
interactions with faculty, involvement in learning through coursework, writing experiences and activities, [and] other curricular and extracurricular activities“ as well as higher estimates of self- reported gains (Aper, 1994).
2.4.4.2 Objective measures of academic performance
Several pre-1990 studies as reviewed by Charles Harrell (1988) investigated grade point average as a predictor of library use, and without exception found a low if significant
relationship. Harrell (1988) in his doctoral dissertation studied the role of several independent variables as predictors of library use and he too found that grade point average was not a significant predictor of library use. Several studies investigating this issue have switched the dependent and independent variables by exploring the linkages between student use of library materials and academic performance and found weak to no correlation. Jane Hiscock (1986) examined the relationship of library use to grades and found only that use of the library catalog was a predictor of academic achievement. James Self (1987) found that use of reserve materials was not a useful predictor of academic performance. Karin de Jager (1997) investigated linkages between student use of materials on reserve for a given class and those students’ final exam grades. In de Jager’s study she compared short loan (reserves) and open shelf circulation records with final exam grades for students in 2 History courses, one economics course, and one group of sociology students. A significant positive correlation was found between open-shelf reading patterns and final exam grades for the two history groups and the sociology group. However,
88
there was no correlation between use of the short loan collection and final exam grades for any of the groups. Shun Han Rebekah Wong and T.D. Webb reported on a large-scale study with a sample of over 8,700 students grouped by major and level of study. In sixty-five percent of the groups, they found a positive relationship between use of books and A/V materials borrowed from the library and GPA (2011). And in a large-scale study, researchers at the University of Huddersfield (Stone, et al., 2011, 2012) using data from eight universities demonstrated correlations between book checkouts and electronic article downloads and graduating with honors.
2.4.4.3 Library skills and information literacy as dependent variables
Libraries have provided instruction to users for generations, but an emphasis on library instruction picked up steam in the 1990s in part to instruct users how to use new digital resources. Some authors argue that more instruction is needed (Head & Eisenberg, 2009) while others argue that as information systems become easier to use instruction will no longer be needed (Wilder, 2005). Interest in information literacy outcomes emerged in the 1990s in parallel or perhaps in response to calls for learning outcomes assessment in higher education institutions (American Library Association, 1989). The Information Literacy Competency Standards for Higher Education issued by the Association of College and Research Libraries (American Library Association, Association for College and Research Libraries, 2000) codified information literacy learning outcomes for academic libraries and established criteria for assessment. Megan Oakleaf (2008) identified three main categories of information literacy assessment gains: fixed choice tests, performance assessments, and rubrics. Affective impacts and changes in behavior are other impacts of information literacy instruction that have been measured.
Fixed choice tests
Changes in students’ information seeking abilities are most often assessed through locally developed tests. A few examples are shared here. Donald Barclay (1993) constructed a local test
89
using free response questions in combination with a survey to evaluate the increase in library use skills and attitudes following library instruction to freshman writing classes. Elizabeth Carter (2002) also used locally developed tests and survey questions to measure the effectiveness of library instruction for students in a Psychology class. Barclay found improvement from pre-tests to post-tests, but Carter did not. Chris Portmann and Adrienne Rousch (2004) conducted similar research using pre-tests and post-tests investigating the utility of a one-hour library instruction for community college students in a Sociology class and found there was no increase in skills
following the session. Heidi Julien and Stuart Boon (2004) assessed through pre-test, post-test, and post-post-test the impact of one-hour library instruction sessions with six groups at five different institutions. Statistically significant increases were found for four groups between the pre-test and post-test, however, one group showed a statistically significant decline in the post- post-test administered three to four months after the instruction.
The high levels of effort and skills required for the local development of reliable tests is a significant obstacle to continuous assessment of library instruction efforts. Standardized tests developed by professional testing agencies should reduce overall costs and increase reliability. Two such tests are worth mentioning, though results of their utility in the field are still emerging. Hosted at Kent State, the Standardized Assessment of Information Literacy Skills (SAILS) is a web-based testing instrument designed to answer three questions: (1) Are students information literate? (2) Does the library contribute to information literacy? and (3) Does information literacy contribute to retention and academic success (Kent State University, 2008). The test questions are based on the ACRL Information Literacy Competency Standards for Higher Education and have been tested and validated (ACRL, 2000). To date however, few institutions have used the tool to assess student learning of information literacy. The Educational Testing Service has developed, iSkills, a standardized test available for subscription by interested institutions
(Educational Testing Service, 2008). Institutions such as the University of Central Florida (Beile, 2008), Colorado State University Libraries (2008), and California State University system
90
(Brasley, 2008) have used the tool to establish baseline data regarding students’ information literacy skills and set information literacy curriculum goals. Researchers in library impact should monitor applications of iSkills and assess it as a valid measure of information literacy outcomes. Performance assessment
Megan Oakleaf (2008) defines performance assessment as “qualitative forms of assessment that require students to perform real-life applications of knowledge and skills” (p. 239). Performance assessment can be supported through the investigation of portfolios of student papers (Barclay, 1993). In this approach, student portfolios of finished academic work are analyzed to determine if the quality of resources cited has improved following library instruction. Unlike testing, this approach investigates the application of information literacy skills in the execution of academic work. Mark Emmons and Wanda Martin (2002) describe a representative “evidence of use” study conducted at the University of New Mexico that explored the
effectiveness of a library instruction program launched in 1999 for a writing course for first-year students. In their study, they analyzed 250 research portfolios from the first-year writing seminar collected from 1996 to 2001. Papers were evaluated by citation distribution (number, format, age, and accuracy) in the first phase of analysis. In the second phase of analysis, papers were evaluated for relevance, credibility (students are critical of sources and their author’s viewpoints), and engagement (student supports arguments with evidence, challenges source’s ideas). Emmons and Martin found no significant improvements over time in the portfolios they assessed and saw the need to revise their instructional methods (Emmons and Martin, 2002). Loanne Snavely and Carol Wright (2003) report on an effort to use ‘research portfolios’ to assess student performance in a semester long library research credit course for honor’s students. The authors found the portfolio enhanced their ability to grade student’s work and students reported that the use of the portfolio offered opportunities to reflect on their work through the semester.
91 Rubrics
Oakleaf (2008) describes rubrics as “descriptive scoring schemes to guide analysis of student work” (p. 245). Rubrics describe expectations for specific tasks associated with a
learning objective and may describe high, medium, and low performance for those tasks. Rubrics are shared with students to provide guidance when completing the assignment and then faculty use rubric to judge the quality of completed student work (Oakleaf, 2008, p. 246). Relatively few librarians have used rubrics to assess information literacy efforts. Elizabeth Choinski, Amy Mark, and Missy Murphy (2003) used a rubric to analyze students’ reflection papers to assess information seeking skills and found students performed poorly on items requiring higher order thinking such as evaluating web sites or differentiating between popular and scholarly resources. Emmons & Martin also used rubrics in their effort reported above. Davida Scharf et al. (2007) describe a project creating and applying an information literacy rubric at the New Jersey Institute of Technology that drew criteria from the ACRL standards for information literacy. Scores on the information literacy rubric were correlated (r= .497, p<.01) with scores on a writing rubric used by faculty who teach the seminar. However, no significant associations were found between their model and SAT scores or GPA. Lorrie Knight and Kimberly Lyons-Mitchell (2006) graded student bibliographies using a rubric tied to the ACRL Information Literacy Standards. The method simplified grading, offered opportunities for collaborating with faculty, and provided students with “up-front clearly defined levels of success.” Challenges though include the amount of time it took to develop the rubric and the amount of time it took to rate each student’s
bibliography.
Affective and behavioral changes
Several studies seek to connect student participation in library instruction to positive affective outcomes and changes in behavior. Heidi Julien and Stuart Boon (2004) for instance followed up their testing with interviews of students who had participated in library instruction. Eighty two % of respondents reported improved confidence when performing research related to
92
either an increase in skills or a willingness to ask questions when needed (p. 130). Though Julien and Boon found no increase in information literacy skill levels among students who participated in a 50-minute library instruction class, those students did report they would be more likely to use the library in the future for their research (2004). While JoAnn Jacoby and Patricia O’Brien’s report (2005) is not strictly a study of library instruction, they did study the impact of reference services to undergraduate students through a survey distributed to students using the Social Sciences library at the University of Illinois at Urbana-Champaign. Students were asked to rate their confidence in their ability to find information independently” before and after the reference encounter on a Likert scale from 1 – 6. Paired t-tests were conducted and the mean difference in confidence (1.65) was found to be statistically significant (t(68) = 8.26, p < 0.0001). Open-ended survey responses reinforced the role the reference encounter played in increasing the students’ confidence and multiple regression analysis revealed that the approachability of the librarians contributed to gains in confidence.
Clearly, students are in need of help in learning how to find, evaluate, and use
information (Head & Eisenberg, 2009) and library instruction should be an essential component of the academic library outreach strategy today (Lewis, 2007). However, promoting information literacy outcomes as the library’s sole contribution to undergraduate student learning is
problematic and sells the library short for several reasons. First, current assessment approaches use cross-sectional approaches to assess changes in skill levels due to exposure to instruction in a single session, over a single semester, or in an academic year. Experts in college impact note that some learning gains may not be manifest until the conclusion of the college experience or even afterwards (Astin, 1973; Gonyea, & Kuh, 2003; Pascarella & Terenzini, 2005). As illustrated above, these efforts have yielded mixed results. Second, the literature suggests that information literacy skills are very closely related to other general education and discipline-specific skills taught and assessed by faculty. If these skills are interrelated, it will be difficult or impossible for a single assessment tool to determine where faculty influence ends and library influence begins.
93
Finally, it is unclear if information literacy skills are universally recognized by stakeholders in higher education. Information literacy outcomes are recognized as such in some standards such as the AAC&U’s list of Essential Learning Outcomes (2007). Yet, as Megan Oakleaf reported in 2011, other standards do not explicitly cite information literacy outcomes but refer to similar skills using different terms such as critical thinking. Laura Saunders found three of six regional accreditation agencies specifically name information literacy as a desired outcome and assert the library’s prominent role in information literacy instruction and assessment of related skills. Others rarely use the term “information literacy” in their standards. Instead, competencies such as “evaluating and using information ethically” appear in these standards as general education outcomes to be taught and assessed throughout the college curriculum (2007). Megan Oakleaf and Neal Kaske (2009) reported similar findings.
2.4.4.4 Exploring self-reported gains in learning and cognitive development
The College Student Experiences Questionnaire (CSEQ), discussed in section 2.1 and 2.2, is a means of measuring from student reports both the amount and quality of effort that students expended in their college coursework (Pace, 1984; Trustees of the University of Indiana Bloomington, 2012a). Several authors have had varying degrees of success using CSEQ data to understand effects of interactions with the library on self-reported gains in learning and cognitive development. Lamont Flowers (2004) explored the self-reported experiences of 7,923 African- American students who completed the CSEQ between 1990 and 2000 and found that using the library as a place to study had a positive effect on gains on the Personal and Social Development scales, Thinking and Writing Skills Scale, and the Vocational Preparation Scale. When they developed a bibliography for a paper, gains were achieved in the Arts and Humanities Scale, Personal and Social Development scale, Thinking and Writing Skills scale, and the Vocational Preparation scale. Browsing in the stacks and locating items referred to by another author were associated with gains in all five scales except the Personal and Social Development scale.
94
Ethelene Whitmire (1998) used CSEQ data gathered in 1992-1993 from 18,157 students to study relationships between students’ use of the library and self-reported gains in critical thinking skills. Use of the library, on its own, was not a significant predictor of gains in critical thinking. However, students engaged in “focused library activities” such as checking citations, reading basic references or documents, browsing the stacks, and checking out books reported significantly higher gains in critical thinking (b = 0.03, p < 0.001). Students more active in engaging faculty regarding their academic work (b = 0.03, p < 0.001) and those engaged in active learning (b = 0.26, p < 0.001) were also more likely to report higher gains in critical thinking scores. Whitmire examined this relationship again during the 1996 IPEDS and CSEQ data sets (Whitmire, 2002b). In this study of 7,958 students from 32 institutions of all types she found two variables that were significant predictors of self-reported critical thinking gains: academic year and engagement in active learning experiences. There was a slight positive correlation between academic library resources and self-reported critical thinking skills at Research Universities, but no relationship was found between the use of library resources and perceived gains in critical thinking skills. This study is weakened by the fact that most of the students in the study were first and second year students, who may not have been given assignments that required use of the library (Whitmire, 2002b).
Robert Gonyea and George Kuh (2003) used CSEQ data to investigate changes in library use over time and the effects of library use on three outcome variables: (1) gains in information