On June 6, the City University of New York (CUNY) held its first library assessment conference. Called Reinventing Libraries: Reinventing Assessment, the event grew from its initial target of 100 attendees to almost twice that many, and positive feedback from many attendees included calls for the conference to be repeated, or even turn into an annual event. In its full sessions and breakouts, the gathering addressed the many strands that make up assessment, measuring topics from ebooks and journals to faculty productivity to physical plant, and ranging from big picture visions to the nitty gritty of turning findings into infographics for effective presentation. Several recurring themes, however, became leit motifs running throughout the day: turning from an emphasis on exclusively quantitative to qualitative assessment, libraries partnering with faculty on instruction, and the intersection of outcomes measurement and predictive analytics in a new granular portrait of individual students’ library use.

Qualitative Insights

“Libraries have taken the plunge into the deep end of library assessment,” said Steve Hiller, director of assessment and planning at the University of Washington Libraries, during his opening keynote address. This is evident both in the accumulation of data regarding library services and performance, and in the rapidly growing number of librarians who have “assessment” or user experience (UX) in their job description or title. The growth of assessment as a field has coincided with a period of transition for academic libraries. The central role that the library played in student learning was once taken for granted, and quantitative data such as circulation and gate counts offered straightforward ways in which to measure library use. But as student needs and habits change, and as the library’s role evolves, quantitative metrics alone are no longer sufficient. Quantitative data still has a significant role to play in assessment, but qualitative research drawn from software and website usability studies, ethnographic field studies, one-on-one interviews, focus groups, or other sources can offer insights that raw data won’t reveal. “For advocacy, narratives combined with data can make a powerful case,” Hiller said, adding that qualitative information is “also an integral part of market research.” Describing the usability studies conducted by human-computer interaction engineer and UX pioneer Jakob Nielsen in the 1990s as “a real game changer,” Hiller explained that qualitative research often does not require a large sample size. “Basically, [Nielsen] said in dealing with usability of online systems you didn’t need to have lots of people,” Hiller said. “You could tell if something was working or not working from the customer perspective with relatively few people—six people, 10 people… It took us away from the quantitative research side with its sample size and response rates and so on.” Combining quantitative data with narratives gleaned from qualitative research will help libraries think of assessment in terms of what their collections and services enable people to do, and to view the library as a partner and facilitator within its institution. “In a sense, we’re moving out of the library “box” and into the institutional environment. That’s a critical thing to remember as we go through this,” Hiller said. “It’s not just about the library. The library plays a critical role, obviously, in education. But how do we enable others within the institution, or how do we contribute with others in the institution to the outcomes we want to see in student learning or research?” Sometimes librarians underestimate what they can offer in terms of academic partnerships, he later added. “We have the big picture, because we serve the entire community,” Hiller said. “We understand higher education, scholarly communication, the institution, and the programs at the institution. We have close links to academic programs through our liaisons which are the envy of others on campus, particularly in the support units. And we’re student centered.” Breakout sessions by Dr. Donna Lanclos, associate professor for anthropological research, J. Murrey Atkins Library, UNC Charlotte; Mariana Regalado, associate professor and head of reference and information services, Brooklyn College, CUNY; and Maura A. Smale, associate professor and coordinator of library instruction, New York City College of Technology, CUNY, offered an in-depth look at some of the ethnographic and observational techniques that are helping libraries develop a deeper understanding of how they are serving their students and their institutions. “Full-time or part-time qualitative projects are facilitating scholarly and policy discussions about the nature of information, the configuration of digital and physical spaces in academia, and I think the changing state of academic work and scholarly communication in the 21st century,” Lanclos said during her presentation titled The Mixed-Method Library: Qualitative Research and the Future of Assessment in Higher Education. Lanclos used observational research to map areas within the library that people were most likely to use for group discussion, quiet study, or other activities, including sleeping. One key discovery was that “we as an institution are asking [students] to work in groups, and we're not giving them the proper space to do this,” she said. So, a recent remodeling of the J. Murrey Atkins Library included the installation of modern conference rooms outfitted with glass walls, large desks for group discussion, whiteboards, and flatscreen televisions for presentations. Regalado and Smale discussed qualitative research that led to insights regarding the environments in which CUNY students study, and how those insights might be applied to improve CUNY libraries in their joint presentation “I like being under those rules here”: Students Using the College Library. The majority of CUNY students commute to their college, and most younger students live at home with their parents. Research involved asking students to take photos of the areas in which they regularly studied. Other students kept track of their daily travel with “mapping diaries.” These photo surveys and mapping diaries revealed that these students deal with distracting workspaces when away from their college. Working on tables used for meals was common, and one student reported working in the hallway outside of her apartment. Understanding these environments helped clarify student complaints at some colleges regarding the use of library computer workstations for non-school-related functions, such as Facebook, or complaints about rules and expectations for noise not being enforced at some libraries. “When I go home, I get distracted easily,” one student wrote. “Here it's very quiet, you can't be loud, you can't talk, so I like being under those rules here.”

Assessing Information Instruction

Among the major themes of the conference was how to most effectively train students in how to use the resources that libraries provide. In one of the day’s first breakout sessions, Lisa Janicke Hinchliffe, Information Literacy Coordinator at the University of Illinois at Urbana-Champaign (UIUC), spoke about the steps her library is taking to better educate students and faculty about how to make the most of the library. The first step to setting up an effective training program, she said, was gathering information on how faculty and students perceive library services and what they need from the library, something UIUC has been doing since 2003, giving staff a decade of data to draw from.

The event was hosted at The William and Anita Newman Conference Center, Baruch College CUNY

One statistic that stuck out like a sore thumb in a series of surveys was that faculty and students largely agreed on one point—every incoming student should have to complete a library skills training program. After all, Hinchliffe pointed out, they are already required to complete a wide variety of similar trainings on issues like drug abuse awareness when they arrive at UIUC. Tellingly, the further along students were in their college career, the more likely they were to agree that library skill training should be mandatory for incoming students, suggesting that, as students move forward in their academic careers, they place more value on campus libraries. At UIUC, librarians don’t do tours anymore, highlighting what UIUC’s instructional services librarian Susan Avery said is a distinction the system has made between orientation and instruction. That doesn’t mean tours are unavailable at UIUC libraries, but they work differently than they have in the past. “If a faculty member or grad student wants a tour for their students, we give them a script,” Avery told the audience. “You don’t need a librarian for that.” Instead of showing people around the physical library, UIUC librarians focus on partnering with faculty, visiting classes as experts on information literacy, and taking a “guest lecturer” approach on how the library can be used for that course, rather than offering the simple library ‘how-tos’ that come standard with most class visits. Avery and Hinchliffe also emphasized the importance of working not just with tenured faculty, but with teaching assistants as well, in order to set a good model for young academics to see librarians as partners throughout their careers. In a later breakout, Indiana University (IU), Bloomington, Information Literacy and Assessment Librarian Brian Winterman discussed how he made the most of partnerships to drive information literacy in the school’s biology department. In such a large department, where communication between staff can leave a lot to be desired, Winterman pointed out that making just a couple of allies on the faculty side can lend library professionals credibility in the eyes of the department as a whole that can make their lives much easier as they develop and implement information literacy programs. In the IU biology department, Winterman and his colleagues set goals first, then worked backwards, doing a gap analysis to see where library training had room to improve. And while he emphasized the importance of goals, rubrics, and other measurement tools, he also reminded the audience that those tools alone can’t force a good program into place. “Sometimes it’s not about hard data,” he said. “It’s about changing the culture of a department and getting buy-in from faculty.” Data can help in that goal, but it is only one tool in a successful program. In her closing keynote, Debra Gilchrist, Vice President for Learning and Student Success at Pierce College, Lakewood and Puyallup, WA, also touched on the need for libraries to embed themselves in the instructional process, asking attendees “How are we proactive, rather than waiting for students to come to us?”, and urging librarians to present their data as integrated with the departments they support, rather than siloed. Gilchrist also emphasized the need for the library to embrace the larger institution’s goals and look for low factors where the library’s contribution can make a measurable impact. In particular, she suggested, libraries can make a major difference to students most at risk of dropping out by helping to boost engagement and persistence. Among the factors contributing to this are several that are right in the library’s wheelhouse: offering opportunities for guided practice and hence, earned success; active learning space; interactions with caring adults, fostering collaboration; and respect for diverse learning styles. Equally important, she called on librarians to document their success in moving those needles and use that data to make the case for their value to administrators, something she did by studying how many more points in a performance-based system (based on successful completion of educational milestones) students earned thanks to library involvement.

Where Outcomes Are Going

Assessing outcomes is, of course, far from a new trend in librarianship, academic or otherwise. But while many past attempts to measure outcomes have been qualitative and incomplete, in part due to privacy concerns, one trend that was repeatedly referenced at the conference was attempts to quantify the results of library services on the student level. These exemplify several of the trends cited by Hiller in his opening remarks, including assessment for advocacy, moving quantitative assessment from counting stuff to accountability, linking narratives closely to customized services, and asking what we need to know to make users successful. On the simplest level, probably replicable by many schools, and presenting minimal privacy concerns, was a poster session from Shawnee State University, Portsmouth, OH. Entitled Reimagining Library Assessment: Partnering for Student Success, Shawnee looked at how checkouts correlated with college readiness, with GPA, and with graduation rate. “All of this was in response to state government funding pressures on our open-access institution,” Katy Mathuews, Access Services Librarian, Clark Memorial Library, told LJ. “We plan to incorporate other variables in addition to checkouts as they become accessible/available so we can build a more robust analysis.” These could include use of electronic resources, computers, and spaces, as well as and instruction/research contact. The library also hopes to analyze library use by academic department and student cohort (first-year, first-generation, and at-risk). Taking it to the next level was the University of Minnesota, Minneapolis and St. Paul, in Now What!?! Exploring the Next Steps for Large-scale Library Assessment Projects. Minnesota is already assessing many of the factors Shawnee hopes to add, including circulation, digital resource use, reference services use, instruction, and workstation use, even attending workshops. Based on that data, plus added demographics and performance info from the school’s office of institutional research, the library has found that students see a .23 GPA increase with just one library use, are 1.54 times more likely to reenroll after one use, and are 7.54 times more likely to reenroll if they take an online workshop. These numbers have proved to be powerful tools in convincing faculty to incorporate the library in their programs and students to avail themselves of library services. However, some data is still missing, including statistics on the library as place—the impact, if any, of students using the library to study—and any traffic carried by VPNs. Much more ambitiously, however, than merely filling in those gaps, the library is grappling with how to balance privacy concerns with helping to improve individual student outcomes by digging deeper into the data they do have. The library is considering telling students’ academic advisors about their level of library use and how it compares to others in their field, as well as debating whether it can and should analyze which books are checked out, databases accessed, etc.—information which would be not only aggregated to provide advocacy info or shape library services, but provided to the individual students and their instructors, programs, and advisors with an eye to intervention. University of Minnesota Undergraduate Services Librarian polled the room to see on which of these points the assembled librarians felt that privacy outweighed data… and said the results were more heavily in favor of data than she’d expected. Gilchrist extended the theme furthest into the speculative future. Individual outcomes assessment was far from the only theme she touched on—performance based funding, competency-based education, a “reframing” of academic freedom, including student outcomes in tenure process, and wrap-around services were some of the other major trends in higher ed which she feels will impact library assessment in the years to come. However, she made no bones about saying “predictive analytics are where we're going,” connecting not just library data and demographics but all the information the whole institution collects, and using sophisticated modeling to determine relevance. The result, she says, will be a system of Early Alert systems and dashboards and “intrusive interventions” in the form of advising, tutoring, and even reference. While such a shift might have advantages for students who need more help, as well as for libraries who can clearly demonstrate their value, audience reaction was skeptical, with one attendee asking how librarians can resist these changes, which he called dystopian, while others on Twitter, hashtag #cunylib2014, characterized Gilchrist’s analogy to Disney’s ‘magic band’ as disturbing, questioned the impact on privacy, and whether students will find intrusive interventions helpful. Echoing Hiller, Gilchrist advised concerned librarians to take ownership of the assessment role for the campus as well as the library, where they can help set the agenda and define the parameters for the institution as a whole.