Machine learning (ML) tools can potentially help patrons discover relevant content and services as they search a library’s catalog. However, ML tools need to be trained with a lot of good data to generate good recommendations, and initially, contextual recommendations generated with high-quality library metadata may be more effective at achieving the same goal. In addition, some patrons may express privacy concerns regarding ML projects that leverage catalog search data. These were a few of the topics discussed during the “Personalization in the Catalog with Machine Learning and Linked Data” online session at the American Library Association’s LibLearnX virtual conference on Sunday, January 23. David Wasserman, online services manager for King County Library System (KCLS), WA, was joined by Natasha Hesch, senior product manager for BiblioCommons, and Francisco Canas, staff software engineer for BiblioCommons, to discuss the results of a recent ML pilot that KCLS conducted with BiblioCommons using Amazon Web Services (AWS) Personalize ML service.

KCLS, along with the Chicago Public Library, had been part of an earlier pilot with BiblioCommons that added contextual promotions to catalog search results using linked metadata. Wasserman explained the earlier pilot with a brief demo, noting that “the user, you can see, has performed a search for ‘pie recipes,’ and in addition to the search results from the catalog, they’re seeing a blog post for ‘Books that Foodies Will Devour.’ This is one of the contextual promotions…which is leveraging the user’s search terms and related metadata to pull back the best promotion” of related content.

Continuing the demo, he showed how a search for “learn Spanish” generated a contextual promotion for the subscription database Mango Languages, and another search for “citizenship” generated a promotion for an upcoming KCLS event about applying for a green card. Separately, he demonstrated how searches that did not have a relevant contextual promotion would surface “spotlight” promotions selected by library staff—resources, content, or events that are “a little more evergreen.”

The goal of the subsequent pilot “was to see if adding machine learning would increase patron engagement by showing patrons personalized content, rather than just contextual or spotlighted content,” Hesch explained. “We wanted to test our hypothesis to see if machine learning promotions would have higher click-through rates than spotlight promotions.”

ML tools have the potential to make recommendations more scalable, and could make the recommendation system less dependent on staff creating and managing spotlight promotions to fill the gaps when no linked metadata-generated contextual recommendations are relevant, Hesch said. “It has the potential to increase patron engagement and patron satisfaction. There’s numerous studies that show customers can feel frustrated when website content isn’t personalized for them.”

Two datasets were used with AWS Personalize for the pilot. “The first dataset was the full set of [KCLS] content, including all of the metadata associated with that content,” Canas said. “The second data set was a collection of all recent interactions where patrons might have clicked on any of that content…. The documents dataset was used to learn similarities between the different types of content, and the interactions dataset was used to learn [in aggregate] patrons’ specific preferences about what type of content [they] might be interested in.”

During the pilot, the recommendations were prioritized in the following order: Relevant contextual recommendations generated by metadata—such as the “learn Spanish” and “pie recipes” examples above—were given top priority. “They tend to be the most relevant to that patron’s specific search,” Canas said. If none of those were available, ML generated recommendations were displayed, and then “if the patron wasn’t logged in, or if that particular patron had no machine learning promotions available, then it would default to displaying a staff-selected spotlight” recommendation, Canas said.

Amazon bills AWS Personalize as the “same machine learning (ML) technology used by Amazon.com for real-time personalized recommendations,” and typically, an ML tool such as AWS Personalize would be updated constantly, tracking every interaction with a consumer website—or in this case, every patron’s interaction with the KCLS catalog—in order to refine its recommendations. Due to both time constraints for the pilot and privacy concerns for patrons, this ML tool was “retrained” with aggregated interaction data (along with any new content added to the catalog) once per week.

From the pilot data, BiblioCommons estimates that “personalized promotions—if we turned it on for…100 percent of traffic to the library’s sites—would result in an approximately six percent increase in content views…and just over [a] five percent increase in the number of patrons who view content,” Canas said.

In a successful implementation, the ML driven personalized promotions might still be supplementary for contextual promotions driven by library generated metadata. During the pilot, those contextual promotions still drove a higher clickthrough rate of 1.31 percent, compared to the ML-generated promotions clickthrough rate of 1.1 percent—both of which are significantly better than the commercial industry 0.34 percent average for ML-generated ads in internet searches.

“Contextual promotions tend to score better across the board,” Canas said. “This is definitely what we would expect, because these promotions are most relevant to the specific search.”

The ML pilot only ran for one month last December, and Canas noted that ML tools improve recommendations over time as they are trained on more and more data. In addition, ML tools would almost certainly improve personalized recommendations if given access to real-time, individualized patron interaction data. However, such access would raise privacy concerns, the panelists agreed.

PROTECTING PATRON PRIVACY

To be as transparent as possible, KCLS launched the initial metadata-driven personalized recommendations pilot in October 2021 with a “highly visible news post that answered the questions we expected to receive about what was going on, what might be different, and information about privacy,” Wasserman said. “We did this as part of all of the promotions, and this predated the machine learning being enabled. We also shared this out on social media…and responded to patrons’ questions,” and continued to do so with the addition of the ML pilot.

During the pilot “on the patron experience side, we want to make sure we’re being transparent,” Hesch said. “So, beside each content promotion in the catalog, we include a ‘Why am I seeing this?’ link, and that explains why the promotion is appearing. Additionally, we also have a privacy statement in the footer of every [catalog] page, and this statement does cover the use of collecting and using anonymized information and activity to improve the quality of features and content for patrons.”

Real-time ML tools might improve personalized recommendations significantly, but as Wasserman noted, early feedback during the pilot included at least one patron who was enthusiastic about the idea, and another who said they would prefer to opt out, presumably for privacy reasons.