Researchers have long contended with a problem with timeliness. Peer-reviewed articles are often published about a year after submission. Likewise, peer-reviewed articles that cite, praise, criticize, or discredit their work won’t appear for at least another year after that. As a result, there can be a lag of three to five years before citations begin offering enough information to indicate the effect that a given piece of research has had on a field. Even then, citations alone may not offer a complete view of the impact of that research.

“Measuring the impact of research through the traditional methods—counting citations in published literature—is important, but it doesn’t tell the whole story,” said Timothy Deliyannides, Director of the Office of Scholarly Communication and Publishing and Head of Information Technology for the University of Pittsburgh.

Pittsburgh recently wrapped up a pilot test with Plum Analytics, one of several new companies in the emerging field of altmetrics. By examining how often a paper is downloaded, mentioned in the news, or linked to on social media sites, altmetrics providers offer researchers, funding agencies, and librarians a more immediate, quantifiable view of the impact an article is having on its field. The company in January announced the official launch of PlumX, a tool that harvests and aggregates alternative impact metrics in five major categories: usage, captures, mentions, social media, and citations.

Altmetrics “gives a whole new layer, looking at additional dimensions that used to be hidden, and it’s so much more timely,” Deliyannides said. “We can start seeing where research has an impact right away.”

The field is just beginning to get on the radar of academic evaluators, he noted. And, it’s in a transitional, experimental phase in which these new metrics are being assessed to see how they translate not only into measures of “buzz” but also of quality.

“It is going to take a little bit of time for people to really understand what these nontraditional measures are really measuring—whether they can be used as a proxy for these traditional measures of quality,” Deliyannides said.

Tale of the Tweet

When explaining altmetrics, the concept of tracking Facebook “likes,” or links on Twitter and other social media is often used as a straightforward point of reference. But social media tracking represents only one aspect of the field, noted Plum Analytics co-founder Mike Buschman. Mentions in news outlets, for example, are tracked separately from social media. And, usage metrics track how often a book has been checked out or how often a paper has been accessed online. And of course, classic citations are included as a metric.

There’s also the notion of “capturing” works. If someone reads an article and then bookmarks it, or saves the bibliographic information to their Zotero account, “that’s a different metric that in some ways is a little more powerful even than usage,” Buschman explained. “It’s the notion that I’ve read it, or I’ve read enough of it that I know I want to come back to it. Or I want someone else to come back to it.”

Currently, the field is still studying how these different metrics interact. An article that generated a large volume of mentions on social media, but was rarely downloaded or bookmarked may have been more controversial than substantive, for example. And weighting these different metrics, for now, can be somewhat subjective.

“What is the relationship to a tweet versus a PDF download versus a slideshow favorite?” Buschman asked.

It’s a field that is ripe for the type of analysis to which librarians are ideally suited, he said. Academic librarians could maintain more relevance in the research process by studying these metrics and helping students and faculty get more impact out of their research output and publications.

“If you see an impact where tweeting actually does result in higher usage and ultimately higher citations, let’s talk about how to do that,” Buschman said as an example.

At the University of Pittsburgh, “I think some researchers weren’t aware of the extent to which their research was talked about in social media,” Deliyannides said. “And then you’ve got communities of scholars developed around services like Mendeley, for instance. And I don’t think researchers are always aware which of their works are most widely downloaded, saved, referred to, or commented on. These are a whole range of measures that are different from the citation counts that still show impact. There were some surprises for some of the researchers there.”

Altmetrics tools also help analyze the impact of a research group in aggregate, he added. While only one or two principal investigators might be mentioned in a journal article or monograph, “you’ve got all kinds of other people on the research team that are talking about the research and sharing it in different venues. So, some of the impact there that was invisible before becomes visible with a tool like Plum Analytics’.”

Although the results have not yet been fully compiled or published, Deliyannides was pleased with the results of the pilot, and is now planning to roll out these tools as a service University-wide.

“We’re working very hard to collect information about all of the intellectual output of the University of Pittsburgh. It’s hard because a lot of this stuff is in the gray literature—stuff that is not published in traditional publishing media. And you’ve got all of the multimedia output, too. It’s difficult to identify all of that for thousands of faculty members at one institution. But…we do plan to have every single researcher at the University of Pittsburgh able to look at their own [altmetrics] profile and share that.”