Measuring the success of an online bioinformatics resource

Like many other journals, PLOS Computational Biology aims to publish research that helps to further the field and develop the community that the journal serves. From time to time, however, we publish a paper that makes a significant impact not as research but as a resource. One such article is a Perspective entitled ‘An Online Bioinformatics Curriculum’ by David Searls, one of our Associate Editors, who presents a bioinformatics curriculum in the form of a virtual ‘course list’, together with editorial commentary, and an assessment of strengths, weaknesses, and likely future directions for open online learning in bioinformatics.

Published in September 2012 as part of the PLOS Computational Biology Education collection, the article has picked up just over 77,000 hits, if you look at both PLOS and PubMed views and downloads combined. Not only that, but it’s been shared widely via social media – 204 times on Twitter and 303 times on Facebook – and saved over 300 times using bookmarking tools. These high article-level metrics (ALMs) – for comparison, see other Education articles published in 2012 – give us a fascinating insight into the specific ways readers have interacted with this article.

Using the traditional citation metric, the article doesn’t particularly stand out, as it has only been cited a handful of times. The high pageviews, saves and social media shares, on the other hand, indicate that people are showing an interest in the article, passing it on via their networks and bookmarking it for future use. The acceptance of these ‘altmetrics’ as measures of impact is not yet well established, perhaps because they are not recognised as having a direct effect on researchers’ careers – though this is changing, according to a Perspective article out last week in PLOS Biology. What these newer ALMs do demonstrate, however, is an awareness within the community that the Searls article is a useful resource to be shared and saved for another day. And for an educational article intended to benefit the community, that’s surely an indicator of success.

So what’s the attraction of this particular article? Well, bioinformatics is a relatively new field and one that hasn’t traditionally been taught widely in universities, on-the-job learning having been the main route into the field.  The existence of an online resource where bioinformatics courses are systematically listed follows a general pattern whereby self-directed learning is becoming available across multiple disciplines, according to Searls’ introduction, and is especially relevant for a field that has embraced an online movement that includes open-access publishing.

As the first of its kind, it’s perhaps not surprising that the article has attracted attention. And, importantly, this is regular, long-term attention: Figure 1, produced using almreports.staging.plos.org, shows the total article views over time. After an initial spike, followed by some ups and downs, you can see that the article is being viewed as frequently over two years on as it was three months after publication. This continuous use suggests the article is well established as a valuable resource for members of the community.

The success of ‘An Online Bioinformatics Curriculum’ prompted Searls to write a follow-up article, which we published in June this year. In ‘A New Online Computational Biology Curriculum’, Searls updates the curriculum he originally provided in response to a surge in the development of course content and availability. He explains in the introduction why the terminology used in the title has changed: ‘the sizeable expansion of courses available[…]has been deemed sufficient to widen the scope of this edition to encompass the more expansive term “computational biology” as opposed to “bioinformatics” (for those who consider the distinction important)’.

In fact, it was the original article’s popularity as indicated through the less traditional ALMs that prompted Searls to write the latest instalment, as he explains below:

“The article is obviously unique in its format, and it was great fun to see how much information I could fit into the framework of a typical university course catalog. The first edition of the “catalog” that I wrote two years ago was an opportunity to let the unusual format spontaneously evolve from the data I wanted to convey, and the level of interest in that initial effort was the major factor that spurred me to do the follow-up. Interestingly, although I had a scattering of personal contacts from readers that were very encouraging, I never would have had an inkling of just how great the uptake was if it hadn’t been for the various article-level metrics that PLOS tracks. It’s obviously not the sort of publication that people are going to cite widely, and even citations would not have provided the granularity and immediacy of feedback that would have inspired me to write the sequel so soon after the initial effort. That speed of turnaround was particularly important given the rate at which the MOOC universe has evolved, and it strikes me that there’s a nice symmetry there between the subject matter of the article and the way it was published and measured — all reflecting the online nature of science and education today. Since the piece appeared, I’ve had several invitations to speak and inquiries from institutions that are interested in using online educational resources systematically, so I have every reason to believe that free online education, like free online publication, will have real staying power.”

Making educational resources as freely available as possible is important for building a future for the computational biology community, and the use of ALMs such as pageviews, social media shares and bookmarks provides a way to measure their success that traditional measures of impact fail to capture.