What makes a journal important?

The scientific community values citations, and with good reason. But, in other publishing arenas, the evaluation is based on how often the publication is read, not how often it is cited. This is true in traditional publishing with best-seller lists, and also in modern web publishing with metrics for visits and click-throughs. So, I was curious what relationship exists between how often works from a journal are cited, and how often articles from those journals are read by our users. Read on to find out:

A few years ago, I wrote a set of python scripts to improve on the services offered by PubCrawler and MyNCBI. Using some enrichment statistics and the Entrez API, I began emailing myself article alerts once a week without the false positive results inherent when searching PubMed for an author name or keyword. And when I first considered expanding my little set of python scripts into what is now PubChase, I began collecting a little polling data. At the department social hour, or at dinner with a seminar speaker, I asked how people found new literature to read. Answers involved RSS feeds and various heuristics for PubMed searches, journal club discussions, and dedicated blog reading. I suspect these days it would involve a fair amount of Twitter-gazing. But the most common answer in these informal surveys was to simply read the table of contents of a few journals each week. To me, the obvious follow-up question was how one should select which few journals of the thousands available to consider. In other words: what makes a journal important?

Now, it’s no mystery which three journals were invariantly listed in the response. After all, there is a pervasive (if dubious) belief that the awards of fellowships, faculty positions, and tenure are all predicated on “SNC” publications1. And in fairness, a lot of folks told me that they also took time to look in one or two other journals, typically specific to their research interests or a “second tier” journal. But the lesson was that a frightening proportion of scientists were limiting themselves to Science, Nature, and Cell, maybe with a smattering of Genetics, Neuron, NSMB, etc.

And it’s also no mystery why these are the journals exalted to the lofty status of “worth-one’s-time-to-read-stuff-in-them.” Articles in these journals are, on average,2 cited more often, and accordingly their Impact Factor is high. Citation count and Impact Factor are really easy statistics to understand: journals with high ones are the “winners.” But, if Impact Factor is the all-mighty reflection of journal quality,3 then why is it that Science, Nature, and Cell are not the three top journals by this metric?

Any good graduate student knows the answer to this question too: the other journals with high impact factors can be dismissed for targeting a clinical audience, for being a “review” journal, or for being a physics journal that got lost in the heavily biology-biased list of “science” journals. Because clinicians don’t win Nobel prizes too much these days. And reviews are only c.v. fodder with no value to the scientific community. And physics has too much math. So, after these dismissals, Nature is indeed the top-ranked journal, with Cell and Science close behind (after stumbling through crags of Nature specialty journals). Voila. Justification attained. We can all resume a frenetic SNC-or-Bust pursuit of scientific external validation.

But, before we do, I thought I’d share with you a few observations from the data comprised by our PubChase user base and the articles they chose to read.

The PubChase algorithm uses a “Bayesian-ish” model to predict which articles will be of interest before they accrue citations. This prognostication is required if you, like me, don’t want to wait around for a year or two to find out what to read. And it considers, basically, things that make sense to consider. These considerations include an estimate of the value of each journal, but unlike Impact Factor, which is based on citations, our model is based on who is reading the articles in a particular journal. And when I take a look at our internal rankings of journals in the model, a couple of things really stand out: 1) Science, Nature, and Cell are not the top 3 journals, and 2) maybe we should hit the breaks on dismissing all those reviews.

Here are, as of today, the top 20 journals according to our model:

1 Nature reviews. Genetics
2 Genetics
3 Genome biology
4 Genome research
5 PLoS biology
6 Genes & development
7 Cell
8 Molecular biology and evolution
9 Nature methods
10 Nature reviews. Molecular cell biology
11 Trends in genetics : TIG
12 Developmental biology
13 Journal of molecular biology
14 Science (New York, N.Y.)
15 Yeast (Chichester, England)
16 Development (Cambridge, England)
17 Current opinion in genetics & development
18 Molecular cell
19 Nature
20 Molecular systems biology

So, first things first: Bravo Cell! You made the top 10, unlike your SNC brethren.

Now, I would like to express this caveat: it’s clear to me from many observations that our users are enriched for the more quantitative sub-disciplines of biology, most notably genomics and population genetics. So, this likely explains the heavy representation of journals starting with the letters “G-E-N” in the list.

Interestingly, PLoS Biology takes one of the top spots, which agrees with an undercurrent of my informal polling suggesting that PLoS was on the cusp of joining the vaunted SNC trinity ">4.

You can also see that developmental genetics is alive and well as a field of research5.

And it’s also quite clear that those c.v. cluttering review articles might actually have some value. I asked how many of the over 22,000,000 articles available in PubMed are from these journals. The answer is 0.92%.6 Then, I asked what percentage of articles read by our users were from these journals. The answer is 4.18%. If you want to know if our users are reading reviews more often than they would be by chance, then you could compare the last two percentages7. And with this many events, it’s not surprising that a difference of roughly 1% and 4% is super-significant.8 So, it turns out that while major advances in science are not published first in review form, a lot of the articles read are reviews. This gets to a bigger question perhaps best saved for conversational pub fodder regarding exactly to whom rewards are given for what in academia, what the roles of discovery versus communication should be, and so forth. But to me at least, this is a clear rejection of the idea writing a review is a valueless endeavor. Lots of people are reading reviews.

So, one last thought: I compared our rankings to journal Impact Factors for last year. I expected some relationship between citation number, which principally drives Impact Factor, and the PubChase score, but I was not sure how strong this relationship would be. In essence, I’m asking what is the relationship between citation events and reading events. Again a caveat: keep in mind this is just a scatter plot of 20 data points, selected from the tail of a distribution. The correlation coefficient for this plot is about 0.2.910 It could be that the PubChase user base is not representative of the biological sciences as a whole. Or, it could be that the relationship between what is read and what is cited is not particularly strong. Or most likely, these data are structured with respect to multiple factors. Nonetheless, the relationship between readership and Impact Factor is mild at best.

Feedback and commentary are certainly welcome.

  1. I was a little horrified, too, to see that some very accomplished researchers actually split out these publications on their lab websites (e.g. this one). It makes me wonder, as a reader, what such authors think of the articles they have published in other journals. []
  2. I’m actually quite mystified why Impact Factor, and basically all other metrics of journal quality continue to use only the mean citations as the only estimator of quality. It seems obvious to me that each journal has a true- and false-positive rate for generating truly high-impact work, and one could, given access to the sort of data that Google Scholar and ISI has amassed, easily explore the other distribution characteristics of citations per article for each journal. It seems intuitively true that the citation counts of articles in any journal are distributed exponentially, thus assuming that the majority of articles are as good as the article of mean quality in a given journal is super duper wrong. I suspect it’s true that in journals of all tiers, there are a minority of good articles that are allowing the remainder of the articles to hitchhike on the backs of the reputations earned by the minority of articles with good citation numbers. []
  3. Even ISI doesn’t think this is true, by the way. In this article from 1994, they explicitly say as much: “Thomson Reuters does not depend on the impact factor alone in assessing the usefulness of a journal, and neither should anyone else.” []
  4. <head spins while contemplating the nested acronym possibilities> []
  5. Though I’ve heard it suggested recently that reading G&D was tres passé. []
  6. I took a quick proxy for how many journals were dedicated to reviews just by asking what percentage contained the word “review” in their title I know this isn’t perfect, but it’s almost certainly an underestimate. The answer is 4.55%. []
  7. Or, rather, their proportions. []
  8. a Fisher’s Exact p-value on these proportions is very closely approximated by the number zero []
  9. I’d be happy to recreate this graph for thousands of journals, if I could get my hands on the full list of Impact Factors, or better yet, Google Scholar citation counts []
  10. Which I tried to do with some foul-minded html scraping, but quickly learned that Google isn’t keen on me scraping Scholar results. []
This entry was posted in PubChase. Bookmark the permalink.

10 Responses to What makes a journal important?

  1. Pingback: What makes a journal important? | Pubchase Blog...

  2. Dennis says:

    Impact Factor:

    According to wikipedia (https://en.wikipedia.org/wiki/Impact_factor#Calculation) they just take the citations of that journal and divide it by the total of ‘citable items’ published in a certain period… so citations per article - which is the mean.

    I keep posting this everywhere, but if you look at the data presented here:
    you will see that using the mean value of article citation is not justified. The data are not normally distributed, so the median should be used. But that wouldn’t put certain journals in a special place any more, would it ;)

  3. James says:

    What if you used an algorithm like google does for search results (PageRank) whereby the number of citations is scaled by the number of times those citations are themselves cited?

    • Matt says:

      A measure with such a second-order estimator would definitely be more robust here, given that “junk” citations are less likely to be propagated. And such a metric could skip journal evaluation entirely and get straight to the quality of an article. As for how this cumulative score would for journals would compare to Impact Factor, I’m not sure, but I bet the correlation would still be high.

  4. Ravi Sachidanandam says:

    Could this data be a reflection of your users, rather than the
    journals? I am guessing the high impact journals are read more widely,
    so despite lower readership in any one group, they win in raw numbers.

    I think there are two kinds of scientists, people who read extensively
    in their own sub-fields while ignoring other areas, and others who
    read broadly but ignore their subfield (maybe because they know it
    well and can guess from the abstract if something is worth reading).

    And potentially your users may skew towards the first group?

    • Matt says:

      That’s an interesting idea. I have no reason to think this would be the case for our users, but it seems like a testable hypothesis. If you’re right about the two types of readers, then I should be able to partition the users into these two groups based on the proportion of their libraries that are from “general” journals. So, then the question is how I should determine which journals are “general” and which are not. Any ideas?

    • Lenny Teytelman says:

      This is certainly possible, but I think unlikely for a few reasons.

      1. Considering the pyramid structure of academic researchers, with most being graduate students and postdocs, the overwhelming majority consists of scientists who are not experts with 30 years of experience and no need to actually read a paper in their field.
      2. Personally, I read papers relevant to my research. I am much less likely to download a PDF of an article totally-unrelated to my research, and instead will simply look at the abstract. Given the number of relevant papers published in non-glam journals, my library is naturally dominated by non-CNS papers. I have no reason to believe that my grad. school or postdoc advisors behave differently, since as a PI, you want to carefully read the papers that directly overlap with the work of the people in your lab.
      3. The strongest argument supporting the idea that what people read and cite is different is from Dennis’s post above.

      “The effect of feedback loops should not be underestimated. According to a recent estimate based on propagation of citation errors, about 80% of all references are transcribed from other reference lists rather than from the original source article. Given this finding, it is hard to escape the suspicion that many authors do not read every paper they cite, and instead tend to cite those papers that appear most often on other authors’ references lists. Indeed, the distribution of citations within the literature as a whole is consistent with this model. “

  5. Pingback: Use PubChase instead of RSS | Pubchase Blog

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>