Today the paper A large-scale analysis of impact factor biased journal self-citations by Caspar Chorus (TU Delft) and Ludo Waltman (CWTS) has appeared in PLOS ONE. In this blog post, the authors discuss their main findings and reflect on the implications of their research.
The journal impact factor (IF) plays a prominent role in many research evaluations. The IF of a journal essentially equals the average number of citations received in a certain year by papers published in the journal in the two preceding years. There has been quite a lot of controversy and debate about the IF, resulting for instance in the San Francisco Declaration on Research Assessment and in a high-profile paper advocating the publication of journal citation distributions. Despite these discussions, many research evaluations remain heavily dependent on the IF. In these evaluations, the IF of the journal in which a paper has appeared is usually seen as a proxy of the quality or impact of the paper.
Given the importance of the IF, it may be tempting for editors and publishers to try to increase the IFs of their journals, and sometimes they may try to do so in questionable ways. For instance, they may publish editorials with many journal self-citations, they may participate in citation cartels, or they may coerce authors to include additional journal self-citations in their papers, in particular self-citations to publications from the past two years.
Besides editors and publishers, authors may also behave strategically. When authors are planning to submit a paper to journal X, they may believe that the opinion of the journal’s editor about their work will be positively influenced by having many citations to recent publications in journal X in their paper. Authors may therefore decide to include additional citations to journal X in their paper, citations which they would not have included if they had submitted their paper to a different journal.
Analyzing journal self-citation malpractices
Are IFs indeed influenced by journal self-citation malpractices such as the ones discussed above, and how large is the effect of these malpractices? Systematic research into this question is scarce. The most important contribution is a paper published in Science in 2012. Based on the results of a survey, this paper shows that coercive citations are an ‘uncomfortably common’ phenomenon in some fields in the social sciences.
In a paper published today in PLOS ONE, we provide new insights on the above question. For all science and social science journals indexed in the Web of Science database, we analyze whether there is an IF bias in journal self-citations. More specifically, we examine whether journal self-citations are biased to publications from the past two years, which are the years based on which IFs are calculated. We consider journal self-citations to be IF biased if the percentage of journal self-citations to publications from the past two years is significantly higher than the percentage of journal self-citations to publications from the five preceding years. For instance, let’s consider citations given in 2015 to publications in journal X. Suppose that 30% of the citations given to publications from 2013 and 2014 are journal self-citations, while 15% of the citations given to publications from the period 2008–2012 are journal self-citations. In this situation, journal X’s percentage of self-citations to the past two years is twice as high as its percentage of self-citations to the five preceding years, and therefore our measure of IF biased self-citation practices (IFBSCP) has a value of 2 for journal X.
Findings
Our analysis shows that the average IFBSCP value of all journals in Web of Science was almost 1.7 in 2015. This means that on average a journal’s percentage of self-citations to the past two years is about 70% higher than its percentage of self-citations to the five preceding years. Furthermore, in 2015, more than 5% of all journals had an IFBSCP value above 3. For these journals, the percentage of self-citations to the past two years is more than three times as high as the percentage of self-citations to the five preceding years.
The figure below, taken from our paper, shows for each year in the period 1987-2015 the percentage of journals with an IFBSCP value above 3. A distinction is made between physical sciences, life sciences, and social sciences journals. High IFBSCP values turn out to be more common in the life sciences than in the physical and social sciences. Moreover, the figure shows that during the past decade there has been a strong increase in the percentage of journals with a high IFBSCP value. In earlier years, this percentage was more or less stable. (Significant fluctuations are visible in the social sciences. This can be explained by the relatively limited number of social science journals.)
Conclusions
What do these findings tell us? There can be different reasons for an IF bias in journal self-citations. Self-citation malpractices are one possible reason. As already mentioned, editors may for instance coerce authors to include in their papers additional journal self-citations to the past two years. Alternatively, before submitting a paper to a journal, authors may strategically decide to include in their paper additional citations to the journal. However, as we discuss in our paper, there are also various perfectly legitimate reasons for an IF bias in journal self-citations. For instance, when recent work is cited, the citing and the cited paper typically can be expected to have a stronger topical relatedness than when older work is cited, and work that is topically related is more likely to appear in the same journal. This mechanism may explain why many journals have an IF bias in their self-citations.
Based on our analysis, it is not possible to distinguish between the various reasons that may explain why journal self-citations have an IF bias. However, looking at the time trend shown in the above figure, we do believe that our findings provide circumstantial evidence of increasing journal self-citation malpractices. The increasing trend shown in the figure follows, with a delay of a number of years, the increasing importance of the IF in research evaluations. The increasing importance of the IF is for instance visible in the increasing number of editorials about the IF. We also note that a relatively large share of all editorials about the IF appear in life sciences journals, suggesting that the IF is especially important in the life sciences. Life sciences journals indeed turn out to be the journals that most often have a large IF bias in their self-citations. Based on these observations, we consider it likely that our findings at least partly reflect journal self-citation malpractices.
When self-citation malpractices become more common, assessments of journals based on the IF will become less and less accurate and will be unfairly biased in favor of journals that engage in self-citation malpractices. More fundamentally, including non-relevant citations in papers harms the integrity of the scientific literature. It introduces noise in the literature, which may confuse and mislead readers. Rather than being fixated on a single metric, such as the IF, research evaluations need to take into account a more diverse set of metrics, thereby reducing the incentive to manipulate the IF. Publishers, editors, and authors should resist the temptation to opportunistically game the IF, and we should all actively remind each other of our ethical responsibilities in this matter.
