Impact Factor: A poor quality indicator

Impact factor is the average (the arithmetic mean) of the number of current citations per a citable item published in a journal during the previous two years. Although it was supposed to be a journal ‘metrics,’ it is commonly misused to evaluate quality of research articles and even performance of researchers. If used as a quality indicator, it would represent the most unscientific method of measuring the quality of science. This is because impact factor itself has a poor quality and thus can’t be an indicator of quality. It is a bogus indicator of quality. Below are the quality attributes of impact factor that justify this claim.

Table: Ten ‘quality’ characteristics of impact factor (IF)

SN Criteria  

Quality

Reason

1 Validity

???…

Number of citations doesn’t follow a normal distribution (Yet impact factor is a MEAN)
2 Coverage

???…

Citations are counted within indexed journals only. And indexing is both inadequate and uneven.
3 Comparability

???…

Number of articles and citations is highly dependent on type of disciplines.
4 Relevance

???…

The purpose of research is not just to get a paper cited by other research. It is for use.
5 Integrity

???…

IF is highly susceptible to manipulation by editorial policies and practices.
6 Timeliness

???…

IF is about articles published at least a year before. Citation period is also limited to a year only.
7 Completeness

???…

IF counts citations only. It provides no information about how or why articles were selected & cited.
8 Adequacy

???…

IF tells nothing about the content of articles or quality of peer-review and editorial processes.
9 Reliability

???…

IF can’t be reproduced. High level of uncertainty is involved. Journals can’t be ranked with precision
10 Attributability

???…

A citable item is the result of authors’ contribution, peer-review, and editorial process.

The quality of a scientific product needs to be evaluated using scientific methods that take in to account the quality of its content. As a result of the unintended impact of impact factor, many good articles published in low impact factor journal are being considered to be less useful. On the contrary, bad articles published in high impact factor journals are being considered are more useful or of higher quality. From the outset, impact factor was not designed for quality assessment purpose. But the scientific community misused it as such.

For effective research translation, the quality of a scholarly product should consider quality of the process used to produce it and quality of content of the product. Many organizations are moving towards this direction. But a significant number of academic and research organizations are still worshipping this false idol. These organizations have to ‘wake up’ and start using impact factor for the primary purpose it was designed for.

About these ads

10 responses to “Impact Factor: A poor quality indicator

  1. Nice analysis. The problem is not with impact factor per se. As you pointed it out, it is with the way it is understood and interpreted–like we have the ‘ecological fallacy’ in epidemiology, there is a sort of fallacy which may be termed ‘journal-impact fallacy’. What applies at the journal level doesn’t necessarily apply at the article level….. Cognizant of this fallacy, now there is a move to other kinds of ‘metrics’ such as article-level metrics. (See: http://www.plosone.org/static/almInfo). No index could be perfect, though.

    But regarding the timeliness, citation is not limited only to a year. It counts the number of citations in the preceding two years, i.e., the IF of a journal for the year 2013 is, for example, based on the average number of citations of articles published on that journal in the the years 2011 and 2012.

    • Yes, that has been the major problem. P2A argues that ‘impact factor’ should never be used as a quality indicator at any level – journal, article, and researcher levels. Because it is not designed for that and citation-based metrics by its nature can’t be a good proxy of quality. Citation is just an indicator of the ‘linkage’ between different knowledge products, and is a means to acknowledge the sources of information. That’s all about citation.Thanks for the comment on timeliness.

  2. Ayalew, IF doesn’t qualify to be considered as an indicator because it doesn’t fulfill the criteria for any indicator. That’s the conclusion from this quality assessment.

    • I think that is a very ‘hard’ conclusion. While the 10 criteria used to assess IF are very important and relevant, you used them in an ‘all-or-none’ fashion. I think the criteria should be seen along a continuum from very ‘weak/low/….’ to very ‘strong/high…’ or something of this sort.

      • That is a good point Ayalew. But I wounder about its practical importance. For some of the ‘quality attributes’ of an indicator, it either ‘miss’ or ‘hit’ a target. Would it make any sense if we grade the degree of ‘hit’ (as low or high) when the target is actually ‘missed?’ If it was for the things that the impact factor measures, it would have made some sense. Unfortunately, this is about the quality of the measuring indicator itself.

  3. Thanks P2A!
    I also read an interesting post which is related to yours in another blog. Please follow the link http://scholarlyoa.com/2013/12/03/a-magical-combination-easy-acceptance-and-an-authentic-impact-factor/#more-2716

  4. Hi, I do think this is a great web site. I stumbledupon
    it ;) I may rreturn once again since I bookmarked
    it. Money and freedom is the greatest way to change, may yyou be
    rich and continue to help others.

  5. Way cool! Some very valid points! I appreciate you penning this post and tthe rest of the website is also very good.

  6. Hey there! I just wish to give you a big thumbs up for your great information
    you’ve got right here on this post. I will be coming back to your site
    for more soon.

Join this discussion

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s