Pros and cons of bibliometrics

Bibliometrics can be useful but they are also contentious. Here we explore their pros and cons:

Pros

• They are a quantitative way of measuring your research impact, so are seen as objective. This also means research impact can be compared more readily than with peer review, which is seen as subjective
• The procedure is transparent and results can be reproduced using the same method
• They are relatively easy to produce and use
• They take relatively little time to produce and use
• They are scalable. You can look at bibliometrics on an individual, institutional, national or international level
• Some familiarity with bibliometrics may help in relation to REF (Research Excellence Framework) cycles, as bibliometrics have been used to support evaluation in some sub-panels.

Cons

• Metrics distinguish between what is cited and what is not cited, not what is necessarily of good quality. It is perfectly possible for articles to be cited a lot but for negative reasons
• Metrics can be mis-used or gamed
  • Self citations - Self-citation is a valid practice; it identifies progression in your research and helps avoid self-plagiarism.  Nevertheless, excessive self-citation will inflate bibliometric measures that may be seen as a marker of research quality.
  • Citation farms, cartels or circles - Citation cartels are defined as groups of authors that cite each other disproportionately more than they do other groups of authors that work on the same subject.
  • Honorary citations - may occur when a submitting author may voluntarily (or be requested) to cite papers of a journal editor or from the journal.
• Some people feel bibliometrics skew research by encouraging people to write papers they think will be cited more, not what is valuable in research terms
• Disciplinary differences need considering as publication frequency and citation cultures differ. It is not reasonable, for example, to compare articles from medicine and the arts
• The Matthew Effect and bandwagon dynamics - This may impact the papers of a famous author or a famous paper; if an article achieves a certain status and visibility, the likelihood is that it will even more citations.
• Publication types - some publication types garner more citations than others: methods papers and review papers typically get more citations than original research papers.  Conversely, replication papers may not receive as many citations.  Any comparison of citations needs to take publication types into consideration.
• Citations and Societal Value - measures of research quality will increasingly also include a societal value dimension. Citations may not reflect the usefulness of research to the wider society.  This leads directly to the field of Alternative Metrics where mentions of research are detected outside the academic literature.
• Citation Databases - the attributes of citation databases (e.g., Scopus, Web of Science, Dimensions, Google Scholar and CrossRef) impact of the number of citations a publication will accrue. If a database is larger a publication will typically get more citations.  Publications from the Arts, Humanities and Social Sciences will typically receive less citations from those citation databases that focus on Science, Technology, Engineering and Mathematics disciplines. Some databases are more tightly curated and you may expect less indexing errors from those databases.