Product Information:-

  • Journals
  • Books
  • Case Studies
  • Regional information
Real World Research - #RealWorldResearch
Request a service from our experts.

How to develop research with impact

Options:     Print Version - How to develop research with impact, part 2 Print view

Article Sections

  1. Introduction
  2. Impact in the academe
  3. Impact outside academe
  4. References

Impact in the academe

Academic measurement of impact

While the process of how to measure the social and economic impact of research has yet to be definitively established, the traditional process of measuring the scholarly impact of research is more firmly entrenched.

The two most widely used measures are:

  1. Published output (the number of publications, and where they are published, "prestigious" journals being preferred).
  2. The number of times a piece of work is cited in other scholarly works.

Benchmarking

Because universities use benchmarking as a way of defining quality, rankings and league tables are important. There are a number of different measures, ranging from length of time to employment after graduating, to teaching and student satisfaction, but, according to researchers Adler and Harzing (2009), the most aspired rankings are those which concern research productivity.

In some countries, such as the UK and Australia, such research benchmarking is directly linked to government funding: the results, and their position in the league table, dictates the funding they receive.

The UK's Research Assessment Exercise for 2008 used published outputs as a measure; but the RAE is due to be replaced by the Research Excellence Framework (REF) in 2014, which will make use of citation analysis to judge the academic quality of research, as well as its social and economic impact.

The Australian Government's Excellence in Research for Australia (ERA) will also be partly based on citation metrics for some disciplines, although not for the social sciences and humanities.

Citation analyses

Citation analysis, a form of bibliometrics, involves examining the frequency and pattern of citation of one work in the work of others, or one scholar by other scholars, in order to gain some measure of the particular impact of a piece of research on knowledge as a whole.

Citations are also used to measure journal quality, and inclusion in the commercial citation databases – and subsequently gaining a high impact factor – is a key target for most journal editors.

The two most commonly used citation index databases, both of which are being used in the REF pilot exercise, are Thomson Reuters' (formerly ISI) Web of Knowledge and Elsevier's Scopus. The latter is the citation supplier for the ERA.

Web of Knowledge is a suite of products comprising abstracting services, citation indexes across arts, humanities, social sciences and medicine, and bibliographic databases. One of its major products, Web of Science, provides citation based searching in over 10,000 journals and 100,000 conference type papers.

Scopus is an abstract and citation database which claims to cover 18,000 titles from more than 5,000 publishers.

Problems with academic measurement

The usual advice given to young scholars is to publish in journals with a high citation impact (see the Emerald author guide on "How to find the right journal", and information professional, Rachel Singer Gordon's column, Publish, don't Perish – Instalment 35).

However, looking at citations and counting authors' publication outputs have been severely criticized as approaches on the grounds that they distort both publishing and research.

There are, according to some, too many journals and too many people wanting to publish in them. The "count your publications" approach to research advancement encourages over-publication (Bauerlein et al., 2010), with research being "salami sliced" and divided up between papers to increase the amount of publication.

In one case, a physics professor divided up the findings of his research so that each of his students recorded his results separately, and each of the 450 resulting papers also contained the professor's name.

The desire to publish work in top journals often means that the research is manipulated towards the known interests of the editor. Targeting what are perceived to be, on the basis of their impact factors, "top" journals means that more specialist journals are avoided, even though these might be more appropriate for the work and have more impact in the field.

Journals have become the main means of communicating research, to the point where book publication is shunned by those looking for career advancement, even though a book might have a more prolonged impact, and be more cited.

Conference proceedings, regarded as less important by the counters of research because they are not "published", nevertheless avoid the time lag of publication and are hence a quicker method of dissemination. In fact, some fast-moving subjects, such as computer science, prefer them simply because the field is advancing so quickly that research will be out of date by the time it is published in conventional form.

"Top" journals may also be discipline specific – accounting, marketing for example – and hence be less welcoming to cross disciplinary research, which may nevertheless address important and complex social issues. It would be hard, for example, to think of a more important issue than the creation of buildings that can withstand disasters such as earthquakes. Emerald's journal, International Journal of Disaster Resilience in the Built Environment, is aimed at those working in the built environment field. Built environment encompasses a wide range of disciplines including architecture, engineering, construction and urbanism.

New journals may likewise be shunned, because Thomson Reuters requires a three-year waiting period before a new journal can be assessed for its database.

In short, the existing system encourages researchers to be conservative, to ask the same questions using the same methodology, to avoid being innovative, take risks, branch out into new areas, and engage in dialogue with other disciplines. This in turn, according to Adler and Harzing (2009), leads to rigorous, but irrelevant, research.

Other measures of impact

Dissatisfaction with the present system has led some scholars to propose other measures of academic research impact, for example usage, i.e. the number of times an article is downloaded. (Read the information management viewpoint on usage on the subject for more information.)

Using citations to help you

However, it is unlikely that usage or network analysis will replace citations as a tool of impact measurement in the near future. Scholars therefore need to develop ways of using citation metrics to help them.

Although the main tools of citation measurement are the commercial databases, they are not the only ones: Google Scholar (GS) also offers powerful citation search facilities.

One researcher who is a strong advocate of GS is Anne-Wil Harzing. Harzing has developed a piece of software that analyses citations, Publish or Perish – discussed below – and further information can be found on Harzing's website.

GS may be a better tool, it is claimed, for the following disciplines (Harzing, 2010):

  • Business, administration, finance and economics.
  • Engineering, computer science and mathematics.
  • Social sciences, arts and humanities.

The first reason for using GS is that it is stronger in these areas than Web of Science, which simply does not have as many journals from these fields as it does in science.

GS works by combing the whole of the Web for academic content, and some claim it has a broader intellectual and international impact (Meho and Yang, 2007; quoted in Harzing, 2008). Although there was some initial criticism of its coverage, its scholarly content has now increased significantly as more and more publishers allow its crawlers to search their databases (Jacsó, 2008a).

It also captures other scholarly output media, such as books, book chapters, and conference proceedings, as well as institutional repositories. And unlike commercial databases, which require a subscription, GS is free.

GS is not without its drawbacks, however, which have been highlighted by Péter Jacsó of the University of Hawaii, writing in Online Information Review (2005; 2006; 2008a; 2008b).

Although coverage has improved since Jacsó described GS as being "as secretive about its coverage as the North Korean Government about the famine in the country" (2005), there are still significant publications missing from its database (2008a). The most significant problems, however, lie with the software, which Jacsó accuses of innumeracy and illiteracy (2008a), and which has the effect of producing inaccurate and inflated results. For example, it does not do Boolean searches: a search for <chicken OR chickens> returns fewer searches than one for chicken (Jacsó, 2006a).

The inaccuracy of the results, inflated by error and also by the occasional inclusion of inappropriate material such as press releases (2006b), renders GS unreliable as a tool of citation metrics, believes Jacsó (2006b; 2008a).

Nevertheless at least one major research organization – the Centre National de la Recherche Scientifique (CNRS) in France – has requested that researchers provide GS-based data in addition to that from Web of Knowledge (Adler and Harzing, 2009).

Statistical measures in citation impact

For those wanting to go a bit deeper into citation impact than merely counting up the number of references to their published output, there are a number of statistical measures, summarized in Table I (Harzing, 2010).

Table I. Statistical measures in citation impact
Hirsch's h-index Aims to assess the cumulative impact of an author's work. Definition = "A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-h) papers have no more than h citations each"
Egghe's g-index Aims to give more weight to highly cited articles.
Definition = "[Given a set of articles] ranked in decreasing order of the number of citations that they received, the g-index is the (unique) largest number such that the top g articles received (together) at least g2 citations"
Contemporary h-index Adds an age-related weighting to each article to give less weighting to older articles
Individual h-index Divides the standard h-index by the average number of contributing authors, in order to mitigate the effects of co-authorship
Individual h-index (Publish or Perish variation) Divides number of citations for each paper by the number of authors for the paper, calculating the h-index of the result
Multi-authored h-index This modification of the h-index is another way of taking account of multiple authorship. It uses fractional paper counts to develop a measure known as the hm-index
Age-weighted citation rate (AWCR) and AW-index Measures the average number of citations for an entire body of work, with adjustments for the age of individual papers

A proposed metric for economics and business

Harzing and van der Wal (2007) proposed a new metric for measuring impact in economics and business based on Hirsch's h-index, which they suggest provides a "more robust and less time sensitive" measure of journal impact, together with GS, which provides wider coverage.

They base their argument on the fact that there is a strong correlation, for those journals which are listed in the Thomson database, between impact factor and GS h-index. This means that in those areas which are not well covered in Thomson Reuters, such as finance and accounting, marketing, general management and strategy, the GS h-index measure could appropriately be used.

On the other hand, Péter Jacsó (2008b) advises against depending on the GS h-index, on grounds of the unreliability of GS's citation matching algorithm, which can lead to rogue results, together with duplication, and inaccuracy in numbers of citations. GS should be used in conjunction with a more reliable subscription-based database such as Web of Science or Scopus.

The issue of the relative merits of Google Scholar and Web of Science is a complex one, and there is no space in this article to give it an adequate coverage. Those who wish to find out more are advised to consult Anne-Wil Harzing's newly released The Publish or Perish Book: Your Guide to Effective and Responsible Citation Analysis (Harzing, 2010b) which provides detailed advice, not only on how to use the software, but also on citation analysis as a whole. It is available from www.harzing.com/popbook.htm.

This book has separate chapters on Google Scholar and Web of Science, as well as a very extensive comparison of GS, ISI and Scopus

Publish or Perish

Publish or Perish (PoP) is a free piece of software, created by Harzing.com and Tarma Research Software Pty Ltd, which retrieves and analyses academic citations. Using GS to obtain the raw citations, it can present the following statistics (Harzing, 2010):

  • Total number of citations.
  • Average number of citations per paper.
  • Average number of citations per author.
  • Average number of papers per author.
  • Average number of citations per year.
  • Hirsch's h-index and related parameters.
  • Egghe's g-index.
  • The contemporary h-index.
  • The age-weighted citation rate.
  • Two variations of individual h-indices.
  • An analysis of the number of authors per paper.

It can search for articles, authors, and journals, and runs on Microsoft Windows, Apple Mac OS X, and GNU/Linux systems. The data can be exported in BibTex, CSV, EndNote Import, and RIS.

There is a detailed help file as well as plenty of useful information on www.harzing.com.

A new version of the software (3.1) has just been published with a tabbed interface, so you can easily navigate between different types of search, and there is also a web browser.

Image: Figure 1. Screenshot of Publish or Perish.

Figure 1. Screenshot of PoP

PoP is widely used as a method for citation analysis (CNRS recommends its use alongside GS), and has been described by Péter Jacsó (2009: p. 1189) as "a swift and elegant tool to provide the essential output features that Google Scholar does not offer".

The fact that it helps compensate for some of GS's errors, for example facilitating identification and removal of duplicates, earns it praise, although Jacsó argues its results are not totally reliable because of the unreliability of GS's data (2009: p. 1190). Jacsó recommends some changes to the software, including the ability to backload cleaned data for recalculating metrics (2009: p. 1190).