How to develop research with impact
By Margaret Adolphus
All researchers want their research to have impact – on other scholars, and on the world at large.
Impact is also important for achieving tenure, promotion, merit rises, grants – and all the other enticing apples on the research tree.
But what is impact and how is it measured? Or even, should it be measured? This article will explore these questions, along with criticisms of, and alternatives to, the most "accepted" ways of measuring research impact.
Impact is like a coin, and has two faces. One looks to the world outside academe, at social and economic impact. The other looks at the narrower, but no less important, impact on scholarship.
Impact in the academe
Academic measurement of impact
While the process of how to measure the social and economic impact of research has yet to be definitively established, the traditional process of measuring the scholarly impact of research is more firmly entrenched.
The two most widely used measures are:
- Published output (the number of publications, and where they are published, "prestigious" journals being preferred).
- The number of times a piece of work is cited in other scholarly works.
Because universities use benchmarking as a way of defining quality, rankings and league tables are important. There are a number of different measures, ranging from length of time to employment after graduating, to teaching and student satisfaction, but, according to researchers Adler and Harzing (2009), the most aspired rankings are those which concern research productivity.
In some countries, such as the UK and Australia, such research benchmarking is directly linked to government funding: the results, and their position in the league table, dictates the funding they receive.
The UK's Research Assessment Exercise for 2008 used published outputs as a measure; but the RAE is due to be replaced by the Research Excellence Framework (REF) in 2014, which will make use of citation analysis to judge the academic quality of research, as well as its social and economic impact.
The Australian Government's Excellence in Research for Australia (ERA) will also be partly based on citation metrics for some disciplines, although not for the social sciences and humanities.
Citation analysis, a form of bibliometrics, involves examining the frequency and pattern of citation of one work in the work of others, or one scholar by other scholars, in order to gain some measure of the particular impact of a piece of research on knowledge as a whole.
Citations are also used to measure journal quality, and inclusion in the commercial citation databases – and subsequently gaining a high impact factor – is a key target for most journal editors.
The two most commonly used citation index databases, both of which are being used in the REF pilot exercise, are Thomson Reuters' (formerly ISI) Web of Knowledge and Elsevier's Scopus. The latter is the citation supplier for the ERA.
Web of Knowledge is a suite of products comprising abstracting services, citation indexes across arts, humanities, social sciences and medicine, and bibliographic databases. One of its major products, Web of Science, provides citation based searching in over 10,000 journals and 100,000 conference type papers.
Scopus is an abstract and citation database which claims to cover 18,000 titles from more than 5,000 publishers.
Problems with academic measurement
The usual advice given to young scholars is to publish in journals with a high citation impact (see the Emerald author guide on "How to find the right journal", and information professional, Rachel Singer Gordon's column, Publish, don't Perish – Instalment 35).
However, looking at citations and counting authors' publication outputs have been severely criticized as approaches on the grounds that they distort both publishing and research.
There are, according to some, too many journals and too many people wanting to publish in them. The "count your publications" approach to research advancement encourages over-publication (Bauerlein et al., 2010), with research being "salami sliced" and divided up between papers to increase the amount of publication.
In one case, a physics professor divided up the findings of his research so that each of his students recorded his results separately, and each of the 450 resulting papers also contained the professor's name.
The desire to publish work in top journals often means that the research is manipulated towards the known interests of the editor. Targeting what are perceived to be, on the basis of their impact factors, "top" journals means that more specialist journals are avoided, even though these might be more appropriate for the work and have more impact in the field.
Journals have become the main means of communicating research, to the point where book publication is shunned by those looking for career advancement, even though a book might have a more prolonged impact, and be more cited.
Conference proceedings, regarded as less important by the counters of research because they are not "published", nevertheless avoid the time lag of publication and are hence a quicker method of dissemination. In fact, some fast-moving subjects, such as computer science, prefer them simply because the field is advancing so quickly that research will be out of date by the time it is published in conventional form.
"Top" journals may also be discipline specific – accounting, marketing for example – and hence be less welcoming to cross disciplinary research, which may nevertheless address important and complex social issues. It would be hard, for example, to think of a more important issue than the creation of buildings that can withstand disasters such as earthquakes. Emerald's journal, International Journal of Disaster Resilience in the Built Environment, is aimed at those working in the built environment field. Built environment encompasses a wide range of disciplines including architecture, engineering, construction and urbanism.
New journals may likewise be shunned, because Thomson Reuters requires a three-year waiting period before a new journal can be assessed for its database.
In short, the existing system encourages researchers to be conservative, to ask the same questions using the same methodology, to avoid being innovative, take risks, branch out into new areas, and engage in dialogue with other disciplines. This in turn, according to Adler and Harzing (2009), leads to rigorous, but irrelevant, research.
Other measures of impact
Dissatisfaction with the present system has led some scholars to propose other measures of academic research impact, for example usage, i.e. the number of times an article is downloaded. (Read the information management viewpoint on usage on the subject for more information.)
Using citations to help you
However, it is unlikely that usage or network analysis will replace citations as a tool of impact measurement in the near future. Scholars therefore need to develop ways of using citation metrics to help them.
Although the main tools of citation measurement are the commercial databases, they are not the only ones: Google Scholar (GS) also offers powerful citation search facilities.
One researcher who is a strong advocate of GS is Anne-Wil Harzing. Harzing has developed a piece of software that analyses citations, Publish or Perish – discussed below – and further information can be found on Harzing's website.
GS may be a better tool, it is claimed, for the following disciplines (Harzing, 2010):
- Business, administration, finance and economics.
- Engineering, computer science and mathematics.
- Social sciences, arts and humanities.
The first reason for using GS is that it is stronger in these areas than Web of Science, which simply does not have as many journals from these fields as it does in science.
GS works by combing the whole of the Web for academic content, and some claim it has a broader intellectual and international impact (Meho and Yang, 2007; quoted in Harzing, 2008). Although there was some initial criticism of its coverage, its scholarly content has now increased significantly as more and more publishers allow its crawlers to search their databases (Jacsó, 2008a).
It also captures other scholarly output media, such as books, book chapters, and conference proceedings, as well as institutional repositories. And unlike commercial databases, which require a subscription, GS is free.
GS is not without its drawbacks, however, which have been highlighted by Péter Jacsó of the University of Hawaii, writing in Online Information Review (2005; 2006; 2008a; 2008b).
Although coverage has improved since Jacsó described GS as being "as secretive about its coverage as the North Korean Government about the famine in the country" (2005), there are still significant publications missing from its database (2008a). The most significant problems, however, lie with the software, which Jacsó accuses of innumeracy and illiteracy (2008a), and which has the effect of producing inaccurate and inflated results. For example, it does not do Boolean searches: a search for <chicken OR chickens> returns fewer searches than one for chicken (Jacsó, 2006a).
The inaccuracy of the results, inflated by error and also by the occasional inclusion of inappropriate material such as press releases (2006b), renders GS unreliable as a tool of citation metrics, believes Jacsó (2006b; 2008a).
Nevertheless at least one major research organization – the Centre National de la Recherche Scientifique (CNRS) in France – has requested that researchers provide GS-based data in addition to that from Web of Knowledge (Adler and Harzing, 2009).
Statistical measures in citation impact
For those wanting to go a bit deeper into citation impact than merely counting up the number of references to their published output, there are a number of statistical measures, summarized in Table I (Harzing, 2010).
|Hirsch's h-index||Aims to assess the cumulative impact of an author's work. Definition = "A scientist has index h if h of his/her Np papers have at least h citations each, and the other (Np-h) papers have no more than h citations each"|
|Egghe's g-index||Aims to give more weight to highly cited articles.
Definition = "[Given a set of articles] ranked in decreasing order of the number of citations that they received, the g-index is the (unique) largest number such that the top g articles received (together) at least g2 citations"
|Contemporary h-index||Adds an age-related weighting to each article to give less weighting to older articles|
|Individual h-index||Divides the standard h-index by the average number of contributing authors, in order to mitigate the effects of co-authorship|
|Individual h-index (Publish or Perish variation)||Divides number of citations for each paper by the number of authors for the paper, calculating the h-index of the result|
|Multi-authored h-index||This modification of the h-index is another way of taking account of multiple authorship. It uses fractional paper counts to develop a measure known as the hm-index|
|Age-weighted citation rate (AWCR) and AW-index||Measures the average number of citations for an entire body of work, with adjustments for the age of individual papers|
A proposed metric for economics and business
Harzing and van der Wal (2007) proposed a new metric for measuring impact in economics and business based on Hirsch's h-index, which they suggest provides a "more robust and less time sensitive" measure of journal impact, together with GS, which provides wider coverage.
They base their argument on the fact that there is a strong correlation, for those journals which are listed in the Thomson database, between impact factor and GS h-index. This means that in those areas which are not well covered in Thomson Reuters, such as finance and accounting, marketing, general management and strategy, the GS h-index measure could appropriately be used.
On the other hand, Péter Jacsó (2008b) advises against depending on the GS h-index, on grounds of the unreliability of GS's citation matching algorithm, which can lead to rogue results, together with duplication, and inaccuracy in numbers of citations. GS should be used in conjunction with a more reliable subscription-based database such as Web of Science or Scopus.
The issue of the relative merits of Google Scholar and Web of Science is a complex one, and there is no space in this article to give it an adequate coverage. Those who wish to find out more are advised to consult Anne-Wil Harzing's newly released The Publish or Perish Book: Your Guide to Effective and Responsible Citation Analysis (Harzing, 2010b) which provides detailed advice, not only on how to use the software, but also on citation analysis as a whole. It is available from www.harzing.com/popbook.htm.
This book has separate chapters on Google Scholar and Web of Science, as well as a very extensive comparison of GS, ISI and Scopus
Publish or Perish
Publish or Perish (PoP) is a free piece of software, created by Harzing.com and Tarma Research Software Pty Ltd, which retrieves and analyses academic citations. Using GS to obtain the raw citations, it can present the following statistics (Harzing, 2010):
- Total number of citations.
- Average number of citations per paper.
- Average number of citations per author.
- Average number of papers per author.
- Average number of citations per year.
- Hirsch's h-index and related parameters.
- Egghe's g-index.
- The contemporary h-index.
- The age-weighted citation rate.
- Two variations of individual h-indices.
- An analysis of the number of authors per paper.
It can search for articles, authors, and journals, and runs on Microsoft Windows, Apple Mac OS X, and GNU/Linux systems. The data can be exported in BibTex, CSV, EndNote Import, and RIS.
There is a detailed help file as well as plenty of useful information on www.harzing.com.
A new version of the software (3.1) has just been published with a tabbed interface, so you can easily navigate between different types of search, and there is also a web browser.
Figure 1. Screenshot of PoP
PoP is widely used as a method for citation analysis (CNRS recommends its use alongside GS), and has been described by Péter Jacsó (2009: p. 1189) as "a swift and elegant tool to provide the essential output features that Google Scholar does not offer".
The fact that it helps compensate for some of GS's errors, for example facilitating identification and removal of duplicates, earns it praise, although Jacsó argues its results are not totally reliable because of the unreliability of GS's data (2009: p. 1190). Jacsó recommends some changes to the software, including the ability to backload cleaned data for recalculating metrics (2009: p. 1190).
Impact outside academe
Describing itself as "a leading publisher of global research with impact in business, society, public policy and education", Emerald prides itself on publishing articles which have an impact both inside and outside the world of academe.
In other words, impact is defined in terms of what one Guardian columnist described as, "making a demonstrable difference in a non-academic context" (Wolff, 2010).
There is hardly an Emerald editor who does not look, alongside rigorous research, for implications for practice. Some articles have a particular section for practical implications; others summarize the research so that working managers can read quickly what they need to know.
Two of Emerald's Awards for Excellence are for non-academic impact:
- Social impact – "research that makes a tangible difference for the good of society" – awarded for the first time in 2010.
- Best practical implications
Emerald is far from being a lone voice crying in the wind. Academics and academic and professional bodies also lament the way that in management, research is often divorced from real impact.
In their paper urging a new look at academic assessment systems (Adler and Harzing, 2009: p. 2), the authors quote the Financial Times as claiming that business views business-school research as irrelevant. They call for academics to be judged not by their star papers, but by research that has made a difference.
" ... in management, as in other professional disciplines, impact needs to be assessed not only among scholars, but rather within both the academic and professional communities of discourse" (Adler and Harzing, 2009: p. 26).
Criticism of the irrelevance of management research became particularly acute around the time of the global financial crisis that began in 2008. Questions such as "Were business schools to blame for not teaching ethics?", and "What about the role of research?" were raised.
At the beginning of 2010, Emerald developed a new framework for evaluating research impact, which looked at the following areas:
- Knowledge (further research). Research will contribute to the body of knowledge. This can be assessed through citation and usage impact factors, as well as the implications for research identified in the research conclusions.
- Practice. Industry and business leaders, practitioners and consultants in both public and private sector organizations are all affected by the outcomes of research. This can be assessed through the implications for practice that are identified in the research conclusions. Evidence that research has been applied successfully in industry and business practice can be gathered to demonstrate usefulness.
- Teaching. Students and faculty in a classroom setting are direct consumers of research. The impact of research in teaching can be assessed through the clarity of the conclusions to aid learning, and the provision of case studies and examples.
- Public policy. Civil servants, politicians, decision makers in public bodies, institutions, and charities draw on research to shape their policies and practice. Implications for policymaking and society can be identified in the research conclusions. Evidence that research has influenced public policy successfully can be gathered to demonstrate usefulness.
- Society. Cultural norms and accepted ways of thinking can and should be challenged by the outputs of research. This will include the impact on the environment (at micro and macro levels), ability to influence social responsibility in industry, business, and public policy, and the incorporation of social values as well as financial values in research outputs. These can also be assessed through implications for society in the research conclusions.
The framework was published as a note from the publisher, "Measuring the impact of research", in the first 2010 issue of many Emerald journals, and concluded with an observation, echoing the "groundswell of opinion", that "research impact needs to be measured in a variety of ways in addition to citation" (Marsh, 2010).
The groundswell of opinion to which Marsh alludes affected government funding bodies, such as the Australian Research Council, and the UK's Higher Education Funding Council (HEFCE), both of which acknowledged non-academic impact as a criteria.
What is harder to determine is how such impact can be measured. The problem is that there are no easy quantifiable measures such as citation or article count that can be used to measure impact in the world of business and practice.
Perhaps the best example of a system designed to capture non-academic impact was the Australian Research Quality Framework (RQF). Initially proposed in 2004, the RQF defined impact as:
"the economic, social, environmental and cultural benefits of research in the wider community".
The RQF recognized, however, that this type of impact was not amenable to precise measurement. It therefore came up with a list of potential indicators (see Figure 2) and required researchers to compile case studies supporting their claim, using the indicators.
Figure 2. Slide showing potential indicators for the RQF (Grant et al., 2009), copyright RAND Europe
Unfortunately, despite having been piloted in three universities (Griffith, Flinders and James Cook), the RQF was abandoned in 2007 by the incoming Australian Labor administration. The robust section on impact has, in the new ERA framework, been replaced by the rather more tame "application":
"Research application is considered on the basis of research commercialization income and other applied measures" (Australian Government and Australian Research Council, 2010).
The RQF model heavily influenced the British REF, where the proposal was to give a 25 per cent weighting to impact. However, this caused a storm of protest, with 13,500 people signing a petition against its inclusion (Corbyn, 2009), and complaints that it would be hard to assess impact in fields such as mathematics and the arts and humanities. A counter proposal was made for 15 per cent and it was decided to delay the setup of the REF to allow more time to find ways of measuring impact.
What qualifies as impact?
Consider the following articles which have won awards with Emerald:
Marketing theory and practice: the case of the Egg Card (Madichie, 2009) describes how Egg sacked 10 per cent of its customers who were either credit risks or otherwise "unprofitable". Not only does the article offer a readable and rigorous piece of research, and a useful case study, but it also proposes a new theme in the study of marketing (deviant customer and marketing behaviour) as well as a new conceptual base. In other words, the research not only has an effect on practice, but also challenges other research to develop a framework for studying deviant behaviour, as this extract describes:
"In the final analysis, this paper has some notable implications for the future of marketing as a discipline in its own right. Two key themes are advanced – first, from the purview of Egg Card's firing of 'unprofitable' customers – either real or perceived – this could set precedence for Jaycustomer behavior. In such situations, deviance may be preempted as customers may begin to feel no longer obliged to meeting their financial commitments as stipulated in their contractual agreements. This is likely to breed a dangerous precedent. Second, the event could also provide the conceptual base for reinterpreting deviant behavior in a new form – the marketers. This is where the term 'Jaymarketer' stems from and could enable researchers to develop a framework for evaluating deviant behavior of firms from corporate scandals to socially irresponsible behavior" (pp. 938-939).
Wikis as a knowledge management tool (Grace, 2009) is likewise well written, and is a multiple case history containing four studies of very different organizations that adopted wikis. Brief descriptions are followed by the outline of a framework which analyses what leads organizations to adopt wikis and what are the challenges. The framework is illustrated with the following very clear diagram:
Figure 3. Wiki selection and implementation framework
Can business learn from the public sector? (Heracleous and Johnston, 2009) also uses case studies – this time longitudinal – to challenge the conventional wisdom that the public sector should learn from its private counterpart. It maintains that the learning process can go both ways, and the article looks at the National Library Board of Singapore, and Singapore Airlines. It is laid out similarly to the previous example, with brief description and then an analysis of the lessons learnt.
What matters to managers?: The whats, whys, and hows of corporate social responsibility in a multinational corporation (Pedersen and Neergaard, 2009) analyses how managers in a multinational corporation experience corporate social responsibility (CSR). Using a single case study, the Danish authors conclude that managers' views of CSR are very heterogeneous – and that this is not a pre-condition of high performance in CSR.
All the above articles are very well written and clearly structured; they also use sound research methodology and are situated in the literature. They all offer a good balance between description and analysis, with just enough of the former to provide context and a concentration on analysis and implications.
All cover themes that are topical and relevant. While the implications do have sound relevance to practice, they also offer a genuinely fresh approach – a new framework, or a challenge to orthodox wisdom. In other words, their findings impact both on research and on practice – there is a genuinely symbiotic relationship between the two.
Strangely enough, all the above articles are written by academics, although one sure way of having real-world impact is to collaborate with a practitioner. Research by Hubbard and Norman (2007) has indicated that work produced by such collaboration is, in the area of marketing at least, as highly cited as that produced by academics on their own.
In the area of business and management, the link between research and practice is like standing in a room with mirrors on both sides. Each side reflects the other, and in turn reflects the other reflecting. Frameworks are important to both practice and research. They help practitioners understand what they are doing, and they should also help researchers understand the world in a way that is helpful to those whose behaviour is being observed.
The academics developing such well-balanced research should have no trouble having impact in either the academic or the professional and social spheres.
Adler, N. and Harzing, A.W. (2009), "When knowledge wins: transcending the sense and nonsense of academic rankings", Academy of Management Learning and Education, Vol. 8 No. 1, pp. 72-95
Australian Government and Australian Research Council (2010), ERA 2010 Evaluation Guidelines, available at: http://www.arc.gov.au/pdf/ERA2010_eval_guide.pdf [accessed 7 September 2010].
Bauerlein, M., Gad-el-Hak, M., Grody, W., McKelvey, B. and Trimble, S.W. (2010), "We must stop the avalanche of low-quality research", The Chronicle of Higher Education, 13 June, available at: http://chronicle.com/article/We-Must-Stop-the-Avalanche-/65890/ [accessed 25 August 2010].
Corbyn, Z. (2009), "REF rivals square off over value of impact proposal", Times Higher Education, 10 December, available at: http://www.timeshighereducation.co.uk/story.asp?sectioncode=26&storycod… [accessed 25 August 2010].
Glen, D. (2009), "Scholars seek better metrics for assessing research productivity", The Chronicle of Higher Education, 16 December, available at http://chronicle.com/article/Scholars-Seek-Better-Metric/62618/ [accessed 25 August 2010].
Grace, T.P.L. (2009), "Wikis as a knowledge management tool", Journal of Knowledge Management, Vol. 13 No. 4, pp. 64-74.
Grant, J., Brutscher, P.B., Kirk, S., Butler, L. and Wooding, S. (2009), Capturing Research Impacts: A Review of International Practice, RAND Europe, available at: http://www.hefce.ac.uk/pubs/rdreports/2009/rd23_09/rd23_09.pdf [accessed 2 September 2010].
Harzing, A.W.K. (2008), "Google Scholar – a new data source for citation analysis", available at: http://www.harzing.com/pop_gs.htm [accessed 25 August 2010].
Harzing, A.W.K. (2010a), "Publish or Perish", available at: http://www.harzing.com/pop.htm [accessed 25 August 2010].
Harzing, A.W.K. (2010b), The Publish or Perish Book: Your Guide to Effective and Responsible Citation Analysis, Tarma Software Research, Melbourne.
Harzing, A.W.K. and Wal, R. van der (2009), "A Google Scholar h-index for journals: an alternative metric to measure journal impact in economics & business?", Journal of the American Society for Information Science and Technology, Vol. 60 No. 1, pp. 41-46.
Heracleous, L. and Johnston, R. (2009), "Can business learn from the public sector?", European Business Review, Vol. 21 No. 4, pp. 373-379.
Hubbard, R. and Norman, A.T. (2007), "What impact has practitioner research had in the marketing academy?", Management Research News, Vol. 30 No. 1, pp. 2-33.
Jacsó, P. (2005), "Google Scholar: the pros and the cons", Online Information Review, Vol. 29 No. 2, pp. 208-214.
Jacsó, P. (2006), "Dubious hit counts and cuckoo's eggs", Online Information Review, Vol. 30 No. 2, pp. 188-193.
Jacsó, P. (2008a), "Google Scholar revisited", Online Information Review, Vol. 32 No. 1, pp. 102-114.
Jacsó, P. (2008b), "The pros and cons of computing the h-index using Google Scholar", Online Information Review, Vol. 32 No. 3, pp. 437-452.
Jacsó, P. (2009), "Calculating the h-index and other bibliometric and scientometric indicators from Google Scholar with the Publish or Perish software", Online Information Review, Vol. 33 No. 6, pp. 1189-1200.
Madichie, N. (2009) "Marketing theory and practice: the case of the Egg Card", Marketing Intelligence & Planning, Vol. 27 No. 7, pp. 926-944.
Marsh, R. (2010), "Measuring the impact of research", Note from the Publisher, published in Issue 1 of many Emerald journals.
Pedersen, E.R. and Neergaard, P. (2009), "What matters to managers?: The whats, whys, and hows of corporate social responsibility in a multinational corporation", Management Decision, Vol. 47 No. 8, pp. 1261-1280.
Wolff, J. (2010), "The impact of research – so hard to pin down", The Guardian, 2 February.