We've blogged previously about the dangers of seeing these systems as the solution for tracking impact. To be successful, any system must have the needs of the people generating the impacts at its heart, and reflect the perceptions of those who are actually using the research. So an online impact tracking tool is probably only going to be one small part of a wider systems that’s embedded at various levels in your institution. If you get this right, then you might have a bunch of people who are motivated to record impacts as they are generated, and who might want an online system to help them keep track of it all.
There are a number of reasons why institutions are increasingly turning to online systems:
· By making the process of recording a impact activities fast and simple, the theory is that researchers will record impacts as they arise, rather than waiting to be prompted
· By increasing the proportion of researchers recording their activities, there is a lower risk that impacts go unrecorded, so a more comprehensive record of impact activity can be built across the institution
· Identify people involved with particular activities and impacts, to target additional support to particular teams to enhance impact
· Enable good practice in knowledge exchange and impact to be shared across the institution
· Give marketing teams a fully searchable list of all the different impact generation activities taking place across the organisation, as well as a repository of media resources (e.g. photographs and video) linked to impacts that can be used to market research activity
· Help researchers build online CVs to promote their own work more effectively, integrating metrics that demonstrate the (social, online) impact of their work
Off-the-shelf or in-house?
If you want to start using an impact tracking system, the first choice you are likely to be faced with, is between buying access to an existing system or creating your own in-house system. The table below summarises some of the pros and cons of each approach.
Table 1: Pros and cons of buying access to an external system versus in-house
The University of Coventry developed their own in-house system. A more stripped-down prototype in-house system has been developed by University of Leeds, and other Universities are developing their own systems. In-house systems offer the ability to tailor the design to the specific needs of the institution, and the capacity to adapt the system to the changing needs and preferences of local users, but this typically comes at a high cost.
For many, the decision will boil down to whether the University has the expertise to develop and maintain a system with the functionality it wants, and whether up-front costs of development and the long-term costs of maintaining, updating and supporting an in-house system are worth it. For some, these sorts of costs may be hidden and make use of existing teams that would otherwise be under-utilised. Although some of the off-the-shelf options may be comparably expensive, some are likely to be far more cost-effective than developing an in-house system. Although these systems might not offer everything you want, given how much money and time you could save, it is worth taking a serious look at what’s already out there.
We decided to road-test three of the most widely used and relevant existing systems for tracking impact. We started by creating accounts and trying the demos that are publicly available. We then sent our evaluations to each company and asked for feedback, which led to some personal tuition in the capabilities of some of the sites. This means you’re able to read the most detailed and accurate assessment to date of these systems’ ability to track research impact.
Table 2 provides a comparison of features from these three sites. It is important to note that each of these systems has been designed for very different purposes, and so this is not a “like for like” comparison. The comparison is based purely on features that are designed to track impact, or in the case of ResearchFish®, we focus a sub-set of impact-related outcomes that are routinely captured in their system.
We have not reviewed systems that are focused solely on collecting details of publications (e.g. Pure and Mendeley), even where these offer collaborative spaces for researchers to interact with each other. Bibliometric measures of impact, even when these related to public interest in research, are fairly crude indicators of actual impacts on society, and would not be eligible for inclusion in most assessments of research impact. However, Altmetrics are growing in popularity and do offer a number of benefits to researchers who want to enhance their public engagement. In addition to ImpactStory, covered here, there are other altmertric websites that measure online impact of research in relation to the amount that articles are viewed, downloaded, discussed and cited, such as Figshare and Altmetric.com. There is also a range of other tools that can enable collaboration between researchers and that may be adapted to collect information about impacts, but these are not considered here as they are not specifically designed for researchers or for tracking impact (for example, Microsoft Office 365 and Yammer, Sugar Customer Relations Management Pro, and LinkedIn Sales).
In a nutshell: Enables staff across an institution to enter and organise information about their impacts
Cost: £2.68-3.00 per month per user for 100-8 users; £3000 per institution (assuming 500 users)
· Researchers can input impacts as they occur
· Flexible interface allows researchers to capture a wide range of impacts and add associated materials in a range of formats to illustrate and further capture impacts
· Tool to create a visual timeline showing the order in which activities took place, from which a story may be constructed with researchers to share with colleagues or for REF
· Impacts are accessible to research managers and marketing teams to identify interesting stories and the people behind them. These can be analysed over time (by creating a visual timeline showing the order in which activities took place), by person (showing who has submitted most/least activities), or by type of activity (via simple true/false statements assessment ensures key details are always captured for every activity) so you can compile statistics across your institutions for key variables you are interested in (Figure 1)
· Collaborative tools so researchers can work together to capture impact from joint work via shared ePortfolios including:
o Attach web links, e-mails, images, videos, PDFs, Office docs and more as supporting evidence
o “Join” button lets users quickly add themselves to other people's activities, avoiding duplication of effort
o Users can leave personal reflections on activities, aiding the collaboration process
· Support team
Figure 1: Top: True/false statements used to capture impacts in Kolola; Bottom: Front page user interface showing different events captured in the system
In a nutshell: According to their website, ImpactStory is, “Your CV, but better”. Provides information about how research publications (and other products) are read, cited, tweeted and bookmarked.
Cost: $60 per person per year (with a free option available on request)
· Publications and other products (such as software, presentations and datasets) are evaluated against a range of metrics, automatically calculated from online evidence about the extent to which each product has been cited, saved, viewed and discussed. It differentiates between academic versus wider public interest and shows how these impacts compare to other products ranked in ImpactStory via the production of percentiles
· Emails automatically notify users of new social media metrics, bookmarks, and citations for their top 10 performing research products, aggregated into a single report
· New products can be added easily via email
· Impacts are summarized in a visually appealing overview, which can act as a CV, or can be viewed on a map, showing the locations where most of the impacts originated (Figure 2)
Figure 2: Screenshot from ImpactStory example profile
In a nutshell: A research outcomes data collection service used by most UK research funders to track and communicate their investments, which can be used by researchers to build a personal portfolio make their research impacts visible to stakeholders.
Cost: Free access for researchers whose funders use the system; Institutions have access to the system for the purposes of compliance or £1000 per year for the ability to access the outputs data and access data analytics and API for integration with other databases.
· Researchers enter data into their own portfolio, linking research outcomes with awards. Outcomes include: publications; collaborations and partnerships; further funding; engagement activities; influence on policy, practice, patients and the public; research databases and models; intellectual property and licensing; medical products; artistic products; spin-outs; and awards
· The site can be used to build CVs for researchers, drawing on reported outcomes in the system
· Data may be inputted on behalf of researchers by delegates, and awards with multiple team members in different institutions can be managed together
· This data can be used be research institutions to:
o Ensure compliance with funder requirements
o Gain insight into activity, outcomes, awards and impact progress
o Reduce the burden and expense of impact reporting
o Analyse research outcomes that have been submitted by researchers
Figure 3: Screenshot of impacts captured in ResearchFish® for a research project
Each of the three products reviewed here fills a different niche:
· Kolola focuses on enabling communication between researchers around knowledge exchange activities and providing institutions with flexible tools for tracking knowledge exchange activities across their institution.
· ResearchFish® focuses providing standardized data analytics to compare specific impact-related research outcomes both across and between institutions.
· ImpactStory is focused primarily on providing data to individual researchers about the reach of their publications.
Both ResearchFish® and Kolola enable researchers to record impacts as they occur, and can be used to prioritize further work to build capacity for impact or develop more detailed narratives about specific impacts. A decision between Kolola and ResearchFish® is therefore likely to be about the importance of creating collaborative spaces for researchers around the recording of knowledge exchange activities that might indicate impacts, versus data analytics to detect impact-related research outcomes.
For institutions, ResearchFish® presents better value for money, with its £1000 per year price tag, and its focus on collecting and analyzing impact-related outcomes. It is also likely to be used by a high proportion of researchers already, and so despite having a less user friendly interface than Kolola, people are likely to be familiar with it and less likely to complain about duplication of effort (if they’re already being mandated to report impacts to research funders via ResearchFish®). The value of Kolola is therefore likely to be for specific projects that anticipate a range of knowledge exchange activities leading to impacts. On a project-by-project basis, Kolola becomes more cost-effective, and it may in many cases be possible to fund subscriptions to the service via research funding. Despite its title, ImpactStory does not capture the societal or economic impacts of research, but can be a motivational tool to help researchers monitor and improve their online presence and the reach of their research publications.
The final verdict: for most institutions, if you’re looking for an off-the-shelf product, you’ll probably want to rely on ResearchFish® to track impacts across your institution and promote KOLOLA on a project-by-project basis to help researchers manage and report on their knowledge exchange activities.
Note: this review was provided to all three companies for comment and developed in response to feedback from them all. The author has no conflict of interest with the companies reviewed and will not benefit in any way from the contents of this blog.