How we judge research outputs when making funding decisions

Robert Kiley, Head of Open Research, and Jim Smith, Director of Science, discuss the steps Wellcome is taking to fulfil the principles of the San Francisco Declaration on Research Assessment (DORA).

Illustration showing question marks around DNA

Credit: Science Photo Library

A researcher’s skills, qualities and attitudes can never be expressed in a single metric.

When we published our open access policy over a decade ago, we made it clear that what counts when we make funding decisions is the intrinsic merit of the work and not the journal or publisher.

Despite significant progress in making our research open access, we know many researchers remain unconvinced that it’s the work that counts and not where it’s published. We’re worried about this and so we’re continuing our efforts to promote best practice in research assessment.

What we’re doing

We were one of the first funders to sign the San Francisco Declaration on Research Assessment (DORA), publicly committing that we will consider all research outputs and look at a broader range of qualitative measures of impact, such as influence on policy and practice. In partnership with other funders and publishers, we are providing funding to DORA to help promote the broad adoption of these principles and to collect evidence of good practice.

We’ve modified our grant application forms so that now we ask researchers for their outputs, which may include (but are not limited to) preprints, datasets, software, research materials and inventions, patents and other commercial activities. Previously, we just asked researchers to cite their research publications.

Our forms also invite applicants to:

We make sure our advisory committees are fully aware of our values during their induction. This is to ensure they focus on the content and quality of publications when reviewing applications, rather than their number or the impact factors of the journals in which they were published.

What we’re going to do next

Although we’ve taken steps to improve how we assess funding applications, we know we need to do more.

A researcher recently lamented in Nature that despite its shortcomings he would "settle for impact factor as the least-bad option" for researcher assessment. The simplicity of using journal impact factor as a tool for researcher evaluation is enticing – and we suspect its simplicity explains its persistence. But researcher assessment requires a far more nuanced understanding of a researcher’s skills, qualities and attitudes, and these can never be expressed in a single metric.

At Wellcome, we recognise this and seek to be transparent in the approach we’re adopting.  

If you'd like to comment on the issues discussed here, email Robert Kiley at r.kiley@wellcome.ac.uk.

Related links