Guidance for research organisations on how to implement the principles of the San Francisco Declaration on Research Assessment

The draft guidance below provides information for Wellcome-funded organisations on how to implement the core principles of the San Francisco Declaration on Research Assessment (DORA).

We want to hear your comments and feedback on this guidance, before we publish an updated and final version in spring 2020. Fill in our short survey by 17:00 GMT, 24 February 2020. 

The DORA principles

DORA identifies two key principles that research organisations should follow when they assess researchers:

  1. be explicit about the criteria used to evaluate scientific productivity, and clearly highlight that the scientific content of a paper is more important than publication metrics or the identity of the journal in which it is published
  2. recognise the value of all research outputs (for example publications, datasets and software), as well as other types of contributions, such as training early-career researchers and influencing policy and practice.

Implementing these principles is a requirement of our open access policy 2021, and supports our broader efforts to build a better research culture and embed good research practice.

What we expect

As a minimum, by January 2021, Wellcome-funded research organisations should have a:

  • statement of commitment to implementing the DORA principles on their website – this should be prominent and accessible
  • plan for implementing the DORA principles, or a clear process in place for developing a plan (with a specified delivery date) 
  • process in place for monitoring and reporting on progress.

The rest of this guidance is not intended to be prescriptive. We recognise that organisations will take different approaches to implementing the DORA principles, that reflect their own values, cultures and ways of working. Organisations should consider other approaches if they feel they would add value. 

Work to implement the DORA principles should be joined-up with the organisation’s broader efforts to enhance research culture and promote research integrity.  

We strongly encourage organisations to be transparent and proactive in sharing their approaches and good practice, and in holding each other to account.

Three areas to consider

1. Statement of commitment

Organisations should have a prominent and accessible statement on their externally-facing website, highlighting their commitment to implementing the DORA principles. This should be in place by January 2021.

Alongside this statement, organisations are encouraged to:

  • specify how the DORA principles are aligned with their overarching mission and values 
  • be explicit on whether they:
    • are a signatory to DORA or a related declaration
    • are actively considering signing up to DORA
    • have decided not to sign up to DORA but will adopt equivalent (or enhanced) principles and procedures.
  • explain the key steps they are taking to implement and embed the DORA principles, linking through to relevant policies, guidance and the implementation plan
  • provide detailed materials, guidance and resources to staff through an internally-facing website as appropriate (while considering making these publicly available wherever possible as exemplars of good practice).

2. Plan for implementing the DORA principles

Organisations should develop a plan for implementing the DORA principles, or have a clear process in place for developing a plan (with a specified delivery date), by January 2021. This should incorporate elements from each of the subsections below. 

Strategy and leadership

The development and delivery of the implementation plan should be resourced sufficiently and have clear ownership and buy-in across the organisation.  

Organisations should consider:

  • making sure one or more senior leaders take ownership for championing the DORA principles and their implementation as part of broader efforts to enhance research culture
  • establishing a cross-organisation working group – with a suitably diverse and representative membership (including across research disciplines and career levels) – to develop the implementation plan and oversee its delivery
  • seeking input and feedback from staff across the organisation during the development and roll out of the implementation plan where appropriate
  • making sure the delivery of the implementation plan is suitably resourced – with one or more dedicated staff members taking forward the agreed workplan as part of their role
  • establishing a way for staff to report confidentially if the DORA principles are breached, and for appropriate remedial action to be taken by senior leaders.

Hiring and promotion practices

Organisations should develop their recruitment, promotion and career-advancement policies and practices in ways that reflect their commitment to the DORA principles.

This might include:

  • making sure that the criteria used for hiring, promotion and other decisions on career advancement are clear and transparent, and specifically reference the DORA principles
  • developing clear guidance for staff involved in recruitment and promotion decisions which:
    • explicitly caution against the inappropriate use of publication metrics
    • encourage them to value a full and diverse range of research outputs and contributions
  • enabling and encouraging candidates to highlight a broader range of research outputs and other contributions, in addition to publications
  • asking candidates to highlight a limited number of key achievements (which may be based on one or several outputs), and to provide a narrative description of their significance and the role they played in the research
  • prohibiting the use of language in job advertisements which refers directly or indirectly to journal title as a proxy for quality (for example, a track record of publication in leading journals)
  • discouraging the use of lists of ‘target journals’ that researchers (at all levels) should aim to publish in.  

Communication, advocacy and raising awareness

Organisations should put an ongoing programme of work in place to make sure that staff are aware of the organisation’s policy and expectations in relation to the DORA principles, and have the guidance and support required to implement them.

This might include:

  • highlighting ongoing work through the organisation’s communication channels – such as websites, newsletters and social media
  • running seminars or workshops to engage staff across the organisation in discussions about the DORA principles, and the opportunities and challenges of implementing them 
  • incorporating information about the DORA principles in staff inductions and in specific training for group leaders and others involved in hiring and promotion
  • identifying ‘champions’ across the organisation who can act as advocates of good practice and work with colleagues to identify and address any challenges and concerns.

3. Monitoring, reporting and sharing learning

Organisations should have a process in place to monitor and report on their progress against implementing the DORA principles by January 2021.

Organisations should consider:

  • gathering a range of quantitative and qualitative data on hiring and promotion decisions, and wider attitudes and practices, to help monitor and report on progress against an initial baseline
  • reviewing and adapting their implementation plan on a regular basis in line with lessons learned and emerging good practice 
  • sharing their experiences with other research organisations and the wider research community (as well as through DORA) – including both successful and unsuccessful approaches.

Background

Since we introduced our open access policy in 2005, Wellcome has adopted a clear position that research publications should be assessed on their intrinsic merit, and not on the journal in which they are published.  

We were one of the first funders to sign the San Francisco Declaration on Research Assessment (DORA) when it was launched in 2013. 

There is a clear and growing consensus over the need to reform research assessment practices to help improve research culture and integrity, which is reflected in the Hong Kong Principles for Assessing Researchers. The adoption of the DORA principles is recognised as a core element of the changes required. 

Many funders, universities and other research organisations have either signed DORA, indicated their support for related declarations (such as the Leiden Manifesto), or otherwise started work to put the principles into practice.

In 2019, the cOAlition S group, of which Wellcome is part, confirmed its commitment to assess research outputs based on their intrinsic merit by making it a core principle of Plan S.

To be successful in driving cultural change, funders and research organisations must reform the assessment processes that underpin decisions relating to research funding, recruitment and career progression. Credible and trusted processes need to be put in place, through which researchers are genuinely judged on the value of their work and not on where they have published, and are empowered to challenge situations where practices fail to reflect these standards.

For this reason, our new open access policy, which comes into force in January 2021, requires the organisations we fund to publicly commit to the DORA principles. We may ask them to show they are abiding by these principles as part of our regular audits.

The cultural change required to embed the DORA principles will take time, and we are committed to working with our funded organisations to drive this change.

What do you think about this guidance?

We want to hear your comments and feedback, before we publish an updated and final version in spring 2020. Fill in our short survey by 17:00 GMT, 24 February 2020. 

More information