Peer-review and assessment

The assessment of research grant proposals is made by appropriate experts drawn from academia, government and industry. Proposals are usually assessed first by individual reviewers and then by peer review panels.

Normally at least two individual reviewers are used, one of whom may be nominated by the applicant, although STFC reserves the right not to use nominated reviewers. Nominated reviewers must not be collaborators; nor should they be from the applicant's or collaborator's organisation. Applicants are given an opportunity to respond to reviewer comments prior to assessment by the peer review panel.

The membership of peer review panels is chosen to span the scientific remit of the panel and is decided in consultation with the community, usually following a call for nominations. STFC attaches great importance to ensuring that conflicts of interest among panel members and reviewers are disclosed and managed. Guidance for managing conflicts of interest is available below.

Guidelines for managing conflicts of interest in the peer review process

The Science and Technology Facilities Council, as a publicly funded organisation, is accountable to Government and the Public for its actions and for the way it conducts its business, which must be undertaken in a way that is transparent and guards against 'potential conflicts of interest' influencing the outcome of decisions.

Introduction

1. These guidelines set out the procedures for managing conflicts of interest that the Council expects peer review panels to follow in the assessment of proposals and the allocation of funds or other resources. The guidelines have been developed in order to protect the integrity of Council and reduce the risk of impropriety or any perception of impropriety in the conduct of peer review business and allocation of funds. The guidelines apply to all individuals involved in any way in the peer review of applications for funding.

2. Individual panel members like others who serve the public are expected to follow the seven principles of public life set out by the Committee on standards in Public Life (the Nolan Committee). The seven principles are listed at Annex A of the “Code of Practice for Council Members”.

Standards

3. In the absence of specific statutory provisions, common law requires that members of public bodies should not participate in the discussion or determination of matters in which they or a member of their family or household have a direct pecuniary or other material interest and where participation in the discussion would suggest a real danger of bias or give rise to a perception of bias.

4. In the context of peer review of project proposals or grant applications, a conflict of interest is normally deemed to arise:

  • when a panel member is named as a participant on the proposal or application under consideration;
  • where a panel member is from the same Research Organisation as the applicant (for Ernest Rutherford Fellowship Panel this includes the current and proposed host institutions);
  • where a panel member has any commercial or financial/pecuniary interest, for example where the reviewer is a member of an organisation that may benefit financially, directly or indirectly, from any decision made;
  • where a panel member has a close personal (including family) and/or close professional (for example, where a recently supervised student or PDRA is named as an Investigator or researcher) link to anyone named on the proposal.

Procedures

5. Pre-panel meeting briefings will be used by the Council staff to bring to the Chair's attention all relevant information on any connection between an application and a panel member, including institutional conflicts of interest. These Council guidelines should be followed in order to identify where a conflict of interest will require an individual panel member to withdraw from the meeting.

6. At the start of a panel meeting, the Chair will ask panel members to make a declaration on conflicts of interest. This declaration will include the requirement for a positive declaration that there is no conflict of interest. The nature of any conflict of interest, including positive declarations, will be recorded in the minutes of the meeting. If an individual panel member finds that a conflict of interest becomes apparent only as the meeting progresses, the nature of the conflict should be declared as soon as practicable to the Chair. The panel can then recommend whether the member has to 1. Leave the meeting during discussions relating to the conflicted application; 2. Stay in the meeting but not participate in the discussion; or 3. Stay in the meeting and participate freely. The office will make the final decision in all circumstances.

7. A conflict of interest in any of the following categories will automatically require that the member withdraw from the meeting for the discussion of that application:

  • the panel member is named as a participant on the proposal or application under consideration;
  • the panel member is a member of the same Research Organisation as the applicant (for Ernest Rutherford Fellowship Panel this includes the current and proposed host institutions);
  • the panel member has a conflict of interest of a personal, professional or commercial nature.

8. If after exploring other possibilities, it is considered that the exclusion of a member under 7(b) above would compromise the panel's ability to make a proper scientific judgement of the proposal, the automatic exclusion may be overridden. However this option only applies where the panel member does not plan to work on the project under consideration. Where such an exception is made, it must be solely on the grounds that the value to the quality and integrity of the peer review process of the panel member in question may be deemed to outweigh any residual suggestion of bias arising from the declared conflict of interest.

9. Notwithstanding any of the above, a panel member may voluntarily choose to withdraw from the discussion of a proposal at any time if s/he wishes to do so. This may arise for a number of reasons as discussion of a proposal develops. The secretary to the panel should record in the minutes the point at which an individual ceases to take part in the discussion and leaves the meeting.

Peer review guidance

Cross-Council guidelines (PDF - link opens in a new window) have been developed in order to protect the integrity of Council and reduce the risk of impropriety or any perception of impropriety in the conduct of peer review business and allocation of funds. The guidelines apply to all individuals involved in any way in the peer review of applications for funding.

Individual panel members like others who serve the public are expected to follow the seven principles of public life set out by the Committee on standards in Public Life (the Nolan Committee).

Reviewers are required to make a statement in every completed report to declare whether they have/have not any conflicts of interest. Failure to make this declaration will invalidate the report.

Names of reviewers are not disclosed to applicants and neither are those of the lead introducer on the grant. There are exceptions to the latter for those grant rounds which operate clarification meetings which are led by the lead introducer or where the lead introducer also acts as a mentor to the applicant.

Applicants for research grants should note that canvassing of members of the Peer Review Panel will lead to disqualification.

This page provides general advice and guidance for reviewers on completing reports for the STFC peer review process. You should also refer to the JeS Helptext when preparing to complete a review.

Reviewers' input is the single most important element in the peer review process, providing advice on the qualities of the many research proposals we receive each year. For the process to work effectively, reviewer comments should be timely, objective, fair and informed.

Reviews are based around a series of assessment criteria:

  • Scientific excellence
  • International competiveness
  • Strategic value within the STFC programme
  • Productivity of grant supported staff (where relevant)
  • Quality of leadership/management
  • Potential for economic impact and quality of the Pathways to Impact section (this will be assessed by the Panel)

In addition some funding schemes have further scheme-specific assessment/selection criteria which can be found in the relevant guidance notes:

Good reviewing

To maximise their value to the peer review process reviewer reports should aim to:

  • Provide clear comments and recommendations
  • Give justification for markings
  • Be consistent between box markings and comments
  • Provide enough information without being over-long
  • Provide constructive criticism
  • Clearly identify strengths and weaknesses
  • Raise concerns in the form of questions for the applicant

It is important to bear in mind how your report will be used. Your report will be fed back anonymously to the applicant, who will then be allowed to respond to factual inaccuracies or questions you raise. Following this, members of peer review panels will be asked to use your reports as a tool for ranking between proposals.

How it works

When you are asked to review a proposal, all the necessary documents will be sent to your Joint electronic Submission (Je-S) account. If you are asked to review a proposal submitted against a particular call for proposals, we will also include a web link to the call document which will set out all the call requirements providing you with background information as to what the applicant was advised.

The proposal you are asked to review includes a case for support. In some instances, the case for support may include a link to a web site containing information on the research proposed. Reviewers are not required to consider this additional information when providing comments on a proposal. If you do choose to look at this information, it is possible that your anonymity to the applicant will be compromised.

All reviews on our behalf are submitted using the electronic forms available through the Research Councils UK Je-S system.

Timescales and confidentiality of comments

If you cannot comment within the indicated timescale, please confirm this immediately so we have time to approach an alternative reviewer or perhaps extend the deadline. In addition, please confirm immediately if you do not feel qualified to comment at all.

Assessment criteria

STFC has adopted the following standard assessment criteria for research grant proposals. Additional criteria may apply to specific funding calls and will be described in the specific call guidance.

More information on assessment criteria

Category 1: Absolute pre-requisite, without which an application will not be recommended for funding

  • Scientific excellence: specific objectives of the project
  • International competitiveness
  • Strategic value within the STFC programme

Category 2: Supporting evidence

Supporting evidence which increases the confidence in a successful outcome. Where any of these are not met the risk and any proposed remedial or mitigation action must be identified. Where any criteria are not met any recommendation for funding would be subjected to close scrutiny by STFC. If approved for funding, STFC is likely to make an award contingent on remedial action to address the concerns highlighted before funds are committed.

  • Productivity of Investigator
  • Productivity of grant supported staff (where relevant)
  • Quality of leadership/management
  • Suitability of Institution/Group

Category 3: Important additional criteria

Important additional criteria, the opportunities and plans for which must be addressed in the application.

  • Potential for economic impact
  • Quality of Pathways to Impact section (where applicable)
  • Applicants are advised to consult the guidance on the above for further information.

Category 4: Ensuring that the health and critical mass in key instrument/construction groups is maintained (consolidated grants only)

  • Sustainability (of key instrument/construction groups)

Applicants should always refer to the specific call guidance for further details.

Pathways to Impact assessment guidance

Pathways to Impact are an important part of STFC and RCUK investment in research. They increase the likelihood that activities pursued during the research cycle have the intended societal and economic impact, add value to the UK and stimulate interest from wider stakeholders, including the general public. Actively demonstrating the impact of research is essential to ensure continued investment in the research base. As from April 2015, responsibility for assessing the Pathways to Impact document will lie with the peer review panel rather than the STFC Office – for guidance on providing this assessment, please see below.

Pathways to Impact are an important part of STFC and RCUK investment in research. They increase the likelihood that activities pursued during the research cycle have the intended societal and economic impact, add value to the UK and stimulate interest from wider stakeholders, including the general public. Actively demonstrating the impact of research is essential to ensure continued investment in the research base.

Research proposals must include clearly thought through and an acceptable Pathways to Impact statement, which outlines the initiatives and plans that applicant groups have/will put in place to engage relevant users and wider stakeholders. It should also detail how applicants will investigate the potential routes to exploitation of the outputs of the research described in the research proposal. If there is no Pathways to Impact statement, or if parts are missing, or if the panel decides that the statement does not adequately address the criteria, then the principal investigator will be asked to address these deficiencies before a grant is awarded.

Guidance on the preparation of Pathways to Impact is available, along with general guidance from RCUK.

Peer review panels are asked to assess whether the Pathways to Impact statement is clearly thought through and acceptable. This statement should be:

  • be project-specific and not generalised (i.e. does it adequately consider the potential of the specific research proposed);
     
  • be flexible and focus on potential outcomes; the PtoI should be flexible so it can respond to changes in events during the duration of the grant i.e. new results in the research or outcomes from PtoI activities.

In addition, applicants are encouraged to:

  • identify and actively engage relevant users of research and stakeholders at appropriate stages;
     
  • articulate a clear understanding of the context and needs of users and consider ways for the proposed research to meet these needs or impact upon understandings of these needs;
     
  • outline the planning and management of associated activities including timing, personnel, skills, budget, deliverables and feasibility;
     
  • include evidence of any existing engagement with relevant end users.

It is expected that being able to describe a pathways to impact will apply for the vast majority of proposals. In the few exceptions where this is not the case, the Pathways to Impact statement should be used to fully justify the reasons why this is not possible.

STFC have taken the decision that evaluation of the Pathways to Impact will not affect the assessment or ranking of the research proposal itself, however, consideration of potential impact of the research should be part of the assessment of the overall value of the research.

Determining whether the statement is acceptable involves an element of judgement. However, some guiding statements follow.

Common characteristics of acceptable Pathways to Impact

  • Identification of the relevant beneficiaries and target groups, with consideration of user needs and tailored activities.
     
  • Clear description of feasible plans with achievable deliverables and milestones.
     
  • Co-production and involvement of beneficiaries and users from the outset.
     
  • Evidence of engagement with the STFC office, University support staff and mechanisms, or external organisations fulfilling a similar impact support role.
     
  • Clear commitment for realising both academic and non-academic research impacts, by identifying resources required to deliver Pathways to Impact activities.
     
  • Briefly noted track record for knowledge exchange and impact generating activities in the context of the specific research project.

An excellent project would address the same points as an acceptable plan, but would outline one or more initiatives that are particularly interesting and innovative and would likely produce particularly high-quality impact.

When assessing Pathways to Impact, introducers should consider the relative scope of the proposal and size of the proposing group. A small group, or a new applicant, is less likely to be able to describe a Pathways to Impact with the same range of activities as a large group, and acceptability of a statement should be considered within this constraint.

Unconscious Bias

A particular equality issue in peer review is unconscious bias. Despite striving to be objective, people often hold implicit or unconscious assumptions that influence their judgement. Examples range from expectations or assumptions about physical or social characteristics associated with gender, ethnicity and age to those associated with certain jobs, academic institutions and fields of study. A briefing note on unconscious bias is available below.

Unconscious Bias Briefing

Despite striving to be objective, people often hold implicit or unconscious assumptions that influence their judgement. Examples range from expectations or assumptions about physical or social characteristics associated with gender, ethnicity and age to those associated with certain jobs, academic institutions and fields of study.

One example reported by Goldin and Rouse1 concerned the hiring of musicians by major symphony orchestras. When the identity of auditioners was withheld from the judging panel and they were placed behind a screen, the orchestras increasingly hired women.

Examples from STEM-related fields have also been reported. Steinpreis et al.2 reported a study where 238 academic psychologists evaluated an early-career CV which had been randomly assigned a male or female name. Both male and female assessors gave the male candidates better evaluations for teaching, research and service and were more likely to employ the male than the female candidate.

Wenneras and Wold3 reported a study of evaluators’ rating sheets for postdoctoral fellowships awarded by the Medical Research Council in Sweden. They found that women candidates need substantially more publications to achieve the same rating as men: the equivalent of three more papers in Nature or Science, or 20 more papers in speciality journals.

Trix and Psenka4 studied over 300 letters of recommendation for medical faculty who were hired by a large American medical school in the 1990s. They found that letters written for women were shorter, raised more doubts and portrayed women more as students and teachers while portraying men more as researchers and professionals.

Similar effects have been reported for race and ethnicity.

  1. Orchestrating Impartiality: The Impact of "Blind" Auditions on Female Musicians, Claudia Goldin; Cecilia Rouse, The American Economic Review, Vol. 90, No. 4. (Sep., 2000).
  2. The Impact of Gender on the Review of the Curricula Vitae of Job Applicants and Tenure Candidates: A National Empirical Study, Rhea E. Steinpreis, Katie A. Anders and Dawn Ritzke, Sex Roles, Vol. 41, Nos. 7/8, 1999.
  3. Nepotism and sexism in peer-review, Christine Wennerås and Agnes Wold, Nature Vol 387, 1997.
  4. Exploring the ColoUr of Glass: Letters of Recommendation for Female and Male Medical Faculty, Frances Trix and Carolyn Psenka, Discourse & Society, Vol. 14, No. 2, 191-220 (2003)

Overcoming Unconscious Bias

The difficulty with unconscious bias is of course that individuals are unaware that bias is influencing their decisions. However, there are some general points of good practice that panel members should adopt in order to minimise the effects of unconscious bias:

  • ensure that evaluation criteria are understood and applied consistently;
  • ensure sufficient time is allowed for the evaluation of each candidate. Reducing time pressures increases the likelihood that all applicants will receive a fair assessment
  • evaluate each candidate’s entire application;
  • be able to defend every decision for eliminating or advancing a candidate against the evaluation criteria.

As part of STFC’s commitment to eliminating unlawful discrimination and promoting equality of opportunity, we collect, analyse and review equality data relating to the people we award funding to and appoint to our boards, committees and panels. This helps us to monitor how successful we are in eliminating discrimination and promoting equality and to identify any issues that need addressing. We use this information for monitoring purposes only, treat it in strictest confidence and process it separately from any activities concerned with the selection or operation of boards, committees and panels. Reporting is in anonymised, aggregate form only.

STFC seeks the best members possible for its advisory bodies but also seeks to ensure that membership reflects the diversity within the relevant research communities and, therefore, particularly welcomes nominees from groups historically under-represented on these bodies. We also welcome nominations of appropriate experts based outside the UK. Where appropriate we take positive action to encourage nominations of people from under-represented groups and to broaden the pool of nominees more generally.

We also encourage members to tell us about any adjustments that would help them to participate in meetings and other activities.

The assessment of research grant proposals is made by appropriate experts drawn from academia, government and industry. Proposals are usually assessed first by individual reviewers and then by peer review panels.

Normally at least two individual reviewers are used, one of whom may be nominated by the applicant, although STFC reserves the right not to use nominated reviewers. Nominated reviewers must not be collaborators; nor should they be from the applicant's or collaborator's organisation. Applicants are given an opportunity to respond to reviewer comments prior to assessment by the peer review panel.

The membership of peer review panels is chosen to span the scientific remit of the panel and is decided in consultation with the community, usually following a call for nominations. STFC attaches great importance to ensuring that conflicts of interest among panel members and reviewers are disclosed and managed. Guidance for managing conflicts of interest is available below.

Security and Data Protection

Applications, independent reviews and PI Responses are available to panel members via the Peer Review Extranet (STFC’s preferred method for sharing data). Strict controls on data security and data handling are currently in place for Government departments and Government-funded organisations, including the Research Councils. Panel members must not save data (on to laptops, discs or hard drives) and if printed copies of any of the documents are made, these must be shredded after use.

Peer Review Framework: Accessing information about Research Council proposals and funding

Research Councils have obligations under both the Freedom of Information Act and Data protections acts. In line with this, a cross Council Peer Review Framework has been agreed to provide guidance on information relating to the peer review process including what is routinely disclosed/published and what is not. The document is designed for use by applicants, research organisations, board/panel members and members of the public. The full framework can be found below.

FAQ

Links

Science and Technology Facilities Council Switchboard: 01793 442000