David Sands

For the last couple of years I have had the honour of chairing the Computer Science Panel of the Swedish Research Council. I am scheduled to chair this committee for a third (and final) period in 2013. In this note I'd like to explain to the potential applicant how the evaluation process works. I'm writing this because many researchers, particularly those who have little contact with people who have previously served on a VR panel, have misconceptions about how their application is handled and what chances they have of receiving funding. Since the way that we work with applications is not intended to be secret, I hope this extended note may be of some interest. Since it is rather long let me point out some key points:

Almost all grants are evaluated by the evaluation panel themselves and not external evaluators
The success rate for project grants has been around 25%, but unfortunately the success rate for project grants in the junior category is aournd half that.

Evaluation Panels and how they are formed

Computer Science is one of 19 specialist areas of the Natural and Engineering Sciences (NT) division of the Science Council. Applications in each specialist area of are handled by a so-called evaluation panel who have the job of suggesting which applications should be funded and by how much. The panel chair usually serves for 3 years, but has to be formally re-elected each year. The chair's first responsibility is to propose a committee for that year's evaluation round.

The Scientific Council of NT have decided that committees are allowed to have up to a maximum of 13 members (including the chair). VR's guidelines for evaluation panels' composition instruct us to strive for a gender balance, with 40-60% of each sex, and they note that: "Foreign researchers may be appointed as evaluation-panel members, and the option of including one or more international members in the evaluation panel is to be borne in mind."

For the Computer Science panel, I strongly believe that it is preferable to have as many international members as possible. With an international panel there seems little danger that Swedish political issues arise, even subconsciously. Recruiting panel members internationally makes it possible (although not easy) to find panel members of a very high calibre. For example in 2012, two of our panel, were ERC advanced grant holders. (To the best of my knowledge there are only two ERC seniors in the Computer Science field in all of Sweden, both of whom are working on topics at the border with Mathematics.) For this reason, in 2011 and 2012 I selected a committee of 11 international researchers and one Swedish member (the co-chair).

Composition of the Panel

The aim in selecting an evaluation panel is to find a good spread of Computer Scientists who bring both specialist knowledge, and an ability to discuss the merits of applications in a broad range of areas, not just their own. The most challenging aspect is perhaps combining this with the requirement for a good gender balance. In 2012 we had four female panel members, and the search process in which around 20 where invited to participate. The selection process takes a considerable amount of time, with the goals of good subject coverage and gender balance making the job very sequential. Each proposed member is discussed with the co-chair and an appointed observer from the NT committee. Forming a committee with such a high number of international participants takes much more time than if one recruits from Sweden. As a Swede (I use this word in the loose sense!) one feels a certain obligation to participate when asked. As a result, in 2011 and 2012 the formation of the committee was not complete until a short while after the submission deadline in April, and not in time for the applicants to see the committee before submitting.

Assignment of Applications to Panels and Panel Members

In May the applications become visible to the chair. At this point applications which are on the boundaries between different panels are discussed, and some applications are moved from one panel to another in an attempt to find the most suitable panel. For Computer Science the main trading partners are in signals and systems, various bio panels, and Mathematics.

The next stage is to assign the applications to the panel members for reviewing. Each application is assigned a "primary reviewer" and two secondary reviewers from the panel. The primary reviewer is responsible for the final overall report on the application, and begins by writing a preliminary review; the two secondary reviewers also grade the application and provide further "evaluation notes" for the primary reviewer.

It has been customary in the VR panels to assign the applications to reviewers manually. When I previously served on the committee in 2005 I found this rather unsatisfactory. As a chair it also make the handling of conflict cases difficult. I have adopted a more civilised approach and use a standard conference management system which allows the panel members to express their preferences (and mark their conflicts) based on the titles and abstracts of the applications. Using this the assignments were made automatically with almost no manual adjustment.

The Evaluation Process (regular grant categories)

The evaluations are made by the panel members themselves; VR's instructions imply that panel members are not permitted to request reviews from "subreferees", and are expected to write their own reviews. A small number of additional external reviews are permitted, but these must be requested via VR. One issue with external reviewers is that they are paid for their reviewing work. For this reason, each panel has a budget for external reviews: a maximum of 10% of applications may be sent out to an external reviewer. In the CS panel we use this possibility to find one or two external reviewers for clusters of applications in areas where we feel a clear lack of expertise on the panel. Often, however, having a small number of applications reviewed by an external reviewer is not as helpful as one might hope in achieving a final ranking of applications.

The Selection Process

At the end of the summer the panel convenes for a two-day physical meeting to discuss the grants. The purpose of the meeting is to form a linear ranking of the better applications in each grant category. At this stage there is a known lower bound on the amount of money that can be awarded in the project and junior grant categories. Based on an estimate of the average grant size at this point we have an initial estimate of the number of applications that can be funded in each category. In a later stage a few (typically 2-4) further applications below the "cut" are granted. This takes place at a separate meeting involving the chairs of a cluster of panels. For 2012, with the support of the panel I strongly advocated the Junior category grants over the senior ones at this stage.

In 2011 and 2012 all of the panel members attended the two-day selection meeting, and conducted a professional and serious evaluation of the applications. During the discussions no attention is paid the geographic distribution of grants across Sweden, or to the distribution of grants by subject area. I felt that panel members were able to provide objective evaluations of their own areas of expertise without acting as advocates for their own fields. As instructed by VR, the distribution of grants by gender is monitored carefully during the meeting. The panel also make a serious attempt to give a fair hearing to applications which border on other disciplines -- although evaluating inter-disciplinary research is recognised as a tricky problem.

With the exception of the Chair and co-chair, there were very few conflicts of interest. The Chair and co-chair were automatically deemed in conflict with all applications from their own department, and where not present in the room for the discussion of those applications.

Special Grant Categories (2012)

In 2012 there were a number of special grant categories -- the so-called directed calls -- including the recurring theme of ICT Framework Grants. Applications in these categories are divided across several panels. Each panel ranks and agrees an overall grade for the applications for which it is responsible, and this output is provided to a cross-panel committee consisting of panel chairs and external members for the final selection process. This upper committee conducts a second stage of evaluation with an eye on the "strategic" aspects of the proposals.

The Final Reports

The primary reviewer for each grant has the responsibility of producing the final written review for the applications that were assigned to him/her. The task here is to combine the material from the internal preliminary reviews, together with the main points that were discussed at the physical meeting. This task is overseen by the chair and the VR member of staff charged with assisting the panel; we both went through the final reports and tried to check that they gave a reasonable reflection of the various opinions expressed by the reviewers.

Statistics

Project Grant success rate 2012 (2011,2010,2009) : 20% (36,21,23)

Project, Junior category * 2012 (2011.2010,2009) : 12% (14,14,15)

(* Refers to all project grants awarded to applicants 0-7 years since PhD; in 2012 this coincided with the Junior grant category)

I would like to note here that the funding rate for the Junior positions in Computer Science is disturbingly low. VR keep to their promise that at least a third of the total funding is directed to this group. Unfortunately, in the Computer Science area the number of applicants in this category is about the same as the number of applicants in the "senior" category. The situation in other panels in NT is, for the most part, much better in this regard.

While it may be tempting for junior researchers to try their luck in the Senior category, it should be noted that the statistics from 2009-2012 show that this is probably not a good strategy. The youngest researchers who are not eligible to apply for junior research grants, namely those with 8-9 years since their PhD, have a very low success rate: out of 39 applications from this group (2009-2012), only 3 were funded.

Concluding Thoughts

I first came to Sweden just over 17 years ago. If my memory serves me well, at that time the Computer Science panel of VR's predecessor was the only panel to have international members. I'm happy to have pushed this a stage further to form the first panel with a (large) majority of non-Swedish members [Edit: I wasn't the first: this job already accomplished by my predecessor, Stefan Arnborg, who had ten international panelists in 2010]. But I can't help thinking that the committee might function even better without any Swedes at all. Perhaps the chair of one of our Scandinavian neighbours might be interested in a job swap?