Tuesday 5 February 2013

Evaluation of Grant Applications within VR's Computer Science Panel: A Report from the Panel Chair

For the last couple of years I have had the honour of chairing the Computer Science Panel of the Swedish Research Council. I am scheduled to chair this committee for a third (and final) period in 2013. In this note I'd like to explain to the potential applicant how the evaluation process works. I'm writing this because many researchers, particularly those who have little contact with people who have previously served on a VR panel, have misconceptions about how their application is handled and what chances they have of receiving funding. Since the way that we work with applications is not intended to be secret, I hope this extended note may be of some interest. Since it is rather long let me point out some key points:
  1. Almost all grants are evaluated by the evaluation panel themselves and not external evaluators
  2. The success rate for project grants has been around 25%, but unfortunately the success rate for  project grants in the junior category is aournd half that.

Evaluation Panels and how they are formed

Computer Science is one of 19 specialist areas of the Natural and Engineering Sciences (NT) division of the Science Council. Applications in each specialist area of are handled by a so-called evaluation panel who have the job of suggesting which applications should be funded and by how much.  The panel chair usually serves for 3 years, but has to be formally re-elected each year. The chair's first responsibility is to propose a committee for that year's evaluation round.

The Scientific Council of NT have decided that committees are allowed to have up to a maximum of 13 members (including the chair).  VR's guidelines for evaluation panels' composition instruct us to strive for a gender balance, with 40-60% of each sex, and they note that: "Foreign researchers may be appointed as evaluation-panel members, and the option of including one or more international members in the evaluation panel is to be borne in mind."

For the Computer Science panel, I strongly believe that it is preferable to have as many international members as possible.  With an international panel there seems little danger that Swedish political issues arise, even subconsciously. Recruiting panel members internationally makes it possible (although not easy) to find panel members of a very high calibre.  For example in 2012, two of our panel, were ERC advanced grant holders. (To the best of my knowledge there are only two ERC seniors in the Computer Science field in all of Sweden, both of whom are working on topics at the border with Mathematics.)  For this reason, in 2011 and 2012 I selected a committee of 11 international researchers and one Swedish member (the co-chair).

Composition of the Panel

The aim in selecting an evaluation panel is to find a good spread of Computer Scientists who bring both specialist knowledge, and an ability to discuss the merits of applications in a broad range of areas, not just their own.  The most challenging aspect is perhaps combining this with the requirement for a good gender balance. In 2012 we had four female panel members, and the search process in which around 20 where invited to participate. The selection process takes a considerable amount of time, with the goals of good subject coverage and gender balance making the job very sequential.  Each proposed member is discussed with the co-chair and an appointed observer from the NT committee. Forming a committee with such a high number of international participants takes much more time than if one recruits from Sweden. As a Swede (I use this word in the loose sense!) one feels a certain obligation to participate when asked. As a result, in 2011 and 2012 the formation of the committee was not complete until a short while after the submission deadline in April, and not in time for the applicants to see the committee before submitting.

Assignment of Applications to Panels and Panel Members

In May the applications become visible to the chair. At this point applications which are on the boundaries between different panels are discussed, and some applications are moved from one panel to another in an attempt to find the most suitable panel.  For Computer Science the main trading partners are in signals and systems, various bio panels, and Mathematics.

The next stage is to assign the applications to the panel members for reviewing. Each application is assigned a "primary reviewer" and two secondary reviewers from the panel. The primary reviewer is responsible for the final overall report on the application, and begins by writing a preliminary review; the two secondary reviewers also grade the application and provide further "evaluation notes" for the primary reviewer.

It has been customary in the VR panels to assign the applications to reviewers manually. When I previously served on the committee in 2005 I found this rather unsatisfactory. As a chair it also make the handling of conflict cases difficult. I have adopted a more civilised approach and use a standard conference management system which allows the panel members to express their preferences (and mark their conflicts) based on the titles and abstracts of the applications. Using this the assignments were made automatically with almost no manual adjustment.

The Evaluation Process (regular grant categories)

The evaluations are made by the panel members themselves; VR's instructions imply that panel members are not permitted to request reviews from "subreferees", and are expected to write their own reviews.  A small number of additional external reviews are permitted, but these must be requested via VR. One issue with external reviewers is that they are paid for their reviewing work.  For this reason, each panel has a budget for external reviews: a maximum of 10% of applications may be sent out to an external reviewer.  In the CS panel we use this possibility to find one or two external reviewers for clusters of applications in areas where we feel a clear lack of expertise on the panel. Often, however, having a small number of applications reviewed by an external reviewer is not as helpful as one might hope in achieving a final ranking of applications.

The Selection Process

At the end of the summer the panel convenes for a two-day physical meeting to discuss the grants. The purpose of the meeting is to form a linear ranking of the better applications in each grant category. At this stage there is a known lower bound on the amount of money that can be awarded in the project and junior grant categories. Based on an estimate of the average grant size at this point we have an initial estimate of the number of applications that can be funded in each category. In a later stage a few (typically 2-4) further applications below the "cut" are granted. This takes place at a separate meeting involving the chairs of a cluster of panels. For 2012, with the support of the panel I strongly advocated the Junior category grants over the senior ones at this stage.

In 2011 and 2012 all of the panel members attended the two-day selection meeting, and conducted a professional and serious evaluation of the applications. During the discussions no attention is paid the geographic distribution of grants across Sweden, or to the distribution of grants by subject area. I felt that panel members were able to provide objective evaluations of their own areas of expertise without acting as advocates for their own fields. As instructed by VR, the distribution of grants by gender is monitored carefully during the meeting.  The panel also make a serious attempt to give a fair hearing to applications which border on other disciplines -- although evaluating inter-disciplinary research is recognised as a tricky problem.

With the exception of the Chair and co-chair, there were very few conflicts of interest. The Chair and co-chair were automatically deemed in conflict with all applications from their own department, and where not present in the room for the discussion of those applications.

Special Grant Categories (2012) 

In 2012 there were a number of special grant categories -- the so-called directed calls -- including the recurring theme of ICT Framework Grants.  Applications in these categories are divided across several panels.  Each panel ranks and agrees an overall grade for the applications for which it is responsible, and this output is provided to a cross-panel committee consisting of panel chairs and external members for the final selection process. This upper committee conducts a second stage of evaluation with an eye on the "strategic" aspects of the proposals.

The Final Reports

The primary reviewer for each grant has the responsibility of producing the final written review for the applications that were assigned to him/her. The task here is to combine the material from the internal preliminary reviews, together with the main points that were discussed at the physical meeting.  This task is overseen by the chair and the VR member of staff charged with assisting the panel; we both went through the final reports and tried to check that they gave a reasonable reflection of the various opinions expressed by the reviewers.

Statistics

Project Grant success rate   2012 (2011,2010,2009) : 20% (36,21,23)
Project, Junior category *    2012 (2011.2010,2009) : 12% (14,14,15)

 (* Refers to all project grants awarded to applicants 0-7 years since PhD; in 2012 this coincided with the Junior grant category)


I would like to note here that the funding rate for the Junior positions in Computer Science is disturbingly low. VR keep to their promise that at least a third of the total funding is directed to this group. Unfortunately, in the Computer Science area the number of applicants in this category is about the same as the number of applicants in the "senior" category. The situation in other panels in NT is, for the most part, much better in this regard. 

While it may be tempting for junior researchers to try their luck in the Senior category, it should be noted that the statistics from 2009-2012 show that this is probably not a good strategy. The youngest researchers who are not eligible to apply for junior research grants, namely those with 8-9 years since their PhD, have a very low success rate: out of 39 applications from this group (2009-2012), only 3 were funded.  

Concluding Thoughts


I first came to Sweden just over 17 years ago.  If my memory serves me well, at that time the Computer Science panel of VR's predecessor was the only panel to have international members.  I'm happy to have pushed this a stage further to form the first panel with a (large) majority of non-Swedish members [Edit: I wasn't the first: this job already accomplished by my predecessor, Stefan Arnborg, who had ten international panelists in 2010]. But I can't help thinking that the committee might function even better without any Swedes at all.  Perhaps the chair of one of our Scandinavian neighbours might be interested in a job swap?

Links



I am indebted to VR's staff, in particular Andreas Augustsson, for outstanding support thoughout the whole process, and in particular for providing the statistics which are summarised above.


8 comments:

  1. I strongly applaud and second the suggestion of a completely non-Swedish panel and to have as wide a coverage as possible of all major areas of Computer Science and Engg (which has conspicuously not been the case in the past).

    ReplyDelete
  2. Thank you very much for the interesting insight - it is very useful for VR applicants like myself to gain access to specific information regarding the CS area. I have a few comments and questions regarding your post:

    You say that "During the discussions no attention is paid the geographic distribution of grants across Sweden, or to the distribution of grants by subject area." Does the particular institution of the applicant weigh in any way in the evaluation? Is preference accorded to applications from "older" institutions (e.g., Uppsala, Lund, KTH), perhaps because they have a more established record in the particular field of the proposal?

    You mention that "as instructed by VR, the distribution of grants by gender is monitored carefully during the meeting." What does this mean exactly? Is there an amount of the budget that is "reserved" for women? If so, is this amount set aside before the application process or is it the same pot of funding? Are there explicit policies you follow in determining the amount of positive discrimination that occurs? I think it is very important to be totally transparent on this issue - if, for instance, VR decides that female applicants should be more likely to get funded, it should be stated explicitly to what extent this is done, and the guidelines that are followed in according preference based on gender should be made public.

    When you speak about final reports, you say that "material from the internal preliminary reviews" is combined to achieve a final report. This suggests that the review process produces material on the proposals that is not shown to the applicants. Is this material available? I say this also because reports are usually very brief and not very informative (i.e., it is often not clear what one should do to improve the application), therefore any further insight on the reasons for a rejection would be greatly appreciated.

    Thank you in advance for any further information you can give us.

    Federico Pecora - Örebro.

    ReplyDelete
  3. Federico, thanks for the interesting question. I'll start with the first one for now and get to the others later.

    Since the committee are non-Swedish it is rather easy for them to ignore the distinctions between old and new universities. The research group in which the applicant works may have some influence (in particular co-applicants) but whether it is an old University or a new one really doesn't matter. From the viewpoint of most international reviewers it's all just Swedish. (As I once remarked to the previous Chalmers president: Chalmers is world-famous ... in Gothenburg).

    The previous track-record of a researcher will certainly be highly relevant. But in the case of a junior applicant who has a good application, the lack of a strong group behind him or her might equally be used as a positive argument. But again, this issue is discussed independently of the University from which the applicant applies.

    ReplyDelete
    Replies
    1. Regarding gender issues, VR's policy is described in depth (in Swedish) below.

      https://www.vr.se/omvetenskapsradet/strategierochriktlinjer/jamstalldhetsstrategi20102012.4.1f599ea412a30327ccf800042.html

      There is no reserved budget for women, but VR have stated the goal that the number of grants awarded to each gender should be in proportion the the number of applications. Gender issues are monitored in the sense that we are reminded (by the VR appointed observer for example) of the number of female applicants that are likely to be funded as the discussion progresses. In some panels there is even a "gender observer" who attends the meeting and makes notes on the way that applications from female researchers are handled (for example how much discussion time is devoted to discussing applications from applicants of one gender vs the other). In the end I believe that positive discrimination may be applied in the "usual way", namely when it is hard to order two applications based on purely scientific merits; in that case gender may be used as a tie-breaker.

      If you want to know more about how things are supposed to operate, then I can also recommend the evaluation handbook.

      http://www.vr.se/download/18.a1218571375c85932a70/1337586115700/Beredningshandbok+NT+2012.pdf

      Delete
  4. Links to VR's evaluation handbook in Swedish and English now added to the Links section.

    ReplyDelete
  5. On Fredrico's final question "This suggests that the review process produces material on the proposals that is not shown to the applicants. Is this material available?"

    The evaluators make notes which are not intended for the reviewers, and so they contain a mixture of useful material (for the applicant) that makes its way into the final report, and possibly other notes that may not be suitable for that purpose. Since this material is internal working matter, it cannot be obtained from VR.

    This is not unlike programme committees and journal reviewers make notes for the committee only. These are never made available to the authors.

    Having said that, I think I would prefer a process more like the one we are used to from conference evaluation, where each reviewer writes a full report intended for consumtion by the author. But this is not how VR choose to work.

    ReplyDelete
  6. Thanks for the interesting blog.

    I have one question regarding the funding decision. You wrote that "Applications in each specialist area are handled by a so-called evaluation panel who have the job of suggesting which applications should be funded and by how much." and "The purpose of the meeting is to form a linear ranking of the better applications in each grant category."

    I noticed the choice of word "suggesting". My main question is: Does the panel also make the final decision? Or, is it the NT board (a group of 9 people) that makes the decisions over all subject areas (of course, hopefully, based on the panels' input)? In the latter case, do the decisions coincide with the recommodations, in your experience?

    Also, could you please clarify the meaning of "better applications" above?

    Thank you.

    ReplyDelete
    Replies
    1. In principle the NT board could decide to reorder the list of applications, but in practice I suspect that this does not happen often, so in effect the panel makes the final decision, with the exception of one or two applications at the tail of the list which are chosen in a second round after comparison with applications from 4 or 5 other panels.

      The "better applications" are not a formal concept. I was referring to those applications for which the preliminary reviews suggest that there may be a chance of awarding funding.

      There is no point in ranking applications which have no chance of getting funding.

      Delete