Thursday, August 19, 2010
I will be sitting on an NSF review panel a few weeks from now. I thought I would use this as an opportunity to describe how the NSF review panels I sit on tend to run.
Let me start by pointing out that this is all based on my experience with panels in the BIO Directorate, that NSF Program Officers (PO's) have a lot of leeway in terms of how these things work, and that I am aware that other panels have different modus operandi.Pre-Panel Selection
The panels held this Fall, including the one I'm on, will be dealing with proposals that were submitted for the July deadlines/target dates. Between then and now the PO's have been busy sorting those proposals into the right areas and finding reviewers. These are distinct from panel members. An NSF proposal is typically sent out to anywhere from two to six (maybe more) reviewers prior to panel formation. These will submit their reviews via the FastLane system before the panel meets.Panel Selection
To the best of my knowledge there are no "standing members" on NSF BIO review panels. Regulars, yes. Permanent members, no. The PO contacts prospective panel members based on the portfolio of proposals under review. The goal is to have a panel of ~15 people whose collective expertise covers the entire portfolio. PO's, IME, most often succeed at meeting this goal. The panel size is limited to about 15. In my opinion this is a good size. It's large enough for their to be a broad range of expertise in the room, yet small enough that everyone can be involved in the discussion without needing to shout to be heard.Post-Panel Selection
Once the PO has assembled their panel they will send out a list of the proposals that will be up for discussion. On the panels I've sat on that's typically 60-90. Each panelist will be required to review ~10 proposals. The panelists sort through the list and indicate which they would feel comfortable reviewing and which, if any, they have a conflict of interest (COI) with. The PO then takes that information and assigns proposals for each panelist to review. Generally each proposal will have at least two, usually three and sometimes more, panelists as reviewers. If you're keeping count, that's a total of four to nine+ reviewers when you add in the outside (non-panel) reviewers. Panelists will be assigned as primary reviewer, secondary reviewer, or scribe (more about that below). Any panelist can review any proposal, even if they're not assigned to it, if they so desire. I haven't seen that happen (on a NSF panel - AHA panel I have, but that's a story for another day), but in principle it could.
All panelists are required to complete their reviews on the FastLane system prior to the meeting.* Once a review is submitted, but not before, the panelist will be able to view all other reviews for that proposal. Most panelists I know, including myself, try to get their reviews completed at least a few days before the meeting so they have time to read and digest the contents of the other reviews. It is important to be well-prepared for the actual meeting.The Panel Meeting
The review panel meetings I've sat on are scheduled to last for almost three days. Meetings are usually scheduled to end at about 3pm on the last day, but often finish by about noon. It's an intense two-and-a-half days. They start at 8am and go until some goal is met (usually half of the proposals are discussed each of the first two days), with just the one break for lunch. The first two days are devoted to discussing proposals, and the last day to ranking them. All
proposals are discussed - there is no triage.
All panelists have a laptop logged in to the FastLane system. This allows everyone to read the various reviews of each proposal, look at the proposal itself, etc. It's a pretty cool way of doing business.
Proposal discussions tend to go as follows: Anyone with a COI leaves the room. Then the primary reviewer gives a short synopsis of the proposal, and describes the salient points of their review. The secondary reviewer will then chime in with any additional points and/or points of disagreement. Then it's the scribes turn. The secondary reviewer or the scribe also gives a description of the reviews submitted by outside reviewers. There is then general discussion to which all panel members can (and often do) contribute. For a panelist this is the fun part. Once some consensus is obtained, the proposal is given some kind of preliminary ranking. There's a big whiteboard in the room on which this ranking is done.
Why is the scribe called a scribe? They write the panel summary. This is done after the discussion and preliminary ranking of the proposal, while the next proposals are being discussed. The scribe writes up a summary and this is submitted to the panel via the FastLane system. All panel members (except those with a COI) can then suggests changes. Eventually the final version is submitted for approval. All panel members (except COI's) must approve the final version.
The final half-day is spent finishing up panel summaries and then re-ranking, where appropriate, the proposals. When the panel is done, the PO's have a rank ordering of the proposals they can then use to decide on funding. Note that this ordering is a recommendation - NSF PO's have a lot of latitude in determining who gets funded. Of course if they deviated too far from panel recommendations panel members would refuse to serve again.
There will be one, sometimes two, PO's who run the meeting. Others will drift in and out depending on whether a proposal in their personal portfolio is being discussed. The PO's, who are always present, keep panels on a tight leash. They will quickly curb discussions that wander off-topic. They will squelch any inappropriate comments. They will defend proposers (and outside reviewers) whom they feel are not receiving appropriate treatment. They will (try to) diffuse arguments between panelists. They will ensure the panel summary reflects the actual discussion.Final Thoughts
I have heard some people complain about the behavior of some review panels they've served on. Panelist's not paying attention. Making uncalled for derisive comments about proposers. Unfairly trashing competitors proposals. PO's not doing their jobs. Etc. This has not been my experience. Every panelist, with one exception*, and PO that I've served with has been very professional. It's always been a fun and educational experience for me. I'm looking forward to attending the meeting in a few weeks.
* I once sat on a panel where one panelist turned up without having submitted a single review. His excuse was that he wanted to hear what the other panelists thought first. I was sitting next to him - it was clear from his laptop screen and his lack of participation in the proposal discussions that he hadn't done any of the reviews prior to the meeting. To say the PO was not amused would be an understatement. This happened about a decade ago - I strongly suspect that panelist is still on a list at the NSF. Not the kind of list you want
to be on.