banner
You are not using a standards compliant browser. Because of this you may notice minor glitches in the rendering of this page. Please upgrade to a compliant browser for optimal viewing:
Firefox
Internet Explorer 7
Safari (Mac and PC)
Post Archive
2017 (0)2011 (1)
January (1)

Et tu Odysseyus?
Tuesday, January 11, 2011
2010 (37)
December (5)

First most influential paper
Tuesday, December 14, 2010

Twelve (minus one) Months of pondering Blather
Tuesday, December 7, 2010

I'll save Tideliar the trouble
Tuesday, December 7, 2010

Thinking differently
Friday, December 3, 2010

Music Warz! - The Maccabees
Thursday, December 2, 2010
November (6)

Standing out in a crowd: An addendum
Monday, November 8, 2010

Standing out in a crowd
Friday, November 5, 2010

Ripping your arms off
Tuesday, November 2, 2010

The Tea Party explained
Tuesday, November 2, 2010

Words of wisdom for prospective graduate students
Monday, November 1, 2010

Grant advice
Monday, November 1, 2010
October (10)

Planet of the Apes
Wednesday, October 27, 2010

Too many postdocs?
Tuesday, October 26, 2010

Oh! Rats! [UPDATED]
Thursday, October 21, 2010

Rethinking Education
Monday, October 18, 2010

Elephant man, rabies and leprosy
Wednesday, October 13, 2010

Over-priced mochas and syphilis
Sunday, October 10, 2010

DonorsChoose - give early and give often. [UPDATED]
Friday, October 8, 2010

Question
Wednesday, October 6, 2010

What I would be doing if I weren't doing science
Tuesday, October 5, 2010

Time spent reviewing
Monday, October 4, 2010
September (6)

Dear PI's who wrote the NSF proposals I am now reviewing...
Wednesday, September 29, 2010

Funding Illusions
Tuesday, September 28, 2010

FIve years ago today
Wednesday, September 22, 2010

A (temporary) cure for vortices of suckitude
Wednesday, September 15, 2010

Things that make Odyssey grumpy
Tuesday, September 14, 2010

What I wish I knew before starting my faculty position
Tuesday, September 7, 2010
August (10)

Flying 101
Tuesday, August 24, 2010

Don't panic!
Monday, August 23, 2010

One to Rule Them All
Friday, August 20, 2010

The NSF review panel process
Thursday, August 19, 2010

Peer review, schmeer review
Friday, August 13, 2010

Hypotheses: The most disposable of lab supplies
Thursday, August 12, 2010

How much do you need to want it?
Monday, August 9, 2010

Bugger
Sunday, August 8, 2010

REPOST: How Many Papers for Tenure?
Thursday, August 5, 2010

Checking it out
Thursday, August 5, 2010
Rate This Post
Total votes: 4
Blogger Profile

Odyssey

I'm a molecular biophysicist in a biochemistry department. In a college of medicine. And I'm funded by the NSF. Not too sure my dean likes that... I'm here to blather on about things that interest me and to raise the average age of the bloggers here by at least 1.2567 years. And I'm Australian.

My posts are presented as opinion and commentary and do not represent the views of LabSpaces Productions, LLC, my employer, or my educational institution.

Blog RSS Feed
RSS Add to My Yahoo Add to Google
Recent Comments

Haha the comment above is funny! Research before hypothesis! . . .Read More
Aug 09, 2012, 1:48pm
Comment by Tim in Peer review, schmeer review

Good article! "Do proponents of this system have any idea just how hard it is to find reviewers under the current system?" I've just stumbled across this artcile, so my comments are. . .Read More
Oct 13, 2011, 5:20pm
Comment by Tideliar in Standing out in a crowd

Rules of the Internets, #3"Don't necro dead threads n00b"Corollary Rule 3.1"If the best you can add is content-lite, don't fucking necro dead threads n00b"Corollary Rul. . .Read More
Jul 06, 2011, 5:48pm

anyone interested in helping my kindergarteners? Check out http://www.donorschoos. . .Read More
Jan 25, 2011, 10:47am
Comment by Anonymous in Standing out in a crowd

Thanks for the post! Very helpful. - Reshmi . . .Read More
Jan 13, 2011, 12:44am
Awesome Stuff
Wikio RSS:

http://www.wikio.com

ResearchBlogging:

researchblogging.org

Site Meter

Locations of visitors to this page

Blogroll
Oct 15, 2014, 6:55am
Jan 26, 2014, 10:58am
Feb 29, 2012, 10:31am
Nov 19, 2011, 4:38pm
Feb 17, 2011, 9:40am
Feb 11, 2011, 8:27pm
Nov 07, 2010, 4:50pm
Thursday, August 19, 2010

I will be sitting on an NSF review panel a few weeks from now. I thought I would use this as an opportunity to describe how the NSF review panels I sit on tend to run.

Let me start by pointing out that this is all based on my experience with panels in the BIO Directorate, that NSF Program Officers (PO's) have a lot of leeway in terms of how these things work, and that I am aware that other panels have different modus operandi.

Pre-Panel Selection
The panels held this Fall, including the one I'm on, will be dealing with proposals that were submitted for the July deadlines/target dates. Between then and now the PO's have been busy sorting those proposals into the right areas and finding reviewers. These are distinct from panel members. An NSF proposal is typically sent out to anywhere from two to six (maybe more) reviewers prior to panel formation. These will submit their reviews via the FastLane system before the panel meets.

Panel Selection
To the best of my knowledge there are no "standing members" on NSF BIO review panels. Regulars, yes. Permanent members, no. The PO contacts prospective panel members based on the portfolio of proposals under review. The goal is to have a panel of ~15 people whose collective expertise covers the entire portfolio. PO's, IME, most often succeed at meeting this goal. The panel size is limited to about 15. In my opinion this is a good size. It's large enough for their to be a broad range of expertise in the room, yet small enough that everyone can be involved in the discussion without needing to shout to be heard.

Post-Panel Selection
Once the PO has assembled their panel they will send out a list of the proposals that will be up for discussion. On the panels I've sat on that's typically 60-90. Each panelist will be required to review ~10 proposals. The panelists sort through the list and indicate which they would feel comfortable reviewing and which, if any, they have a conflict of interest (COI) with. The PO then takes that information and assigns proposals for each panelist to review. Generally each proposal will have at least two, usually three and sometimes more, panelists as reviewers. If you're keeping count, that's a total of four to nine+ reviewers when you add in the outside (non-panel) reviewers. Panelists will be assigned as primary reviewer, secondary reviewer, or scribe (more about that below). Any panelist can review any proposal, even if they're not assigned to it, if they so desire. I haven't seen that happen (on a NSF panel - AHA panel I have, but that's a story for another day), but in principle it could.

All panelists are required to complete their reviews on the FastLane system prior to the meeting.* Once a review is submitted, but not before, the panelist will be able to view all other reviews for that proposal. Most panelists I know, including myself, try to get their reviews completed at least a few days before the meeting so they have time to read and digest the contents of the other reviews. It is important to be well-prepared for the actual meeting.

The Panel Meeting
The review panel meetings I've sat on are scheduled to last for almost three days. Meetings are usually scheduled to end at about 3pm on the last day, but often finish by about noon. It's an intense two-and-a-half days. They start at 8am and go until some goal is met (usually half of the proposals are discussed each of the first two days), with just the one break for lunch. The first two days are devoted to discussing proposals, and the last day to ranking them. All proposals are discussed - there is no triage.

All panelists have a laptop logged in to the FastLane system. This allows everyone to read the various reviews of each proposal, look at the proposal itself, etc. It's a pretty cool way of doing business.

Proposal discussions tend to go as follows: Anyone with a COI leaves the room. Then the primary reviewer gives a short synopsis of the proposal, and describes the salient points of their review. The secondary reviewer will then chime in with any additional points and/or points of disagreement. Then it's the scribes turn. The secondary reviewer or the scribe also gives a description of the reviews submitted by outside reviewers. There is then general discussion to which all panel members can (and often do) contribute. For a panelist this is the fun part. Once some consensus is obtained, the proposal is given some kind of preliminary ranking. There's a big whiteboard in the room on which this ranking is done.

Why is the scribe called a scribe? They write the panel summary. This is done after the discussion and preliminary ranking of the proposal, while the next proposals are being discussed. The scribe writes up a summary and this is submitted to the panel via the FastLane system. All panel members (except those with a COI) can then suggests changes. Eventually the final version is submitted for approval. All panel members (except COI's) must approve the final version.

The final half-day is spent finishing up panel summaries and then re-ranking, where appropriate, the proposals. When the panel is done, the PO's have a rank ordering of the proposals they can then use to decide on funding. Note that this ordering is a recommendation - NSF PO's have a lot of latitude in determining who gets funded. Of course if they deviated too far from panel recommendations panel members would refuse to serve again.

There will be one, sometimes two, PO's who run the meeting. Others will drift in and out depending on whether a proposal in their personal portfolio is being discussed. The PO's, who are always present, keep panels on a tight leash. They will quickly curb discussions that wander off-topic. They will squelch any inappropriate comments. They will defend proposers (and outside reviewers) whom they feel are not receiving appropriate treatment. They will (try to) diffuse arguments between panelists. They will ensure the panel summary reflects the actual discussion.

Final Thoughts
I have heard some people complain about the behavior of some review panels they've served on. Panelist's not paying attention. Making uncalled for derisive comments about proposers. Unfairly trashing competitors proposals. PO's not doing their jobs. Etc. This has not been my experience. Every panelist, with one exception*, and PO that I've served with has been very professional. It's always been a fun and educational experience for me. I'm looking forward to attending the meeting in a few weeks.









* I once sat on a panel where one panelist turned up without having submitted a single review. His excuse was that he wanted to hear what the other panelists thought first. I was sitting next to him - it was clear from his laptop screen and his lack of participation in the proposal discussions that he hadn't done any of the reviews prior to the meeting. To say the PO was not amused would be an understatement. This happened about a decade ago - I strongly suspect that panelist is still on a list at the NSF. Not the kind of list you want to be on.

This post has been viewed: 8226 time(s)

Blog Comments

Genomic Repairman
Rate Post:

Like 0 Dislike
Nice walkthrough of the NSF process. How late do you guys run into the evening? Also does the NSF give special consideration for reviewers (rolling grant acceptances) like the NIH does for their reviewers?

Odyssey
Rate Post:

Like 0 Dislike
Usually we're done by 7pm, but I have heard of panels going until 11pm. It depends on how many proposals are being reviewed. With ~60 we can be done by 6ish. With 90, much, much later.

There are no special considerations (e.g. later submission dates or special review panels) for reviewers or panelists. But I don't think PO's ask people to serve on panels if they have a proposal pending. Could be wrong on that...

Prof-like Substance
Rate Post:

Like 0 Dislike
Thanks for writing this, Odyssey. It'll certainly be helpful for me come October, when I head to DC.

Evie
Rate Post:

Like 0 Dislike
Is there like a peer review that decides what would be a COI? Or do you rely on each member to decide that for themselves?

Odyssey
Rate Post:

Like 0 Dislike
@Evie:
It's pretty much up to the reviewer to decide what is and what isn't a COI. The PO's are more than happy to help out if you're not sure. Generally it's things like reviewing grants from collaborators, old friends, people at your institution (even if you don't know them), etc. I would like to hope panel members are honest about this. When we agree to review each proposal we are legally bound to declare conflicts.

Brian Krueger, PhD
Columbia University Medical Center
Rate Post:

Like 0 Dislike

Odyssey
Rate Post:

Like 0 Dislike
LOL, this is the 7th result on Google for "NSF Review Panel" searches...

Wow!
BHR

Guest Comment

Excellent explanation of the review process. Thanks a lot for your time in writing this post.


Odyssey
Rate Post:

Like 0 Dislike

I hope it's useful.

Add Comment?
Comments are closed 2 weeks after initial post.
Friends