Monday, October 23, 2017

Statistics Sunday: How Does the Grant Review Process Work?

While I've posted my Statistical Sins posts late a few times, this is the first time I missed Statistics Sunday. Unfortunately, once NaNoWriMo starts up, this will become more likely, so I'm hoping to sit down in the coming days and write a few Statistics Sunday posts ahead of time. There's still time for suggestions if there are any topics you want to see here!

Today, I thought I'd follow up on a post from the other day, about a proposal for a new grant review process in the National Science Foundation, a federal agency that supports scientific research in many areas. Though NSF is funded by taxpayer money, it is an independent agency, which is meant to shield it from political influence. And in an America where scientific issues, like vaccination and climate change, become more and more politicized, that division is very important.

I realized, as I was reading about this proposal, that people might not be familiar with how peer review occurs for grants. So I thought I'd quickly outline what that process looks like. I should add a disclaimer that the process I'm about to outline is how things are done for Veterans Affairs merit-based grant reviews, but I'm told the process is similar to what they do in the National Institutes of Health (NIH) (and I would imagine NSF as well).

First, within any institute awarding grants, there are multiple panels. These panels reflect different themes within that area of research. Experts in that area of research are invited to serve on these panels; this is not their full-time job, but a service they do for their profession. Usually, they are researchers themselves, but they might also be practitioners in the field. Each panel also has a program manager (or similar title) who oversees the review process, organizes meetings, assigns proposals, and so on.

When you submit a grant, you can often request which panel your proposal goes to. This information will be taken into account, but there is absolutely no guarantee your proposal will go to the panel you request. If there are too many conflicts of interest (people on the panel who have to recuse themselves when it comes to discussing your proposal), or the proposal doesn't seem to fit that panel, it would probably go somewhere else.

Once those decisions are made, the program manager will send the information on the proposals to the members of the panel. Panel members tell the program manager which proposals they cannot review (because of a conflict of interest). The program manager then assigns each proposal a primary, secondary, and tertiary reviewer.

These reviewers read the proposal ahead of time, and use a scoring template to assign ratings to each part of the proposal. They also note strengths and weakness of the proposal. Basically the job of these reviewers is to know the proposal very well. And here's why:

When the full panel meets, it is the job of those three reviewers to present the research to the rest of the panel. (Note: Anyone who has to recuse themselves wouldn't be involved in the discussion.) The primary reviewer will go first, being given the most time to summarize the proposal and then give his or her initial assessment. The secondary reviewer is then asked if he or she has anything to add to the primary reviewer's summary and assessment. The tertiary reviewer goes last, once again being asked for additional comments. The secondary reviewer gets less time than the primary reviewer, and the tertiary reviewer less time than the secondary.

Next, the full panel discusses the project, asking questions of the reviewers (such as, "Does the proposal talk about X?") and discussing the strengths and weaknesses of the proposal. While it seems the opinion of the three reviewers would have a lot of weight when it comes to decisions about whether to fund the proposal (spoiler alert: funding decisions aren't up to anyone on the panel), the rest of the panel can (and do) identify strengths and weaknesses (more likely) the reviewers missed. The reviewers are able to change their scores and the comments in their review as a result of the discussion.

The panel does not decide which proposals get funded. Let me say that again, for emphasis: The panel does not choose which proposals will receive funding. They have a good idea which proposals they think should be funded or not, but that's not their call. Their job is for each proposal to be scored. Decisions are made later - and by someone else - what the funding cut scores are. That is, based on the distribution of scores on the proposals, and how much money is available to award, a score is chosen and anything above that score is funded. Lots of very strong proposals will find themselves on the wrong side of the funding line. And there's nothing anyone on the panel can do about that.

My point is that, despite what people outside of the research and/or soft money settings think, the possibility of abuse of the system is small. If I wanted to guarantee my project is funded through influence, I'd have to 1) make sure my proposal makes it to the panel I want, 2) make sure at least one of the people I can influence is a reviewer (and also make sure there won't be any obvious conflicts of interest that anyone else on the panel can point out), 3) get my influenced reviewer to convince the other reviewers to give my proposal a good score, 4) get those reviewers to convince the panel that any flaws in my study aren't dealbreakers (and also make sure no one on the panel is aware that one of my reviewers should have recused him- or herself), and 5) get my proposal scored so that it will be above the funding line, which is not known until later.

Dude, it would just be easier to put my energy into writing a strong proposal.

And if the reviewers give your proposal a bad score to begin with, your proposal likely won't even be discussed in the meeting. The focus will be on those proposals that could be fundable, depending on where the cutoff is set. Program managers take notes during the meeting, and can usually share some information about what was discussed with the principal investigator of the proposal. This information is useful if the PI wants to revise their proposal and submit it at the next grant cycle. There are usually limits to how many times you can submit the same proposal. And if your proposal isn't discussed at the meeting (that is, it was "triaged"), you sometimes can't revise and resubmit. You can likely recycle pieces of the proposal, but basically have to submit as a brand new proposal.

There's a great deal of uncertainty when it comes to grant money. This is the main reason I left VA; as much as I loved the people, the mission, and the job itself, I didn't want to live my life from grant cycle to grant cycle. It was a very personal choice, and I have nothing but admiration and respect for my colleagues still fighting the good fight.

2 comments:

  1. Upon seeing the headline, I had thought this entire post would just read, “It doesn’t.”

    ReplyDelete
    Replies
    1. Touché. Perhaps "How is a Grant Review Process Supposed to Work?"

      Delete