I’ve Got My Pilot Data. … Now What?

Amid the cutthroat competition for federal research funding, new investigators can feel daunted by the prospect of jumping headlong into the scope of work required for larger federal grants. In applying for funding, they may face conflicting demands from the agencies who fund the research and the scientists who review it. Agencies tend to want high-reward, paradigm-shifting research that proposes grand advancements in science, technology, health or education. Conversely, reviewers may look for evidence of promise, potential or proof-of-concept that suggests a sure bet over a longshot.

Small, university-sponsored grants can help “de-risk” higher-risk research through the process of gathering pilot data, but investigators often struggle to move from pilot work to a competitive proposal—prompting the question: I’ve got my pilot data. … Now what? In this article, I review common conventions and common errors in presenting preliminary data before providing a template for new (and even experienced) researchers to make the most of their preliminary work to improve their chances of federal funding.

Planning

So what is pilot data and how can it be used to support bold new research ideas? Investigators tend to think of pilot data as limited to preliminary results generated from small, self-contained studies that assess feasibility, refine methods, and inform the development of larger research projects. In truth, pilot data is anything that serves to reduce the level of risk or uncertainty a reviewer might assign to a proposal. As such, pilot data need not be limited to self-contained studies—or even add-ons to larger, tangentially related studies. For purposes of grant proposals, pilot data can include any research that demonstrates the technical and scientific promise of your approach. This can include access to particular populations or datasets, a track record of meaningful collaboration among investigators, evidence of procedural rigor and feasibility, proof of instrument validity, or even evidence of conceptual and theoretical merit.

Yet, if pilot data can be used to reduce uncertainty across such a range of domains, the question becomes, how does a researcher determine what data to highlight? Leveraging pilot data requires investigators to first ask themselves what elements of their own proposal are most likely to raise eyebrows among reviewers, and what elements are most likely to prompt skepticism without targeted justification or preliminary evidence. For example, an investigator proposing a sharp departure from prevailing assumptions, the repurposing of existing technologies in novel settings, or innovative but unproven data-collection methods can reduce perceived risk by drawing on their own prior studies that have successfully taken comparable approaches.

Next, investigators will want to collect any and all pilot studies and materials that support the likely success of those innovative departures into a list of raw materials. These raw pilot materials can include all publications for which any member of the study team is an author. More importantly, those pilot data can include a range of materials such as published and unpublished results, tables, figures, other project deliverables like reports and datasets, qualitative interviews and interview quotes, population pools, community partnerships, and collaborations. When first assembling materials, investigators should err on the side of inclusivity—pilot materials are generally more comprehensive than writers tend to assume.

Generally speaking, pilot data summaries should address only the riskiest elements of the proposed work (if it’s not risky—it’s a literature review). Because of this, it is important to prune the pilot materials to support only those risky elements.

Writing

Having identified the risky elements of a grant proposal alongside the data on hand that could be used to reduce reviewers’ perceptions of risk, it is time to ask a few key questions:

1. Does it make more sense to include a single section to discuss all pilot data or should a separate pilot data section be included under each individual research aim or objective?
2. Should the pilot data section be structured study-by-study or concept-by-concept?
3. How much space should be allocated to the pilot data section overall—and to each study within the pilot data section?

There is no optimal strategy that works for every proposal, so these questions must be answered according to the parameters, needs and risks of each individual project.

Nevertheless, there are specific conventions in pilot data sections—but how closely these are followed is determined largely by the specific needs of a proposal and pragmatics, like space availability. For example, a sharp pilot data section will generally begin with an introductory paragraph that acknowledges the risky and transformative elements of a proposal—these are a proposal’s strengths and where the proposed research stands to make a contribution to scientific advancement. The introduction should further provide a thesis statement that offers insights into the aggregate contribution of the pilot data, making the argument that the whole is greater than the sum of its parts. And finally, the introduction should include a roadmap—a sentence that tells the reader how the section will proceed, be it section-by-section or study-by-study.

For each concept or study to be addressed in the pilot data section, investigators should include a short title to identify the study or the concept being introduced. Next, that sub-section should include a short description of the study’s goals or objectives—generally no longer than a sentence or two. In the event that this study was funded by federal, foundation or pilot grant funding, investigators may want to not only mention that the study was funded, but also refer to the grant number. For example, “With our university-funded seed grant, we studied …” After that, investigators should include a short-form description of the study characteristics. These include data participants, core data collection procedures, analytic methods, and results. Once again, avoid the temptation to be exhaustive—and be sure to allocate space as a function of a project’s importance and its ability to provide evidence of proof of concept. Provide enough detail that investigators can adequately assess the scope and scale of a project.

Most importantly, each subsection should end by addressing the concept or study’s relevance to the proposed project. Investigators should make an explicit connection between the pilot data presented and the risks it is designed to address. Specifically, investigators should note whether the pilot data speaks to either:

1. the methodological feasibility of the proposed study (e.g., recruitment, retention, adherence to treatment, task completion, study timeline or data quality). An example might be, “Our 90 percent retention rate between waves 1 and 2 shows robust follow-up capacity and evidence of participant engagement.”
2. the promise or proof-of-concept (e.g., preliminary associations and effect sizes, validated measures or model output). For example, “Preliminary analysis showed that participants who completed the intervention reported consistent reductions in perceived stress, suggesting expected directionality and informing power estimates for aim 1.”

Finally, investigators may want to end their pilot data section with a conclusion paragraph to address opportunities for growth. Most notably, this section should answer the question: What was it about your previous work that kept it from providing the definitive answer to your research question?

At a presentation of this information to scholars at a recent workshop, new investigators were curious to know when it was time to make the jump to applying for federal funding. The honest answer is that pragmatics—approaching deadlines, encouragement to apply from department heads, etc.—play a large role.

Nevertheless, there are a few dos and don’ts to guide that transition. Specifically, pilot data are less about completeness than judgment. Use pilot data to anticipate where reviewers are most likely to question feasibility, rigor or plausibility, and select evidence that directly addresses those points of risk. Don’t treat the pilot data section as a repository for everything you have done to date, or as a substitute for a clearly articulated research question and study design. When used strategically, pilot data do not eliminate uncertainty, but they can demonstrate that investigators understand where their proposal departs from the status quo and have taken concrete steps to manage the risks that follow.