Measuring Impact, Beyond Publications

How can we account more fully for the impact of a faculty member’s research?

There are of course the obligatory publications in respected journals with a high impact rating. Official awards from agencies, institutions or societies are also obvious indicators of impact, as are significant grants and positions. However, there are many other outcomes of impactful research that can be discussed and noted. They often go unstated and fly under the radar.

First, a caveat. In most cases, you cannot simply wish impact into being or create it via self-praise or self-promotion.

Still, there may be ways your research is having an impact that you’re not accounting for. To that end, I have itemized more than four dozen potential indicators of impact. In the following, I tried to organize the wide variety of impacts into nine groupings by theme, but I do not intend any implied priority or ranking between the themes or within a theme. Consider them random.

The most common indicators of true impact include:

• The use of research results. This includes the commonly used measure of citation count, but also evidence of the work actually being used by others and incorporated into others’ research, not just cited.
• Authorship of handbook chapters or being cited by others in handbook chapters or encyclopedia entries.
• International recognition and participation—as conference chair, program chair, on editorial boards, etc., and participation in keynotes, panels, etc.
• Long-term, international collaboration with leading researchers from well-regarded institutions.

The paragraphs below discuss the fuller set of indicators. While there is some overlap between themes, together they paint a comprehensive picture of the different kinds of impact your research could have.

First, there is the use of research results. This goes beyond citations by others. Do others actually use your research as a foundation to work from? To leverage? Do they teach and discuss the research results in their courses? Does the work appear in some form on assessments? While not all research can be directly applied in practice, some can: Has the research been recognized by practitioners and used “in the wild”? Or perhaps in research labs as part of their work? These examples reflect more than a simple nod of the head in one’s direction: They are indicators that the research results are important enough to actually be used, to be part of the foundation for further research and practice. Another indicator to consider is if others have replicated the research with perhaps stronger methods, strengthening the foundation. The use of research results is perhaps the strongest indicator of value and impact. Words are cheap: actions speak.

The second theme involves recognition for “expert” consultative positions within the field, positions in which your advice is actively sought. Requests to be an external examiner nationally or internationally for graduate work examinations is one such indicator. Another is being asked by prestigious institutions to review candidates for promotion and tenure or by significant granting agencies to review applications. Being asked to help develop a research strategy for a granting agency or participating in program/curriculum reviews at other institutions also falls into this category.

Another indicator would be participation on award panels. It is also possible to be viewed as the expert’s expert—as when other academics suggest you as a reference when someone is seeking deep insights into a topic or when organizations such as think tanks seek the advice of a subject expert. Requests from visitors wishing to seek an audience of perhaps a few days or few weeks to discuss the field, not simply to collaborate on ongoing research, is also an excellent indicator of impact.

Third, there is the recognition that is manifest when additional dissemination is requested. Examples include invitations to give keynote speeches, to contribute papers to special issues or to write a “view of the field” or the “future of” chapter for a text or handbook or encyclopedia. Unsolicited requests are especially powerful indicators of impact.

A fourth area of impact speaks to your role as a leader and innovator in your field. Initiating/founding a workshop or conference is certainly noteworthy. Other examples include creating a special issue, being one of the primary editors of a new handbook or curating a program stream at a significant conference. How a faculty member is introduced in a handbook introduction or when giving a keynote speech is potentially of significance: Being introduced as someone who inspires, is a founder, pioneer, game changer or as someone who redefined the field, are things to keep track of. It is not enough for the faculty member to claim they are a pioneer or leader: It is important to provide evidence in the form of arm’s-length praise

Other examples of such praise would be fellow researchers characterizing your work as “seminal” or significant,” or describing it in other “good” papers as being at the forefront of the field. Another good indicator would be multiple citations for different contributions you’ve made in a “state-of-the-art review.”

Fifth, there are senior fellow indicators such as being an editor or associate editor of well-regarded journals. Conference and program chair positions are in a similar category, as is being invited to author handbook chapters intended to summarize a body of work or to write a review chapter for a text or handbook or encyclopedia. These types of activities indicate peer valuation and respect.

The sixth theme involves the faculty member’s network. Where you visit, are invited to visit or spend time during a sabbatical can be supportive of a claim related to impact. Adjunct or visiting appointments are important. Hosting visitors is also possibly noteworthy, particularly if the visits happen at repeated intervals and the collaborator is a big name in the field. Repeated collaborations are suggestive of a sustained value proposition: While a single visit or collaboration might be more related to happenstance, if someone of note works with you on multiple occurrences, this is the classic example of being judged (favorably) by the company you keep. It might be possible to fool someone once, but repeated collaboration? That is what we are talking about!

The seventh domain captures outside-the-norm publications. These could include white papers, texts of original work, interpretative pieces for practitioners or many more examples. These publications are not necessarily peer-reviewed and might be looked at with a different lens than more traditional journal articles and the like. Just because they are not peer-reviewed does not mean that the results are not robust and valid; then again, they might not be, and of course at one time the peer view was that the earth was flat! In any case, such publications can be fairly discussed if there is evidence of the work having impact and being valued by respected practitioners or others.

Eighth, consider the trajectory of the research. It might have been that you beat the clock, so to speak, achieving as much in fewer years compared to others in your peer group. If a faculty member has been early in meeting one or more of the typical milestones for promotion, this is a signal of outstanding performance, significantly above the norm.

Ninth, and finally, we come to progeny. It is also possible to look at the students a faculty member has supervised and consider those students’ contributions. Your contribution might also be shown by the quality of your supervision and mentorship.

That’s about it. There are many different types of possible impacts to consider and show evidence for. It should not be the expectation that all full professors should have all or even a majority of items on the list. It is likely that many associate professors have only a few on the list, but that might be enough—it will depend on the peer group and institutional expectations.

Throughout a career, the traditional publication count and grants may ebb and flow for a variety of reasons, and being aware of other forms of impact can help you complete the picture.