How Can We Measure Commitment to Undergraduate Teaching?

Here's what U.S. News & World Report has to say about its methodology for its widely circulated 2016 Best Undergraduate Teaching rankings:

"The rankings for Best Undergraduate Teaching focus on schools where faculty and administrators are committed to teaching undergraduate students in a high-quality manner. College presidents, provosts and admissions deans who participated in the annual U.S. News peer assessment survey were asked to nominate up to 10 schools in the Best Colleges ranking category with a strength in undergraduate teaching. The Best Undergraduate Teaching rankings are based solely on the responses to this separate section of the 2016 peer assessment survey."

The emphasis in bold is mine.

The U.S. News rankings are based solely on the opinions of college presidents, provosts and admissions deans. The returned surveys of these academic leaders' judgment of their peer institutions demonstrate “an especially strong commitment to undergraduate teaching.”

What improvements to the U.S. News methodology for ranking schools for their undergraduate teaching quality might we suggest?

Really? What other variables could be included beyond indicators of peer reputation that could capture an institution’s “commitment” to excellence in teaching and learning?

Before we choose our measures, we first need to figure out our parameters. We should be clear about what we want our indicators to capture. Our methodology should conform to our goals, as well as our abilities to actually carry out such rankings.

This is one area that is so troublesome about the U.S. News ranking for Best Undergraduate Teaching. As far as I can find, the publication does not engage in a critical analysis of its own methods (at least for the Undergraduate Teaching ranking).

It is not enough to criticize U.S. News. We need to do better. The teaching and learning community needs to do better.

Indicators of Success

My thinking on this is that our indicators should be measurable, input-based and actionable.   

First, any measure that we choose must actually be measurable; otherwise, it is not useful. But measuring the level of commitment around excellence in undergraduate teaching is incredibly difficult. For example, how does one measure intangibles such as culture and leadership? For this reason, any indicator that we choose to measure commitment will be flawed.  

Our goal should be to find measurable proxies for commitment. We should not let "the perfect" be the enemy of "the good" in choosing our measures, because if we look for the perfect measures, we will be stuck with the existing reputation rankings.

Second, I’d argue that to measure commitment to high-quality undergraduate teaching, that we should stay with inputs rather than outputs. It is necessary to account for the variation in student preparedness across institutions. Schools should not be penalized for admitting students who are poorly prepared when entering for college-level work. In the same way, schools that are more selective and can admit highly prepared students should not receive an advantage in a ranking of commitment to teaching.

This decision will be controversial, as what we want to measure is learning (or learning progress) and not really teaching.

The best we can do is own up to the flaws in an approach that focuses on inputs, and do the best we can to choose the right ones.

Finally, we should choose measures that are actionable and that will drive investments in teaching excellence that we judge as worthy organizational outcomes. This means that we need to make our values explicit in constructing our ranking; that is, we see a ranking around teaching quality as a lever to direct institutional (and industry-wide) investment towards teaching and learning.  

We should abandon the stance that rankings are value-neutral exercises. Rather, we should embrace rankings that have the power to change policies and motivate investments -- and be clear about where our values lie.  

What do You Think?

Can you expand, argue with or dispute these three parameters for developing a better ranking of institutional commitment to excellence in undergraduate teaching?

Have other researchers, publications or organizations offered an alternative ranking solely around commitment to quality teaching?

Next week in Inside Digital Learning I’ll provide some ideas for what variables and measures that I’d include in constructing our teaching commitment ranking.

Please let me know -- either in the comments section or privately by email if you prefer -- your ideas for constructing a ranking of institutional teaching commitment beyond reputation. Your thoughts are important. I want to hear from you.