A Ranking That Would Matter

As graduate students work their way through courses, dissertations and defenses, many have an ideal goal for the end: a job at a research university. But when would-be graduate students decide which programs to apply to, or which to enroll in, they often base their decisions on a program's general reputation, or the reputation of the university as a whole. In theory, programs with good reputations would have good job placement records. Maybe not.

The journal PS: Political Science & Politics has just published an analysis that suggests that there is not a direct relationship between the general reputation of a department and its success at placing new Ph.D.'s; some programs far exceed their reputation when it comes to placing new Ph.D.'s while others lag. The analysis may provide new evidence for the "halo effect" in which many experts worry that general (and sometimes outdated) institutional reputations cloud the judgment of those asked to fill out surveys on departmental quality. And while the analysis was prepared about political science, its authors believe the same approach could be used in other fields in the humanities and social sciences, with the method more problematic in other areas because fewer Ph.D. students aspire to academic careers.

Benjamin Schmidt, one of the co-authors and a Ph.D. candidate in history at Princeton University, said he first started thinking about the issue when he was applying to graduate programs. Too many of the available resources were "reputational," he said, rather than "output" oriented, and for a potential graduate student, the output of people into good jobs is the most important output of all. His co-author is Matthew M. Chingos, a Ph.D. candidate in government at Harvard University. Although Schmidt is not in political science, they focused on that field because of the availability of data.

Their approach was to take the Ph.D.-granting programs and see which ones placed the most new Ph.D. graduates (from 1990-2004) in faculty jobs at Ph.D.-granting programs. Recognizing that not all departments that award Ph.D.'s are alike, their formula gave more weight to those that were more successful at placing more of their students in top departments. Chingos compared the process to the way Google ranks Web pages -- where not all links are alike. (Details on the methodology, along with examples, may be found here.) And then they applied a per capita factor on graduate programs, so that departments placing many students just because they were large wouldn't do as well as small departments where most students were getting great jobs.

That final weighted analysis showed dramatic ups and downs for some institutions (the top 20 is reproduced at the end of this article). The University of Rochester was ranked 11th in the last National Research Council analysis of political science departments, but its per capita record at placing students in top departments is 4th. Other departments significantly exceeding their NRC rankings include Duke University (14th in NRC, but 7th at placing new Ph.D.'s), Northwestern University (22nd in NRC and 9th in placing Ph.D.'s), Michigan State University (26th in NRC and 17th in job placement), and Emory University (36th in NRC and 19th in job placement).

Of course other universities have better reputations than records at placing new Ph.D.'s. Yale University was ranked 3rd by the NRC, but its track record with new Ph.D.'s landed it 12th. Princeton was ranked 7th by NRC, but 13th when it comes to jobs for its new Ph.D.'s. The University of Wisconsin at Madison was ranked 10th by the NRC, but 35th in placing Ph.D.'s. The University of Minnesota's numbers: 13th vs. 32nd.

Schmidt said of the analysis: "One of the things I found most disappointing about it is that a lot of the programs with very good reputations that attempt to turn out large numbers of Ph.D.'s often tended to not have as good placement rates as their reputations of their faculty or other things might imply."

Chingos sees several factors at play. While declining to identify institutions, he thinks some programs "have slipped," but the professors filling out reputational surveys may be basing their views on a department's status 10 or more years ago. In contrast, the decision about whom to hire is a result of intense scrutiny and reflects the current quality of a department -- in attracting talent to a graduate program and preparing students for academic careers. Chingos also said that size appears to be a factor in reputation more than program quality when it comes to placing students in top departments.

"Rochester is a perfect example," he said, in that it's not as well regarded by those filling out surveys but bests most of the competition when it comes to producing Ph.D.'s who get tenure-track jobs at top universities. Chingos said that data like this paper could be most helpful to potential graduate students in flagging a university like Rochester as worth considering above better known institutions.

The Ph.D. program at Rochester is small -- enrolling only about 8 new students a year, less than half of the number enrolled as new students annually by many top departments. But in the last four years, Rochester has placed new Ph.D.'s in tenure track jobs at Harvard, Michigan State, Princeton, Texas A&M (2), the University of California at Riverside, the University of Minnesota-Twin Cities, the University of North Carolina at Chapel Hill, Washington University in St. Louis, Wayne State University, and Yale University (2). The department features the names of those graduates and other placement information on its Web site.

Randall Stone, director of graduate studies and an associate professor in the department, sees several reasons for the department's success with placement, as a small program. One reason is that there is a focus in terms of philosophy. "Rochester has been known for decades as a program that takes a particular approach to political science, which is very rigorous, very mathematical, emphasizes the decision making by rational agents as being the first cut we take to explain phenomenon," he said.

But in terms of a philosophy of graduate education that could be applied elsewhere, Stone said that the key was providing more guidance than he thinks is the norm elsewhere. As graduate director, he prepares a letter for each graduate student each year, with details on how to move ahead and identifying deadlines for dealing with problems. The letter is based on a faculty meeting in which the entire department faculty participates "and everyone knows all of the students." Graduate students also must prepare, in addition to standard Ph.D. requirements, a second-year paper that must meet the test of "contributing to new knowledge," Stone said.

While this work may not be publishable, it represents the idea of providing "pre-professional experiences" and exposing graduate students early to to "withering criticisms they are going to get" as they advance. In addition, he said, Rochester insists on more instruction in statistical methods and formal theory than is the case in many other programs. "Our students speak a common language," he said.

Stone, who earned his Ph.D. at Harvard, said that "a number of top programs get very good students, and then don't pay much attention to them, and the students get lost intellectually."

Chingos acknowledged that there is a major bias in his analysis: the belief that success means landing a job at another research university. He said that he was not trying to denigrate those who work at community colleges or liberal arts colleges or outside academe -- but to reflect the reality that so many entering Ph.D. programs tell themselves that, whatever they read about the job market, they will be the ones to get jobs at graduate departments.

He said that those interested in a broader range of institutions in fact have more options. But one of the things most striking to him in examining the data was that those seeking a job at a Ph.D.-granting institution have relatively few options. The dropoff from top institutions in job placement to other institutions is sharp, he said, and there are many places awarding Ph.D.'s in political science where a new doctoral recipient's odds of getting a job at a doctoral institution are "essentially zero."

The following table shows the rankings produced by the study (based on per capita placement in top programs) compared to the NRC rankings, for the top 20 in the new study.

Rankings Based on Placing Political Science Ph.D.'s in Jobs vs. National Research Council Survey