About one month ago, President Obama announced plans for sweeping changes in higher education. In short, he wants the system to be much more efficient, affordable, and timely. Numerous reports have indicated the cost of higher education has increased at rapid rates. Bloomberg indicated that since I started college in 1985, the cost has risen by 500 percent.
This is a complex problem. Our health care provider told us to expect a 19 percent increase this year. Technology upgrades mean additional costs. The reality is while we are a nonprofit, nothing is slowing the for-profits interested in greater profit margins. But I understand the president’s concerns.
Yet as I heard President Obama share ideas about measuring the effectiveness of institutions as a solution, I was concerned. I agree that assessment is essential. We need to make sure we are delivering on what we promise. But my concern is how will these metrics be developed, and will they really be able to consider all of the factors that impact student success and institutional performance?
As Secretary of Education Arne Duncan and his team begin their work, I would like to propose a competitive diving approach to college assessment. In diving, you receive a raw score from 1 to 10 based on dive execution. That score is averaged by the judges, then multiplied by the degree of difficulty for the overall score.
Most of the rankings that exist, particularly those of U.S. News & World Report, measure inputs dependent upon wealth so that quality is determined by whom you serve rather than what you do with them. Essentially, the fewer Pell Grant, part-time, nontraditional and students of color you serve, the better your outcomes.
Elite colleges, which educate those who received the best high school educations and who frequently have plenty of money, serve students who have the right inputs ,which almost guarantee high retention and graduation rates, low debt, and high employment.
But, in order to be fair, any new rating system must calculate the degree of difficulty when examining the metrics. For example, reviewing data for the last three years available, the smaller a share of the student body made of Pell Grant eligible students a college has, the better the graduation rate.
In fact, decades of research prove this point. The difference is significant as well. For 2011, as an example, the graduation rate for baccalaureate nonprofit colleges was 52 percent. For colleges with fewer than 20 percent Pell grant recipients (generally households earning less than $40,000 a year), the graduation rate was 79 percent. It dropped to 56 percent for colleges with 21-40 percent Pell students, and then to 42 percent for institutions with 41-60 percent Pell students. For those where more than 60 percent were Pell grant recipients, the graduation rate was 31 percent.
Colleges with less than 20 percent Pell students had few part-time, nontraditional and underrepresented students of color. Colleges with more than 60 percent Pell students had twice as many part-timers, five times as many nontraditionals, and almost six times as many underrepresented students of color.
And yet most rankings have lauded the first group for providing a great education. They essentially have done simple dives -- forward in a tuck position off a 1-meter springboard which has a degree of difficulty of 1.2 (based on USA diving). Meanwhile, many colleges attempt a back 4 1/2 somersaults in a tuck position off a 3-meter springboard, degree of difficulty 4.6. The problem is, we don’t get the degree of difficulty factored in. Only the raw score is calculated and we’re determined to be lesser institutions.
If President Obama’s plan to overhaul higher education is to have any credibility, there must be a degree of difficulty factor. In fact, there should be some other factors as well if there is to be any equity in this process. If colleges will be evaluated on the earnings of graduates, will the methodology take into account that women earn 77 percent of what men do, and this would disproportionately penalize women’s colleges and those with high proportions of women? Will the rankings factor in students who had to leave college because the government changed Parent PLUS loan eligibility?
The skepticism is widespread because we’ve watched numbers being used without proper context. For example, the highly touted White House College Scorecard was launched in February as a great step in accountability. For my institution, the graduation rate was listed as 24 percent. That looks atrocious. And yet, nothing on that webpage indicated that rate was based on freshmen who started in August of 2005, a few weeks before Hurricane Katrina made the campus unusable for almost one year. Eight years later we are finally opening all previously closed buildings.
We lost half that freshman class after one year, and large numbers of sophomores and juniors. The simple analysis presented on the scorecard paints a damaging picture. If consequences are then attached without all factors being weighed, this becomes an attack on a college.
The point is there has to be serious analysis with the broadest range of institutions at the table as this rating is developed. If all the factors are not considered, we end up with a simplistic one-size-fits-all that harms many institutions and their students. I know President Obama does not want that to be part of his legacy.
We are diving into a new territory to rate colleges. I just hope we’ll use diving's scoring as well.
Walter M. Kimbrough is president of Dillard University.
Understand one fact about the president's speech in Buffalo August 22 and the White House's plans to reform college financial aid: President Obama's proposed tie of financial aid prospects to ratings relies on an overcomplicated, implausible set of mechanisms to accomplish a simple task. The admirable goal: target financial aid on the colleges and universities that make it easier for students of moderate means to attend and complete undergraduate programs.
We should not be surprised that the president's goal is mixed up with the fantasy of an algorithm that can judge the merit of colleges and universities. Ratings and rankings are the crack cocaine of today's generation of education reformers and have been since the Reagan administration concocted a "wall chart" attempting to compare state educational performance. Our current president, his White House advisers, Secretary of Education Arne Duncan, former Florida Governor Jeb Bush, and many others cannot get the idea out of their heads -- that if we just find the right (magical) formula, we can push the education system to perform better.
Sometimes algorithms are important and helpful, and I understand why dreams of a perfect educational judgment system are so appealing. Brookings Institution researchers Matthew Chingos and Beth Akers have the best short description of the potential benefits of a ratings system for higher education, written from the perspective of two algorithm advocates. Alas, they assume both the existence of meaningful, comprehensive, nonmanipulable data, and the ability of an algorithm to spot high and low performers with accuracy. We know from both college ranking systems and the history of elementary and secondary school accountability that such attempts generally fail. Cedar Reiner and Timothy Burke have explained some of the concerns I have about applying blind faith in bad formulas for higher education... and I am afraid that the president's promise, "We'll figure out the right formula in the next two years!" is as comforting as other similar claims have been. Such a system is inevitably a Rube Goldberg machine.
But we do not need to feed politicians' dependency on school and college ratings. Mr. President, tear down that wall chart! You can accomplish the same goal with much simpler, more robust tools -- and even better, you would not need Congress to amend the Higher Education Act to improve the federal government's financial aid system. Here is my Five-Step Program to break politicians' addiction to ratings systems, at least in higher education:
1. Distribute a large chunk of college aid directly to colleges and universities based on Pell graduates, if not exactly as proposed. The basic idea is good: institutions that effectively serve poor students should have some support to continue and expand their work. But we do not know whether it would be best to have a flat payment per graduate or weight it by financial aid received by the student. There are some potential advantages of weighting the reward by the size of Pell Grant received while enrolled, and possibly other financial aid: addressing transfer issues in a rational (if not perfect) way, giving some advantage to institutions with the poorest students, and giving institutions a significant incentive to help students keep Pell grants and other aid year-to-year. Instead of making one decision at the federal level, Congress could distribute funds to states, tell them they must distribute funds based on associate or bachelors degrees earned by those who received Pell grants, and let states partially or fully weight those rewards based on federal and/or state and institutional financial aid received by the graduates. Letting states determine if the rewards are weighted by financial aid may appeal to governors and state legislators, as well as allowing states to include state and institutional financial aid in the weights.
2. Cap student loans not just by students but also by institutions, with the cap tied to the number of recent graduates who carried federal student loans at the institution. For example, if the cap for four-year colleges was hypothetically set at $10,000 times the total loan-carrying graduates in the past three years, then an institution with a consistent 40 percent graduation rate would have a much lower loan cap for all its students than an institution with a consistent 60 percent graduation rate. (For example, a college that admits 1,000 students who take outloans every year would graduate 400 of them annually, 1,200 every three years, with a total loan cap of $12 million. If it graduated 600 students carrying loans instead every year, it would have a total loan cap of $18 million.) Institutions with lower graduation rates would either have to have lower net costs, raise their graduation rates, or stop recruiting students who need large student loans.
Capping loans at an institution by the absolute number of loan-carrying students would not need a difficult-to-calculate graduation rate but would realistically address the capacity of an institution to educate students. The formula could be generous at the start and focus on limiting loans for the worst actors in higher education, those whose business plans rely on both the federal financial-aid system and also the gullibility of prospective students.
3. Cap not only personal student loans but also loan-forgiveness and so-called PLUS loans available to parents and graduate students, to circumvent individual loan caps. A comprehensive all-system cap that sets annual, per-degree, and lifetime loan and forgiveness caps would end the structure that has allowed Georgetown Law School to game graduate student PLUS loans so that neither Georgetown nor its students pay (in net) anything to the federal government. A comprehensive family-based cap on college-related loans would also address concerns about the exploitation of students and their families by tuition-dependent colleges and universities of all sectors. This subject is sensitive because of the claims by many tuition-dependent colleges and universities that they serve the public interest even while graduating a small minority of their students. The latest round of debates on PLUS loans and private historically black colleges and universities has gone to HBCUs, with what appears to be largely a reversal of the Obama administration's efforts to tighten credit-worthiness criteria for parents. The solution to the dilemma of the nonselective tuition-dependent college should not be the exploitation of families but the direct support of valuable institutions, which is why two more steps are necessary.
4. Use the Experimental Sites waiver provision of Title IV (the portion of the Higher Education Act with the rules for most student aid programs) to let public and nonprofit colleges work together on the student-services and business sides of their work, creating consortiums that draw on common services for some critical supports. Using Experimental Sites to improve student services and tighten the business operations of colleges is far more likely to help students than using Experimental Sites for MOOCs. In May, Southern New Hampshire University President Paul LeBlanc complained that federal rules prohibited nonprofits from working together on bundled services, even as many used commercial (and more expensive) corporations such as Academic Partnerships. Now, LeBlanc thinks that Experimental Sites might allow that bundling of services in the nonprofit college world. While LeBlanc would like SNHU to provide bundled services for online, competency-based education, we can extend the concept to consortiums of nonprofits with common interests, such as relatively nonselective HBCUs, and encourage such consortiums to address issues that most directly affect student persistence and completion and that can reduce costs if shared between institutions.
5. Directly support consortiums of tuition-dependent colleges and universities, contingent on sharing of relevant data on student progress and completion. If the federal government provides technical assistance grants supporting both instruction and student persistence/completion efforts to consortiums of nonselective public and private colleges, it could make data-sharing a condition of such grants. This could be accomplished through the Fund for the Improvement of Post-Secondary Education (FIPSE).
Together, these five steps address the general goals the president is targeting but likely without needing the Higher Education Act to be amended. None of them require the assumption of finely tuned ratings, and none should require a huge amount of statistical work by institutions beyond what Title IV colleges and universities must track today. The greatest difficulty is likely in using the appropriations process for the first and last steps -- and my guess is that it is easier to persuade Congress to send money to the states to reward colleges that graduate poor and moderate-income students than to ask members of Congress to give up their FIPSE earmarks.
But as long as no statutory change is required, there is some hope of addressing the fundamental dilemma with allowing loans to prop up tuition-dependent colleges and universities. The controversy over HBCUs and student/family loans is one form of a broader question untouched by the ideas above: How can we address the needs of tuition-dependent private colleges that admit students with weaker records -- when it is difficult to separate exploitative institutions from institutions that have an historical record of serving the public good? In the last century, black college graduates have represented a disproportionate number of African-American professionals after college. That history does not justify indefinite indirect subsidies by the federal government through student and family loans, especially given the inability to discharge educational debt in bankruptcy. But it does justify an understanding that some private institutions have low graduation rates while meeting a legitimate public purpose. In that regard, HBCUs constitute the canaries in the coal mine for tuition-dependent institutions more generally.
Advocates of a federal ranking/rating system argue that they can adjust measures of student success by the difficulties of serving the institution's population. I am skeptical of that claim, in part based on the experience with K-12 accountability algorithms that repeatedly fail to demonstrate sensitivity or specificity in identifying weakly performing schools, and in part based on the substantial disconnect between the available database (IPEDs) and even the best social scientific attempts to quantify entering-student needs. My experience and common sense about statistics lead me to believe that a federal ranking/rating system would not be worth all the candlepower that the White House is going to put into it. Or fairy dust sprinkled into the computers.
Instead of trusting a magical-algorithm approach, we should provide some direct support to such institutions in a way that allows them to be a little more efficient business-wise, boosts capacity in supporting students, and provides a little more accountability. The last two suggestions above focus on those goals. For tuition-dependent private colleges and universities that are allergic to sharing records, they could join together in consortiums without getting federal assistance. Those that most desperately need direct support and technical assistance would access to resources, with the understanding it comes bundled with accountability and sharing of data.
I am highly skeptical that the president's desire for a rating system will do much good. But it is not enough for skeptics of college ratings/rankings to point out all their flaws. We need concrete policy alternatives, ways to get to the same end without them. The above is my not-so-modest attempt to tackle that goal. It eliminates the Perfect Algorithmic Mechanism (that is DOA anyway) in favor of simpler, more robust mechanisms. And for tuition-dependent nonprofits that have a claim of serving the public good by serving especially needy students, it has the option of providing modest direct support in return for sharing of data.
But my attempt may not be the best option. What is your proposal for targeting aid to the most needy students in a way that realistically could happen in the next few years?
Sherman Dorn is a historian of education at the University of South Florida in Tampa. He wrote Accountability Frankenstein (Information Age Publishing, 2007) and blogs here.
Submitted by Ben Miller on September 3, 2013 - 3:00am
After a month of speculation, President Obama unveiled his plan to “shake up” higher education last week. As promised, the proposal contained some highly controversial elements, none greater than an announcement that the U.S. Department of Education will begin to rate colleges and universities in 2015 and tie financial aid to those results three years later. The announcement prompted typical clichéd Beltway commentary from the higher education industry of “the devil is in the details” and the need to avoid “unintended consequences,” which should rightfully be attributed as, “We are not going to outright object now when everyone’s watching but instead will nitpick to death later.”
But the ratings threat is more substantive than past announcements to put colleges “on notice,” if for no other reason than it is something the department can do without Congressional approval. Though it cannot actually tie aid received directly to these ratings without lawmakers (and the threat to do so would occur after Obama leaves office), the department can send a powerful message both to the higher education community and consumers nationwide by publishing these ratings.
Ratings systems, however, are no easy matter and require lots of choices in their methodologies. With that in mind, here are a few recommendations for how the ratings should work.
Ratings aren’t rankings.
Colleges have actually rated themselves in various forms for well over a hundred years. The Association of American Universities is an exclusive club of the top research universities that formed in 1900. The more in-depth Carnegie classifications, which group institutions based upon their focus and types of credentials awarded, have been around since the early 1970s. Though they may not be identified as such by most people, they are forms of ratings — recognitions of the distinctions between universities by mission and other factors.
A federal rating system should be constructed similarly. There’s no reason to bother with ordinal rankings like the U.S. News and World Report because distinguishing among a few top colleges is less important than sorting out those that really are worse than others. Groupings that are narrow enough to recognize differences but sufficiently broad to represent a meaningful sample are the way to go. The Department could even consider letting colleges choose their initial groupings, as some already do for the data feedback reports the Department produces through the Integrated Postsecondary Education Data System (IPEDS).
It’s easier to find the bottom tail of the distribution than the middle or top.
There are around 7,000 colleges in this country. Some are fantastic world leaders. Others are unmitigated disasters that should probably be shut down. But the vast majority fall somewhere in between. Sorting out the middle part is probably the hardest element of a ratings system — how do you discern within averageness?
We probably shouldn’t. A ratings system should sort out the worst of the worst by setting minimum performance standards on a few clear measures. It would clearly demonstrate that there is some degree of results so bad thatit merits being rated poorly. This standard could be excessively, laughably low, like a 10 percent graduation rate. Identifying the worst of the worst would be a huge step forward from what we do now. An ambitious ratings system could do the same thing on the top end using different indicators, setting very high bars that only a tiny handful of colleges would reach, but that’s much harder to get right.
Don’t let calls for the “right” data be an obstructionist tactic.
Hours after the President’s speech, representatives of the higher education lobby stated the administration’s ratings “have an obligation to perfect data.” It’s a reasonable requirement that a rating system not be based only on flawed measures, like holding colleges accountable just for the completion of first-time, full-time students. But the call for perfect data is a smokescreen for intransigence by setting a nearly unobtainable bar. Even worse, the very people calling for this standard are the same ones representing the institutions that will be the biggest roadblock to obtaining information fulfilling this requirement. Having data demands come from those keeping it hostage creates a perfect opportunity for future vetoes in the name of making perfect be the enemy of the good. It’s also a tried and true tactic from One Dupont Circle. Look at graduation rates, where the higher education lobby is happy to put out reports critiquing their accuracy after getting Congress to enact provisions that banned the creation of better numbers during the last Higher Education Act reauthorization.
To be sure, the Obama administration has an obligation to engage in an open dialogue with willing partners to make a good faith effort at getting the best data possible for its ratings. Some of this will happen anyway thanks to improvements to the department’s IPEDS database. But if colleges are not serious about being partners in the ratings and refuse to contribute the data needed, they should not then turn around and complain about the results.
Stick with real numbers that reflect policy goals.
Input-adjusted metrics are a wonk’s dream. Controlling for factors and running regressions get us all excited. But they’re also useless from a policy implementation standpoint. Complex figures that account for every last difference in institutions will contextualize away all meaningful information until all that remains is a homogenous jumble where everyone looks the same. Controlling for socioeconomic conditions also runs the risk of just inculcating low expectations for students based upon their existing results. Not to mention any modeling choices in an input-adjusted system will add another dimension of criticism to the firestorm that will already surround the measures chosen.
That does not mean context should be ignored. There are just better ways to handle it. First and foremost is making ratings on measures based on performance relative to peers. Well-crafted peer comparisons can accomplish largely the same thing as input adjustment since institutions would be facing similar circumstances, but still rely on straightforward figures. Second, unintended consequences should be addressed by measuring them with additional metrics and clear goals. For example, afraid that focusing on a college's completion rate will discourage enrolling low-income students or unfairly penalize those that serve large numbers of this type of students? The ratings should give institutions credit for the socioeconomic diversity of their student body, require a minimum percentage of Pell students, and break out the completion rate by familial income. Doing so not only provides a backstop against gaming, it also lays out clearer expectations to guide colleges' behavior, something the U.S. News rankings experience has shown that colleges clearly know how to do with less useful measures like alumni giving (sorry, Brown, for holding you back on that one).
Mix factors a college can directly control with ones it cannot.
Institutions have an incentive to improve on measures included in a rating system. But some subset of colleges will also try to evade or “game” the measure. This is particularly true if it’s something under their control — look at the use of forbearances or deferments to avoid sanctions under the cohort default rate. No system will ever be able to fully root out gaming and loopholes, but one way to adjust for them is by complementing measures under a college’s control with ones that are not. For example, concerns about sacrificing academic quality to increase graduation rates could be partially offset by adding a focus on graduates’ earnings or some other post-completion behavior that is not under the college’s control. Institutions will certainly object to being held accountable for things they cannot directly influence. But basing the uncontrollable elements on relative instead of absolute performance should further ameliorate this concern.
Focus on outputs but don’t forget inputs.
Results matter. An institution that cannot graduate its students or avoid saddling them with large loan debts they cannot repay upon completion is not succeeding. But a sole focus on outputs could encourage an institution to avoid serving the neediest students as a way of improving its metrics and undermine the access goals that are an important part of federal education policy.
To account for this, a ratings system should include a few targeted input metrics that reflect larger policy goals like socioeconomic diversity or first-generation college students. Giving colleges “credit” in the ratings for serving the students we care most about will provide at least some check against potential gaming. Even better, some metrics should have a threshold a school has to reach to avoid automatic classification into the lowest rating.
Put it together.
A good ratings system is both consistent and iterative. It keeps the core pieces the same year to year but isn’t too arrogant to include new items and tweak ones that aren’t working. These recommendations present somewhere to start. Group the schools sensibly — maybe even rely on existing classifications like those done by Carnegie. The ratings should establish minimum performance thresholds on the metrics we think are most indicative of an unsuccessful institution — things like completion rates, success with student loans, time to degree, etc. They should consist of outcomes metrics that reflect their missions—such as transfer success for two-year schools, licensure and placement for vocational offerings, earnings, completion and employment for four-year colleges and universities. But they should also have separate metrics to acknowledge policy challenges we care about — success in serving Pell students, the ability to get remedial students college-ready, socioeconomic diversity, etc. — to discourage creaming. The result should be something that reflects values and policy challenges, acknowledges attempts to find workarounds, and refrains from dissolving into wonkiness and theoretical considerations that are divorced from reality.
Ben Miller is a senior policy analyst in the New America Foundation's education policy program, where he provides research and analysis on policies related to postsecondary education. Previously, Miller was a senior policy advisor in the Office of Planning, Evaluation, and Policy Development in the U.S. Department of Education.