Understand one fact about the president's speech in Buffalo August 22 and the White House's plans to reform college financial aid: President Obama's proposed tie of financial aid prospects to ratings relies on an overcomplicated, implausible set of mechanisms to accomplish a simple task. The admirable goal: target financial aid on the colleges and universities that make it easier for students of moderate means to attend and complete undergraduate programs.
We should not be surprised that the president's goal is mixed up with the fantasy of an algorithm that can judge the merit of colleges and universities. Ratings and rankings are the crack cocaine of today's generation of education reformers and have been since the Reagan administration concocted a "wall chart" attempting to compare state educational performance. Our current president, his White House advisers, Secretary of Education Arne Duncan, former Florida Governor Jeb Bush, and many others cannot get the idea out of their heads -- that if we just find the right (magical) formula, we can push the education system to perform better.
Sometimes algorithms are important and helpful, and I understand why dreams of a perfect educational judgment system are so appealing. Brookings Institution researchers Matthew Chingos and Beth Akers have the best short description of the potential benefits of a ratings system for higher education, written from the perspective of two algorithm advocates. Alas, they assume both the existence of meaningful, comprehensive, nonmanipulable data, and the ability of an algorithm to spot high and low performers with accuracy. We know from both college ranking systems and the history of elementary and secondary school accountability that such attempts generally fail. Cedar Reiner and Timothy Burke have explained some of the concerns I have about applying blind faith in bad formulas for higher education... and I am afraid that the president's promise, "We'll figure out the right formula in the next two years!" is as comforting as other similar claims have been. Such a system is inevitably a Rube Goldberg machine.
But we do not need to feed politicians' dependency on school and college ratings. Mr. President, tear down that wall chart! You can accomplish the same goal with much simpler, more robust tools -- and even better, you would not need Congress to amend the Higher Education Act to improve the federal government's financial aid system. Here is my Five-Step Program to break politicians' addiction to ratings systems, at least in higher education:
1. Distribute a large chunk of college aid directly to colleges and universities based on Pell graduates, if not exactly as proposed. The basic idea is good: institutions that effectively serve poor students should have some support to continue and expand their work. But we do not know whether it would be best to have a flat payment per graduate or weight it by financial aid received by the student. There are some potential advantages of weighting the reward by the size of Pell Grant received while enrolled, and possibly other financial aid: addressing transfer issues in a rational (if not perfect) way, giving some advantage to institutions with the poorest students, and giving institutions a significant incentive to help students keep Pell grants and other aid year-to-year. Instead of making one decision at the federal level, Congress could distribute funds to states, tell them they must distribute funds based on associate or bachelors degrees earned by those who received Pell grants, and let states partially or fully weight those rewards based on federal and/or state and institutional financial aid received by the graduates. Letting states determine if the rewards are weighted by financial aid may appeal to governors and state legislators, as well as allowing states to include state and institutional financial aid in the weights.
2. Cap student loans not just by students but also by institutions, with the cap tied to the number of recent graduates who carried federal student loans at the institution. For example, if the cap for four-year colleges was hypothetically set at $10,000 times the total loan-carrying graduates in the past three years, then an institution with a consistent 40 percent graduation rate would have a much lower loan cap for all its students than an institution with a consistent 60 percent graduation rate. (For example, a college that admits 1,000 students who take outloans every year would graduate 400 of them annually, 1,200 every three years, with a total loan cap of $12 million. If it graduated 600 students carrying loans instead every year, it would have a total loan cap of $18 million.) Institutions with lower graduation rates would either have to have lower net costs, raise their graduation rates, or stop recruiting students who need large student loans.
Capping loans at an institution by the absolute number of loan-carrying students would not need a difficult-to-calculate graduation rate but would realistically address the capacity of an institution to educate students. The formula could be generous at the start and focus on limiting loans for the worst actors in higher education, those whose business plans rely on both the federal financial-aid system and also the gullibility of prospective students.
3. Cap not only personal student loans but also loan-forgiveness and so-called PLUS loans available to parents and graduate students, to circumvent individual loan caps. A comprehensive all-system cap that sets annual, per-degree, and lifetime loan and forgiveness caps would end the structure that has allowed Georgetown Law School to game graduate student PLUS loans so that neither Georgetown nor its students pay (in net) anything to the federal government. A comprehensive family-based cap on college-related loans would also address concerns about the exploitation of students and their families by tuition-dependent colleges and universities of all sectors. This subject is sensitive because of the claims by many tuition-dependent colleges and universities that they serve the public interest even while graduating a small minority of their students. The latest round of debates on PLUS loans and private historically black colleges and universities has gone to HBCUs, with what appears to be largely a reversal of the Obama administration's efforts to tighten credit-worthiness criteria for parents. The solution to the dilemma of the nonselective tuition-dependent college should not be the exploitation of families but the direct support of valuable institutions, which is why two more steps are necessary.
4. Use the Experimental Sites waiver provision of Title IV (the portion of the Higher Education Act with the rules for most student aid programs) to let public and nonprofit colleges work together on the student-services and business sides of their work, creating consortiums that draw on common services for some critical supports. Using Experimental Sites to improve student services and tighten the business operations of colleges is far more likely to help students than using Experimental Sites for MOOCs. In May, Southern New Hampshire University President Paul LeBlanc complained that federal rules prohibited nonprofits from working together on bundled services, even as many used commercial (and more expensive) corporations such as Academic Partnerships. Now, LeBlanc thinks that Experimental Sites might allow that bundling of services in the nonprofit college world. While LeBlanc would like SNHU to provide bundled services for online, competency-based education, we can extend the concept to consortiums of nonprofits with common interests, such as relatively nonselective HBCUs, and encourage such consortiums to address issues that most directly affect student persistence and completion and that can reduce costs if shared between institutions.
5. Directly support consortiums of tuition-dependent colleges and universities, contingent on sharing of relevant data on student progress and completion. If the federal government provides technical assistance grants supporting both instruction and student persistence/completion efforts to consortiums of nonselective public and private colleges, it could make data-sharing a condition of such grants. This could be accomplished through the Fund for the Improvement of Post-Secondary Education (FIPSE).
Together, these five steps address the general goals the president is targeting but likely without needing the Higher Education Act to be amended. None of them require the assumption of finely tuned ratings, and none should require a huge amount of statistical work by institutions beyond what Title IV colleges and universities must track today. The greatest difficulty is likely in using the appropriations process for the first and last steps -- and my guess is that it is easier to persuade Congress to send money to the states to reward colleges that graduate poor and moderate-income students than to ask members of Congress to give up their FIPSE earmarks.
But as long as no statutory change is required, there is some hope of addressing the fundamental dilemma with allowing loans to prop up tuition-dependent colleges and universities. The controversy over HBCUs and student/family loans is one form of a broader question untouched by the ideas above: How can we address the needs of tuition-dependent private colleges that admit students with weaker records -- when it is difficult to separate exploitative institutions from institutions that have an historical record of serving the public good? In the last century, black college graduates have represented a disproportionate number of African-American professionals after college. That history does not justify indefinite indirect subsidies by the federal government through student and family loans, especially given the inability to discharge educational debt in bankruptcy. But it does justify an understanding that some private institutions have low graduation rates while meeting a legitimate public purpose. In that regard, HBCUs constitute the canaries in the coal mine for tuition-dependent institutions more generally.
Advocates of a federal ranking/rating system argue that they can adjust measures of student success by the difficulties of serving the institution's population. I am skeptical of that claim, in part based on the experience with K-12 accountability algorithms that repeatedly fail to demonstrate sensitivity or specificity in identifying weakly performing schools, and in part based on the substantial disconnect between the available database (IPEDs) and even the best social scientific attempts to quantify entering-student needs. My experience and common sense about statistics lead me to believe that a federal ranking/rating system would not be worth all the candlepower that the White House is going to put into it. Or fairy dust sprinkled into the computers.
Instead of trusting a magical-algorithm approach, we should provide some direct support to such institutions in a way that allows them to be a little more efficient business-wise, boosts capacity in supporting students, and provides a little more accountability. The last two suggestions above focus on those goals. For tuition-dependent private colleges and universities that are allergic to sharing records, they could join together in consortiums without getting federal assistance. Those that most desperately need direct support and technical assistance would access to resources, with the understanding it comes bundled with accountability and sharing of data.
I am highly skeptical that the president's desire for a rating system will do much good. But it is not enough for skeptics of college ratings/rankings to point out all their flaws. We need concrete policy alternatives, ways to get to the same end without them. The above is my not-so-modest attempt to tackle that goal. It eliminates the Perfect Algorithmic Mechanism (that is DOA anyway) in favor of simpler, more robust mechanisms. And for tuition-dependent nonprofits that have a claim of serving the public good by serving especially needy students, it has the option of providing modest direct support in return for sharing of data.
But my attempt may not be the best option. What is your proposal for targeting aid to the most needy students in a way that realistically could happen in the next few years?
Sherman Dorn is a historian of education at the University of South Florida in Tampa. He wrote Accountability Frankenstein (Information Age Publishing, 2007) and blogs here.
Submitted by Ben Miller on September 3, 2013 - 3:00am
After a month of speculation, President Obama unveiled his plan to “shake up” higher education last week. As promised, the proposal contained some highly controversial elements, none greater than an announcement that the U.S. Department of Education will begin to rate colleges and universities in 2015 and tie financial aid to those results three years later. The announcement prompted typical clichéd Beltway commentary from the higher education industry of “the devil is in the details” and the need to avoid “unintended consequences,” which should rightfully be attributed as, “We are not going to outright object now when everyone’s watching but instead will nitpick to death later.”
But the ratings threat is more substantive than past announcements to put colleges “on notice,” if for no other reason than it is something the department can do without Congressional approval. Though it cannot actually tie aid received directly to these ratings without lawmakers (and the threat to do so would occur after Obama leaves office), the department can send a powerful message both to the higher education community and consumers nationwide by publishing these ratings.
Ratings systems, however, are no easy matter and require lots of choices in their methodologies. With that in mind, here are a few recommendations for how the ratings should work.
Ratings aren’t rankings.
Colleges have actually rated themselves in various forms for well over a hundred years. The Association of American Universities is an exclusive club of the top research universities that formed in 1900. The more in-depth Carnegie classifications, which group institutions based upon their focus and types of credentials awarded, have been around since the early 1970s. Though they may not be identified as such by most people, they are forms of ratings — recognitions of the distinctions between universities by mission and other factors.
A federal rating system should be constructed similarly. There’s no reason to bother with ordinal rankings like the U.S. News and World Report because distinguishing among a few top colleges is less important than sorting out those that really are worse than others. Groupings that are narrow enough to recognize differences but sufficiently broad to represent a meaningful sample are the way to go. The Department could even consider letting colleges choose their initial groupings, as some already do for the data feedback reports the Department produces through the Integrated Postsecondary Education Data System (IPEDS).
It’s easier to find the bottom tail of the distribution than the middle or top.
There are around 7,000 colleges in this country. Some are fantastic world leaders. Others are unmitigated disasters that should probably be shut down. But the vast majority fall somewhere in between. Sorting out the middle part is probably the hardest element of a ratings system — how do you discern within averageness?
We probably shouldn’t. A ratings system should sort out the worst of the worst by setting minimum performance standards on a few clear measures. It would clearly demonstrate that there is some degree of results so bad thatit merits being rated poorly. This standard could be excessively, laughably low, like a 10 percent graduation rate. Identifying the worst of the worst would be a huge step forward from what we do now. An ambitious ratings system could do the same thing on the top end using different indicators, setting very high bars that only a tiny handful of colleges would reach, but that’s much harder to get right.
Don’t let calls for the “right” data be an obstructionist tactic.
Hours after the President’s speech, representatives of the higher education lobby stated the administration’s ratings “have an obligation to perfect data.” It’s a reasonable requirement that a rating system not be based only on flawed measures, like holding colleges accountable just for the completion of first-time, full-time students. But the call for perfect data is a smokescreen for intransigence by setting a nearly unobtainable bar. Even worse, the very people calling for this standard are the same ones representing the institutions that will be the biggest roadblock to obtaining information fulfilling this requirement. Having data demands come from those keeping it hostage creates a perfect opportunity for future vetoes in the name of making perfect be the enemy of the good. It’s also a tried and true tactic from One Dupont Circle. Look at graduation rates, where the higher education lobby is happy to put out reports critiquing their accuracy after getting Congress to enact provisions that banned the creation of better numbers during the last Higher Education Act reauthorization.
To be sure, the Obama administration has an obligation to engage in an open dialogue with willing partners to make a good faith effort at getting the best data possible for its ratings. Some of this will happen anyway thanks to improvements to the department’s IPEDS database. But if colleges are not serious about being partners in the ratings and refuse to contribute the data needed, they should not then turn around and complain about the results.
Stick with real numbers that reflect policy goals.
Input-adjusted metrics are a wonk’s dream. Controlling for factors and running regressions get us all excited. But they’re also useless from a policy implementation standpoint. Complex figures that account for every last difference in institutions will contextualize away all meaningful information until all that remains is a homogenous jumble where everyone looks the same. Controlling for socioeconomic conditions also runs the risk of just inculcating low expectations for students based upon their existing results. Not to mention any modeling choices in an input-adjusted system will add another dimension of criticism to the firestorm that will already surround the measures chosen.
That does not mean context should be ignored. There are just better ways to handle it. First and foremost is making ratings on measures based on performance relative to peers. Well-crafted peer comparisons can accomplish largely the same thing as input adjustment since institutions would be facing similar circumstances, but still rely on straightforward figures. Second, unintended consequences should be addressed by measuring them with additional metrics and clear goals. For example, afraid that focusing on a college's completion rate will discourage enrolling low-income students or unfairly penalize those that serve large numbers of this type of students? The ratings should give institutions credit for the socioeconomic diversity of their student body, require a minimum percentage of Pell students, and break out the completion rate by familial income. Doing so not only provides a backstop against gaming, it also lays out clearer expectations to guide colleges' behavior, something the U.S. News rankings experience has shown that colleges clearly know how to do with less useful measures like alumni giving (sorry, Brown, for holding you back on that one).
Mix factors a college can directly control with ones it cannot.
Institutions have an incentive to improve on measures included in a rating system. But some subset of colleges will also try to evade or “game” the measure. This is particularly true if it’s something under their control — look at the use of forbearances or deferments to avoid sanctions under the cohort default rate. No system will ever be able to fully root out gaming and loopholes, but one way to adjust for them is by complementing measures under a college’s control with ones that are not. For example, concerns about sacrificing academic quality to increase graduation rates could be partially offset by adding a focus on graduates’ earnings or some other post-completion behavior that is not under the college’s control. Institutions will certainly object to being held accountable for things they cannot directly influence. But basing the uncontrollable elements on relative instead of absolute performance should further ameliorate this concern.
Focus on outputs but don’t forget inputs.
Results matter. An institution that cannot graduate its students or avoid saddling them with large loan debts they cannot repay upon completion is not succeeding. But a sole focus on outputs could encourage an institution to avoid serving the neediest students as a way of improving its metrics and undermine the access goals that are an important part of federal education policy.
To account for this, a ratings system should include a few targeted input metrics that reflect larger policy goals like socioeconomic diversity or first-generation college students. Giving colleges “credit” in the ratings for serving the students we care most about will provide at least some check against potential gaming. Even better, some metrics should have a threshold a school has to reach to avoid automatic classification into the lowest rating.
Put it together.
A good ratings system is both consistent and iterative. It keeps the core pieces the same year to year but isn’t too arrogant to include new items and tweak ones that aren’t working. These recommendations present somewhere to start. Group the schools sensibly — maybe even rely on existing classifications like those done by Carnegie. The ratings should establish minimum performance thresholds on the metrics we think are most indicative of an unsuccessful institution — things like completion rates, success with student loans, time to degree, etc. They should consist of outcomes metrics that reflect their missions—such as transfer success for two-year schools, licensure and placement for vocational offerings, earnings, completion and employment for four-year colleges and universities. But they should also have separate metrics to acknowledge policy challenges we care about — success in serving Pell students, the ability to get remedial students college-ready, socioeconomic diversity, etc. — to discourage creaming. The result should be something that reflects values and policy challenges, acknowledges attempts to find workarounds, and refrains from dissolving into wonkiness and theoretical considerations that are divorced from reality.
Ben Miller is a senior policy analyst in the New America Foundation's education policy program, where he provides research and analysis on policies related to postsecondary education. Previously, Miller was a senior policy advisor in the Office of Planning, Evaluation, and Policy Development in the U.S. Department of Education.
In his address on higher education last week, President Obama encouraged innovation, specifically mentioning the groundbreaking work being done to award college credit based on learning, not simply on how much time is spent in a classroom. Under the direct assessment method, colleges and universities like Capella, which recently became the first university to receive U.S. Department of Education approval to offer competency-based bachelor’s- and master’s-degree programs that utilize direct assessment, can deliver high-quality education in a way that allows students to advance through their programs at their own pace and on their own schedule. As President Obama said, this allows students to "learn material faster, pay less and save money."
Using this model, the future of higher education can align specific competencies with the needs of employers as well as critical societal needs, while allowing busy students the flexibility to complete a degree.
However, there is still a lot of work to be done.
Even with the excitement around this new higher education delivery model, colleges and universities wishing to operate programs through direct assessment are still forced to work within the confines of an antiquated financial aid delivery system built upon concepts like the credit hour. Here is where President Obama’s higher education plan has an opportunity to enact real, lasting change that will spark innovation – and change the way we think about education.
Some of the necessary updates to federal financial aid programs are obvious. For example, requirements around weeks of instructional time simply do not work with a direct assessment model that focuses on what the student is learning, not the number of weeks it takes them to do so. Additionally, an examination of artificial, time-based barriers to completion highlights the need to reinstate year-round Pell Grant funding and explore the elimination of annual loan limits. The current funding rules around both the Pell Grant program and the Stafford Loan program prevent ambitious students from moving more quickly through their programs and increase the likelihood that students will have to pause their education for a term or more in order to gain additional aid eligibility.
In addition, policymakers should explore how a student’s academic progress can be measured apart from standards like traditional letter grades. Ultimately, federal funding rules need to reflect the move toward tying financial aid to student outcomes.
Obviously, there is a lot of work to be done to move the dial on a true direct assessment model. One way to address this is for Congress to authorize a demonstration project, or for the Department of Education to launch an experimental site initiative that will allow programs operating under direct assessment to examine the issues listed above, as well as others that arise. This will open the door to a better understanding of what federal financial aid changes make sense, and at the same time, allow institutions like Capella and Southern New Hampshire University to continue to innovate with direct assessment. It is shortsighted to believe that the current financial aid model supports the full potential of direct assessment. At the same time, it would be irresponsible to enact immediate, sweeping changes without the benefit of learning from those innovative institutions that are approved to operate direct assessment programs.
The emergence of direct assessment is a great step forward for higher education, but this is just the beginning. Colleges offering direct assessment cannot perpetually retrofit this delivery model into a traditional student funding model, nor can simply eliminating the credit hour requirement solve the issues that currently exist. Congress has already started the important process of investigating barriers that impede higher education innovation and efficiency. We should all work together to explore the opportunities President Obama outlined and pursue substantial, innovative change in higher education.
Deb Bushway is chief academic officer at Capella University.
On his education bus tour, President Obama is urging, among other suggestions, a new rating system to ensure that more families are able to afford higher education. I think we can all (well, almost all of us) agree that the rising costs of a bachelor’s degree need to be constrained, and we must find ways that facilitate middle- and lower-income students entering and graduating from college. The value proposition matters, and “debt without diploma” is unacceptable.
What is vastly harder to agree upon is how to address the problem, rather than just wringing our hands over it -- which we have been doing for far too long.
Let’s start with the president’s idea of rating colleges based on graduation rates and prospective earnings, among other variables. To be sure, given the president’s reference to U.S. News rankings in his speech today at the University of Buffalo, one wonders whether “ratings” are similar to or different from rankings – apart from using different variables.
More on the Obama Plan
Performance Funding Goes Federal: Obama gets aggressive in proposing college rating system. Making it happen won't be easy, but higher ed leaders said they'll play ball.
Disappointed, Not Surprised: Professors overwhelmingly voted for Obama twice. But five years into his presidency, few faculty leaders are surprised that they disagree strongly with his plan for higher ed reform.
Enjoying White House Attention: For colleges and other organizations promoting alternative paths to degrees, the president's speech was validation they have wanted.
On the surface, these two data points may seem easy to calculate. And advising families on how to compare and contrast college offers seems wise. But devising a quality rating system will require deep insight into how the world of higher education actually works – on the ground, in the trenches. As the president noted, Secretary Duncan needs to garner suggestions from a wide range of educational constituencies.
First, we know that more-elite institutions that serve Pell-eligible students have higher graduation rates than open-access institutions that enroll Pell eligible students. What accounts for this disparity is subject to debate, but arguably, part of the answer is that the richer institutions “cream skim” and only take the “best” among the low-income students.
For example, students who are selected to be Posse Scholars graduate from college (largely highly selective institutions) at a rate of 90 percent -- which is stunningly good. But, it is worth remembering that the 640 Posse Scholars enrolling each year are selected from approximately 15,000 applicants.
This means that elite institutions, absent some adjustment, would rank higher than non-elite institutions on graduation rates without any explanation as to why that is occurring. And the lower graduation rate of less-elite institutions may be at least partially explained by the lack of preparedness of their students. For some students and their colleges, a graduation rate of 40 percent is success, not failure.
Second, if we only calculate graduation rates of true first-year, full-time cohorts, we will be missing the mark in terms of who is actually enrolling in college today. Students with previous credits, transfer students, adults returning, part-time students and veterans would not be counted in the calculation, although at least some of these data points will be included as IPED’s data are improved over time.
Third, earnings are certainly occupation-based. Graduates who become teachers and nurses and police officers earn less than students who are employed by investment banks or hedge funds. Clearly, success in higher education cannot be measured based on earnings alone.
Yes, college graduates should not be underemployed or employed in fields that do not take advantage of their education. But how we calculate “sufficient” earnings is critically important, and more earnings are not necessarily better for the public good.
Finally, there is a built-in assumption that students and their parents will pay attention to and use the ratings effectively. Experience suggests otherwise. Despite transparency in the realm of consumer protection, consumers still make irrational and unwise choices, as behavioral economists have noted.
Indeed, as scholars point out, consumption decision-making is often based on non-economic determiners. And we already have early evidence that the current scorecard has not worked as expected – despite best efforts to share its availability. Moreover, the income-based repayment program – also publicized – has not had the expected uptake among students who could benefit from it, as the president himself noted. We need to make disclosure “smart.” We also need to focus on how to engage families in conversations about money. And educational institutions need to see that their obligations to advise students about loan repayment extends beyond graduation, particularly since initial payments often commence six months post-degree.
So if we proceed with graduation rates and earnings as indicators, we need to be cautious in terms of how we calculate both and be aware that even the best ratings may not help the very audiences we seek to persuade.
Indeed, possible key users of the ranking system are high school guidance counselors. But, as a recent report from the Public Agenda notes, this group of professionals is struggling to counsel students for college effectively. Thus, their caseload and training may make their uptake of any new ratings problematic, absent major changes in their education and training.
As an additive or alternative to the president’s suggestions, I think we would be wise to make change where the “default” position benefits students and their families. So, as one example, what about enacting legislation, through an amendment to the Bankruptcy Code, that enables students and parents to discharge burdensome private and public loans through bankruptcy?
A recent study by the Center for American Progress suggested the dischargeability of select public and private loans (with a robust definition of what constitutes nondischargeable qualified student loans.) The Consumer Financial Protection Bureau and the Department of Education issued a report in 2012 suggesting reconsideration of the nondischargeability of private student loans.
To anticipate the suggestion that easing bankruptcy’s discharge will create a moral hazard, my experience over 30 years of working with debtors and consumer finance suggests that this common concern is not supported by the evidence.
The availability of bankruptcy and the opportunity for dischargeability of specified debt has not led to a wave of abusive bankruptcy filings. As I always have said, most people do not wake up in the morning and say, “Yippee. I get to file bankruptcy today, having failed at America’s rags-to-riches dream.”
Surely the president has latched onto an issue that matters – a college education for the betterment of individuals and their families and society at large. This is because, at the end of the day, we need an educated citizenry to preserve our democracy. The real issue is how we make that accurate idea a reality. As with most difficult issues, the devil remains in the details.
Karen Gross is president of Southern Vermont College. She served as a senior policy adviser to the U.S. Department of Education during 2012 and is now a consultant to the department. The views presented here are her own and do not represent the position of the government, including the Department of Education.