Policy making is difficult and complex; evaluating the effects of policy can also be quite difficult. Nevertheless, it is important that researchers and policy analysts undertake the hard work of asking difficult questions and doing their best to answer those questions.
This is what we attempted to do when we undertook a yearlong effort to evaluate the effects of performance funding on degree completions. This effort has culminated in two peer-reviewed papers and one policy brief which summarizes the results of those papers. Our policy brief was widely distributed and the results were discussed in a recent Inside Higher Ed article.
Recently, Nancy Shulock (of California State University at Sacramento) and Martha Snyder (of HCM Strategists, a consulting firm) responded to our policy brief with some sharp criticism in these pages. As academics, we are no strangers to criticism; in fact, we welcome it. While they rightly noted the need for stronger evidence to guide the performance funding debate, they also argued that we produced “a flawed piece of research,” that our work was “simplistic,” and that it merely “compares outcomes of states where the policy was in force to those where it was not.”
This is not only an inaccurate representation of our study, but it shows an unfortunate misunderstanding of the latest innovations in social science research. We see this as an opportunity to share some insights into the analytical technique Shulock and Snyder are skeptical of.
The most fail-proof method of determining whether a policy intervention had an impact on an outcome is an experimental design. In this instance, it would require that we randomly assign states to adopt performance funding while others retain the traditional financing model. But because this is impossible, “quasi-experimental” research designs can be used to simulate experiments. The U.S. Department of Education sees experimental and quasi-experimental research as “the most rigorous methods to address the question of project effectiveness”, and the American Educational Research Association actively encourages scholars to use these techniques when experiments are not possible to undertake.
We chose the quasi-experimental design called “differences-in-differences,” where we compared performance-funding states with non-performance funding states (one difference) in the years before and after the policy intervention (the other difference). The difference in these differences told us much more about the policy’s impact than could traditional regression analysis or descriptive statistics. Unfortunately most of the quantitative research on performance funding is just that – traditional regression or descriptive analysis – and neither strategy can provide rigorous or convincing evidence of the policy’s impacts. For an introduction to the method, see here and here.
Every study has its limitations and ours is no different. On page 3 of the brief (and in more detail in our full papers) we explain some of these issues and the steps we took to test the robustness of our findings. This includes controlling for multiple factors (e.g., state population, economic conditions, tuition, enrollment patterns, etc.) that might have affected degree completions in both the performance funding states and the non-performance funding states. Further, Shulock and Snyder claim that we “failed to differentiate among states in terms of when performance funding was implemented,” when in fact we do control for this as explained in our full papers.
We do not believe that introducing empirical evidence into the debates about performance funding is dangerous. Rather, we believe it is sorely missing. We also understand that performance funding is a political issue and one that is hotly debated. Because of this, it can be dangerous to promote expensive policies without strong empirical evidence of positive impacts. We wish this debate occurred with more transparency to these politics, as well as with a better understanding of the latest developments in social science research design.
The authors take issue with a second point that requires a response – their argument that we selected the wrong performance funding states. We disagree. The process of identifying these states required painstaking attention to detail and member-checks from experts in the field, especially when examining a 20-year period of time (1990-2010). In our full studies, we provide additional information beyond what is included in our brief (see endnote 8) about how we selected our states.
The authors suggested that we misclassified Texas and Washington. With Texas, our documents show that in 2009, SB 1 approved “Performance Incentive” funding for the biennium. Perhaps something changed after that year that we missed, and this would be a valid critique, but we have no evidence of that. The authors rightly noticed that our map incorrectly coded Washington state as having performance funding for four-year and two-year colleges when in fact it is only for two-year colleges. We correctly identified Washington in our analysis and this is displayed correctly in the brief (see Table 2).
All of these details are important, and we welcome critiques from our colleagues. After all, no single study can explain a single phenomenon; it is only through the accumulation of knowledge from multiple sources that allows us to see the full picture. Policy briefs are smaller fragments of this picture than are full studies, so we encourage readers to look at both the brief and the full studies to form their opinions about this research.
We agree with the authors that there is much that our brief does not tell us and that there are any number of other outcomes one could choose to evaluate performance funding. Clearly, performance funding policies deserve more attention and we intend to conduct more studies in the years to come. So far, all we can say with much confidence is that, on average and in the vast majority of cases, performance funding either had no effect on degree completions or it had a negative effect.
We feel that this is an important finding and that it does “serve as a cautionary tale.” Policy makers would be wise to acknowledge our findings in the context of other information and considerations when they consider whether to implement performance funding in their states, and if so, what form it might take.
Designing and implementing performance funding is a costly endeavor. It is costly in terms of the political capital expended by state law makers; the time devoted by lawmakers, state agency staff, and institutional leaders; and in the amount of money devoted to these programs. Therefore, inserting rigorous empirical analysis to the discussion and debate is important and worthwhile.
But just as the authors say performance funding “should not be dismissed in one fell swoop,” it should not be embraced in one fell swoop either. This is especially true given the mounting evidence (for example here, here, here, and here) that these efforts may not actually work in the same way the authors believe they should.
Claiming that there is “indisputable evidence that incentives matter in higher education” is a bold proposition to make in light of these studies and others. Only time will tell as more studies come out. Until then, we readily agree with some of the author’s points and critiques and would not have decided to draft this reply had they provided an accurate representation of our study’s methods.
David Tandberg is assistant professor of higher education at Florida State University. Nicholas Hillman is an assistant professor educational leadership & policy analysis at the University of Wisconsin at Madison.
A recent research paper published by the Wisconsin Center for the Advancement of Postsecondary Education and reported on by Inside Higher Ed criticized states' efforts to fund higher education based in part on outcomes, in addition to enrollment. The authors, David Tandberg and Nicholas Hillman, hoped to provide a "cautionary tale" for those looking to performance funding as a "quick fix."
While we agree that performance-based funding is not the only mechanism for driving change, what we certainly do not need are impulsive conclusions that ignore positive results and financial context. With serious problems plaguing American higher education, accompanied by equally serious efforts across the country to address them, it is disheartening to see a flawed piece of research mischaracterize the work on finance reform and potentially set back one important effort, among many, to improve student success in postsecondary education.
As two individuals who have studied performance funding in depth, we know that performance funding is a piece of the puzzle that can provide an intuitive, effective incentive for adopting best practices for student success and encourage others to do so. Our perspective is based on the logical belief that tying some funding dollars to results will provide an incentive to pursue those results. This approach should not be dismissed in one fell swoop.
We are dismayed that the authors were willing to assert an authoritative conclusion from such simplistic research. The study compares outcomes of states "where the policy was in force" to those where it was not -- as if "performance funding" is a monolithic policy everywhere it has been adopted.
The authors failed to differentiate among states in terms of when performance funding was implemented, how much money is at stake, whether performance funds are "add ins" or part of base funding formulas, the metrics used to define and measure "performance," and the extent to which "stop loss" provisions have limited actual change in allocations. These are critical design issues that vary widely and that have evolved dramatically over the 20-year period the authors used to decide if "the policy was in force" or not.
Treating this diverse array of unique approaches as one policy ignores the thoughtful work that educators and policy makers are currently engaged in to learn from past mistakes and to improve the design of performance funding systems. Even a well-designed study would probably fail to reveal positive impacts yet, as states are only now trying out new and better approaches -- certainly not the "rush" to adopting a "quick fix" that the authors assert. It could just as easily be argued that more traditional funding models actually harm institutions trying to make difficult and necessary changes in the best interest of students and their success (see here and here).
The simplistic approach is exacerbated by two other design problems. First, we find errors in the map indicating the status of performance funding. Texas, for example, has only recently implemented (passed in spring 2013) a performance funding model for its community colleges; it has yet to affect any budget allocations. The recommended four-year model was not passed. Washington has a small performance funding program for its two-year colleges but none for its universities. Yet the map shows both states with performance funding operational for both two-year and four-year sectors.
Second, the only outcome examined by the authors was degree completions as it "is the only measure that is common among all states currently using performance funding." While that may be convenient for running a regression analysis, it ignores current thinking about appropriate metrics that honor different institutional missions and provide useful information to drive institutional improvement. The authors make passing reference to different measures at the end of the article but made no effort to incorporate any realism or complexities into their statistical model.
On an apparent mission to discredit performance funding, the authors showed a surprising lack of curiosity about their own findings. They found eight states where performance funding had a positive, significant effect on degree production but rather than examine why that might be, they found apparent comfort in the finding that there were "far more examples" of performance funding failing the significance tests.
"While it may be worthwhile to examine the program features of those states where performance funding had a positive impact on degree completions," they write, "the overall story of our state results serves as a cautionary tale." Mission accomplished.
In their conclusion they assert that performance funding lacks "a compelling theory of action" to explain how and why it might change institutional behaviors.
We strongly disagree. The theory of action behind performance funding is simple: financial incentives shape behaviors. Anyone doubting the conceptual soundness of performance funding is, in effect, doubting that people respond to fiscal incentives. The indisputable evidence that incentives matter in higher education is the overwhelming priority and attention that postsecondary faculty and staff have placed, over the years, on increasing enrollments and meeting enrollment targets, with enrollment-driven budgets.
The logic of performance funding is simply that adding incentives for specified outcomes would encourage individuals to redirect a portion of that priority and attention to achieving those outcomes. Accepting this logic is to affirm the potential of performance funding to change institutional behaviors and student outcomes. It is not to defend any and all versions of performance funding that have been implemented, many of which have been poorly done. And it is not to criticize the daily efforts of faculty and staff, who are committed to student success but cannot be faulted for doing what matters to maintain budgets.
Surely there are other means -- and more powerful means -- to achieve state and national goals of improving student success, as the authors assert. But just as surely it makes sense to align state investments with the student success outcomes that we all seek.
Nancy Shulock is executive director of the Institute for Higher Education Leadership & Policy at California State University at Sacramento, and Martha Snyder is senior associate at HCM Strategists.
The endless conversation about big-time football appears to be reaching a point of decision and change. While some would like to see big-time football disappear from the collegiate enterprise, this is not likely or desirable. We like our football, our constituents like our football, the nation tunes into our football, and for many universities, nothing is more visible or engaging about their operations than their football.
So we will have football. Success in the big-time has created a variety of conflicts between ideal models for amateur competition and real behavior that the current system finds increasingly difficult to manage. Much of the difficulty stems from the problem of money: too much of it. The money is not going away because we like our football, but we need a better model to manage the money and the game.
Part of the difficulty comes from trying to reconcile the notion of a college-based student activity with the exceptionally high profile and revenue of a previously unimaginably popular and commercial viable enterprise. Football at the top level of American’s institutions, exemplified by the five major athletic conferences, is a big business that depends for its success on the recruitment, retention, and development of superb athletes who must also function as students.
Among the many issues and controversies (academic, ethical, financial, and operation) that complicate our current operating arrangements, refined over many years, is the mismatch in the eyes of our constituents between the revenue scale of the football enterprise and the compensation provided football players (especially the celebrity players of the most successful teams). The current model limits compensation to direct payments for tuition and other college expenses and the indirect compensation by providing players with a high-cost platform for the development of their potential professional value and their possible future value from a professional contract.
Boxed in by our definition of amateur student-athlete, we have found it difficult to construct imaginative ways of reflecting market circumstances that affect the players. We sometimes think that without the strict amateur definition, the college football enterprise would collapse into an uninteresting minor league activity divorced from its academic sponsorship. This may well underestimate the potential creativity of the university, which has, in other contexts, developed mechanisms that could provide a useful model for the football dilemma.
Any model would need to (1) address the market issue of the value of football players within the five major conferences, (2) redefine the organic connection between college education and the student-athletes in the football enterprise, and (3) retain the connection between football and the rest of the college intercollegiate athletic enterprise.
Fortunately, universities have mechanisms for dealing with similar issues that might be adapted to meet the needs of the football enterprise. Think, for a minute, about the university medical center hospital. In many cases this is a separate not-for-profit enterprise, affiliated with the university. Its relationship with the parent university is contractual, and transactions that involve university and hospital are done not through university internal governance mechanisms but by contract that specifies how the hospital economy and the university economy will interact.
The agreements also specify how the academic activities of the university in medical education and research will engage the hospital and how the hospital activities related to patient care and other services will engage the university.
But the two enterprises are financially, legally and operationally separate organizations. They may well have interlocking boards of directors/trustees, but the labor and financial structure of the hospital is not the same nor is it constrained by the circumstances of the university, and the university’s labor and financial structure is not the same nor is it constrained by the circumstances of the hospital.
Adapting this notion to football, imagine that we spin off the football enterprise out of our university and athletic department into a private not-for-profit corporation affiliated with the university, let’s call it the University Football Corp, or UFC (and we could substitute the name of the University for each institution’s football not-for-profit). We license our name and trademarks to our UFC, we lease our football-related sports facilities to the UFC, we contract for various management services that the university may provide the UFC. The coach and other athletic personnel who operate the football activity will be employees of the not-for-profit UFC, and will not be constrained or managed by the university. The financial structure of the football enterprise will require that it be self-supporting. This should not be a problem for the football programs in the five major conferences since almost all of them do indeed make a profit, even if their universities’ intercollegiate sports programs over all lose money.
Students who perform as football players will be employed by the UFC not-for-profit to perform football duties, but requirements for a football player employee will include an age limit between 18 and 24, eligibility limits, enrollment in the university, maintenance of academic good standing, and progress toward a degree. This is not unusual for other student employees of the university.
The football employee will receive a two-part compensation. The first part will be equivalent to the full cost of attendance at the university, to match the requirement that the football player be a student. This amount will be paid by the UFC not-for-profit to the university that will award the financial aid as it would for any student. The second part will be variable and will depend on the market value of the football player to the UFC not-for-profit. This second amount can vary by season and the market for college-age football players, and might well follow norms and procedures established in the National Football League.
This model bears a close resemblance to what we do for medical students, for faculty physicians, and for other university people who have duties and obligations associated with an independent hospital affiliated with the university.
Where is the NCAA in this model? Like hospitals, the UFC not-for-profit will be regulated by an external agency, in this case the NCAA, that will establish the game rules as it does now, and specify the academic eligibility requirements for football players to be considered students, but the NCAA will not regulate payments to players. These will be managed by each institution’s UFC, but probably in accord with rules established by the five major conferences. The NCAA may well require these organizations to have transparent and independently audited financial records so that the public is clear about the way in which the athlete who is also a student is being paid and managed and clear about the financial arrangements between each UFC and its parent university.
Football players must be students in good standing and making appropriate progress toward a degree and can only have four years of eligibility, but they can test their value in the commercial sports marketplace at any time and choose to leave for professional work.
Once hired for professional play, of course, they will no longer be eligible to participate in the university-related not-for-profit UFC.
Athletes within the UFC can also contract for commercial endorsements and other sports-related (but not sports competition) activities and earn stipends or fees, but they must do so through the UFC not-for-profit so that the organization can identify conflicts of interest or commitment. Similarly, employees of the UFC (coaches, athletic directors, and others) can earn outside income related to sports but must report this income and receive approval for outside commitments from their institution’s UFC (again to prevent conflicts of interest and commitment). Alumni and other fans can contribute to the football enterprise, either to support players or to subsidize athletic facilities, but again, always through the institutional UFC to ensure transparency.
The UFC not-for-profit will create various funds and arrangements for player health and safety and compensation for injuries or other insurance-related functions. Whether it decides that it is better to hire the football players as employees or deal with their football participation as independent contractors will be an issue to be resolved as the market for football players indicates. Either solution would work, although of course the players are likely to emulate the professional marketplace and create a union to represent their interests. In some universities this would mirror the union representation of other student employees. Players can have agents, lawyers or other advisers to help them negotiate the contracts that govern their college-related football participation, although not the academic requirements that define them as students. The five major conferences may well establish salary caps and other financial constraints proved useful in the professional marketplace.
Because each UFC football enterprise is affiliated and ultimately controlled by its parent university, if indirectly through its board appointments, and because it is required to manage its enterprise through contracts with the university that are publicly available, the university can reap the benefits of big-time football without the constraints of trying to fit the football juggernaut into the university’s normal academic infrastructure. However, the university can require that the football UFC provide a significant payment to the university for the use of its name and other marks, a payment that will serve to subsidize the university athletic program for other sports as happens currently.
Finally, because this arrangement puts the academic scholarships for football players who must be students inside the university (but paid by the UFC not-for-profit), the current commitment to women’s sports driven by Title IX requirements to keep scholarships reasonably equivalent will remain.
Any university can create this model and participate, although they will need conference support, television revenue, and other characteristics of big-time programs to succeed. Such an opportunity may well help institutions decide that they want good football but not big-time football.
While endless details will need to be worked out, as were required to create the arrangements that govern independent not-for-profit hospitals affiliated with major research universities, the model offers an approach to the growing difficulty of managing big-time football within the current university context.
John Lombardi is president emeritus of the University of Florida and served most recently as president of the Louisiana State University System. He is the author of the forthcoming How Universities Work (Johns Hopkins University Press).