Innovation in higher education, I sometimes think, is a bit like the weather. Everybody talks about it, but nobody does anything about it.
Every six months or so, as some new conference or other on the future of higher education heaves into view, I’ll get a call asking if I can list any and all recent innovations in higher education. The people on the other end of the line seem to feel that these innovations must surely be out there, so they make phone calls looking for them. But they always seem disappointed when I resort to listing the usual suspects: online universities, open educational resources, commercial ventures looking to partner with institutions. That’s not innovation, the people on the other end of the line seem to be saying. And in many respects, I agree with them. We haven’t yet seen anything truly game-changing, have we?
In recent months, the focus on innovation in higher education has turned to “disruptive innovation,” that concept originally formulated over a decade ago by Harvard Business School professor Clayton Christensen to describe change and innovation across numerous industries, but which he has more recently begun applying to education. Now everybody wants to know where the disruptive innovations in higher education are hiding.
For his part, Christensen points to online learning. But even by the standards established by Christensen’s own theory (where disruptive innovations are easier to use, cheaper, and serve new audiences), the case for online learning as a disruptive innovation is equivocal.
Is it a simpler, easier-to-use product? In some respects, but not all.
Is it less expensive to deliver? Outside of a few grant-program case studies, not particularly; the potential may be there, but it has yet to be fully realized.
Is it reaching a new audience? Probably, yes, but the evidence is mostly indirect and approximate.
Of course, there’s a reason why we don’t actually see much in the way of real innovation in higher education, and Christensen understands this. Incumbent leaders in mature industries engage in what Christensen calls “sustaining innovation” – the development of new features and benefits that make a product or service more useful, but not dramatically so. Think, for example, of the addition of a camera on the iPad2. With an increase in benefits, prices typically rise as well. What keeps pricing in balance in most industries, however, are those disruptive innovations – think of the personal computers that supplanted the mainframes decades ago. These cheaper and easier to use tools attract new audiences to the category and refashion the economics of the industry’s business model.
While colleges and universities may well engage in some sustaining innovations (the high-rise dormitories, the state-of-the-art fitness centers, the not-entirely-mythical rock climbing walls, not mention the world-class science labs and other high-tech investments), the fact is that they face little in the way of disruptive innovation because they have a lock on the market – it’s called accreditation – and thus there’s little opportunity for new entrants to come in and offer something less expensive or simpler to use.
To my mind, if you’re looking for an innovation opportunity, technology is just a part of the story. The real innovation – in price, in ease of use, in access – will occur when our colleges and universities face some real competition, and that will only come when we allow some new, entrepreneurial providers into the market.
If you want innovation, I say, remove the barriers.
To that end, I’d like to propose that the U.S. Department of Education establish a new “demonstration program,” not unlike the Distance Learning Demonstration Program of the past. That former program allowed institutions that delivered a majority of their programs online to distribute Title IV funds. Twenty-four institutions – a mix of nonprofits and for-profits – participated in the program. Along the way, we learned something important about the potential for scale within online learning; and today, one in four college students has taken at least one course online.
Now we need something a little different, but based on the same model – call it the “Innovation Demonstration Program.” In this case, the program will charter new organizations to offer degrees and distribute Title IV funds – even if they lack accreditation. That has the potential to open up real innovation within multiple segments of the marketplace.
Commercial organizations that offer tutoring services, curriculum, or learning technologies could get into the degree granting business and even make federal financial aid available to their students.
At the same time, established institutions might see this as a terrific opportunity to build new degree-granting organizations adjacent their own traditional campuses – unencumbered by the regulatory and governance hurdles that currently stymie their attempts to reach new markets, deliver new programs, or otherwise rethink how they do business.
It will, of course, be necessary to guard against the potential for new diploma mills entering the market and targeting federal dollars, but that’s where the regulatory apparatus becomes useful. It can both foreclose fraud and stimulate innovation at the same time. Under the kind of close supervision that a federal demonstration program would require, a few dozen experiments of this sort could teach us a great deal about what’s really possible when it comes to innovation in higher education.
If you think this sounds absurd, consider the case of the Relay Graduate School of Education, granted a charter by the New York State Board of Regents earlier this year to offer master’s degrees to teachers in New York. Founded by three charter school organizations – KIPP, Achievement First, and Uncommon Schools – the Relay Graduate School of Education was purpose-built to meet the education and professional development needs of those schools’ own teachers.
Along the way, Relay did something innovative. It tore up the semester model. In its place, Relay delivers 60 discrete learning modules. Students learn in the context of the schools in which they teach, and online curriculum is augmented by cohort discussions within the schools, all under the supervision of on-site mentors. This is a very different way of thinking about delivering education – and it’s innovative.
What makes it innovative isn’t that there’s technology involved – it’s really a very people-centered learning model – it’s that the organization is free to rethink the “why” and “how” of teacher professional development. Equally important, the oversight of the Board of Regents puts Relay on a level playing field with all of the other traditional providers of master’s degrees in education within the state of New York. Now ask yourself why the same thing shouldn’t be happening in disciplines such as business, engineering, computer science, health care, and numerous other fields, and on a national scale.
There is, after all, another key element in Christensen’s theory of disruptive innovation. It happens at the margins, and it happens within organizations that are free from the obligations of established incumbents. One of the great misunderstandings regarding Christensen’s theory, in my view, is that we can all disruptively innovate ourselves. But Christensen himself points out that the only companies that have successfully accomplished that feat have done so by establishing separate, discrete R&D units free of the pressures of the parent organization’s business model, customer demands, profit targets and more. The reality is, more often than not, that disruptive innovations put established incumbents out of business. That, after all, is what makes them disruptive.
If traditional higher education wants to innovate – if it realizes that it must – then that innovation will have to take place in the margins, free from the demands of traditional culture, regulation, and financial models. An Innovation Demonstration Program would allow us a chance to see just how much invention is in us, and how far we can go in lowering prices, increasing access, and educating the nation.
Peter Stokes is executive vice president of Eduventures, a higher education consulting firm.
Do majors matter? Since students typically spend more time in their area of concentration than anywhere else in the curriculum, majors ought to live up to their name and produce really major benefits. But do they?
Anthony P. Carnevale, the Director of Georgetown’s Center for Education and the Workforce, had recently provided a clear answer. Majors matter a lot -- a lot of dollars and cents. In a report entitled “What’s it Worth,” he shows how greatly salaries vary by major, from $120,000 on average for petroleum engineers down to $29,000 for counseling psychologists.
But what if one asked whether majors make differing contributions to students’ cognitive development? The answer is once again yes, but the picture looks very different from the one in the Georgetown study.
A few years ago, Paul Sotherland, a biologist at Kalamazoo College in Michigan, asked an unnecessary question and got not an answer but a tantalizing set of new questions. It was unnecessary because most experts in higher education already knew the answer, or thought they did: as far as higher-order cognitive skills are concerned, it doesn’t matter what you teach; it’s how you teach it.
What Sotherland found challenged that conventional wisdom and raised new questions about the role of majors in liberal education. Here’s what he did. Kalamazoo had been using the Collegiate Learning Assessment (CLA) to track its students’ progress in critical thinking and analytical reasoning. After a few years it become clear that Kalamazoo students were making impressive gains from their first to their senior years. Sotherland wondered if those gains were across the board or varied from field to field.
Since gains in CLA scores tend to follow entering ACT or SAT scores, they “corrected” the raw data to see what gains might be attributed to instruction. They found significant differences among the divisions, with the largest gains (over 200 points) in foreign languages, about half that much in the social sciences, still less in the fine arts and in the humanities, least of all in the natural sciences .
How was this to be explained? Could reading Proust somehow hone critical thinking more than working in the lab? (Maybe so.)
But the sample size was small and came from one exceptional institution, one where students in all divisions did better than their SAT scores would lead one to expect, and where the average corrected gain on CLA is 1.5 standard deviations, well above the national average. (Perhaps Inside Higher Ed should sponsor the “Kalamazoo Challenge,” to see if other institutions can show even better results in their CLA data.)
The obvious next step was to ask Roger Benjamin of the Collegiate Learning Assessment if his associates would crunch some numbers for me. They obliged, with figures showing changes over four years for both parts of the CLA -- the performance task and analytical writing. Once again, the figures were corrected on the basis of entering ACT or SAT scores.
The gains came in clusters. At the top was sociology, with an average gain of just over 0.6 standard deviations. Then came multi- and interdisciplinary studies, foreign languages, physical education, math, and business with gains of 0.50 SDs or more.
The large middle cluster included (in descending order) education, health-related fields, computer and information sciences, history, psychology, law enforcement, English, political science, biological sciences, and liberal and general studies.
Behind them, with gains between 0.30 and 0.49 SDs, came communications (speech, journalism, television, radio etc.), physical sciences, nursing, engineering, and economics. The smallest gain (less than 0.01 standard deviations) was in architecture.
The list seemed counterintuitive to me when I first studied it, just as the Kalamazoo data had. In each case, ostensibly rigorous disciples, including most of the STEM disciplines (the exception was math) had disappointing results. Once again the foreign languages shone, while most other humanistic disciplines cohabited with unfamiliar bedfellows such as computer science and law enforcement. Social scientific fields scattered widely, from sociology at the very top to economics close to the bottom.
When one looks at these data, one thing is immediately clear. The fields that show the greatest gains in critical thinking are not the fields that produce the highest salaries for their graduates. On the contrary, engineers may show only small gains in critical thinking, but they often command salaries of over $100,000. Economists may lag as well, but not at salary time, when, according to “What’s It Worth” their graduates enjoy median salaries of $70,000. At the other end majors in sociology and French, German and other commonly taught foreign languages may show impressive gains, but they have to be content with median salaries of $45,000.
But what do these data tell us about educational practice? It seems unlikely that one subject matter taken by itself has a near-magical power to result in significant cognitive gains while another does nothing of the sort. If that were the case, why do business majors show so much more progress than economics majors? Is there something in the content of a physical education major (0.50 SDs) that makes it inherently more powerful than a major in one of the physical sciences (0.34 SDs)? I doubt it.
Since part of the CLA is based on essays students write during the exam, perhaps the natural science majors simply had not written enough to do really well on the test. (That’s the usual first reaction, I find, to unexpected assessment results -- "there must be something wrong with the test.") That was, however, at best a partial explanation, since it didn’t account for the differences among the other fields. English majors, for example, probably write a lot of papers, but their gains were no greater than those of students in computer sciences or health-related fields.
Another possibility is that certain fields attract students who are ready to hone their critical thinking skills. If so, it would be important to identify what it is in each of those fields that attract such students to it. Are there, for example, “signature pedagogies” that have this effect? If so, what are they and how can their effects be maximized? Or is it that certain pedagogical practices, whether or not they attract highly motivated students, increase critical thinking capacities – and others as well? For example, the Wabash national study has identified four clusters of practices that increase student engagement and learning in many areas (good teaching and high-quality interactions with faculty, academic challenge and high expectations, diversity experiences, and higher-order, integrative, and reflective learning).
Some fields, moreover, may encourage students to “broaden out” -- potentially important for the development of critical thinking capacities as one Kalamazoo study suggests. Other disciplines may discourage such intellectual range.
One other hypothesis, I believe, also deserves closer consideration. The CLA is a test of post-formal reasoning. That is, it does not seek to find out if students know the one right answer to the problems it sets; on the contrary, it rewards the ability to consider the merits of alternative approaches. That suggests that students who develop the habit of considering alternative viewpoints, values and outcomes and regularly articulate and weigh alternative possibilities may have an advantage when taking the CLA exam, and quite possibly in real-life settings as well.
Since the study of foreign languages constantly requires the consideration of such alternatives, their study may provide particularly promising venues for the development of such capacities. If so, foreign languages have a special claim on attention and resources even in a time of deep budgetary cuts. Their "signature pedagogies," moreover, may provide useful models for other disciplines.
These varying interpretations of the CLA data open up many possibilities for improving students’ critical thinking. But will these possibilities be fully utilized without new incentives? The current salary structure sends a bad signal when it puts the money where students make very small gains in critical thinking, and gives scant reward to fields that are high performers in this respect . (For example, according to the College & University Professional Association for Human Resources, full professors in engineering average over $114,000, while those in foreign languages average just over $85,000.
Isn’t it time to shift some resources to encourage experimentation in all fields to develop the cognitive as well as the purely financial benefits of the major?
W. Robert Connor
W. Robert Connor is senior advisor to the Teagle Foundation.
In my work as Oregon’s college evaluator, I am often asked why state approval is not "as good as accreditation" or "equivalent to accreditation."
We may be about to find out, to our sorrow: One version of the Higher Education Act reauthorization legislation moving through Congress quietly allows states to become federally recognized accreditors. A senior official in the U.S. Department of Education has confirmed that one part of the legislation would eliminate an existing provision that says state agencies can be recognized as federally approved accreditors only if they were recognized by the education secretary before October 1, 1991. Only one, the New York State Board of Regents, met the grandfather provision. By striking the grandfather provision, any state agency would be eligible to seek recognition.
If such a provision becomes law, we will see exactly why some states refuse to recognize degrees issued under the authority of other states: It is quite possible to be state-approved and a low-quality degree provider.Which states allow poor institutions to be approved to issue degrees?
Here are the Seven Sorry Sisters: Alabama (split authority for assessing and recognizing degrees), Hawaii (poor standards, excellent enforcement of what little there is), Idaho (poor standards, split authority), Mississippi (poor standards, political interference), Missouri (poor standards, political interference), New Mexico (grandfathered some mystery degree suppliers) and of course the now infamous Wyoming (poor standards, political indifference or active support of poor schools).
Wyoming considers degree mills and other bottom-feeders to be a source of economic development. You’d think that oil prices would relieve their need to support degree mills. Even the Japanese television network NHK sent a crew to Wyoming to warn Japanese citizens about the cluster of supposed colleges there: Does the state care so little for foreign trade it does not care that 10 percent of the households in Japan saw that program? You’d think that Vice President Dick Cheney and U.S. Senator Mike Enzi, who now chairs the committee responsible for education, would care more about the appalling reputation of their home state. Where is Alan Simpson when we need him?
In the world of college evaluation, these seven state names ring out like George Carlin’s “Seven Words You Can’t Say On Television,” and those of us responsible for safeguarding the quality of degrees in other states often apply some of those words to so-called “colleges” approved to operate in these states -- so-called “colleges” like Breyer State University in Alabama and Idaho (which “State” does this for-profit represent, anyway?).
There are some dishonorable mentions, too, such as California, where the standards are not bad but enforcement has been lax and the process awash in well-heeled lobbyists. The new director of California’s approval agency, Barbara Ward, seems much tougher than recent placeholders -- trust someone trained as a nurse to carry a big needle and be prepared to use it.
The obverse of this coin is that in some states, regulatory standards are higher than the standards of national accreditors, as Oregon discovered when we came across an accredited college with two senior officials sporting fake degrees. The national accreditors, the Accrediting Commission of Career Schools and Colleges of Technology and the Accrediting Bureau of Health Education Schools, had not noticed this until we mentioned it to them. What exactly do they review, if they completely ignore people’s qualifications?
The notion that membership in an accrediting association is voluntary is, of course, one of the polite fictions that higher education officials sometimes say out loud when they are too far from most listeners to inspire a round of laughter. In fact, losing accreditation is not far removed from a death sentence for almost any college, because without accreditation, students are not eligible for federal financial aid, and without such aid, most of them can’t go to school – at least to that school.
For this reason, if Congress ever decoupled aid eligibility from accreditation by one of the existing accreditors -- for example, by allowing state governments to become accreditors -- the “national” accreditors of schools would dry up and blow away by dawn the next day: They serve no purpose except as trade associations and milking machines for federal aid dollars.
The Libertarian View of Degrees
One view of the purpose and function of college degrees suggests that the government need not concern itself with whether a degree is issued by an accredited college or even a real college. This might be considered the classic libertarian view: that employers, clients and other people should come to their own conclusions, based on their own research, regarding whether a credential called a “degree” by the entity that issued (or printed) it is appropriate for a particular job or need. This view is universally propounded by the owners of degree mills, who become wealthy by selling degrees to people who think they can get away with using them this way.
The libertarian view is tempting, but presupposes a capacity and inclination to evaluate that most employers have always lacked and always will, while of course an average private citizen is even more removed from that ability and inclination. Who will actually do the research that the hypothetical perfect employer should do?
Consider the complexities of the U.S. accreditation system, the proliferation of fake accreditors complete with names nearly identical to real ones (there were at least two fake DETCs, imitating the real Distance Education Training Council, in 2005), phone numbers, carefully falsified lists of approved schools, Web sites showing buildings far from where the owners had ever been and other accoutrements.
To the morass of bogus accreditors in the U.S., add the world. Hundreds of jurisdictions, mostly not English-speaking, issuing a bewildering array of credentials under regimens not quite like American postsecondary education. Add a layer of corruption in some states and countries, a genial indifference in others, a nearly universal lack of enforcement capacity and you have a recipe for academic goulash that even governments are hard-pressed to render into proper compartments. In the past 10 days my office has worked with national officials in England, Sweden, The Netherlands, Canada and Australia to sort out suspicious degree validations. Very few businesses and almost no private citizens are capable of doing this without an exhausting allocation of time and resources. It does not and will not happen.
Should state governments accredit colleges?
State governments, not accreditors or the federal government, are the best potential guarantors of degree program quality at all but the major research universities, but only if they take their duty seriously, set and maintain high standards and keep politicians from yanking on the strings of approval as happens routinely in some states. Today, fewer than a dozen states have truly solid standards, most are mediocre and several, including the Seven Sorry Sisters, are quite poor.
If Congress is serious about allowing states to become accreditors, there must be a reason. I can think of at least two reasons. First, such an action would kill off many existing accreditors without having their work added to the U.S. Department of Education (which no one in their right mind, Democrat, Republican or Martian, wants to enlarge). This would count as devolutionary federalism (acceptable to both parties under the right conditions).
The second reason is the one that is never spoken aloud. There will be enormous, irresistible pressure on many state governments to accredit small religious schools that could never get accredited even by specialized religious accreditors today. The potential bounty in financial aid dollars for all of those church-basement colleges is incalculable.
Remember that another provision of the same proposed statute would prohibit even regionally accredited universities from screening out transfer course work based on the nature of the accreditor. Follow the bread crumbs and the net result will be a huge bubble of low-end courses being hosed through the academic pipeline, with the current Congressional leadership cranking the nozzle.
The possibility of such an outcome should provide impetus to the discussions that have gone on for many years regarding the need for some uniformity (presumably at a level higher than that of the Seven Sorry Sister states) in standards for state approval of colleges. We need a “model code” for state college approvals, something that leading states can agree to (with interstate recognition of degrees) and that states with poor standards can aspire to.
The universe of 50 state laws, some excellent and some abysmal, allows poor schools to venue-shop and then claim that their state approval makes them good schools when they are little better than diploma mills. We must do better.
Should states accredit colleges? Only if they can do it well. Today’s record is mixed, and Congress should not give states the power to accredit (or allow the Department of Education to give states the power) until they have proven that their own houses are in order. That day has not yet come.
Alan L. Contreras
Alan L. Contreras has been administrator of the Oregon Office of Degree Authorization, a unit of the Oregon Student Assistance Commission, since 1999. His views do not necessarily represent those of the commission.
The new Carnegie classifications have emerged from gestation, showing a great deal of thought and energy, which is too bad. Once again we are classifying the boxes and not the fruit.
The education establishment works very hard to say that the classifications are not intended to represent a pecking order among institutions, but the rest of the world instantly uses it that way. The gold rush mentality causes perfectly respectable regional colleges (e.g., Western Oregon University, in my neck of the woods) to wriggle and stretch through all manner of political hoops to become “Universities,” even faux flagships such as the magically relabeled Missouri State University (another perfectly respectable regional college all tarted up with nowhere to go).
There are other classification systems. Although the nation’s system of college accreditation and state approval is not exactly a college classification system, in some ways it functions as one. There are overlapping hierarchies of academic seraphim, cherubim and what Jack Aubrey, in one of Patrick O’Brien’s novels, calls “ordinary foremast angels.” Regional accreditors, national accreditors, state agencies and licensing boards all watch with proprietary care the shifting Cassini divisions between their roles and jurisdictions.
It is time to recognize that these boxes, too, are not that different in their basic descriptions except at the lowest levels, and that what matters is the quality of programs colleges contain as related to their mission.
All college degree programs are not created equal, nor are they equal today. This may be obvious to my friends who hold senior faculty positions at the University of Oregon, Illinois, Northwestern and elsewhere in the upper strata of research institutions. These, after all, are major research universities, formally authorized to condescend by their role as the top layer in the Carnegie classification system. Likewise, my friends who hold positions at Washington & Jefferson, Reed, Davidson and other fine liberal arts colleges can nod politely from their elfin perch in the canopy layer, content to consort with fine young minds.
The distinction is less obvious to those who work in and attend the great bulk of American colleges and universities, but it is nonetheless true. All colleges glaze the clay that they are given. The clay is largely formed by the time it reaches college, but the nature of both the formed clay and the available glaze differs widely, and society expects the resultant china to perform differently under different conditions. Let us recognize this reality and stop comparing unlike things.
Meandering through the pages of any college catalog looking at degree programs is much like walking the streets of an old western ghost town (or a movie set of one). All of the programs are excellent, leading their field and cutting edge -- apparently all the faculty trained at Lake Wobegon U. The main drag of programs consists of an impressive array of two-story buildings, all of similar appearance on the front. Some have two stories of solid building behind them, full of rooms and people. Others are mainly false fronts, behind which awaits what amounts to a conveyor belt: “this way to the Egress.”
It is time to stop classifying colleges and start classifying degree programs. Today this is done on an occasional basis for certain doctoral programs by the National Research Council, but other programs are largely ignored except by specialized accreditors in certain fields. All college degrees issued in the United States should be formally classified according to the nature of the work necessary to obtain them. Classification should be mandatory and no college degree should be exempt from it.
Such a classification scheme would allow students, employers and all other interested people to decide whether a particular college degree is what they want, either as a learning experience or in an employee, co-worker or colleague. There are other classification schemes already in existence, but they do not provide the right kind of information needed by most students, potential students and employers.
Each degree-granting program operating legally in the United States should be classified according to the strength of its program as determined by experts in its field. This system should not be applied to colleges, only to degree programs individually, because there is so much variation among programs at each school except the very best and the abysmal.
Note that this system says nothing about admission standards, only about program quality. There is no reason for a program to adjust its quality and expectations based on who enrolls in it: programs should decide what level they should most sensibly be at and stay there. Students will, for the most part, self-select based on program type just as they do now. In this system, programs would be classified as follows, as determined by peers in the field, with U and G representing undergraduate and graduate programs:
Honors (U). The best undergraduate programs, maintaining the highest expectations of students, and using the most difficult and complex curricula. Intended to provide superlative undergraduate learning for its own value and, secondarily, to prepare students to study in Research programs.
Research (G). The highest level graduate programs, intended to train professional researchers and faculty for colleges and universities, exclusive of licensed professional fields. There is no such thing as a “research institution,” there are only research-level programs, and it is time for the higher education establishment to admit this.
Professional (U, G). Programs that train students to practice in licensed professions. Programs of this nature can most effectively be evaluated by professionals in the field, in part using professional licensure rates and reputational surveys within licensed professions. The Carnegie system already has a similar category.
Standard (U, G). Programs designed for a wide variety of students, but not as challenging as Honors programs, with less ambitious expectations. These programs are not designed to prepare students to obtain doctoral degrees in Research programs, although some top students may succeed in such programs.
Basic (U, G). Programs that meet the basic expectations of a college-level degree program but do not meet the requirements for a Standard designation owing to some academic deficiencies.
Nonstandard (U, G). Programs that do not meet the basic expectations of a college-level degree program, or which decline to be evaluated.
New. Designation of “New” can be applied to any program, only at its own request, during its first five years of operation, as a qualifier for any other classification. Few programs show their true colors right out of the box.
In another of Patrick O’Brien’s novels, Stephen Maturin reminds us that “the kinds of happiness cannot be compared.” In some ways, neither can the kinds of degrees. The first necessary step, however, is to recognize that differences exist and to acknowledge them, rather than pretending that all regionally accredited colleges produce the same kind of degree-earning experience, or that degrees issued for one purpose are comparable to those issued for another. This is fiction.
Let us stop using institutional classifications of dubious meaning and start classifying academic programs using a system that is honest, based on evaluation by faculty and helps people understand college degree programs, as well as pointing out which emperors are naked and which paupers wear cloth of gold.
Alan L. Contreras
Alan L. Contreras has been administrator of the Oregon Office of Degree Authorization, a unit of the Oregon Student Assistance Commission, since 1999. His views do not necessarily represent those of the commission.
At the annual meeting of one of the regional accrediting agencies a few years ago, I wandered into the strangest session I’ve witnessed in any academic gathering. The first presenter, a young woman, reported on a meeting she had attended that fall in an idyllic setting. She had, she said, been privileged to spend three days “doing nothing but talking assessment” with three of the leading people in the field, all of whom she named and one of whom was on this panel with her. “It just doesn’t get any better than that!” she proclaimed. I kept waiting for her to pass on some of the wisdom and practical advice she had garnered at this meeting, but it didn’t seem to be that kind of presentation.
The title of the next panel I chose suggested that I would finally learn what accrediting agencies meant by “creating a culture of assessment.” This group of presenters, four in all, reenacted the puppet show they claimed to have used to get professors on their campus interested in assessment. The late Jim Henson, I suspect, would have advised against giving up their day jobs.
And thus it was with all the panels I tried to attend. I learned nothing about what to assess or how to assess it. Instead, I seemed to have wandered into a kind of New Age revival at which the already converted, the true believers, were testifying about how great it was to have been washed in the data and how to spread the good news among non-believers on their campus.
Since that time, I’ve examined several successful accreditation self-studies, and I’ve talked to vice presidents, deans, and faculty members, but I’m still not sure about what a “culture of assessment” is. As nearly as I can determine, once a given institution has arrived at a state of profound insecurity and perpetual self-scrutiny, it has created a “culture of assessment.” The self-criticism and mutual accusation sessions favored by Communist hardliners come to mind, as does a passage from a Credence Clearwater song: “Whenever I ask, how much should I give? The only answer is more, more!”
Most of the faculty resistance we face in trying to meet the mandates of the assessment movement, it seems to me, stems from a single issue: professors feel professionally distrusted and demeaned. The much-touted shift in focus from teaching to student learning at the heart of the assessment movement is grounded in the presupposition that professors have been serving their own ends and not meeting the needs of students. Some fall into that category, but whatever damage they do is greatly overstated, and there is indeed a legitimate place in academe for those professors who are not for the masses. A certain degree of quirkiness and glorious irrelevance were once considered par for the course, and students used to be expected to take some responsibility for their own educations.
Clearly, from what we are hearing about the new federal panel studying colleges, the U.S. Department of Education believes that higher education is too important to be left to academics. What we are really seeing is the re-emergence of the anti-intellectualism endemic to American culture and a corresponding redefinition of higher education in terms of immediately marketable preparation for specific jobs or careers. The irony is that the political party that would get big government off our backs has made an exception of academe.
This is not to suggest, of course, that everything we do in the name of assessment is bad or that we don’t have an obligation to determine that our instruction is effective and relevant. At the meeting of the National Association of Schools of Art and Design, I heard a story that illustrates how the academy got into this fix. It seems an accreditor once asked an art faculty member what his learning outcomes were for the photography course he was teaching that semester. The faculty member replied that he had no learning outcomes because he was trying to turn students into artists and not photographers. When asked then how he knew when his students had become artists, he replied, “I just know.”
Perhaps he did indeed “just know.” One of the most troubling aspects of the assessment movement, to my mind, is the tendency to dismiss the larger, slippery issues of sense and sensibility and to measure educational effectiveness only in terms of hard data, the pedestrian issues we can quantify. But, by the same token, every photographer must master the technical competencies of photography and learn certain aesthetic principles before he or she can employ the medium to create art. The photography professor in question was being disingenuous. He no doubt expected students to reach a minimal level of photographic competence and to see that competence reflected in a portfolio of photographs that rose to the level of art. His students deserved to have these expectations detailed in the form of specific learning outcomes.
Thus it is, or should be, with all our courses. Everyone who would teach has a professional obligation to step back and to ask himself or herself two questions: What, at a minimum, do I want students to learn, and how will I determine whether they have learned it? Few of us would have a problem with this level of assessment, and most of us would hardly need to be prompted or coerced to adjust our methods should we find that students aren’t learning what we expect them to learn. Where we fall out, professors and professional accreditors, is over the extent to which we should document or even formalize this process.
I personally have heard a senior official at an accrediting agency say that “if what you are doing in the name of assessment isn’t really helping you, you’re doing it wrong.” I recommend that we take her at her word. In my experience -- first as a chair and later as a dean -- it is helpful for institutions to have course outlines that list the minimum essential learning outcomes and which suggest appropriate assessment methods for each course. It is helpful for faculty members and students to have syllabi that reflect the outcomes and assessment methods detailed in the corresponding course outlines. It is also helpful to have program-level objectives and to spell out where and how such objectives are met.
All these things are helpful and reasonable, and accrediting agencies should indeed be able to review them in gauging the effectiveness of a college or university. What is not helpful is the requirement to keep documenting the so-called “feedback loop” -- the curricular reforms undertaken as a result of the assessment process. The presumption, once again, would seem to be that no one’s curriculum is sound and that assessment must be a continuous process akin to painting a suspension bridge or a battleship. By the time the painters work their way from one end to the other, it is time to go back and begin again. “Out of the cradle, endlessly assessing,” Walt Whitman might sing if he were alive today.
Is it any wonder that we have difficulty inspiring more than grudging cooperation on the part of faculty? Other professionals are largely left to police themselves. Not so academics, at least not any longer. We are being pressured to remake ourselves along business lines. Students are now our customers, and the customer is always right. Colleges used to be predicated on the assumption that professors and other professionals have a larger frame of reference and are in a better position than students to design curricula and set requirements. I think it is time to reaffirm that principle; and, aside from requiring the “helpful” documents mentioned above, it is past time to allow professors to assess themselves.
Regarding the people who have thrown in their lot with the assessment movement, to each his or her own. Others, myself included, were first drawn to the academic profession because it alone seemed to offer an opportunity to spend a lifetime studying what we loved, and sharing that love with students, no matter how irrelevant that study might be to the world’s commerce. We believed that the ultimate end of what we would do is to inculcate both a sensibility and a standard of judgment that can indeed be assessed but not guaranteed or quantified, no matter how hard we try. And we believed that the greatest reward of the academic life is watching young minds open up to that world of ideas and possibilities we call liberal education. To my mind, it just doesn’t get any better than that.
Edward F. Palm
Edward F. Palm is dean of social sciences and humanities at Olympic College, in Bremerton, Wash.
College officials and members of the public are watching with intense interest -- and, in some quarters, trepidation -- the proceedings of the U.S. Secretary of Education's Commission on the Future of Higher Education. Given that interest, the following is a memorandum that the panel's chairman, Charles Miller, wrote to its members offering his thinking about one of its thorniest subjects: accountability. As always on Inside Higher Ed, comments are welcomed below.
To: Members, The Secretary of Education’s Commission on the Future of Higher Education
From: Charles Miller, Chairman
Dear Commission Members:
The following is a synopsis of several ongoing efforts, in support of the Commission, in one of our principal areas of focus, "Accountability." The statements and opinions presented in the memo are mine and are not intended to be final conclusions or recommendations, although there may be a developing consensus.
I would appreciate feedback, directly or through the staff, in any form that is most convenient. This memo will be made public in order to promote and continue an open dialogue on measuring institutional performance and student learning in higher education.
As a Commission, our discussions to date have shown a number of emerging demands on the higher education system, which require us to analyze, clarify and reframe the accountability discussion. Four key goals or guiding principles in this area are beginning to take shape.
First, more useful and relevant information is needed. The federal government currently collects a vast amount of information, but unfortunately policy makers, universities, students and taxpayers continue to lack key information to enable them to make informed decisions.
Second, we need to improve, and even fix, current accountability processes, such as accreditation, to ensure that our colleges and universities are providing the highest quality education to their students.
Third, we need to do a much better job of aligning our resources to our broad societal needs. In order to remain competitive, our system of higher education must provide a world-class education that prepares students to compete in a global knowledge economy.
And finally, we need to assure that the American public understand through access to sufficient information, particularly in the area of student learning, what they are getting for their investment in a college education.
Commission Meeting (12/6/05)
At our Nashville meeting, the Commission heard three presentations from a panel on “Accountability.” Panelists represented the national, state and institutional perspectives and in the subsequent discussion, an informal consensus developed that there is a critical need for improved public information systems to measure and compare institutional performance and student learning in consumer-friendly formats, defining consumers broadly as students, families, taxpayers, policy makers and the general public.
Needs for a Modern University Education
The college education needed for the competitive, global environment in the future is far more than specific, factual knowledge; it is about capability and capacity to think and develop and continue to learn. An insightful quote from an educator describes the situation well:
“We are attempting to educate and prepare students (hire people in the workforce) today so that they are ready to solve future problems, not yet identified, using technologies not yet invented, based on scientific knowledge not yet discovered.”
--Professor Joseph Lagowski, University of Texas at Austin
Trends in Measuring Student Learning
There is gathering momentum for measuring through testing what students learn or what skills they acquire in college beyond a traditional certificate or degree.
Very recently, new testing instruments have been developed which measure an important set of skills to be acquired in college: critical thinking, analytic reasoning, problem solving, and written communications.
The Commission is reviewing promising new developments in the area of student testing, which indicate a significant improvement in measuring student learning and related institutional performance. Three independent efforts have shown promise:
A multi-year trial by the Rand Corporation, which included 122 higher education institutions, led to the development of a test measuring critical thinking, analytic reasoning and other skills. As a result of these efforts, a new entity called Collegiate Learning Assessment has been formed by researchers involved and the tests will now be further developed and marketed widely.
A new test measuring college level reading, mathematics, writing and critical thinking has been developed by the Educational Testing Service and will begin to be marketed in January 2006. This test is designed for colleges to assess their general education outcomes, so the results may be used to improve the quality of instruction and learning.
The National Center for Public Policy and Higher Education developed a new program of testing student learning in five states, which has provided highly promising results and which suggests expansion of such efforts would be clearly feasible.
An evaluation of these new testing regimes provides evidence of a significant advancement in measuring student learning -- especially in measuring the attainment of skills most needed in the future.
Furthermore, new educational delivery models are being created, such as the Western Governors University, which uses a variety of built-in assessment techniques to determine the achievement of certain skills being taught, rather than hours-in-a-seat. These new models are valid alternatives to the older models of teaching and learning and may well prove to be superior for some teaching and learning objectives in terms of cost effectiveness.
There are constructive examples of leadership in higher education in addressing the issues of accountability and student learning, such as the excellent work by the Association of American Colleges and Universities.
The AAC&U has developed a unique and significant approach to accountability and learning assessment, discussed in two recent reports, “Our Students’ Best Work” (2004) and “Liberal Education Outcomes” (2005).
The AAC&U accountability model focuses on undergraduate liberal arts education and emphasizes learning outcomes. The primary purpose is to engage campuses in identifying the core elements of a quality liberal arts education experience and measuring students’ experience in achieving these goals -- core learning and skills that anyone with a liberal arts degree should have. AAC&U specifically does not endorse a single standardized test, but acknowledges that testing can be a useful part of the multiple measures recommended in their framework.
In this model, departments and faculty are expected to be given the primary responsibility to define and assess the outcomes of the liberal arts education experience.
Federal and State Leadership
The federal government currently collects a great deal of information from the higher education system. It may be time to re-examine what the government collects to make sure that it’s useful and helpful to the consumers of the system.
Many states are developing relevant state systems of accountability in order to measure the performance of public higher education institutions. In its recommendations about accountability in higher education, the State Higher Education Executive Officers group has endorsed a focus on learning assessment.
Institutional Performance Measurement
What is clearly lacking is a nationwide system for comparative performance purposes, using standard formats. Private ranking systems, such as the U.S. News and World Report “Best American Colleges” publications, use a limited set of data, which is not necessarily relevant for measuring institutional performance or providing the public with information needed to make critical decisions.
The Commission, with assistance of its staff and other advisors and consultants, is attempting to develop the framework for a viable database to measure institutional performance in a consumer-friendly, flexible format.
Historically, accreditation has been the nationally mandated mechanism to improve institutional quality and assure a basic level of accountability in higher education.
Accreditation and related issues of articulation are in need of serious reform in the view of many, especially the need for more outcomes-based approaches. Also in need of substantial improvement are the regional variability in standards, the independence of accreditation, its usefulness for consumers, and its response to new forms of delivery such as internet-based distance learning.
The Commission is reviewing the various practices of institutional and programmatic accreditation. A preliminary analysis will be presented and various possible policy recommendations will be developed.