First they chopped off our hands. Figuratively, of course. The U.S. Department of Education, over the past decade, has made it clear that it expects accrediting agencies seeking recognition to judge student learning and institutional quality on the basis of rubrics, metrics and measured student learning outcomes. This, rather than on the basis of peer review -- the intense, direct interaction with students and faculty members that enabled site visitors to observe directly the nature, quality and level of learning taking place at an institution.
By now, there is wide recognition that relying on these proposed quantitative measures has weakened accreditation, with collateral damage. Thus, colleges that were focused on a financial bottom line rather than on student learning found it easy to produce numbers that satisfied quantitative guidelines, but said little or nothing about the learning taking place.
Sadly, the focus on such proxies remains. Graduate earnings, for example, are still being proposed as an indicator of quality. Low student debt and gainful employment are other proxies that say nothing about actual learning -- nothing about rigor, challenge, intellectual growth or any of the myriad outcomes that characterize the transformation we associate with a postsecondary education. Because these outcomes cannot be described quantitatively, they have been forced off the accreditation agenda.
Resistance proved fruitless. One small agency, facing the federal panel that recognizes accreditors in 2007, argued strenuously that its own intense focus on critical thinking and scholarship was more effective than the recording of quantitative measures. To no avail -- the agency’s recognition depended on compliance. Ever since, it has been requiring its accredited schools to gather irrelevant numerical outcome data in addition to its continued emphasis on qualitative outcomes.
So if accreditation has been falling down on the job, we now know why. Hopefully this will change with the forthcoming reauthorization of the Higher Education Act. Perhaps this paper -- and others like it -- will engender a new attitude in fashioning legislation and regulation. Accreditors will no longer have to be contortionists, and we should see a renewed emphasis on the searching and effective peer review.
The Peer Review -- A Primer
Peer review is based on site visitors who collectively have the experience and expertise to examine an institution comprehensively. A brief illustration of the thinking of a site visitor examining a college’s laboratory program will help explain peer review.
The visitor would probably think about the number of students working at a piece of equipment. If there are more than two, he will watch to see whether all are engaged -- building, adjusting, measuring, changing -- or whether one person is active and the others are simply taking notes.
The visitor will also note whether the lab instructor is sitting behind a desk or whether she is an involved part of the learning process -- helping, challenging, explaining and working with different groups of students throughout the period. The experienced peer reviewer will watch what happens near the end of the period. Will students hand in their assignments and leave -- or will they continue to work with the equipment to see what else they can learn?
These and other observations will contribute to the in-depth and comprehensive site visitors’ report, which will prove invaluable in enabling the accrediting agency to judge the quality of the institution and help its officials address deficiencies in an effort to improve.
On the other hand, an emphasis on quantitative metrics (amount of money spent on lab equipment, number of students served per day…) reveals little of the reality of a school or its lab.
Are There Alternatives to Traditional Accreditation?
Possibly, but those proposed are so far untested and unverified. Ideas that do have a “feet on the ground” validity somehow turn out to look like accreditation.
It’s helpful to visualize what an alternative strategy for the assessment of colleges and universities might look like. Clearly we would want to examine each institution against its own mission, and to do so we would need an understanding of the student demographics and the kind of learning and teaching taking place there. To be able to determine whether students are progressing successfully, we would certainly want to have a discipline expert in our group, and we would judge the institutional resources to ensure that a supportive educational atmosphere results. We would judge the faculty’s background to ensure that teachers have requisite content knowledge, and we would examine financial and other considerations that make an institution run smoothly.
We shouldn’t be surprised that this approach resembles accreditation. Accreditation has been honed by well over a century of practice at more than 6,000 institutions in the United States, and increasingly, throughout the rest of the world.
Licensure and certification pass rates have been proposed as indicators of quality for colleges offering occupational and vocational programs. But we must consider that if students learn only what is needed to pass a licensure exam, their preparation could be obsolete by the time they graduate. Everyone needs to learn how to learn, to adapt, to cope with changes in a given field, and colleges whose programs are limited to enabling students to pass licensure exams cannot be deemed quality. A far more intense examination is called for, and that’s of course the role of the peer review and accreditation.
Examining Proposed Modifications to Traditional Accreditation
(i) Should There Be More Public Member Participation?
There are public members who add immeasurably to accreditation decision-making meetings and just as many who do not. It’s also important to recognize that public members do not represent any special sector of the American people, bring no special expertise and offer no unique perspective. They are representatives from the public, not of the public.
Agencies have discovered that finding a public member prepared to serve, vetting him or her, gathering relevant documents, ensuring that the broad membership approves of this person, and training him or her constitutes a burden that sometimes is not commensurate with the benefit.
Until there is clear, experimentally validated evidence that additional public participation is warranted, no legislative or regulatory change to the current requirements should be made.
(ii) Should Accreditation Be More Transparent?
Accreditation is not a secretive, mysterious process, and on occasion legislators and journalists have been invited to participate in site visits to see for themselves. At a college preparing for a visit, dozens of people are involved. At the visit itself, dozens of others participate. This broad participation is not a recipe for secrecy and a lack of transparency.
Confidentiality, however, is important for decision making, because this enables institutions to be open and forthright with site visitors in describing their limitations. In fact this is a characteristic that distinguishes between accreditation and regulation. Because schools know their internal confidential documents and problems will not be widely published, they trust accreditors. This is why regulation rarely, if ever, leads to disclosure or improvement while accreditation does.
Transparency is neither disclosure nor exposure. Complete disclosure of, for example, all the part numbers of an automobile says nothing to the consumer about its functionality. Nor is transparency exposure. Releasing accreditation reports about institutions to the public ensures that schools will hesitate to discuss their limitations and challenges; it will occasion site visitors to write defensively and result in a distorted picture of an institution being presented to the public.
Transparency properly relates to the needs of the public and makes it possible for members of the public to receive the information that will make a difference, without at the same time compromising the accreditation process.
This is worth reiterating. The strength of accreditation lies in the fact that colleges are prepared to disclose, to be frank, to share limitations, to open the door to examination of all aspects of their activity, including the most embarrassing situations that might subject them to disfavor.
Accreditors are not regulators. Regulators discover only what they happen to see during their visit. Trusted accreditors discover everything.
(iii) Should There Be a Common Language?
Every step taken to unify different accrediting agencies is a step toward compromising the very diversity that makes American higher education so strong. Of particular concern is the fact that anything that is defined as common practice inevitably slides into becoming required practice. American higher education flourishes partly because there is competition; there are differences and there is an opportunity to examine the impact of different approaches in different situations.
We must ever be alert to the dangers of a unitary, “ministerial” form of higher education. Our diversity, competition and freedom to try new and different things should never be limited or compromised.
Alignment is not an overriding imperative, nor is comparability. Sometimes these considerations are helpful, but they should not be permitted to lead to a homogeneity that precludes responsible change and innovation in accreditation.
(iv) Should Accreditation Be More Proactive to Innovation?
Most so-called disruptive innovations are focused on inputs. MOOCs, experiential learning, competency-based education and technology of all kinds speak largely to different means of delivering education -- that is, to inputs.
Calling for innovation says little about quality or outcomes, and this is precisely why accreditation is so necessary in fostering innovation. It is only if an innovative approach is measured against standards, and performs well against a template of excellence, that we can agree that learning is taking place. Innovation, without the careful review associated with accreditation, risks harming students because of deficiencies that can emerge after graduation.
An innovation that does meet the standards of an appropriate accrediting body moves much more quickly and effectively into the mainstream than would otherwise be the case. In this respect accreditation can be seen to be fostering innovation.
(v) Should There Be Different Levels of Accreditation?
Perhaps. At the moment, standards operate to establish a floor, delineating the minimum quality necessary for institutional or program acceptability. It might be desirable to create gradations of accreditation status. Will such gradations make a difference? No one can tell, and careful experimentation is called for before any permanent changes are made.
Accreditation as Gatekeeper to Student Financial Aid
Accreditation as gatekeeper has become a target, and it’s important to outline the reasons that keeping this feature is extremely important to a well-functioning postsecondary educational system.
(i) Maintaining Order and Fostering a Culture of Improvement
The gatekeeping function helps create a sense of order in American higher education. All reputable institutions are recognized by agencies that, in turn, satisfy the regulations of the Department of Education and the standards of the Council for Higher Education Accreditation.
There are problems with this structure, but because of the role of Congress, the tendency of the Department of Education to overreach can be checked. Every reauthorization of the Higher Education Act has helped remind the department that it serves education; it does not guide or control it.
Now visualize a situation in which student financial aid would flow to colleges and universities whether or not they are accredited.
Since they would remain eligible for aid without accreditation, elite institutions with strong, well-established reputations will not find it necessary to seek to be accredited. Some might join together in an exclusive club, but there is not likely to be a pretense of seeking to improve or to adhere to any external standards. Such schools will continue to be able to attract excellent students and donor support on the strength of their reputations alone.
The very lowest-performing colleges in the country are not likely to seek accreditation, either. At best, some will gather to create an accrediting body with indifferent standards, as a marketing feature. Lost in all of this will be the culture of improvement that is integral to accreditation. Schools do respond to site visit recommendations, and if site visit reports are not as incisive and comprehensive as they used to be, they are still effective in charting a path for improvement.
(ii) Diploma Mills Will Proliferate
As noted, the gatekeeper function ensures that all reputable schools seek accreditation from agencies that have Education Department recognition. Indeed, the department maintains a list online of all schools accredited by recognized agencies. Diploma mills are not on this list, nor are they eligible for student financial aid. If accreditors are no longer gatekeepers, many legitimate schools will not seek accreditation yet will receive financial aid. So will diploma mills. People who create diploma mills are extremely resourceful, and will be able to meet any paper requirements needed to become eligible for SFA funds. It’s only accreditation as gatekeeper under the Education Department that prevents them from doing so now.
Further, without the department's recognition function, there will be nothing to prevent a dishonest operator from creating an accrediting body (such as the “Northeast and Southwest Association of Colleges and Schools”), listing several hundred legitimate schools as being recognized by this association, and adding a number of diploma mills to its list.
True, this fraudulent accrediting body would not be admitted to the Council for Higher Education Accreditation, but the unsophisticated and unsuspecting enrollee in a diploma mill would not know the difference. And if necessary, the same printing press that created schools out of air, and accrediting agencies out of fairy dust, can readily create a CHEA-like body as well.
(iii) Transfers Will Be More Difficult
Colleges hesitate to accept transfers from community colleges and weaker institutions without reassurance from an external body. As difficult as it is now for students to transfer credits, it is still possible to move from a weak accredited school to a stronger one. In the future, with no recognized accreditor to offer lose control over their practices, colleges will be free to institute arbitrary and unduly restrictive transfer policies.
(iv) Innovation Will Be Stifled
As noted above, innovation makes sense only if it is responsible and consistent with the standards of higher education. It is useful to students only if their efforts in an innovative mode are subsequently recognized by conventional schools and employers. Accrediting agencies, even at the risk of being accused of being obstructionist, have insisted that innovation not result in any diminution in the quality of education.
Innovators seek accreditation, because once approved, a changed structure, strategy or delivery mechanism can be assured of acceptability and wide implementation. Without a broad-based, unified accreditation structure, tied together by a gatekeeping function, acceptance of innovation will become more difficult.
(v) Public Protection
There are many emerging areas of public activity, particularly in the field of health services, where there is no government oversight and much potential for public harm. Access to financial aid encourages accreditors to seek federal recognition. This is why responsible educators and practitioners in a given field organize themselves around an accrediting structure that will meet Education Department standards. This helps create a level of assurance of quality, which serves the public.
The advent of the Internet and the ease with which false and misleading structures can be created makes it important to keep accreditation intact, particularly since there have been no structured experiments and pilot programs to examine proposed alternatives. Conversation is healthy, but nothing should be done to dismantle the existing gatekeeping/accrediting relationship until it is clear that a replacement is at least as good as the original.
Bernard Fryshman is a professor of physics and former accreditor.
With all the extensive consultation about the Postsecondary Institutions Ratings System during the past 18 months, all the meetings and the many conversations, we know almost nothing about its likely impact on accreditation, our all-important effort by colleges, universities and accrediting organizations working together to define, judge and improve academic quality.
All that the U.S. Department of Education has officially said to date is that the system will “help inform” accreditation -- and we do not know what this means.
This is worrisome. Ratings create, in essence, a federal system of quality review of higher education, with the potential to upend the longstanding tradition of nongovernmental accreditation that has carried out this role for more than 100 years. And establishing the system may mean the end of more than 60 years of accreditation as a partner with government, the reliable authority on educational quality to which Congress and the Education Department have turned.
Accreditation is about judgment of academic quality in the hands of faculty members and academic administrators. It is about the commitment to peer review -- academics reviewing academics yet accountable to the public -- as the preferred, most effective mode of determining quality. It is about leadership for academic judgment when it comes to such factors as curriculum, programs, standards and strategic direction remaining in the hands of the academic community.
In contrast, a ratings system is a path to a government model of quality review in place of the current model of academics as the primary judges of quality.
First introduced by President Obama in August 2013 and turned over to the Education Department for development, the ratings system is on track for implementation in 2015-16. Based on the still incomplete information the department has released to the public, the system is intended to rate (read: judge) colleges and universities based on three indicators: access, affordability and student outcomes. Institutions will be considered either “high performing,” “low performing” or “those in the middle.” Ultimately, the amount of federal student aid funding a college or university receives is intended to be linked to its rating.
A federal ratings system is both an existential and political challenge to accreditation.
First, there is the challenge of a potential shift of ownership of quality. Second, new key actors in judging quality may be emerging. Finally, the relationship between accreditation and the federal government when it comes to quality may be shifting, raising questions about both the gatekeeping role of accreditation in eligibility for federal funds and the agreement about distribution of responsibilities among the parties in the triad -- the federal government, the states and accreditation.
A ratings system means that government owns quality through its indicators and its decisions about what counts as success in meeting the indicators. The indicators replace peer review.
It means that government officials are key actors in judging quality. Officials replace academics. With all respect to the talent and commitment of these officials, they are not hired for their expertise in teaching and learning, developing higher education curriculum, setting academic standards, or conducting academic research. Yet using a ratings system calls for just these skills.
A ratings system means that the relationship between accreditors and the federal government, with the accreditors as dominant with regard to quality judgments, may give way to a lesser role for accreditation, perhaps using performance on the ratings system as a key determinant of eligibility for federal funds -- in addition to accreditation. Or, it is not difficult to envision a scenario in which ratings replace accreditation entirely with regard to institutional eligibility for access to federal financial aid.
We need to know more about what we do not know about the ratings system. Going forward, we will benefit from keeping the following questions in mind as the system -- and its impact on accreditation -- continues to develop.
First, there are questions about the big picture of the ratings system:
Has a decision been made that the United States, with the single most distinctive system of a government-private sector partnership that maximizes the responsible independence of higher education, is now shifting to the model of government dominance of higher education that typifies most of the rest of the world?
What reliable information will be available to students and the public through the ratings system that they do not currently have? Will this information be about academic quality, including effective teaching and learning? What is the added value?
Second, there are questions about the impact of the ratings on accredited institutions:
Are the indicators to serve as the future quality profile of a college or university? Will the three indicators that the system uses -- access, affordability and outcomes -- become the baseline for judging academic quality in the future?
Will it be up to government to decide what counts as success with regard to the outcomes indicators for a college or university -- graduation, transfer of credit, entry to graduate school and earnings?
To claim quality, will colleges and universities have to not only provide information about their accredited status, but also their ratings, whether “high performing,” “low performing” or “in the middle”?
Will institutions be pushed to diminish their investment in accreditation if, ultimately, it is the ratings that matter -- in place of accreditation?
Finally, there are questions about how ratings will affect the day-to-day operation of accrediting organizations and their relationship to the federal government:
Will accreditors be required to collect/use/take into account the information generated by the ratings system? If so, how is this to influence their decisions about institutions and programs that are currently based on peer review, not ratings?
Will performance on the ratings system be joined with formal actions of accrediting organizations, with both required for accredited status and thus eligibility of institutions for federal funds -- in contrast to the current system of reliance on the formal actions of accrediting organizations?
How, if at all, will the ratings system affect the periodic federal review of the 52 accrediting organizations that are currently federally recognized? Will the government review now include the ratings of institutions as part of examination and judgment of an accreditor’s effectiveness?
While we cannot answer many of these questions at this time, we can use them to anticipate what may take place in the approaching reauthorization of the Higher Education Act, with bills expected in spring or summer.
We can use them to identify key developments in the ratings that have the potential to interfere with our efforts to retain peer review and nongovernmental quality review in preference to the ratings system.
Judith S. Eaton is president of the Council for Higher Education Accreditation.
Today, leaders of colleges and universities across the board, regardless of size or focus, are struggling to meaningfully demonstrate the true value of their institution for students, educators and the greater community because they can't really prove that students are learning.
Most are utilizing some type of evaluation or assessment mechanism to keep “the powers that be” happy through earnest narratives about goals and findings, interspersed with high-level data tables and colorful bar charts. However, this is not scientific, campuswide assessment of student learning outcomes aimed at the valid measure of competency.
The "Grim March" & the Meaning of Assessment
Campuswide assessment efforts rarely involve the rigorous, scientific inquiry about actual student learning that is aligned from program to program and across general education. Instead, year after year, the accreditation march has trudged grimly on, its participants working hard to produce a plausible picture of high “satisfaction” for the whole, very expensive endeavor.
For the past 20-plus years, the primary source of evidence for a positive impact of instruction has come from tools like course evaluation surveys. Institutional research personnel have diligently combined, crunched and correlated this data with other mostly indirect measures such as retention, enrollment and grade point averages.
Attempts are made to produce triangulation with samplings of alumni and employer opinions about the success of first-time hires. All of this is called “institutional assessment,” but this doesn’t produce statistical evidence from direct measurement that empirically demonstrates that the university is directly responsible for the students’ skill sets based on instruction at the institution. Research measurement methods like Chi-Square or Inter-rater reliability combined with a willingness to assess across the institution can demonstrably prove that a change in student learning is statistically significant over time and is the result of soundly delivered curriculum. This is the kind of “assessment” the world at large wants to know about.
The public is not satisfied with inferentially derived evidence. Given the cost, they yearn to know if their sons and daughters are getting better at things that matter to their long-term success. Employers routinely stoke this fire by expressing doubt about the out-of-the-box skills of graduates.
Who Owns Change Management
Whose responsibility is it to redirect the march to provide irrefutable reports that higher education is meeting the needs of all its stakeholders? Accreditors now wring their hands and pronounce that reliance on indirect measures will no longer suffice. They punish schools with orders to fix the shortfalls in the assessment of outcomes and dole out paltry five-year passes until the next audit. They will not, however, provide sound, directive steps for the marchers about how to systematically address learning outcomes.
How about the government? The specter of more third-party testing is this group’s usual response. They did it to K-12 and it has not worked there either. Few would be happy with that center of responsibility.
Back to the campus. To be fair, IR or offices of institutional effectiveness have been reluctant to get involved with direct measures of student performance for good reasons. Culture dictates that such measures belong to program leaders and faculty. The traditions and rules of “academic freedom” somehow demand this. The problem is that faculty and program leaders are indeed content experts, but they are no more versed in effective assessment of student outcomes than anyone else on campus.
This leaves us with campus leaders who have long suspected something is very wrong or at least misdirected. To paraphrase one highly placed academic officer, “We survey our students and a lot of other people and I’m told that our students are ‘happy.’ I just can’t find anyone who can tell me for sure if they’re ‘happy-smarter’ or not!” Their immersion in the compliance march does not give them much clue about what to do about the dissonance they are feeling.
The Assessment Renaissance
Still, the intelligent money is on higher ed presidents first and foremost, supported by their provosts and other chief academic officers. If there is to be deep change in the current culture they are the only ones with the proximal power to make it happen. The majority of their number has declared that “disruption” in higher education is now essential.
Leaders looking to eradicate the walking dead assessment march in a systematic way need to:
Disrupt. This requires a college or university leader to see beyond the horizon and ultimately have an understanding of the long-term objective. It doesn’t mean they need to have all the ideas or proper procedures, but they must have the vision to be a leader and a disrupter. They must demand change on a realistic, but short timetable.
Get Expertise. Outcomes/competency-based assessment has been a busy field of study over the past half-decade. Staff development and helping hands from outside the campus are needed.
Rally the Movers and Shakers. In almost every industry, there are other leaders without ascribed power but whose drive is undeniable. They are the innovators and the early adopters. Enlist them as co-disruptors. On campuses there are faculty/staff that will be willing to take risks for the greater good of assessment and challenge the very fabric of institutional assessment. Gather them together and give them the resources, the authority and the latitude to get the job done. Defend them. Cheerlead at every opportunity.
Change the Equation. Change the conversation from GPAs and satisfaction surveys to one essential unified goal: are students really learning and how can a permanent change in behavior be measurably demonstrated?
Rethink your accreditation assessment software. Most accreditation software systems rely on processes that are narrative, not a systematic inquiry via data. Universities are full of people who research for a living. Give them tools (yes, like Chalk & Wire, which my company provides) to investigate learning and thereby rebuild a systematic approach to improve competency.
Find the Carrots. Assume a faculty member in engineering is going to publish. Would a research-based study about teaching and learning in their field stand for lead rank and tenure? If disruption is the goal, then the correct answer is yes.
Assessment is complex, but it’s not complicated. Stop the grim march. Stand still for a time. Think about learning and what assessment really means and then pick a new proactive direction to travel with colleagues.