News
A Call for Assessment -- of the Right Kind
As the intensity of discussion and conflict about higher education accountability and assessment of student learning have grown in recent years, the Association of American Colleges and Universities has often sought to play the role of Switzerland. To the politicians and policy makers clamoring for colleges to prove their value, the liberal education group has cautioned against oversimplified measurements and the importance of institutional diversity.
But to college leaders who've advocated the rope-a-dope strategy, doing nothing while awaiting the passing of a fad or the end of a certain presidential administration, AACU has persistently prodded academic leaders to take seriously the need to look within and improve the teaching and learning that takes place in their classrooms.
As leaders of the group sought to say something now about where the accountability and assessment movement should go next, they found that a near-perfect statement of their views already existed -- and that they had made it five years ago.
Does the fact that the statement the AACU released Wednesday, "Our Students' Best Work: A Framework for Accountability Worthy of Our Mission," is based largely on a 2004 statement mean that nothing of note has happened in the intervening years? That there has been so little progress that the issues have not changed? That AACU was ahead of its time?
Perhaps all of the above. The initial statement came out at a time when state legislatures and the Bush administration, fresh off the imposition of the standardized-test-based No Child Left Behind law in elementary and secondary education, were raising the idea of taking a similar approach in higher education, Carol Geary Schneider, the association's president, said during a telephone news conference Wednesday. It was important at that time, she said, for the association to clearly state "that there was no way that standardized testing alone could provide a full picture" of the many kinds of learning colleges seek to impart to their students, and that was one of the 2004 report's two major themes, Schneider said.
But the other theme was just as important: making it clear that it would be a mistake for colleges to ignore the demand for more and better information about student outcomes, and that "too many institutions and programs still are unable to answer legitimate questions about what their students are learning in college."
Five years later, higher education has been through what Schneider called a "shallow discussion about assessment and accountability," prompted by the work of Education Secretary Margaret Spellings' Commission on the Future of Higher Education. That discussion left a divide at the extremes, she said, between some higher education leaders who believe college-level learning is too complex to be gauged and those who take a "send a number to somebody soon" approach, arguing that colleges must find some results from a nationally or at least widely comparable test (or tests) they can report publicly to satisfy the thirst for proof of their satisfactory performance.
AACU, in contrast, believes that colleges must "actually use assessment to improve student learning," and its re-released report, updated and reinforced in several ways, is meant to be a roadmap to help both the higher education establishment and individual colleges figure out how to do that. By arguing both for the idea that colleges must measure student learning and strongly against the use of "mass testing," which it argues would be an "enormous misuse of time and scarce resources," the AACU statement seeks to walk a middle path in the contentious debate over whether and how colleges should measure and report student learning.
The AACU Framework
Most of the statement will probably succeed in building a consensus. "The public has questions about the quality of education that colleges and universities are providing, and it deserves to know how well students are doing," the report says. "It is time for leaders of education to embrace a set of highly valued and widely affirmed educational goals, establish high standards for each, and assess their achievement across the curriculum."
That cannot be done, AACU says, through the use of "a particular kind of standardized testing -- multiple-choice, 'one-best-answer' tests" -- that advocates (and the testing industry) often champion, because "[a]ssessing what students have learned in colleges and universities requires a sophisticated understanding both of context and of how knowledge and skills are to be used.... Even if better tests continue to be developed, standardized tests alone are an inadequate and inappropriate strategy to foster advanced learning and accountability in higher education."
AACU's preferred approach, it writes, on which it (and other groups like the Teagle Foundation) have been working in recent years, is a several-step process in which (1) educators would define the "essential outcomes" that students should be expected to master from liberal education and the standards; (2) individual institutions would create a set of rubrics to establish goals and levels of performance for themselves and their students in achieving those outcomes, at the institutional, department and program levels; (3) colleges would build those goals and standards into their curriculums, particularly to be met by the time students do "culminating work" "that will both cultivate advanced knowledge and skill and demonstrate students' cumulative learning," most likely through the use of electronic portfolios of their academic work; and (4) create processes for making public the goals, performance expectations, and levels at which students performed.
It is on the last point -- the extent to which AACU envisions colleges using their student learning assessment as a way of proving their performance to their various publics -- that those who have been pushing for a more quantitative (and comparable) method of judging higher education are most likely to find fault with association's statement, which AACU officials more or less acknowledge.
"There are different purposes for assessment and AAC&U's focus is primarily on the 'improving learning and programs' and 'helping students be able to demonstrate their achievement for employment' purposes rather than the purpose of 'comparing institutions to make funding decisions or determine a 'return on investment' of public funds," Debra Humphreys, the association's vice president for communications and public affairs, said in an e-mail message. "We do, however, believe that it would be possible to use the methods we are recommending to create a public report that could go to a legislator and present a picture of aggregate accomplishment that a legislator could, indeed, use to compare to other institutions -- in the same way they could use data [from the National Survey of Student Engagement] to compare the existence of high-impact practices across institutions."
For instance, the report notes, a campus could, based on the standards it has set, determine that "75 percent of the students in the college of arts and sciences met a proficient standard for analytical skill and collaborative problem-solving;" it could then make that finding public along with information about how it collected the information and reached that conclusion.
"Like standardized testing, this method will allow for summarizing the outcomes of student learning with a few scores," the statement says. "But unlike tests based on quick responses to multiple-choice questions, these will be summaries of higher-order skills such as communication, analytic ability and integration of knowledge, and it will reflect meaningful educational projects judged by professionals."
Left Unsaid
The statement says nothing about two of the biggest developments that have occurred on the higher education assessment and accountability front since the 2004 statement: the emergence of the Collegiate Learning Assessment, in which small, representative groups of students typically complete a series of exercises to measure critical thinking, analytic thinking and written communication to measure how much an institution adds to their learning over the course of their careers there, and the creation of the Voluntary System of Accountability, which two groups of state universities have established to try to make their performance on a range of fronts more transparent to the public.
Despite the statement's silence, it implicitly criticizes both endeavors. While it is not the sort of "multiple-choice, 'one-best-answer' tests" directly panned in the AACU statement, the CLA is a standardized measure, in that it is designed to be used across institutions in ways that allow them to be compared. And by requiring participating colleges to use either the CLA or comparable standardized measures created by the Educational Testing Service and ACT, the Voluntary System of Accountability can be seen as an attempt to try to satisfy politicians' thirst for accountability -- to "send a number to somebody soon," as Schneider put it.
When asked about the VSA during the news conference, she went out of her way to praise the accountability system as "a good framework," to note that AACU is working with the two public university groups on a major Education Department grant, and to say that her colleagues at the associations "would be first to say that one test is not enough" to fully measure student learning.
David E. Shulenburger, vice president for academic affairs at the National Association of State Universities and Land-Grant Colleges, one of the VSA's sponsors, said in a brief e-mail message (sent while he was traveling abroad) that the groups encourage participants to "use a complete package of assessments," and thus "does not encourage universities to rely 'solely' on a single test." But he also noted, importantly, that "VSA's use of an outcomes test creates a rough comparability among participating universities and enhances public accountability" -- suggesting that the public university groups clearly see that as a more central priority than does the AACU statement.
The statement's emphasis on assessment for internal improvement purposes rather than external accountability purposes was also singled out as its biggest weakness by Charles Miller, the Texas businessman who headed the Spellings Commission and has argued the need for more higher education accountability.
Miller praised the AACU for continuing to raise these issues and credited its report for "accepting the idea that you can make measurements" of student learning. But he said he had trouble understanding from the statement how reams of term papers and other student work from electronic portfolios, much of which would be assessed subjectively, could be turned into the sort of "quantitative and comparable" evidence of student learning that would really serve the "public purposes" he sees.
"How are they going to tell us how well they did?" Miller said. "You may be able to use it internally for improving teaching and learning, but unless it helps everybody else -- the public, legislators, taxpayers -- it’s not going to answer the question. Defining the skills, setting the goals, measuring the performance -- that's all a step forward, for them. But without some kind of comparability and some kind of quantitative, specific results, it doesn't help the rest of us."
Miller said he was struck by the statement's repeated condemnation of standardized measures, given that colleges lean heavily on standardized tests to decide whom to enroll and whom to admit to graduate school. "They are used throughout the academic process for life and death decisions -- about students," he said. "But they refuse to consider the use of those kinds of tests to measure their own performance. Ironic."
