Assessment

RosEvaluation Conference 2010 Assessment for Program and Institutional Accreditation

Date: 
Fri, 04/09/2010 to Sat, 04/10/2010

Location

Terre Haute , Indiana
United States

You Say You Want a Revolution?

It seems everybody is talking revolution in higher ed these days.

How many times have I read in the higher ed news of the coming revolution in classroom instruction, in the major, in the tenure system, in governance?

Google "higher education revolution" and you find radical reform rising in every direction. Many are sparked by the billions state systems are losing as our economy lurches out of the tank, others by the increasing commodification of the college degree. Some promise to "transform" the American university as they have transformed -- egad! -- the American newspaper. New models of for-profit education promise a revolution in the higher education business model that is already threatening the viability of traditional colleges across the country.

But I can't help wondering if we've spirited all our revolutionary rhetoric for another day at the office.

We tend to talk ourselves right past revolutions in higher education. Our burning impulse to revitalize learning often concludes with a return to the status quo: we end up arguing, say, over our respective roles in shared governance, or over the turf we'd have to give up for genuine improvement in learning.

We can do better.

At a recent conference, I had a glimpse into how the real transformation might unfold. The Teagle Foundation brought together professors, administrators and researchers from across the country to discuss with its board members key questions the foundation has been addressing in recent years:

  • How might we make systematic improvements in student learning?
  • What evidence is there that we’re using what we know about student learning to reform academe?

These, of course, were the very same questions asked by the ill-fated Spellings Commission. Teagle has found success by engaging the strengths of the academy -- and especially the talents and creativity of faculty--by supporting liberal arts college in piloting solutions to the challenges before academe. In doing so, they have started transformative efforts that will deepen student learning while also balancing resources.

With the public university system in crisis -- Clark Kerr's master plan for California has been set adrift along with the strategies for renewal in state after state -- a focus on liberal arts colleges could seem to some like a boutique project. The Teagle Foundation's great insight has been that the nation's liberal arts colleges remain a bellwether for the health of the academy and that small colleges have a great opportunity to model what the 21st century higher education might become.

Teagle has funded dozens of collaborative efforts at liberal arts colleges over the past six years supporting faculty-driven, ground-up assessment projects of student learning outcomes at colleges and universities across the country.

The work that colleges are doing in these Teagle pilots tests the basic assumptions of a college education. Some have examined the meaning and value of general education, exploring radical revision of the ways in which general education might come to be embedded in helping students to think about the ways they will live their lives. One project brought four colleges together to assess how effectively undergraduate students acquire and refine the spiritual values that lie at the heart of their institutional missions. Another explores effective models of community-based learning efforts at three prominent colleges.

Such work aims to deepen student learning and growth at colleges across the country. As importantly, it will help small colleges to think about ways to distinguish themselves in a landscape that increasingly sees no difference between a liberal arts college degree and a degree from, say, the University of Phoenix. Liberal arts colleges must, to use Robert Zemsky's phrase, be "market-smart and mission-centered," and the pilots that Teagle has funded in recent years point us toward solutions to drifting missions and to struggling finances alike.

At Augustana College, we are taking seriously the Teagle Foundation's charge to find ways to use what we know about student learning for reform. Working in a Teagle-funded collaborative of seven colleges across the Midwest -- Alma, Augustana, Illinois Wesleyan, Luther, Gustavus Adolphus, Washington and Jefferson, and Wittenberg -- over the past five years, we have begun to question the 100-year-old credit model system that is at the heart of the American baccalaureate. Our consortium of colleges has begun to ask whether we can still justify the existence of a system that was brought into being mostly to serve the needs of our business offices.

Will federal pressure for transferability of credit only make more secure a system that is now straining under the weight of new understanding of learning and the new pedagogies that follow? In an era when we ask faculty to be deeply engaged with students through interdisciplinary education, undergraduate research, international study, and other high impact practices, can we continue to justify a credit system that has remained unchanged for a century? We are questioning whether the course unit as now constituted -- that three- or four-hour sliver of a college degree or the correlating seat time -- is the best means of measuring student learning.

My colleagues at Augustana and I have begun other pilots that will explore the other hard questions before our college, and all colleges: how will we make better use of vital resources while demonstrating the value of a liberal education to parents, employers, and graduate schools?

We have developed a series of experiments that may answer the question. Our faculty have created a senior capstone program -- Senior Inquiry -- by using a backward design model to re-envision nearly every major on campus, ensuring that all Augustana students will have the sort of hands-on, experiential learning opportunity that will demonstrate their skills to employers and graduate schools alike (even as it provides us with a great chance to evaluate all they have done in four years here). We have redefined scholarship in the Boyer model, embracing the scholarship of teaching and learning. We are piloting new partnerships with universities, community colleges and high schools; we are asking how technology might deepen the advantages of traditional classroom learning models. And we have built our newest program- - Augie Choice -- around the idea that experiential learning -- through research, international study and internships -- ought to be the heart of a liberal arts education.

We don't yet know where all of these experiments will lead us. But, in our 150th year at Augustana, we have learned from the Teagle Foundation that pilots may help us to ensure that we will thrive for the next 150 years.

That, I'm certain, is revolution enough.

Jeff Abernathy is vice president and dean of the college at Augustana College, in Illinois. This summer, he will become president of Alma College, in Michigan.

'Design Thinking' and Higher Education

As an advocate for the position that higher education benefits from studying the lessons of business and selectively implementing those ideas that help corporate and non-profit entities to prosper, I was pleased to come across Inside Higher Ed’s report on the publication of the multi-part work The Business of Higher Education (Praeger), edited by John C. Knapp and David J. Siegel. The author observed, correctly, that “many college and faculty leaders bristle at the suggestion that the institutions -- and their students -- would be better off if only institutions operated more like their counterparts in the private sector.”

That’s why I propose a model that may meet with the approval of those who think higher education is just fine ignoring business models: design thinking. For starters, it’s an idea with origins as remote from business as design itself. While their work is hardly nonprofit, designers are rarely found destroying the competition, maximizing profit margins and exploiting their employees. Few of the designers I know personally would fit the negative perception of corporate America held by many academicians. Design thinking is about helping people and organizations to solve their problems for long-term satisfaction, not achieving efficiency for short-run gains.

While it is true that more businesses are adopting design thinking as a model for achieving better results, enhanced innovation and improved service to customers, as evidenced by several new books about innovation design and design thinking targeted for the business market, the ideas behind design thinking emerged from the methods that are common to nearly all design fields, be it industrial, graphic, instructional or any other design profession. These basic operating principles constitute a process that might be expressed most simply as the way that designers approach problems and achieve solutions. Designers think of themselves as problem finders more so than problem solvers because their solutions start with a deep understanding of the problem requiring a solution.

What can design thinking offer to higher education? In a word, change. Not just change for the sake of creating change or trying the latest fad, but thoughtful change for the higher education institution that wants to position itself to better withstand the challenges presented by both old and new competitors. Change not just for technology’s sake, but change based on better understanding students and putting into a place a mechanism for institution-wide innovation. (I’ll provide some examples later.)

The seminal work on design thinking, The Art of Innovation (Currency/Doubleday, 2001) came from a business outsider, Tom Kelley, then general manager of IDEO, one of the world’s leading design firms. Those interested in learning more about design thinking are well advised to start with Kelley’s book, as it introduces the “IDEO Method,” a five-step approach to understanding how designers think. In a nutshell, the process requires its practitioners to internalize the following:

  • Understand: be an empathic thinker and put yourself in the shoes of your student or whomever it is that you provide a service to.
  • Observe: watch people in real-life situations to better understand how they really use a service or product and those things that both please and frustrate them.
  • Visualize: brainstorm with colleagues to identify new ideas and concepts that will give those you serve or teach a better (learning) experience.
  • Prototype: take time to explore multiple iterations of an idea before exposing those you serve or teach to a potential solution or enhancement.
  • Implementation and Evaluation: be thoughtful about when and how to implement a new idea and invest time to evaluate its impact, and then re-design as needed

For those who need a faster introduction to design thinking, take 22 minutes to watch "The Deep Dive," an episode of “Nightline” that profiled how the staff at IDEO tackle a new problem and develop a solution. As one learns from this video, the designers at IDEO are experts in using the design thinking process to identify and approach problems and then develop elegant solutions to them. That’s how IDEO has designed everything from the mouse you use nearly every day to NASA equipment to toothpaste dispensers and microwave ovens.

As the design thinking method gained popularity, IDEO added organizational consulting to its product design business, and now works with health care and K-12 education systems on restructuring and re-engineering workflows to eliminate dysfunctional practices and improve user experiences. One recent book about design thinking, Change by Design, by Tim Brown, CEO of IDEO, reads more like Zen philosophy than it does a how-to for businesspeople out to rule the world.

But even those who count themselves among higher education’s anti-business faction may benefit from another new book on design thinking authored by -- shudder -- a business school dean. The Design of Business, by Roger Martin, dean of the Rotman Business School at the University of Toronto, is a good example of a business book that even the most business-phobic humanist could enjoy reading. To be certain, there are a number of case studies profiling businesses that achieved success with design thinking, but there is still much food for thought for those who think their school or department could do better.

For example, Martin elegantly explains how businesses emerge and evolve, through a multi-stage process he describes as the “knowledge funnel.” It begins with a mystery in which the fledgling innovator seeks to build a better mousetrap, such as how to organize all the world’s information. The business creates a heuristic or an intuitive sense of how to solve the mystery that allows it to offer an initial product or service. As it moves out of the exploration stage, it develops an algorithm to operate the business so that the core solutions are efficiently exploited.

To illustrate this end stage of the knowledge funnel he points to companies like McDonald’s. What started as mostly a guess that Americans would eat fast food became a highly mechanized process that is easily replicated with great efficiency. McDonald’s has no interest in stimulating employee creativity or innovation; just keep the burgers and fries coming. But blindly adhering to algorithms can cost dearly when a competitor, like Subway, brings new thinking and imagination to the same mystery.

The problem, according to Martin, is that some organizations are operated primarily by intuition while others are rigidly controlled by algorithms. His core message is that organizations guided by design thinking achieve a balance between the two so that both intuition and algorithms merge to keep the organization searching for and solving new mysteries while avoiding the extreme exploitation that leads to obsolescence. When they “satisfice” for exploiting old ideas, businesses are ultimately confronted by upstart competitors exploring new mysteries. Thus emerge disruptive innovations offering products or services that better meet people’s needs.

It’s a cycle that endlessly repeats itself, and higher education is equally susceptible. Consider higher education’s long reign using the same delivery and organizational structures after hundreds of years. It now is pressured by new competitors, many of them for-profit businesses offering low-cost, convenient options that leverage advanced educational technologies. The new mystery is how to deliver higher education in ways that are both affordable and sustainable, and that meet the needs of a new generation of both traditional and nontraditional learners.

Are there ways in which design thinking could help America retain its place as the crown jewel of the world’s higher education system? Admittedly, higher education is a unique industry owing to the vast independence of its primary employees, the faculty. Each faculty member is in his or her own way an independent agent trusted with the responsibility to deliver learning to the students and pursue a highly individualized research agenda. Learning is not a McDonald’s hamburger that can be manufactured on demand guided by a scientific algorithm designed to assure a predetermined outcome. What faculty do in classrooms is largely guided by intuition; there is no algorithm for great teaching.

Despite the differences that distinguish colleges and universities from the corporations that Martin profiles in his book, design thinking is a potential solution by which higher education institutions could create the balance between intuitive and algorithmic methods. But to make that happen both faculty and administrators need to take a closer look at what design thinking can do for organizations.

Martin’s book offers a case study that provides a good example: the turnaround at Procter & Gamble. In 2000, when A.G. Lafley was appointed CEO, P&G had lost market leadership to newer competitors across a wide range of its consumer products. Like higher education in 2010, P&G’s expenses were soaring while Walmart and others introduced cheaper, lower quality private-label products that attracted consumers away from P&G’s more expensive branded goods. Lafley needed to boost innovation at P&G while simultaneously becoming more efficient; a blending of the intuitive and algorithmic sides of the organization.

In 2001 he appointed Claudia Kotchka to turn P&G into a design thinking organization. He invited outside designers to assist with the development of new products, a strategy not previously invoked at P&G, and by 2006 about 35 percent of P&G’s new products had origins outside the company. Perhaps the most critical change was obtaining a deeper understanding of the company’s consumers. Hair care product team members began to visit salons and homes to see how the products were actually used, and listened to the suggestions and complaints of consumers. Within three years of Lafley’s arrival P&G was achieving growth and recapturing market share in nearly every brand category.

The parallels between P&G at its weakest and the plight of many contemporary colleges and universities seem strong enough to suggest that design thinking is an idea worth considering. What might it look like to do so?

To follow the Procter & Gamble example, in higher education students endlessly evaluate courses, but what’s lacking is a committed effort to observe students as they learn and to then listen to their concerns. In a design thinking culture, faculty and administrators would empathically put themselves in the place of the students at their own institution and elsewhere to fully understand how to improve what happens in and beyond the classroom.

Consider the perplexing conundrum presented by scholarly publishing. Faculty members produce research that, in order to achieve tenure, they give away to journal publishers. Publishers, particularly in science, technology and medicine, edit and package faculty’s intellectual property and sell it back to higher education institutions at high prices that require constant increases to library budgets. Despite years of discussion about the scholarly communications crisis, we still have only partial and little used potential solutions.

The scholarly communications crisis presents what Martin would describe as a “wicked problem”, one that requires more than analytical or intuitive thinking. In his book The Opposable Mind, Martin states that when neither option A nor B works, design thinkers must create option C that blends A and B, offering a new and completely untested solution. Proposals to solve the scholarly communications crisis tend to fall into two camps: different pricing models and open access. The former attempts an analytical solution by transferring the existing system to a new price model so money changes hands differently. The latter attempts an intuitive solution by encouraging scholars to distribute their manuscripts through free (to the reader) distribution systems. Could design thinkers develop a C solution?

The design thinker would start by unraveling the real problem that fuels the crisis, which might well be the nature of the research and tenure system itself. Identifying the problem is paramount. The design thinker (or design team) would talk to all the parties involved and learn as much as possible about the scholarly communication process from the experts, both authors and publishers.

Next, the design team would bring back for analysis all the pieces of information, and process it in a brainstorming (“deep dive”) session. Out of the brainstorming would emerge prototypes for a new or modified system of scholarly communications. The design team would implement the prototypes deemed to have the most promise. The prototype that most closely approaches a “C” solution would emerge for implementation. That C solution might be some combination of a change in the tenure process and what counts as scholarship, a de-emphasis on publication in high-impact journals, an editing and publication process in which some publishers could participate, and options for self-publishing and archiving that are simpler, with clear benefits to faculty. In other words, some combination of existing practices and untested ideas that offers a completely new solution.

Design thinking is no panacea for all that ails higher education. Resolving challenges such as low retention and graduations rates, escalating textbook costs, an overdependence on adjunct faculty, lean budgets, for-profit competitors and myriad other problems will take more than a business as usual approach. However, higher education is not a business, and faculty and students will always respond caustically if they believe corporate solutions are being foisted on them by an unsympathetic administration.

This is where design thinking can make a difference. It’s more than a short-term strategy for boosting profits. It’s a roadmap for future-proofing one of society’s most valued resources. And since it involves no acquisition of or investment in sophisticated new technology, only a desire to try a new way of identifying and tackling institutional challenges, it’s right for the times. Those who want to engage with these ideas can begin with the Deep Dive video mentioned above or choose from a host of blogs written by experts in design thinking and user experience.

In 1972 Cohen, March and Olsen introduced the Garbage Can Theory of decision making as an effort to create a predictive model for how decisions are made in higher education organizations. The model describes colleges and universities as “organized anarchies” that make their decision by heaping multiple solutions into garbage cans. The detached solutions in the garbage present no utility until a problem presents itself to which one of the solutions could be attached. For too long the organized anarchy label has proven itself relatively accurate in describing what derails progress in higher education.

Design thinking, based on the premise of correctly identifying the problem before developing solutions, is as far removed from the garbage can theory as a decision making model can be. What is relatively the same since Cohen, March and Olsen devised their model is that higher education still confronts what Martin calls the “wicked problem,” a challenge that is not merely complex but is characterized by ambiguity, shifting qualities and no clear solution. Design thinking may be just what higher education needs to clean up its garbage can.

Steven Bell is associate university librarian at Temple University and co-author of the book Academic Librarianship by Design. He blogs at Designing Better Libraries and From the Bell Tower.

Accreditation 2.0

After years of dialogue, debate and deliberation, we are at the beginning of the next generation of accreditation. An “Accreditation 2.0” is emerging, one that reflects attention to calls for change while sustaining and even enhancing some of the central features of current accreditation operation.

The emerging consensus stems from three major national conversations, all focused on accreditation and accountability, all with roots in much older discussions and intensified in the face of the heightened national emphasis on access and attainment of quality higher education. Taken together, these conversations, despite their differences, provide the foundation for the future and a next iteration: Accreditation 2.0.

Three Conversations

The first major conversation is led by the academic and accreditation communities themselves. It focuses on how accreditation is addressing accountability, with particular emphasis on the relationship (some would say tension, or even conflict) between accountability and institutional improvement. The discussion frequently includes consideration of common expectations of general education across all institutions as well as the need to more fully address transparency. This conversation takes place at meetings of higher education associations and accrediting organizations and has been underway since the 1980s, when the assessment movement began.

The second conversation is led by critics of accreditation who question its effectiveness in addressing accountability, with some who even want to jettison the public policy role of accreditation as a gatekeeper or provider of access to federal funds. These critics often argue that conflicts of interest are inherent in accreditation as a result of peer review and the current funding and governance of the enterprise. The most recent version of this conversation was triggered by the 2005-6 Spellings Commission and continues today in various associations and think tanks.

The third conversation is led by federal officials who also focus on the gatekeeping role of accreditation. In contrast to the call in the second conversation to eliminate this function, attention here is on expanding use of the gatekeeping role of accreditation – to enforce expanding accountability expectations at the federal level.

Convergence

As different as the three conversations are, they reflect some shared assumptions or beliefs about quality in higher education and the role of accreditation. All acknowledge that accreditation provides value in assuring and improving quality, though views differ about how much value and in what way. All are based on a belief that accreditation needs to change, though in what way and at what pace is seen differently. All accept that accountability must be addressed in a more comprehensive and robust way – though they disagree about how to go about this.

The elements common to these conversations provide a foundation, an opportunity, for thinking about a next generation of accreditation or an “Accreditation 2.0.” They provide a basis to fashion the future of accreditation by strengthening accountability and enhancing service to the public while maintaining the benefits of quality improvement and peer review.

Some Thoughts About an Accreditation 2.0

The emerging Accreditation 2.0 is likely to be characterized by six key elements. Some are familiar features of accreditation; some are modifications of existing practice, some are new:

  • Community-driven, shared general education outcomes.
  • Common practices to address transparency.
  • Robust peer review.
  • Enhanced efficiency of quality improvement efforts.
  • Diversification of the ownership of accreditation.
  • Alternative financing models for accreditation.

Community-driven, shared general education outcomes are emerging from the work of institutions and faculty, whether through informal consortiums, higher education associations or other means of joining forces. The Essential Learning Outcomes of the Association of American Colleges and Universities, the Collegiate Learning Assessment and the Voluntary System of Accountability of the Association of Public and Land-grant Universities all provide for agreement across institutions about expected outcomes. This work is vital as we continue to address the crucial question of “What is a college education?” Accreditors, working in partnership with institutions, assure that these community-driven outcomes are in place and that evidence of student achievement is publicly available as well as used for improvement.

Common practices to address transparency in Accreditation 2.0 require that accredited institutions and programs routinely provide readily understandable information to the public about performance. This includes, for example, completion of educational goals, including graduation, success with transfer, and entry to graduate school. Second, accrediting organizations would provide information to the public about the reasons for the accredited status they award in the same readily understandable style, perhaps using an audit-like instrument such as a management letter. A number of institutions and accreditors already offer this transparency. Accreditation 2.0 would mean that it becomes standard practice.

Robust peer review -- colleagues reviewing colleagues -- is a major strength of current accreditation, not a weakness as some critics maintain. It is the difference between genuine quality review and bureaucratic scrutiny for compliance. Peer review serves as our most reliable source of independent and informed judgment about the intellectual development experience we call higher education. In the current environment, peer review can be further enhanced through, for example, encouraging greater diversity of teams, including more faculty and expanding public participation. As such, peer review has a prominent place in Accreditation 2.0, just as it plays a major role in government and other nongovernmental organizations in research, medicine and the sciences, among other fields.

Enhanced efficiency of quality improvement efforts builds on the enormous value of the “improvement” function in current accreditation. Improvement is about what an institution learns from its own internal review and the peer review team that prompts it to make changes to build on strengths or address perceived weaknesses. This is the dimension of accreditation to which institutions and programs most often point when speaking to the value of the enterprise.

However, for the limited number of institutions that are experiencing severe difficulties in meeting accreditation standards but remain “accredited” for a considerable number of years, there can be a downside for students and the public. Students enroll, but may have trouble graduating or meeting other educational goals because of weaknesses of the institution that were identified in the accreditation review, even as the institution is trying to improve and remedy these difficulties. Accreditation 2.0 can include means to assure more immediate institutional action to address the weaknesses and prevent their being sustained over long periods of time.

Diversification of the ownership of accreditation can provide for additional approaches to the process and even additional constructive competition, as well as provide a response to allegations of conflict of interest. At present, most accrediting organizations are either owned and operated by the institutions or programs they accredit or function as extensions of professional bodies. However, there is nothing to stop other parties interested in quality review of higher education from establishing accrediting organizations and obtaining the legal authority to operate. Accreditation 2.0 can encourage exploration of this diversification that can be a source of fresh thinking about sustaining and enhancing quality in higher education. Private foundations or nonprofit citizen groups, for example, can make excellent owners of accrediting organizations.

Alternative financing models for accreditation call for separating the reviews of individual institutions and programs from the financing of an accrediting organization. In Accreditation 1.0, most accreditors are funded through the fees they charge individual institutions and programs for their periodic accreditation review and for the annual operating costs of the accrediting organization – with the latter a condition of keeping accredited status. This mode of financing is viewed by some as an inappropriate enticement to expand the organization’s numbers of accredited institutions and programs and by others as a conflict of interest or disincentive to impose harsh penalties on institutions that might diminish membership numbers. It can create problems for some accreditors, especially smaller operations.

In Accreditation 2.0, an “accreditation bank” might be established by a third party, neither the accrediting organization nor the party seeking accreditation. Institutions and programs interested in investing in the accreditation enterprise would pay into the bank annually, independent of individual reviews. Alternative sources of financing include third parties such as private foundations and endowments.

*****

Accreditation 2.0 builds on the emerging consensus across the major national conversations about accreditation and accountability. It is one means to strengthen accreditation, but not at the price of some of Accreditation 1.0’s most valuable features. It keeps key academic decisions in the hands of institutions and faculty. It strengthens accountability, but through community-based decisions about common outcomes and transparency. It maintains the benefits of peer review, yet opens the door to alternative thinking about the organization, management and governance of accreditation.

Judith Eaton is president of the Council for Higher Education Accreditation, which is a national advocate for self-regulation of academic quality through accreditation. CHEA has 3,000 degree-granting colleges and universities as members and recognizes 59 institutional and programmatic accrediting organizations.

Institute for the Development of Excellence in Assessment Leadership (IDEAL)

Date: 
Mon, 08/02/2010 to Fri, 08/06/2010

Location

Baltimore , Maryland
United States

Webinar: Defining Learning Outcomes

Date: 
Wed, 03/10/2010

Location

Pages

Subscribe to RSS - Assessment
Back to Top