assessmentaccountability

Research Project on Performance-Based Funding

An ongoing study conducted by Research for Action, a Philadelphia-based nonprofit research organization, looked at the effect of performance-based funding policies in higher education across three states: Indiana, Ohio and Tennessee. The group released early results from the work over the weekend at the annual meeting of the American Education Research Association.

The project takes into account key differences in the type of policies as well as variations in state funding that were tied to them. Initial findings showed consistent positive effects on the numbers of bachelor's degrees awarded under the policies. But the study did not find evidence of a positive effect on graduation rates.

Series of studies seeks to gauge higher ed effectiveness, defined broadly

Smart Title: 

New volume of research examines various aspects of higher education performance, going well beyond labor market outcomes to include academic quality and socioeconomic equity.

Essay on how fixation on 'inane' student learning outcomes fails to ensure academic quality

In a recent Century Foundation essay, I raised a concern that accreditors of traditional colleges are allowing low-quality education to go unaddressed while insisting, in a misguided attempt to prove they care about learning, that colleges engage in inane counting exercises involving meaningless phantom creatures they call student learning outcomes, or SLOs.

The approach to quality assurance I recommend, instead, is to focus not on artificially created measures but on the actual outputs from students -- the papers, tests and presentations professors have deemed adequate for students to deserve a degree.

I got a lot of positive feedback on the essay, especially, as it happens, from people involved in some of the processes I was criticizing. Peter Ewell, for example, acknowledged in an email that “the linear and somewhat mindless implementation of SLOs on the part of many accreditors is not doing anybody any good.”

This story began in the 1990s, when reformers thought they could improve teaching and learning in college if they insisted that colleges declare their specific “learning goals,” with instructors defining “the knowledge, intellectual skills, competencies and attitudes that each student is expected to gain.” The reformers’ theory was that these faculty-enumerated learning objectives would serve as the hooks that would then be used by administrators to initiate reviews of actual student work, the key to improving teaching.

That was the idea. But it hasn’t worked out that way. Not even close. Here is one example of how the mindless implementation of this idea distracts rather than contributes to the goal of improved student learning. When a team from the western accreditor, the WASC Senior College and University Commission, visited San Diego State University in 2005, it raised concerns that the school had shut down its review process of college majors, which was supposed to involve outside experts and the review of student work. Now, 10 years have passed and the most recent review by WASC (the team visit is scheduled for this month) finds there are still major gaps, with “much work to be done to ensure that all programs are fully participating in the assessment process.”

What has San Diego State been doing instead of repairing its program review process? It has been writing all of its meaningless student learning outcome blurbs that accreditors implemented largely in response to the Spellings Commission in 2006. San Diego State reported its progress in that regard in a self-review it delivered to WASC last year:

Course Learning Outcomes (CLOs) are required for all syllabi; curricular maps relating Degree Learning Outcomes (DLOs) to major required courses are now a required component for Academic Program Review; programs are being actively encouraged to share their DLOs with students and align DLOs with CLOs to provide a broader programmatic context for student and to identify/facilitate course-embedded program assessment.

All this SLO-CLO-DLO gibberish and the insane curriculum map database (really crazy, take a look) is counterproductive, giving faculty members ample ammunition for dismissing the idiocy of the whole process. The insulting reduction of learning to brief blurbs, using a bizarre system of verb-choice rules, prevents rather than leads to the type of quality assurance that has student work at the center.

The benefits of, instead, starting with student work as the unit of analysis is that it respects the unlimited variety of ways that colleges, instructors and students alike, arriving with different skill levels, engage in the curriculum.

Validating colleges’ own quality-assurance systems should become the core of what accreditors do if they want to serve as a gateway to federal funds. Think of it as an outside audit of the university’s academic accounting system.

With this approach, colleges are responsible for establishing their own systems for the occasional review of their majors and courses by outside experts they identify. Accreditors, meanwhile, have the responsibility of auditing those campus review processes, to make sure that they are comprehensive and valid, involving truly independent outsiders and the examination of student work.

SLO madness has to stop. If accreditors instead focus on the traditional program-review processes, assuring that both program reviews and audits include elements of random selection, no corner of the university can presume to be immune from scrutiny.

Robert Shireman is a senior fellow at the Century Foundation and a former official at the U.S. Department of Education.

Essay on value of student learning outcomes in measuring and ensuring academic quality

Robert Shireman is right. The former official at the U.S. Department of Education correctly wrote recently that there is little evidence that using accreditation to compel institutions to publicly state their desired student learning outcomes (SLOs), coupled with the rigid and frequently ritualistic ways in which many accreditation teams now apply these requirements, has done much to improve the quality of teaching and learning in this country.

But the answer, surely, is not to abolish such statements. It is to use them as they were intended -- as a way to articulate collective faculty intent about the desired impact of curricula and instruction. For example, more than 600 colleges and universities have used the Degree Qualifications Profile (DQP). Based on my firsthand experience with dozens of them as one of the four authors of the DQP, their faculties do not find the DQP proficiency statements to be “brief blurbs” that give them “an excuse to dismiss the process,” as Shireman wrote. Instead, they are using these statements to guide a systematic review of their program offerings, to determine where additional attention is needed to make sure students are achieving the intended skills and dispositions, and to make changes that will help students do so.

As another example, the Accreditation Board for Engineering Technology (ABET) established a set of expectations for engineering programs that have guided the development of both curricula and accreditation criteria since 2000. Granted, SLOs are easier to establish and use in professional fields than they are in the liberal arts. Nevertheless, a 10-year retrospective study, published about two years ago, provided persuasive empirical evidence that engineering graduates were achieving the intended outcomes and that these outcomes have been supported and used by faculties in engineering worldwide.

Shireman also is on point about the most effective way to examine undergraduate quality: looking at actual student work. But what planet has he been living on to not recognize that this method isn’t already in widespread use? Results of multiple studies by the National Institute for Learning Outcomes Assessment (NILOA) and the Association of American Colleges and Universities (AAC&U) indicate that this is how most institutions look at academic quality -- far exceeding the numbers that use standardized tests, surveys or other methods. Indeed, faculty by and large already agree that the best way to judge the quality of student work is to use a common scoring guide or rubric to determine how well students have attained the intended proficiency. Essential to this task is to set forth unambiguous learning outcomes statements. There is simply no other way to do it.

As an example of the efficacy of starting with actual student work, 69 institutions in nine states last year looked at written communications, quantitative fluency and critical thinking based on almost 9,000 pieces of student work scored by faculty using AAC&U’s VALUE rubrics. This was done as part of an ongoing project called the Multi-State Collaborative (MSC) undertaken by AAC&U and the State Higher Education Executive Officers (SHEEO). The project is scaling up this year to 12 states and more than 100 institutions. It’s a good example of how careful multi-institutional efforts to assess learning using student work as evidence can pay considerable dividends. And this is just one of hundreds of individual campus efforts that use student work as the basis for determining academic quality, as documented by NILOA.

One place where the SLO movement did go off the rails, though, was allowing SLOs to be so closely identified with assessment. When the assessment bandwagon really caught on with accreditors in the mid-1990s, it required institutions and programs to establish SLOs solely for the purpose of constructing assessments. These statements otherwise weren’t connected to anything. So it was no wonder that they were ignored by faculty who saw no link with their everyday tasks in the classroom. The hundreds of DQP projects catalogued by NILOA are quite different in this respect, because all of them are rooted closely in curriculum or course design, implementing new approaches to teaching or creating settings for developing particular proficiencies entirely outside the classroom. This is why real faculty members in actual institutions remain excited about them.

At the same time, accreditors can vastly improve how they communicate and work with institutions about SLOs and assessment processes. To begin with, it would help a lot if they adopted more common language. As it stands, they use different terms to refer to the same things and tend to resist reference to external frameworks like the DQP or AAC&U’s Essential Learning Outcomes. As Shireman maintains, and as I have argued for decades, they also could focus their efforts much more deliberately on auditing actual teaching and learning processes -- a common practice in the quality assurance approaches of other nations. Indeed, starting with examples of what is considered acceptable-quality student work can lead directly to an audit approach.

Most important, accreditors need to carefully monitor what they say to institutions about these matters and the consistency with which visiting teams “walk the talk” about the centrality of teaching and learning. Based on volunteer labor and seriously undercapitalized, U.S. accreditation faces real challenges in this arena. The result is that institutions hear different things from different people and constantly try to second-guess “what the accreditors really want.” This compliance mentality is extremely counterproductive and accreditors themselves are only partially responsible for it. Instead, as my NILOA colleagues and I argue in our recent book, Using Evidence of Student Learning to Improve Higher Education, faculty members and institutional leaders need to engage in assessment primarily for purposes of improving their own teaching and learning practices. If they get that right, success with actors like regional accreditors will automatically follow.

So let’s take a step back and ponder whether we can realistically improve the quality of student learning without first clearly articulating what students should know and be able to do as result of their postsecondary experience. Such learning outcomes statements are essential to evaluating student attainment and are equally important in aligning curricula and pedagogy.

Can we do better about how we talk about and use SLOs? Absolutely. But abandoning them would be a serious mistake.

Peter Ewell is president of the National Center for Higher Education Management Systems (NCHEMS), a research and development center.

Editorial Tags: 

Parents and students pay a high price for college remediation, study finds

Smart Title: 

An inadequate high school education can get expensive for students when they need to take remedial courses in college, according to a new report.

Education Department Drafts Standards for Accreditors

The Obama administration is proposing new standards that govern how and when college accreditors have to alert the U.S. Department of Education about troubled institutions under the accreditors’ purview.

The department is soliciting public comments on a letter it plans to send to accreditors that will outline the circumstances under which they must notify federal officials of actions they take against a college.

Federally recognized college accrediting agencies are already required to provide certain information to the Education Department. As part of the administration’s executive actions on accreditation, the Education Department is now looking to standardize how that process works.

The letter outlines, for instance, uniform definitions for the various terms that accreditors use to describe similar types problems at a college (such as “denial” of accreditation or “suspension” of accreditation). Under the new policy, accreditors would also have to triage information they send to the department by level of severity to help federal regulators more quickly sort out serious problems from more routine changes in a college’s accreditation status.

As the department collects more robust information from accreditors about troubled institutions, it also plans to make such information publicly available, according to the letter. The department will accept public comments until June 6.

Paper on Standardized Assessments of Learning

A new report from New America, a think tank, looks at research on the assessment of college learning. The paper by Fredrik DeBoer, a lecturer at Purdue University, tracks the overall push for more assessment and data collection in higher education. It also describes the resulting backlash.

"Effective assessment of student learning in any context represents a significant challenge," the paper said, "and controversies persist at all levels of education about which methods of data collection and analysis are most effective and appropriate."

DeBoer argues that faculty members and local college administrators must be welcomed into the assessment process. He also writes that standardized assessments from testing firms should be subject to external validation.

"Researchers must vet these instruments to determine how well they work, and what the potential unforeseen consequences are of these types of assessments, for the good of all involved," writes DeBoer.

The challenges and tensions of being a board member in difficult times (essay)

In the two short months since the New Year, headlines about college and university boards and governance have abounded. While the headlines paint one picture, those of us who try and keep a thoughtful eye on governance also read the comments sections of the stories reported here and elsewhere. Some of those comments are well formulated and advance the conversation about good governance; others are misinformed or just nasty.

Look at the comments on Inside Higher Ed about the board-president tension at Suffolk University, the actions and inactions at Mount St. Mary’s University, and what is happening at the University of Missouri. While the stories give renewed attention to the power and role of governance and call out some of the tensions, the comments suggest that much more understanding is needed about the role and function of lay boards of trustees, part of our historical structure since America’s first colonial college.

Governance can be arduous, as we will explain. Being a trustee is one difficult volunteer role, and boards often find themselves in “damned if you do, damned if you don’t” situations. In the midst of the vitriol and mudslinging that occurs, our intent is not to play the metaphorical violin and feel sorry for trustees, but we do want to point out a few harsh realities. Boards today must cope with:

Responding to tricky, if not impossible, decisions thrust upon them. Boards do not get to choose the issues that come before them. Some problems and quandaries that end up on board agendas are conceivable -- student concerns (and in some places protests) regarding race and equity are a recent example. Yet, even if foreseeable, the board never knows how an issue will play out in real time, and there often are no right answers or simple paths forward.

Other issues simply cannot be predicted. It is not as if the boards in Louisiana planned for the $940 million budget deficit, or the Mount Saint Mary’s board anticipated the accreditation issues or other fallout from its president’s remarks, including picking up the pieces from his resignation, or the University of Missouri’s Board of Curators wanted to deal with the actions of a faculty member, acting on her own accord and captured on video, that resulted in criminal charges being brought against her and decisions about her employment.

Balancing both immediate and long-term concerns. Taking the long view sometimes seems nearly impossible in a culture obsessed with speed and desirous of instant gratification, especially on college campuses, which have: 1) students who are there for only a short time (in the scheme of things) and are not timid about issuing immediate demands, 2) four generations of faculty with competing interests and who have seen their jobs change a great deal over the years, and 3) administrators who must grapple with crises on a nearly daily basis. Boards find themselves at once caught up in the demands of the immediate while needing to never lose sight of the long term. But, as Harvard sociologist David Reisman said, the role of boards is to “protect the future from the present.”

Boards have three basic, yet sometimes difficult, responsibilities that require them to balance the needs of today with those of future generations:

  1. Duty of care: competence and diligence, the care that an ordinarily prudent person would exercise in a like position and under similar circumstances. Board members have the duty to exercise reasonable care when making decisions as stewards of the institution; they are expected to actively participate in organizational planning and decision making and to make sound and informed judgments.
  2. Duty of loyalty: allegiance. Board members must give undivided allegiance when making decisions. Board members may not use information obtained as a member for personal gain and must act in the best interests of the institution. When acting on behalf of the institution, board members must put institutional interests before any personal or professional concerns and avoid potential conflicts of interest.
  3. Duty of obedience: staying true to mission. Board members are not permitted to act in a way that is inconsistent with the central goals -- the mission -- of the institution. A basis for this rule lies in the public’s trust that the institution will oversee assets (financial, physical and otherwise) to fulfill its mission. Board members must ensure that the institution complies with all applicable federal, state and local laws and regulations.

While it is hoped that all individual members of the campus community would act in such ways, no other group has the same legal and ethical requirement to do so. Board members are not employees and have a fundamentally different relationship with the college, university or state system.

Deciding in the spotlight. Boards are governing in difficult times of heightened scrutiny; in fact, public boards need to govern in front of the public. Imagine trying to having a thoughtful, candid, difficult conversation about controversial issues with your spouse or entire family surrounded by invested onlookers -- and then covered by the press to boot. This is the experience of public boards.

For instance, we would bet that most, if not all, boards should be seriously grappling with how to best deal with the long-term financial health of their institutions. Even well-endowed institutions face economic issues. In January 2013, Moody’s downgraded the entire American higher education sector to negative, an outlook continued in 2014 and 2015. As one trustee put it, “We have a relatively undercapitalized institution, offering a largely undifferentiated product, into an increasingly price-sensitive market, characterized by declining demographics.” Who would be bullish on that? How can boards explore complex and contentious issues, engage in dialogue, and ponder and wonder out loud, when the news media may cover their every thought? Our next point is related.

Balancing competing interests. Because the board needs to take the long view (but also provide counsel and make decisions regarding immediate challenges), and because of its duties of care, loyalty and obedience, it must try to balance the competing interests of a range of stakeholders. They include, among others, faculty members from multiple disciplines, staff members, current students, future students, alumni, community and civic groups, neighborhood associations, policy makers, and boosters. In addition to long-term versus short-term views, on all of our campuses, other paradoxical issues are at hand. Should the board drive change or work to help maintain stability? Focus on core businesses or new businesses? Save and build the endowment or innovate and invest? And in the meantime, the public wants everything better, cheaper. Every single decision a board makes is going to please some and upset others. That’s reality.

Acting as a group. We all know how hard it is to make certain decisions alone, right? While some college and university boards are small, the average size of private university boards is 29. Boards must deliberate issues, hear all sides, seek optimal solutions and come to decisions that will be made public as a collective. Imagine that you have to come to a mutually acceptable decision with competing interests, fast, in a group and under the spotlight (and by the way, with interim leadership, as is often the case). We think we can all agree: this is a tough job.

Dealing with challenges of accountability. Board work is difficult because it lacks natural systems of accountability. Who is watching the board, and to what extent does the board see itself as accountable? Boards work well when they take their own accountability seriously, but too often they do not. Yes, the faculty can vote no confidence in the board, and the state attorney general can intervene, as was the recent case at Cooper Union. But for the most part, boards must develop the ability and conscientiousness to establish their own mechanisms for accountability.

Where Good Governance Reigns

Finally, we must not forget that board members are volunteers, the preponderance of whom are members of the general public, not of the academy. Board members are appointed (public boards), elected (alumni, faculty or student representatives, or by general election in some states), or they volunteer when asked. Most college and university boards are composed of influential individuals who care deeply about the institutions they serve. It’s true that some come with an agenda, but, by and large, that’s not the case. They want to do good work, and they want their institutions to succeed. These volunteer boards can and do add value to the institutions they govern by bringing collective expertise, insight and wisdom.

Strong boards share some important ways of thinking about governance that go beyond size and structure. They:

  • recognize that the stakes are high, and have perhaps never been higher, for the institutions they serve;
  • realize that all trustees and boards have room to improve -- and make a commitment to seeking feedback, reviewing their work and learning;
  • have a certain positive restlessness that keeps them always striving to do better;
  • are self-aware, think about their collective impact, are cognizant of complexity and the paradoxes surrounding them;
  • pay attention to substance, structure, culture, process and boardroom dynamics; and
  • can adapt to changing circumstances rather than get trapped in stale routines (however comfortable they may be). They have, as one president said recently, “the ability to pivot.”

Recommendations for Boards

This essay is not an apologist’s perspective on governance today. Given the complexity of the world in which boards must govern, we realize that we run the risk of ridicule for oversimplifying and making broad-brush statements (and fully expect comments below to those effects). And we realize that there are no panaceas, no right answers and no silver-bullet solutions to governance. Still, we offer here six ideas boards might consider to help ensure that they are ready when (not if) the messy issues arise.

  1. With the administration, and within the parameters set forth by your bylaws, develop an explicit understanding of good governance. Communicate it to the faculty.
  2. Practice (“scrimmage”), when times are good, on easier issues. Boards cannot predict what difficult issues will surface and must be well practiced to take on the most unexpected challenges.
  3. Review cases in the news and ask, “What can we learn? What if that were us?”
  4. Consider the perspectives of multiple stakeholders by asking, “Who are the stakeholders for this decision? What’s at stake for them, particularly the faculty and students? Why?”
  5. When appropriate, seek input from key stakeholders, being especially cognizant of your institution’s shared governance expectations.
  6. Communicate not just the decision outcomes but also the deliberations. Help those affected stakeholders understand the different viewpoints that were broached in the boardroom.

Boards will likely face the “damned if you do, damned if you don’t” context for the foreseeable future. The better they are prepared to address the big challenges ahead, the better our institutions will be.

Cathy Trower is president of Trower & Trower Inc., a board governance consulting firm, and a trustee at Wheaton College, Mass. Peter Eckel is a senior fellow and the director of leadership programs in the Alliance for Higher Education and Democracy in the University of Pennsylvania’s Graduate School of Education and a trustee at the University of La Verne.

 

Editorial Tags: 

ACE Weighs in on Credentialing and Competencies

The American Council on Education on Thursday released two new papers that call for a less fragmented credentialing system in higher education and for better communication about the value of students' competencies. The group, which is higher education's umbrella organization, is hosting a live stream from an open meeting about the two papers on Thursday morning.

The range of credentials issued has grown in recent years, with more associate degrees, certificates and state-issued licenses being awarded.

“The diversity of credentials is not always meeting the needs of students, educational institutions and employers, and unfortunately the proliferation of credentials is causing confusion,” one of the papers said. “There is a lack of shared understanding about what makes credentials valuable, how that value varies across different types of credentials for different stakeholders, what constitutes quality and how credentials are connected to each other and to opportunities for the people who have earned them.”

In addition, ACE said students can benefit from more effective communication to employers about what they know and can do with that knowledge.

"Employers need to understand the competencies of applicants in order to make appropriate hiring and promotion decisions, thereby increasing the value and effectiveness of their organizations,” the second paper said.

Study Finds Increase in College Completion in 15 States

A new report from the National Student Clearinghouse Research Center found that 15 states have improved college completion rates for students in four-year public institutions, despite overall national declines.

Nationally the completion rate dropped from 62.9 percent to 61.2 percent from 2008 to 2009, but states like California, Connecticut, Iowa and South Carolina saw increases of 1-2 percent.

The report also found that 32 percent of students who started at a two-year institution completed at a different institution. This was particularly true in California and Texas, the report found, where more than 40 percent of all completions occurred at a different institution than the one where students first enrolled.

Pages

Subscribe to RSS - assessmentaccountability
Back to Top