Institutional research

Why assessment isn't about certainty (essay)

When I first floated the idea of writing a weekly column from my perch as director of institutional research and assessment at my college, everyone in the dean’s office seemed to be on board.  But when I proposed calling it “Delicious Ambiguity,” I got more than a few funny looks.  
Although these looks could have been a mere byproduct of the low-grade bewilderment that I normally inspire, let’s just say for the sake of argument that they were largely triggered by the apparent paradox of a column written by the measurement guy that seems to advocate winging it. But strange as it may seem, I think the phrase “Delicious Ambiguity” embodies the real purpose of Institutional Research and Assessment. Let me explain why.
This particular phrase is part of a longer quote from Gilda Radner – a brilliant improvisational comedian and one of the early stars of “Saturday Night Live.” The line goes like this:
“Life is about not knowing, having to change, taking the moment and making the best of it, without knowing what’s going to happen next. Delicious Ambiguity.”
For those of you who chose a career in academia specifically to reduce ambiguity – to use scholarly research methods to discover truths and uncover new knowledge -- this statement probably inspires a measure of discomfort.  And there is a part of me that admittedly finds some solace in the task of isolating statistically significant “truths.”  I suppose I could have decided to name my column “Bland Certainty,” but – in addition to single-handedly squelching reader interest – such a title would suggest that my only role is to provide final answers – nuggets of fact that function like the period at the end of a sentence.
Radner’s view of life is even more intriguing because she wrote this sentence as her body succumbed to cancer.  For me, her words exemplify intentional – if not stubborn – optimism in the face of darkly discouraging odds. I have seen this trait repeatedly demonstrated in many of the faculty and staff members I know over the last several years as you have committed yourself to helping a particular student even as that student seems entirely uninterested in  learning.
Some have asserted that a college education is a black box; some good can happen, some good does happen – we just don’t know how it happens. On the contrary, we actually know a lot about how student learning and development happens – it’s just that student learning doesn’t work like an assembly line.  
Instead, student learning is like a budding organism that depends on the conduciveness of its environment; a condition that emerges through the interaction between the learner and the learning context.  And because both of these factors perpetually influence each other, we are most successful in our work to the degree that we know which educational ingredients to introduce, how to introduce them, and when to stir them into the mix.  The exact sequence of the student learning process is, by its very nature, ambiguous because it is unique to each individual learner.
In my mind, the act of educating is deeply satisfying precisely because of its unpredictability.  Knowing that we can make a profound difference in a young person’s life – a difference that will ripple forward and touch the lives of many more long after a student graduates – has driven many of us to extraordinary effort and sacrifice even as the ultimate outcome remains admittedly unknown.  What’s more, we look forward to that moment when our perseverance suddenly sparks a flicker of unexpected light that we know increases the likelihood – no matter how small – that this person will blossom into the lifelong student we believe they can be.
The purpose of collecting educational data should be to propel us – the teacher and the student – through this unpredictability, to help us navigate the uncertainty that comes with a process that is so utterly dependent upon the perpetually reconstituted synergy between teacher and student. The primary role of institutional research and assessment is to help us figure out the very best ways to cultivate – and in just the right ways – manipulate this process.  
The evidence of our success isn’t a result at the end of this process.  The evidence of our success is the process.  And pooling our collective expertise, if we focus on cultivating the quality, depth, and inclusiveness of that process, it isn’t outlandish at all to believe that our efforts can put our students on a path that someday just might change the world.
To me, this is delicious ambiguity.

Mark Salisbury is director of institutional research and assessment at Augustana College, in Illinois. This essay is adapted from the first post on his new blog.

Editorial Tags: 

WGU pushes transfer students to graduate community college first

Smart Title: 

Western Governors U. pushes graduation even before students enroll by offering financial perks for associate degree holders and, at WGU Texas, through partnerships with community colleges.

Gaining and maintaining momentum is key to student completion (essay)

Newton’s First Law of Motion states that an object at rest tends to stay at rest and an object in motion tends to stay in motion, and once in motion, that is when it develops momentum. It will tend to stay in motion unless acted upon by an external force.

Elucidated by Newton in 1687, the first law of motion can also be applied to study of student completion, for like objects, students at rest tend to stay at rest and students in motion tend to stay in motion. Once they gain momentum (that is, acquire more degree credits), they are more likely to stay in motion unless acted upon by an external force.

Gaining and maintaining momentum is key to student completion. Students who progress more quickly through the curriculum are considerably more likely to complete their degrees than those who do not.

This is but one reason why a number of states have begun to focus on the importance of student momentum to completion. The Washington State Board for Community and Technical Colleges, for instance,utilized the analysis of the transcripts of more than 87,000 first-time community and technical college students who entered the Washington system in the 2001–2 academic year to identify key points in the curriculum, referred to as momentum points or milestones, whose timely attainment was associated with student progress to degree completion.

For most institutions, these intermediate points of attainment include the successful completion of developmental coursework, the timely declaration of a major, and the earning, within a particular time period, of a number of degree credit hours. These momentum points were then folded into the state’s funding formula such that institutions are now rewarded when they improve the number of students attaining those points of intermediate achievement.  Other states have or will soon follow suit with similar models of funding that center on the importance of student momentum to completion.

Identifying intermediate points of attainment is one thing. Helping students gain momentum and attain them in a timely fashion is another. Unfortunately, not all students are able to do so. Take the case of students who begin college academically under-prepared. Too many spend too much time on coursework for which they earn no college credit. It some cases it may take some students two or more years to complete basic skill requirements, if they are able to do so at all.

This is but one reason why an increasing number of colleges, such as the Community College of Baltimore County, are turning to accelerated learning programs for those students who begin just one level below college-level work. In this case, rather than being placed in a stand-alone basic skills course for which students do not earn college credit, they are placed in the college-level course to which that course would have provided entry together with a study skills course that is directly connected to that course. In this manner, students earn college credit while acquiring needed basic skills.

Similarly, colleges such as the Community College of Denver have condensed what would otherwise be a two-semester sequence of either developmental math or developmental English into one semester in their FastStart program. By adopting interactive teaching and learning strategies, contextualization of developmental coursework, and cohort-based models, they have been able to substantially increase the percentage of students who complete their developmental coursework and continue in college.

A number of other institutions have taken a different approach to speeding up student progress through developmental coursework by revising the way students’ skill levels are assessed at entry. Tarrant County Community College, for instance, employs ALEKS and MyMathLab not only to assess student math skills bur also provide students an online vehicle to address those skills that require improvement.

Rather than categorizing students into three math levels, each of which requires an individual course to address, Tarrant officials identify 15 math skill modules and ask students to take only the specific modules in which they need help.  Using Computer Assisted Instruction, they have greatly accelerated students’ movement through developmental math and in turn reduced institutional costs. Other institutions, such as Capital Community College and Kapi’olani Community College, have successfully employed summer bridge programs that enable underprepared students to get a head start of their first year of college and therefore move more quickly to earning college credits.

Gaining momentum toward degree completion requires that students not only earn college credits but also do so in ways that lead to degree completion. Yet many students begin college undecided or change their majors, sometimes several times. This is but one reason for the growing emphasis on intrusive first-year advising merged with career counseling. In addition to the front loading of such advising and the use of first year student success courses in which advising and counseling are embedded, as they are in Florida, a number of institutions have employed web-based solutions to help students establish career and educational goals in a timely manner.

Programs such as Valencia Community College’s LifeMap and Century College’s GPS LifePlan, now widely used in Minnesota, have used such programs to increase goal setting and in turn retention and completion.  Other institutions, such as Saddleback College, utilize predictive analytics to construct real-time on-line advising systems that responds directly to student advising needs as they progress through the institution.

Unfortunately,student progress is frequently constrained if not halted by the incoherent array of courses that typify most college offerings. Lacking any clear structure, students tend to wander through the curriculum in ways that undermine their ability to make timely progress. Some leave in frustration and others amass more credits than they need for program completion, that is, if they are ever able to do so. It is for this reason that a number of colleges seeking to improve rates of completion have turned their attention to curricular structure and coherence. Under the auspices of the Bill & Melinda Gates Foundation’s Completion By Design initiative, consortia of community colleges in four states -- Florida, North Carolina, Ohio, and Texas -- are working to develop coherent course pathways whose structure enable, if not require, students to move more quickly through the curriculum to the certificate or degree completion. 

In these and other ways, institutions and states are coming to recognize the wisdom of Newton’s First Law of Motion and the importance of student momentum to college completion. Hopefully these and other efforts will take on a life of their own and gain sufficient momentum to transform how institutions approach the task of improving college completion.

Vincent Tinto is Distinguished University Professor at Syracuse University.

Education Department changing graduation rate measurements

Smart Title: 

The Education Department plans to change widely disparaged federal definition of completion rate to include transfers and nontraditional students.

Pulse podcast explores uses of Starfish Retention Solutions

Smart Title: 

This month's edition of The Pulse podcast features a conversation with David Yaskin, CEO and founder of Starfish Retention Solutions.

TFA's 8th Annual Student Retention & Recruitment Workshop

Mon, 07/16/2012 to Thu, 07/19/2012


222 Saint Paul Place Tremont Plaza Hotel & Grand Historic Venue
Baltimore , Maryland 21202
United States

Researcher creates new tool for timely data on attainment

Smart Title: 

Researcher's new tool for producing timelier data on college degree completion shows 6.5 percent increase in number of credentials awarded in U.S. in 2011.

Ashoka U Exchange

Fri, 02/10/2012 to Sat, 02/11/2012


699 S Mill Ave
Tempe , Arizona 85281
United States

Cavanaugh essay how accreditation must change in era of open resources

Imagine yourself emerging from the Way Back Machine in London, England. It’s 1526. Henry VIII is on the throne. You furtively duck into a shop, and quickly head to the back room. You’ve come to buy an English translation of the New Testament. The mere possession of this book is punishable by death.

In the 1520s, having open access to books (knowledge) was a dangerous game. It threatened the establishment. It meant that ordinary people could see for themselves what the elite had guarded so closely.

Enter Thomas Cranmer, Archbishop of Canterbury, who commissioned the publication of the “Great Bible,” in English, making it available to every church where it was chained to the pulpit to ensure it was accessible and didn’t “disappear.” While the publisher paid for this access with his life, in three short years readers were provided so that everyone, even the illiterate, could hear the Word of God proclaimed in their native English.

Fast-forward 472 years. You’re a college student. You’ve taken advantage of some amazing opportunities in the online world. You’ve listened to Nobel laureates discuss the Eurozone crisis and explain how current difficulties relate (or not) to classical theories of economics. You’ve worked through the underlying physics and chemistry for nearly every episode of "MythBusters." You regularly watch the TED lectures. And you’ve even taken courses from the Open Learning Initiative and from OpenCourseWare at MIT. Now you want the academic credit for those forms of learning.

Although you won’t actually be burned at the stake as Cranmer was, you have a very good chance of experiencing the modern version of this torture because it is equally threatening to the elite. It goes something like this.

First, you’ll be asked to produce the sacred document, otherwise known as a transcript, indicating that you officially took the course. No transcript you say? Sorry — your learning is then considered “illegitimate,” and you’re then often cast out into the night where there is weeping and gnashing of teeth as you stumble back to the very beginning of college to start over.

While an exaggeration, today -- through such outlets as TED, various open-source course initiatives, and primary sources through digital content providers -- we all have access to the knowledge that previously was the province of academia. In the same way that access to the New Testament gave otherwise uneducated English people access to the very heart of Christianity, that access is “dangerous.” It threatens the central notion of what a college or university exists to do, and so, by extension, threatens the very raison d’etre of faculty and staff.

Threats to a well-entrenched status quo are not well-received. But the funny thing about many of them — whether books or ideas — is that they often quickly become the mainstream.

Vu Déjà 

Higher education is facing the very situation that confronted our colleagues in the P-12 world when home schooling threatened the world order. Initially considered a fringe activity of substandard quality, the sector figured out that if appropriate standards (i.e., learning outcomes) were agreed upon and stated clearly, it didn’t really matter what path students took to get to the knowledge destination.

Higher education needs to take a lesson from that experience and work much harder on specifying our analog of the Common Core State Standards. The tools are there, and have been there for a very, very long time. It just has not been in our self-interest to develop and agree on them. But we’d better, and we’d better do it now. Otherwise, it will be done to us.

What Thomas Cranmer figured out was that it was impossible to execute people fast enough to stem the desire to access the new sources of knowledge. So he wholeheartedly adopted the reform, and made it his own. What we need to learn from that is to accept the reality that anyone can access the same information we academics used to carefully mete out, so the best approach is to adapt and make that reality our own. We need to create a higher educational system that embraces competency-based achievement, realign the milestones by which we gauge increasing levels of knowledge/competence, and redefine degrees on this basis.

We have an instructive example. Standard 14 of the Middle States’ Characteristics of Excellence pertains to the assessment of student learning. The standard requires that students be told what “…knowledge, skills, and competencies [they] are expected to exhibit upon successful completion of a course, academic program, co-curricular program, general education requirement, or other specific set of experiences… .”

As stated in the Standard, the objective is “…to answer the question, ‘Are our students learning what we want them to learn?’ ” Such assessment is “an essential component of the assessment of institutional effectiveness” (Characteristics of Excellence,, p. 63). The description then goes on to discuss how learning outcome assessment should be designed and its results used. Nowhere is there a discussion of credits.

Given that we already have an accreditation system based on the assessment of student learning (i.e., knowledge/competence acquisition), then it is a rather straightforward matter of taking the existing approach to the next step to complete the conversion process from one grounded on credit accumulation (irrespective of learning) to one based on demonstrated learning outcomes.

More specifically, we need to adopt the approach already taken in many professions of clearly articulating what students are supposed to know and be able to demonstrate at various levels of educational attainment, and create accreditation standards and metrics that reflect it. This would put real teeth in the assessment of student learning outcomes by putting consequences on not doing it well, as well as put the focus on where the content comes from and its quality assurance that underlies the knowledge/competence we expect students to acquire.

When that happens, the recognition of prior learning becomes very straightforward, and its source becomes irrelevant as long as the appropriate competencies are shown. In other words, we already have all the basic elements necessary to take the Cranmeresque step of moving from banning the immediate and unquestioned acceptance of demonstrated knowledge/competence to creating the postsecondary equivalent of the Book of Common Prayer.

The Brave New World

Adopting an accreditation system predicated on the authentic assessment of student learning outcomes liberates faculty to serve a much more important role — that of academic mentor and guide for the student’s learning and knowledge/competence acquisition process. In a way, this will return us to the past, whereby through the judicious use of technology faculty will be able to provide far more individualized instruction to many more students than the current system could ever possible allow or support. In another way, it means that the kind of individualized attention we give to doctoral students can be extended to all. This would be a major improvement for students and faculty alike.

To continue to have legitimacy, accreditation must focus on the core issue — student learning. Accreditation must begin certifying that students actually learn, and that what they learn matches the stated objectives of a course, an academic program, or a specific set of objectives (such as in general education). In short, accreditation must move from certifying that an institution claims that it is doing what it is supposed to do to certifying that students are learning and progressing in their acquisition of knowledge/competence.

Because people can simply wander around the Web and pick up content that is neither amalgamated by a content provider nor verified for accuracy, it will become necessary for some entity to engage in quality assurance in terms of learning outcomes. The job of verifiying bona fide knowledge/competencies and establishing where along the continuum of knowledge/competence acquisition a student falls can become the province of organizations that resemble, or even broader entities.

In both cases (i.e., using content offered by an certified provider or doing it on your own with no official guidance), a credential or type of certification would be provided each time a new level of knowledge/competence is reached. The student would then deposit those credentials or certifications into a credential bank for future reference. The student -- not the registrar’s office -- owns the credential

Degrees Deconstructed and Decoupled

We get to this alternate accreditation world in two ways: by clearly defining what each degree means and aligning accreditation with content providers (not institutions that confer degrees).

This requires that we come to quick agreement on what different types of degrees mean. In the United States the TuningUSA effort is just beginning the work of more clearly articulating what knowledge/competencies a student is supposed to demonstrate before being awarded a postsecondary degree.

This is in contrast with the current practice of awarding degrees based on a student's spending a specified minimum amount of clock-defined time amassing an arbitrary number of credits and obtaining a minimum set of grades. Nothing in the current definition says anything about what knowledge or competencies the student actually demonstrates. We need to test it — look up the degree requirements for English literature degrees across a variety of institutions and compare them. This loose approach is in contrast to efforts in other parts of the world, such as Europe, where degree qualifications discussions have been ongoing for over a decade.

Once the accreditation focus is placed on student learning outcomes for real, accreditation becomes tied to learning and is decoupled from institutions granting degrees. Accreditation then becomes aligned with entities that provide content and the parcels or “courses” in which they are delivered. The seal of accreditation would then be placed on the separate pieces of content offered by content providers who demonstrate that the content offered comes with embedded authentic assessment of learning. To be sure, most of these providers will still be postsecondary institutions, but the accreditation umbrella is extended more broadly to reflect the current reality that content comes from many sources.

In such a system, regional accreditation no longer gives thumbs-up or thumbs-down only on the traditional degree-granting institution. Rather, it focuses on what is provided by any entity that wants to claim it’s in the business of offering content. If and only if that content meets certain standards would it be “accredited.”

Shifting the focus from the institutional level to the content level would strengthen the link between accreditation and federal financial aid eligibility. If and only if a student was using content from an accredited source would the student be able to apply for and receive federal financial aid. Likewise, if the student has amassed knowledge/competencies from self-instruction or from noncertified sources and wants to convert that into “certified learning,” then federal financial aid could be spent only at accredited entities in that business.

Charting a Future Course

The possible future I have described here is both scary and exciting. We can choose to sit down in the captain’s chair and help chart our own course by fully embracing new opportunities while really being serious about quality as defined as authentic assessment of the acquisition of knowledge/competence. Or we can put up the shields, claim that the way we provide access to knowledge now is to remain immutable for all time and that change will bring our world crashing down and condemn us to eternal damnation, and have a modern equivalent of Thomas Cranmer bring it all crashing down.

It’s up to us. Shields will not work. We have only one real option if we want to build on the true legacy and meaning of education: to boldly go where accreditation has never gone before.

John C. Cavanaugh is Chancellor of the Pennsylvania State System of Higher Education. This essay is adapted from a speech he gave Tuesday at the annual meeting of the Middle States Commission on Higher Education.

NSSE 2011 measures student engagement by major

Smart Title: 

This year's National Survey of Student Engagement broke down learning by major, and while some students are working more than others, they're not necessarily more prepared.


Subscribe to RSS - Institutional research
Back to Top