Advertisement

News, Views and Careers for All of Higher Education

Between a Rock and a Hard Place

On many campuses, institutional research directors have long been the point people for assessing both how successfully students are learning and, by extension, where their colleges can do better still. They believe deeply in the importance of producing and sharing data that can help professors teach, and students learn, more effectively, and they take pleasure in seeing how small changes in curriculums or practices — incorporating extra public speaking in a capstone course, for instance — can significantly improve students’ performance.

Yet as pressure has grown in recent years for colleges both to gauge their student learning outcomes with more objective measures and to report their results broadly — assessment for external accountability, as opposed to internal improvement — institutional researchers have increasingly found themselves caught in the middle.

On one end are politicians and other policy makers accusing college administrators and especially professors of dodging accountability because they are wary of using measurement tools that would allow for easy comparison among institutions. On the other are the faculty members themselves, the more moderate of whom warn that oversimplified assessment will provide no answers, while others do, per the easy stereotype, resent any intrusion into the classrooms where, they argue, they know in their hearts what works.

Dozens of institutional researchers gathered at the Educational Testing Service’s conference center this week for a meeting, co-sponsored by the Association for Institutional Research, focused on that tension and how IR officers can navigate it.

“It’s hard enough to help our institutions develop institutional assessment capacity, and it’s hard enough to answer calls for accountability from policy makers,” Victor Borden, associate vice president for university planning, institutional research and accountability at Indiana University at Bloomington, said in a lunch speech entitled “Measuring Success: Living Between a Rock and a Hard Place.” “Trying to do both is very conflicting, and they can get in the way of each other.”

“It’s a hard gap to build bridges between,” he added.

Yet that’s just what institutional researchers are charged with doing, and at ETS on Monday they shared with each other, with a minimum of griping, strategies and advice for doing so — including by using the external calls for accountability to their advantage internally.

David G. Payne, associate vice president for college and graduate programs at ETS, got a lot of heads nodding when he opened the day’s discussion with the views about assessment from a variety of faculty “types,” including the “historians” who’ve seen previous calls for accountability and think they can wait this one out and those who think it is somebody else’s job — like the institutional researcher.

Despite the easy stereotypes, which resonated with the IR folks for a reason, some of the assessment experts said it was unfair to portray professors as opponents of measuring their performance and that of their students.

“Many of our faculty have been measuring for a long, long time,” said Dawn G. Terkla, associate provost for institutional research and evaluation at Tufts University. And with many other professors, she said, “it’s not that they don’t want to do it; sometimes they just need to be shown how.”

Institutional researchers offered different strategies for involving faculty successfully. “Faculty ownership” is essential, said Beth Jones of West Virginia University, who noted that when the institution first sought to set up a committee for assessing learning outcomes in general education, administrators selected the faculty participants and, surprise surprise, “people didn’t come to the meetings.” The second time around, Jones said, departments solicited volunteers, and the result was far more engagement. “You need to find people who are interested, and directly involved in deciding what is the best approach to take.”

Trudy Bers, executive director for research, curriculum and planning at Illinois’s Oakton Community College, said that institutions seeking to involve faculty members in assessment work should “break the work into manageable tasks” unless they’re prepared to provide rewards — reduced teaching loads or other benefits — “for faculty who take on really significant roles.” About a third of institutions said they provided such incentives.

Even at institutions where professors are sold on the value of measuring their own performance, the goals are changing in response to accreditors, who in some cases are responding to pressure they are feeling from the federal government,

“We’ve been doing a lot but the accreditors are forcing us to be more purposeful,” said Turkla of Tufts. Added Karen Froslid Jones, of American University: “We’re really aware of external pressures on us. We know that if we don’t do a good job with the assessment, it will come back to us one way or the other.”

Regis University has found itself facing “major issues around measuring student learning” with its accreditor, the North Central Association of Colleges and Schools, said Kimberly Thompson, director of assessment and college research there. The problem is not that faculty were not measuring their performance; “we had pockets of excellence, certain departments that were doing an excellent job of assessing student learning, using multiple, direct measures of learning,” Thompson said.

The pockets were just that, though — islands unto themselves, she said. “They didn’t have an opportunity to share what they knew and what they were doing with other departments. Our big challenge is taking the information and know-how we currently possess and sharing that with all our departments, and doing so in such a way that our faculty are driving assessment.”

Bers said the challenge at Oakton — and other institutions, too — is to move the level of assessment “beyond the individual faculty member to the course or program level, or even the institutional level.”

But as colleges and universities seek to respond to the pressure to find measures of student learning that will reflect outward, allowing for comparisons to other institutions, they are likely to run up against (often legitimate) criticism from professors and others that assessment measures that allow ready comparisons among differing institutions are likely to be simplistic to the point of meaninglessness, suggested Borden, of Indiana.

Borden urged the institutional researchers not to let external demands for accountability overwhelm their primary obligation to expand their own colleges’ capacities to assess the quality of student learning — but he also encouraged them to use the outside pressure to their advantage, one of several “points of leverage” they might use with recalcitrant faculty members.

“Use the external mandates to push the question,” Borden said. “Ask, ‘If these are not appropriate measures, what are? What data should we use?’ “

Doug Lederman

Got something to say?


Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Advertisement

Comments

Assessment

When SACS says that all syllabi should be uniform, and that a professor cannot stray out of any assessment measures stated therein in advance, and therefore should not cover any further material regardless of the ability of the class, we only dumb down college into High School Part 2. For example, we will no longer be able to throw out any quiz scores from the dozens given over the semester, unless the number and identity of each quiz so treated are already preordained in said syllabus.

Lawyers will sniff out anything which will line their pockets, and helicopter parents will be eager to employ them. (Perhaps these lawyers will be referred to as Helicopter Chasers.) Administrations will therefore practice the time-honored bureaucratic tradition of Cover Your Ass.

So begins the demise of academic freedom.

DFS, at 7:35 am EDT on July 22, 2008

Community Service Modelling

As a longtime neighbor of Tufts, the host to the National and State Campus Compacts, sponsor of the Tisch College of Citizenship and Public Service, it’s fascinating to see how remarkably provincial a leading university’s concept of community impact can be. Tufts hosts both the PACE Center — which challenges the narrow measures of SAT/defined student learning quite effectively — and the Positive Deviance Institute — which demonstrates a remarkable sensitivity to building new capacity for community change from successful, but unrecognized local innovation. Yet neither of these superb intellectual and academic capacities have had any outlet whatsoever in the communities immediately adjacent to academe. Instead, the university is known for it’s biotesting and drunken undergraduates.

That town-gown contrast — also marked by your report, last month, of the lowest graduation rate in this Congressional District, which has the highest number of undergraduate residents of any in the nation — ought to have caught SOME institutional researcher interest, SOMEWHERE.

Joe Beckmann, at 7:55 am EDT on July 22, 2008

Where’s the Faculty Governance?

I was disappointed to see that several of the institutional research professionals in this article hold a paternalistic view of faculty, portraying them as adversarial and as obstacles to assessment activities.

I appreciated Ms. Jones’ comments about involving faculty in the process, and this highlights a glaring omission in the curricula of AIR-funded Institutional Research Certificate programs. What seems to be desperately needed is a course on faculty governance to balance the vocationally-oriented research methods courses.

As a result, institutional research professionals can better understand the importance of “faculty ownership” in designing campus-wide assessment programs.

DrRingDing, at 9:20 am EDT on July 22, 2008

Tick-tock, people

Bad news for the “academic freedom and tenure do not include monitoring and evaluating my performance” crowd. There’s this thing called the Internet. Students and their families use it to communicate about your performance (e.g., Rate My Professor). De facto, faculty performance and outcomes are being evaluated.

IMO, that’s how some colleges get “hot” — and others that ought to close their doors, frankly. The “hot” ones have faculty who are relevant, engaged, and accountable.

To quote the poet C. Carter: “The truth is out there.” And students know how to find it.

L.L., at 9:50 am EDT on July 22, 2008

Further exploration

It may be useful to know that these issues are covered in depth in the just-published entire issue (Vol. 16, no.2) of the journal On The Horizon that is titled “Accountability for educational results.”

David Shupe, eLumen Collaborative, at 9:50 am EDT on July 22, 2008

“IMO, that’s how some colleges get “hot” — and others that ought to close their doors, frankly. The “hot” ones have faculty who are relevant, engaged, and accountable.

To quote the poet C. Carter: “The truth is out there.” And students know how to find it.”

Thanks for your enlightenment. I was wondering how Sanjay managed to last so long on “American Idol".

So lets see, we ask students, who have not mastered the subject matter yet, and who have never taught a course to evaluate the effectiveness of faculty.

What they are really evaluating is how well they were able to learn concepts and their interest level in the subject matter, which is a measure of themselves not the teacher.. Or perhaps how well they “like” the instructor.

Kind of like asking a virgin to evaluate how well a pregnant women delivered her child.

Lets put the cart before the horse. Does logic not count for anything anymore?

dundermifflin, at 11:15 am EDT on July 22, 2008

SACS Requirements

DFS writes “... SACS says that all syllabi should be uniform, and that a professor cannot stray out of any assessment measures stated therein in advance, and therefore should not cover any further material regardless of the ability of the class..” and “we will no longer be able to throw out any quiz scores from the dozens given over the semester, unless the number and identity of each quiz so treated are already preordained in said syllabus”

I’m interested in these claims. Does SACS really require this kind of ‘uniformity’ and hewing to the syllabus? MidStates does not. Is such ’stringency’ peculiar to SACS?

cts, at 11:15 am EDT on July 22, 2008

Regarding the comments by DFS and CTS about SACS requirements — I don’t believe you are correct in saying that SACS has strict, uniform requirments on syllabi, etc. Check SACS accreditation standards and policies at www.sacscoc.org and see if you can really find such requirements. TEB

TEB, at 4:20 pm EDT on July 22, 2008

Rate My Professor

LL: do you seriously think Rate My Professor is an adequate measure of student learning outcomes? It’s a (rather poor) measure of faculty popularity at best.

Faculty Person, at 4:40 pm EDT on July 22, 2008

RMP is not only evaluation tool

” .. do you seriously think Rate My Professor is an adequate measure ..? It’s a (rather poor) measure .. at best.”

First — who said RMP is the only evaluation tool used by students and their families? It was cited as just o-n-e example.

Second — IMO, RMP’s quality depends on the number of respondents and number of outliers. I’ve checked a few colleagues on RMP — brutally honest.

Ignore the public at thy peril. Great peril.

BTW: it is S-a-n-j-a-y-a. Ten points marked-off for spelling error.

L.L., at 7:35 pm EDT on July 22, 2008

Student learning assessment

In the early 90’s I completed a case study of student satisfaction measurement in Ontario (i.e., Canadian province) universities by surveying 13 of the 16 institutions in Ontario. One of the areas of interest was the evaluation of faculty effectiveness by students. The instruments used within each university varied greatly and were difficult to compare to each other. Needless to say, a comparison between institutions in this area was even more problematic.

I believe, as did Alexander Astin, that a measurement of student satisfaction is one of the key indicators of effectiveness in higher education. When combined with student learning outcomes data this creates a very strong package for evaluation of many things at many levels.

“Between a Rock and a Hard Place” brought back memories of my research in this area and reminded me of just how challenging this work is.

Theron A. Craig, Dr, at 7:35 pm EDT on July 22, 2008

You are right, TEB

My apologies. I hyperventilated about SACS re syllabi. I should have made it clear that it is my institution who is in the process of developing such uniformity in anticipation of SACS review, not the SACS guidelines per se.

I am bemoaning this slow but steady march to micromanaged grading measures in the class. In a developmental class, I believe that a student should be allowed to “develop.” This means that some earlier stumbles can be ignored, as long as true development to standards indeed took place.

I agree with the remark that a virgin cannot speak intelligently about how well another delivered a baby. All the student can do is remark about what the student understands. An open-door policy throughout one’s chain of command should be sufficient to address any real problem with the way the class was conducted.

On a more macro-level, grade distributions should suffice to assess an institution’s “performance.”

We can all improve, but must we do so by the delicate caress of a sledge-hammer?

Again, my apologies to all.

DFS, at 9:40 am EDT on July 23, 2008

Close the Door; They’re Comin’ in the Windows

My institution recently achieved regional accreditation status and it was partially due to the efforts of the faculty to take an active role in the assessment process, both for institutional evaluation purposes, but also to examine where we needed to make changes to our existing programs and courses in order to make holistic improvements. While we had an IR specialist involved, the burden was on the faculty. I don’t particularly like the tone of some of the IR gurus (in agreement with the other comment) that faculty and researchers seem to be adversarial. In our case, we are the ones providing the assessment research for him to process; we also are the ones making the implementations and rollout plans to improve the school. I think this is what should be happening. Prescriptive change without due examination and understanding of the in-class variables is not only counterproductive, but can lead to that adversarial relationship. Who needs that?

Douglas Anke, The Art Institute of Pittsburgh, at 10:00 am EDT on July 23, 2008

DFS, you would be right about grade distributions if grades were really a true reflection of student mastery. However, they are not and cannot be without clear, transparent, public rubrics for what constitutes an A, B, etc. Instead, student grades are inflated for a variety of reasons (e.g., for “effort,” to improve faculty evaluations for tenure and promotions, and to reduce student and parent complaints to professors and dean). For those who grade on a curve (the complete lack of well-defined, clear standards for mastery if ever there was one), a theoretical curve is applied to a small population that probably isn’t bell-shaped. For these reasons, grades are imperfect indicators of how well students are learning and how well professors are teaching. Any other assessment process that can be implemented will be imperfect as well. However, one would hope they would be better than simple grade distributions because disciplinary standards would be clear, well-defined, and public.

IHE Reader, at 10:10 am EDT on July 23, 2008

TEB and DFS

Thanks for the clarifications re SACS. Now, however, I’m alarmed by what DFS’s institution is doing. I argues [against a senior colleague!] years ago when he wanted to impose a very specific syllabus on anyone teaching a particular service course. I just did not think it made educational sense — nor that it was likely to make for happy instructors. This does seems to be a ’sledge-hammer’ approach. On the other hand, this is not what assessment needs to be. It can be done intelligently so that we can be better at what we do. I have to agree that grade distribution is not a good indicator; I’ve heard/seen enough horror stories of easy As at my own college to trust grades.In defense of the IR folks (I’m not one), it should be said that many faculty are very hostile/reactive. THe IR people are just expressing their fear of faculty resistance, as based on induction.

cts, at 4:30 pm EDT on July 23, 2008

Sad News for IR — You are not in charge of assessment!

It is worth considering this article from a different perspective with respect to authority and responsibility. IR directors are actually rarely, not usually, responsible for learning assessment. This important part of teaching and learning is always a faculty responsibility and it is guided by the same principles of higher education. I have never heard of university faculty reporting to an IR director instead of a dean. As such, the IR director is, at its best, a colleague, consultant and advisor — but never the person who answers for learning assessment and certainly not the one who is “caught in between.” I work with faculty and oordinate and assist in the learning assessment process. It is a daunting task for us to respond constructively and in a balanced way to this important challenge and the principles of “standards,” given that we are faced with unending mantras demanding constant and ever-growing IMPROVEMENTS without regard for whether ceaseless improvement is needed in every course, every term, every year. I work with extremely well-motivated faculty who want the best for their students. Our greatest challenges are (a) the added workload of assessment, and (b) the critics of faculty who fail to put this work in a context. I salute all faculty who are taking this challenge seriously and there are many who can be so acknowledged.

Nancy Hedlund, Assoc VP Planning & Assessment at Hawai’i Pacific University, at 5:40 am EDT on July 27, 2008

Advertisement

 Jobs Related to Between a Rock and a Hard Place

or search for jobs directly.

Senior Research Associate — Position # 139601
Miami Dade College

Job Description: The Senior Research Associate conducts quantitative and qualitative research projects and produces routine ... see job

Clinical Research Coordinator
University of Colorado at Denver and Health Sciences Center-Downtown Denver

Posting Description: The University of Colorado Denver Cancer Center has an opening for a full-time Clinical ... see job

Blanket Posting Research Associate; Sr. Research Associate; Research Fellow; Sr. Research Fellow
University of Minnesota, Twin Cities

The University of Minnesota is a premier employer and a talent magnet attracting leading faculty and staff from around the ... see job

Technician III
Cornell University

Cornell University, located in Ithaca, New York, is an inclusive, dynamic, and innovative Ivy League university and New ... see job

Coordinator of Accreditation, Director of Assessment and Evaluation
James Madison University

Join one of the finest regional universities in the nation. James Madison University, home to 18,000 + students, welcomes you ... see job

Data Manager & Analyst (09-350)
Berklee College of Music

Berklee is committed to increasing the diversity of the college community and the curriculum. Candidates who can contribute ... see job

Institutional Research and Planning — Assistant Director
Western Illinois University

Recognized as one of the nation’s “Best Value” institutions and one of the “Best in the Midwest Colleges” by the ... see job

Director, Office of Institutional Accreditation, Strategic Planning and Effectiveness (OIASPE)
University of Colorado at Denver and Health Sciences Center-Anschutz Medical Campus in Aurora

Posting Description: The University of Colorado Denver is seeking nominations and applications for the ... see job

Renal/Cardiovascular Research Scientist
University of Colorado at Denver and Health Sciences Center-Ninth and Colorado

Posting Description: The candidate will pursue her or his own research interests and will be an independent ... see job

Assistant VP for Strategic Development
University of Texas, San Antonio

Description of Job Duties: Provide leadership in strategic expansion of UTSA research Develop new & expanded ... see job