odds and ends

“Well, that’s easy for him to say. Binghamton is a flagship university in the SUNY system. It can pick and choose from among the best students across New York State and nationwide, most of whom come from relatively well-off backgrounds and enroll full time, living on-campus or nearby. Binghamton’s median SAT scores are high, funding levels generous, and scholarly reputation strong, leading U.S. News & World Report to rank it as the 37th best public university in America — sorry, 34th best, up three from last year, which Binghamton proudly announced on August 22nd. In a press release.”

Kevin, I am pretty sure you just made his case for him. I believe you just acknowledged that colleges, there students and missions are all different and that variety is something that has to be taken into account.

How then are CLA scores valid, if they can not be used in a standardized, objective way because they are measuring different conditions?

Observer, at 8:25 am EDT on September 4, 2008

Here’s what’s wrong with the CLA:

It attempts to measure “critical thinking” in a way that is more sophisticated, to be sure, than other standardized tests, but is still far more simplistic than the critical thinking students need in the real world. In the real world, problems are ill-structured; information is incomplete or contradictory; choices come with unknown consequences; competing interests skew information with great skill; tacit values and assumptions influence thinking. Making sense of the mess of the real world — and arriving at the best possible decisions given available information — is what students must prepare for. It requires the ability to think in complex ways, a desire to seek available information, a critical eye for evaluating sources, an ability to look at problems from multiple perspectives, and a mind open to re-evaluating one’s position when new information becomes available. If you’re familiar with King and Kirchener’s work on Reflective Judgment, the goal is Level 7 thinking.

The CLA, by contrast, presents the student with six simply-crafted documents (not real documents from the real world) and asks them to pick out the flaws in reasoning carefully planted in the documents, like children searching for prizes in a sandbox. It is not messy or complex, because messy and complex problems are impossible to score on a mass basis.

What’s the problem with testing some basic, lower-level skills if the higher-level skills cannot be tested? The problem is that institutions with success stories, like UNO, will brag about their scores, and other institutions will want their students, alumni, boards, and legislatures to hold their heads high with pride also, and they’ll begin pouring resources into raising those scores on simplistic measures, leaving the hardest and most important work of the faculty — getting students to really think — high and dry.

It’s happening all over K-12 education. Don’t think it won’t happen in higher ed.

I’m not saying that quality assessments of higher-level skills cannot be done, just that it will require a whole lot more effort (and probably expense) than a standardized test.

Critic of CLA, at 10:10 am EDT on September 4, 2008

To Critic of CLA

Critic of CLA, your critique is spot on. Where we part ways is where you throw the baby out with the bath water. It doesn’t follow that we must not use assessment tools because they are incomplete or imperfect. They will all be flawed because there is no “right answer” when it comes to critical thinking. But the onus is on higher education to demonstrate it is successful. We purport to teach critical thinking skills, but there is scant evidence that we do so successfully on a broad scale. If higher education doesn’t take on assessment itself, then it will be foisted upon us from the outside. Given my druthers, I’d prefer to put all pretense aside and just admit that higher education is a credentialing system that ensures a modicum of knowledge and ability among those who successfully make it through.

IHE Reader, at 11:05 am EDT on September 4, 2008

CLA problems

This article does not indicate how the CLA has been administered at UNO -which is a shame, as it’s been poorly used at some schools. The CLA is designed to be given to the same students at the start and end of their college careers, thus measuring what they have learned. But lots of schools have found that too cumbersome and have instead given it to new students and seniors in the same year, and then contrasted those two sets of data, as if they in anyway measured what students have learned. That’s a terrible methodology as it just gets snapshots of what two groups of students know, rather than measuring what a group has learned over time. CLA is also cumbersome because as it’s not tied to any course assessment, students must have some inducement to take it — usually a payment of money for seniors — and that is a disincentive for schools to use it properly. A shame that Kevin doesn’t reveal whether UNO used it correctly or not.

Mark, at 11:20 am EDT on September 4, 2008

“Critic of CLA, your critique is spot on. Where we part ways is where you throw the baby out with the bath water. It doesn’t follow that we must not use assessment tools because they are incomplete or imperfect.”

I agree that most of CLA Critic’s comments are correct, and I also agree that it IS a good idea to use assessment tools.

But CLA Critic still has a good point, in that the tools must be used correctly. And an example of an incorrect use would be, as described in CLA Critic’s original posting, schools boasting about their high CLA gain scores, leading to other schools taking steps to raise their own scores so they can start boasting, and the competition is on.

Which would be fine if they schools were competing to educate students betters. But higher CLA gain scores do not equate to better education. But once schools started getting ranked by CLA scores, that crucial subtlety would be lost, and the arms race would be on, with schools teaching to the test.

Stop the arms race now. Use CLA (and NSSE) results responsibly, for internal improvement rather than for external public relations.

Would you choose a high school based on how well its students do in driver’s ed and in road tests to get their driving license? Those drivers’ tests are even better than CLA at measuring what they are trying to measure — but it’d be a dumb way to rank high schools or to choose high schools.

mkt, at 9:05 pm EDT on September 4, 2008

When I was teaching at Truman State University in Kirksville, Missouri (1987-1996), it was during that decade the school gained its regional and national reputation for “value-added” liberal arts education and was touted far and wide for its gains in student learning. Conversely, it was also discredited at the time with the same reasoning of those detractors who dismiss the CLA and UNO’s #1 ranking.

No one assessment is perfect, and no one measurement should be used to peg the totality of any school’s “value-added” gains.

Those institutions who would chase after prestige to be selected “Best” in the US News and World Report rankings (and you know who you are)are the same ones who will sniff hypocritically at schools who earn distinction in another “contest.”

Once when Truman State had been featured in the national media for its assessment model and value-added gains in learning, the President of Notre Dame at that time sarcastically criticized Truman’s recognition by saying, “At Notre Dame, we don’t need to test our students to know if they are learning. We know.” Oh, really.

David Cicotello, Director, New Student Enrollment Services at University of Nebraska-Omaha, at 4:00 pm EDT on September 6, 2008