CLA

Learned Colleague,I need your opinion.

Representing Springfield College — a new member of “The CLA Consortium,” I just returned from Raleigh, NC where I attended a CLA conference-workshop.

When I reviewed and scored a CLA retired or sample Performance Test — a scenario dealing with crime prevention — I was rather amused. (Had I gotten some sleep the night before and on my flight, I would have been shocked.)

Admittedly, I examined and scored only part of the test — “The Performance Task.” However, I found it too demanding for most college students. I informed the CLA presenters — who’ve done post-doctoral work but who’ve done little or no classroom teaching — that the CLA exam will surely make a great assessment tool — not for students — but for new faculty hires!

What are seasoned classroom professors saying, if anything, about whether the CLA Exam is too rigorous for college students? (I dearly hope that my institution never decides to give the CLA to the faculty as a pre-condition of their continued employment.)

I’m not concerned about the other objections CLA critics have raised. Rather I’m asking whether the CLA is properly geared to assess realistically and fairly a student’s cognitive and communicative skill sets?

My best...jim ragonnet

james ragonnet, Professor of English (rhetoric) at Springfield College, at 11:15 am EDT on October 25, 2008

And what of the employers?

Nice, well-meaning words. Notes from the employment front-line —

* Is the work in the portfolio, truly the student’s? Is she/he prepared to reproduce it, on the spot?

* As to “standards” and “rubrics” in disciplines — thanks, we’ll pass on group-think and autonomic responses.

We’re more concerned about basic skills (e.g., writing, reading, math) and common sense (table manners, politeness, work ethic, time management).

L.L. Berry, at 8:16 am EST on January 26, 2007

It shouldn’t surprise anyone, other than government bureaucrats, that standardized (IQ) testing doesn’t work very well in measuring the value of a college education. Native intelligence is just that, and it doesn’t change much. And to the extent that it does seem to increase, it’s probably due to an improvement in basic reading and math skills brought about by practice or remedial studies, not by learning, say, political science. What’s being measured in an IQ test is not knowledge, and knowledge is the main thing a college or university is selling. That, and a good time.

Putting more programs on a computer makes it more useful, but it doesn’t make the computer any “smarter.”

Now, if we really want to measure “value added” by a traditional college education, we need to be looking at job placement rates, like the private career schools have been doing for years. Such measurements and comparisons are not without their own problems, but if one assumes that there is a relationship between knowledge and employability, then at least we would be measuring something that education can impart, and not something that it can’t.

Glenn Bogart, at 10:05 am EST on January 26, 2007

A balanced approach to assessment

At Syracuse University’s School of Education we are using Sakai and the Open Source portfolio as a tool to develop a platform that will allow the balance between coursework and testing and student centered portfolios to as a balanced approach to assessment and program improvement. I worry about too much of either approach at this stage, but a blend of assessment strategies certainly would seem to be best suited for capturing the breadth of student growth and effectiveness of the institution.

Sean Keesler, Project Manager, The LIving SchoolBook at Syracuse University, at 11:05 am EST on January 26, 2007

Subject Testing

We’ve been under some pressure here to use ETS subject tests on our seniors. When I say “under pressure,” I mean that — against all expectations — the administration is willing to pay for the testing without taking it out of our budget, and was pretty insistent even after we pointed out the serious disjunctions between our program and the ETS instrument.

Jonathan Dresner, at 1:21 pm EST on January 26, 2007

Standardized exams work well in S&T

By “works well” in the sciences or specifically chemistry means either you know thermodynamics, reaction kinetics, organic mechanisms, statistics, how to interpret mass spectra or you don’t. If you are to be competent chemist in industry, academia, or government labs at the BS, MS PhD levels you must have requisite knowledge. At some point any university’s grads will be measured by against others in the competition for jobs.

Would you consult a physician who failed their board exam? How safe do you feel if you know the new 1/4 mile twin span bridge in your county was designed by a team of engineers who failed the PE exam repeatedly? How seriously do you take the legal advice of someone who failed their bar exam?

Standardize exams are a good idea for SOME disciplines.

Frank, at 2:55 pm EST on January 26, 2007

Standardized Exams in S&T

Yes, my point was to make it perfectly clear that discipline specific exams in S&T (technology) are appropriate and really nothing else.

We need to have nation-wide standardized exams in S&T applied ASAP as my personal experience is that the university GPA has become a meaningless value of late.

Frank, at 4:55 pm EST on January 26, 2007

As to “common sense"...there is no such thing. What we consider common sense is learned sense, definitely cultural, often classist.

It’s nice to know all of this about S&T, where I was situated before changing to the arts, but how to test in the humanities and arts? How to figure one’s gains? How to figure, even more importantly, the role of the university versus the role of any particular professor? And how to assess the student who learned despite the teaching, despite the poor situation? Indeed, how do we assess the role played by direct experience (extra-collegiately)?

Could it be that we are looking in the wrong place to come up with standards of measurment?

Job placement and education do not correlate because one must always take into consideration social conditions, things which “we” cannot control...like, unemployment, need, availability, likes/dislikes of hiring bodies, politics.

Thought: considering the success of the MMPI and its variants, could not this style/form be adapted to educational needs? Sometimes, as with alternative teaching techniques, it is a good idea to NOT focus on what is wanted (being taught) but on the characteristic that is common.

If you don’t know where you’re going, if you don’t know what you want, how can you test for it? “An accepted level of educational attainment” (or whatever phrase) is totally meaningless.

James L. Secor, Ph.D. at Sun Yat-sen University, at 7:50 am EST on January 27, 2007

Employers II

I suggest you check your spelling prior to throwing stones.

When did universities become responsible for teaching table manners, and being polite and respectful? These were values formed in my home. I expect my students to come to my institution with these skills fully formed.

Nicholas R. Santilli, Director, Planning and Assessment at John Carroll University, at 11:25 am EST on January 27, 2007

Of course

U.S. students don’t need SATs, GREs, GMATs, MCATs, LSATs. The U.S. is this great super-power that dominates over other economies. And the U.S. is not a debtor nation to communist powers. And employers will never administer an on-the-spot exam of basic skills.

And everyone wonders why those who think so highly of the non-standardized approach to evaluation — since it is so superior — why they don’t start their own private consulting services? And make them and their clients millionaires?

And when those students go to other countries, they don’t need their manners corrected. And their grammer.

L.L. Berry, at 8:00 am EST on January 29, 2007

I hope that this essay, or others like it, get circulated widely because Prof. Banta makes excellent points.

To address some comments about portfolios: “Oh, and re. the portfolio thing...you have a lot better chance of having some confidence that a student prepared an electronic portfolio these days than you do that they wrote any given paper.”

Yes, that’s one of the advantages of portfolios over tests such as the CLA. Every school that I know of that has administered the CLA has reported problems getting students to even take the test. The biased samples of CLA test-takers are a more potentially severe problem than verifying the authenticity of portfolios.

“It’s just that nobody knows quite what to do with a portfolio, much less umpity thousand of them.”

Two answers here: for assessment purposes, you don’t need or want to look at ALL of the portfolios. You simply select a random sample of them. Hence unlike the CLA and other tests, there’s no problem with sample bias. As for what to do with the portfolios, that’s an evolving process, but consortia of schools providing outside readers to each other, to read each others’ portfolios using rubrics, is a possibility which comes readily to mind. Some work has already been done in this area, and a small handful of schools have all along had outside readers for their senior theses or honors theses. (Not that we’d want every senior thesis to be read by an outside reader, but it’s an example of direct assessment of student work.)

Mike Tamada, Prof. Banta’s essay should be circulated widely at Occidental College, at 4:10 am EST on January 30, 2007

“Who pays for this? Are faculty supposed to be investing significant amounts of time in an activity that does little or nothing for them professionally”

To be sure, few schools or profs will *want* to spend time assessing portfolios. But, if we can get portfolios to be used as a “direct measure” of student learning, rather than tests such as CLA, MAPP, etc., then schools and profs will be doing it all right.

Because they will HAVE TO. The Feds have made it clear that the accreditors have to require direct measures of student learning (ie. not indirect measures, ie. not students’ self-reported assessments of their education).

Accreditation is typically an activity whose cost runs into at least six figures, when you count the time of administrators and faculty. So yes, this scenario of using portfolios is not cheap — but neither is re-accreditation.

My guess is that this assessment of portfolios wouldn’t have to be done every single year, although that might not be a bad idea. Once the infrastructure is set up (admittedly, not a trivial task), then snagging say 10 faculty for a day of portfolio assessment could be done at a cost of several thousand dollars. For comparison, administering the CLA costs $6,300 or more.

Mike Tamada, Occidental College, at 12:10 pm EST on February 1, 2007

Assessment and analysis is QA

Faculty are responsible for delivering high quality instruction and education. Students are responsible for utilizing what the faculty and institution provides them. Assessment of learning is difficult to be sure, however, there are a number of valid approaches. Student outcomes are certainly not perfectly correlated with instructor effectiveness but there is a correlation. In many studies it rises to r=0.7. This is sufficient to identify _patterns_ in teaching style, course content, delivery mode, student attributes...etc that are more likely to result in student learning. In another article someone made a comment student learning is unlike creating a bottle of wine or scotch and certainly opposite to manufacturing a widget. Learning is very subjective and ellusive in many respects but successful academic progress certainly holds substantive and identifiable characteristics.

Cheers,

Kelly

Kelly Deal, at 2:45 pm EST on March 9, 2007