Good Show Professor Fryshman

And while we’re sitting here in our easy chairs conversing, I suppose you’ll want me to recite my university’s mission statement? I ask that because I assume any conversation about institutional assessment or accreditation must start with and be couched in the language of that college’s or university’s mission, values, and guiding principles.

In that light, here are the defining principles of Bethel University as documented in our mission statement ...

“Boldly informed and motivated by the Christian faith, Bethel educates and energizes men and women for excellence in leadership, scholarship, and service. We prepare graduates to serve in strategic capacities to renew minds, live out biblical truth, transform culture, and advance the gospel ...”

“As a university, Bethel is positioned to carry out this vision as never before. To Christians around the world who seek the finest preparation for work and witness, the name “Bethel University” better conveys our broad scope of faith-based undergraduate, master’s, and doctoral programs ...”

“In short, Bethel University is taking the next step to change the world.”

“Rapid growth is both behind and ahead of Bethel University. Even so, our long-standing values remain at work in every class, chapel service, and campus experience, calling up and shaping the God-given potential of each student.”

Now, Professor Fryshman, our intentions are clear ... perfectly clear. I assume any assessment of Bethel will be customized to ask questions and assess our accomplishments in that context. After all, that its the raison d’état of Bethel University, and it should be central to our assessment and our accreditation.

Frizbane Manley, at 7:50 am EDT on August 7, 2008

We Do Assess Learning...Sitting and Standing

Bernard Fryshman has done a wonderful job of thinking through the reasons for why many common assessment approaches to student learning are not useful, especially most of the measures that campuses are currently being urged to adopt. However, we as faculty and academic professionals do not have a choice to NOT assess student learning and to share our findings — it is after all part of our work as educators.

Fryshman is also on point when he grounds the assessment of learning in the individual — of course, each student is our unit of analysis — bravo!

We at the Association of American Colleges and Universities are engaged in a pilot project with faculty and campuses across the country in developing an assessment process that relies upon the work of students through their courses and co-curricula that can be collected in e-portfolios (or portfolios)and assessed through shared expectations for learning as articulated in rubrics for the learning outcomes. This approach allows for individuals to be the units of analysis, the individuals can be aggregated to inform programmatic or institutional needs, and the students can judge their own progress in learning based on the shared expectations of their faculties.

Fryshman’s thoughtful piece leaves us without a course of action for being able to engage in assessment (which we have to do) in a manner that can be useful to faculty, students and others, AND provide valuable assessment of learning for diverse students, outcomes and contexts. Those alternatives are being developed.

Terrel Rhodes, AAC&U, at 10:30 am EDT on August 7, 2008

No cop-outs, please

Who said teaching & learning were simple? Complexity is not the point, and no informed person will argue that against the idea that different majors need different outcomes. Our collective problem in higher ed is that achieving the bachelor’s degree means anything one’s alma mater happens to choose it to mean. Years ago the book Cultural Literacy argued that educated persons have mastered a certain body of knowledge. When Jay Leno can walk around a campus at graduation time, asking newly minted degree holders about the hometown of Shakespeare (Huh?) and Fred Flintsone (Bedrock!), then we know we have a problem. We owe it to students, parents, & taxpayers to state clearly first what a college graduate is to have acquired in knowledge, skill, and ability, and second what means of verifying (assessing) that learning we have taken and what proof we have collected that the degree-holder merits the award. Sure there’s a million variables — for that matter, a competent cobbler is going to tell you the same thing about making shoes. So let’s not condescend just because our ‘product’ is better people rather than something tangible.

An Old Goat, at 10:40 am EDT on August 7, 2008

Yes, but...

OK, let’s stipulate that those who work in academia are artistes whose work cannot be measured by bureaucrats holding rulers.

But let’s also admit that there are some professors, and some institutions, that are doing a damn poor job.

So, how can those — students and taxpayers — who pay for higher education know that their dollars are being well-used?

Prof. Challenger, at 12:10 pm EDT on August 7, 2008

MISSION STATEMENTS

An anthropologist, I am interested in a range of contemporary magical practice including much of today’s assessment ritual both within academia and beyond. But today I’m interested in the origins of the term “mission statement” (which sets the bounds for such magical assessment). The OED’s earliest usage dates to 1967, in a sentence about the military—which one might suspect is the model bureaucracy for assessors. Is there better data about the cultural history of the mission statement out there somewhere?

Lamont Lindstrom, Professor at U. Tulsa, at 1:35 pm EDT on August 7, 2008

Had It Up To Here ...

Okay, here are my basic assumptions ...

1. At least within the boundaries of the United States, a global program of assessment/accreditation (A/A) will be forced upon us (higher education) ... and the stated rationale for it will be “accountability,” whatever that means. A/A is a made-to-order flavor-of-the-month that satisfies a great many politicians and higher ed research types. That’s a potent force, but, fortunately for all, A/A will be subsumed by the next flavor-of-the-month well before 2015.

2. There will be an intellectually vacuous debate about (i) the knowledge constructs that should be defined, (ii) the collection of measurable indicators for each of the constructs, and (iii) the actual measurements that will form the basis of our analysis. It will be decided that “higher-order” learning objectives cannot be encompassed by such a system (the measurement procedures will be technically difficult, and the measurement process will be very expensive on top of that), so we will concentrate on the most trivial of learning objectives (e.g. we can test knowledge of multiplication tables, but “the extent to which we have increased the student’s commitment to life-long learning” will be beyond our grasp).

3. There is no question that the unit of measurement will be the individual student, but that presents three very difficult problems. First, how do you aggregate the multivariate dimensions of each student – after all, each student is a bundle of hir knowledge of mathematics, literature, biology, political science, etc. – to get an overall measure of knowledge for that particular student? Second, how do you aggregate information about students’ knowledge to obtain an overall measure of excellence of the university at which the student matriculates? Okay, you can separate the dimensions of learning at the student level and aggregate them over all students (say, by averaging). But then you have to aggregate those measures at the university level. I suppose U.S. News & World Report will tell us how to pull that off. Whew!

4. And continuing from 3, third, is it the student’s “absolute knowledge” at the conclusion of hir undergraduate experience or the so-called knowledge value-added that will be, somehow or other, aggregated to tell us whether we’re getting our money’s worth from our colleges and universities? I’m wagering that we’ll side with the elites in this matter. Poor Shenandoah Valley University argues, “Our graduates aren’t wonderfully brilliant upon graduation, but they learned a lot during their undergraduate experience here.” The elite Dukemouth, however, says, “That’s unfair! Our incoming freshmen are brilliant to begin with, so expecting us to add a great deal of value is unrealistic.” [Aside: Colleges, college faculty, and students will swiftly see the wisdom of abandoning higher-level objectives in favor of “teaching to the tests.”]

5. I wish I could tell you how this would affect the ubiquitous grade inflation problem, but my guess is that it will have little impact. While our legislators will be Hell bent on institutional accountability, they will go to great lengths – even by passing legislation – to maintain the confidentiality of students’ scores.

6. Either the federal government will assume responsibility for this A/A program or else “someone” – and I think you know who that is – will pay the regional accrediting agencies who will subcontract the task to the College Board to pull this off.

And here is my solution: In the spirit of A/A we have ...

THE MANLEY PLAN: We will convene nine panels of experts, one each in mathematics, the physical sciences, the earth sciences, the biological sciences, philosophy, world literature, world history, geography, and civics and government. Each panel will construct a test bank of between 1,000 and 1,500 questions, and, with all of their college or university requirements satisfied, students who hope to graduate at the end of a certain term will be required to “pass” a test of 50 randomly selected questions from each of the test banks, and with a grade in excess of 65% on each test. A tenth panel will grade a three-page essay in which the prospective graduate’s vocabulary, spelling, and composition will be judged. At this point, each student presenting hir credentials for graduation can be characterized as a point in a ten-dimensional Euclidean space (each dimension is a score on one of the ten “tests”).

Subsequently, we can (1) “aggregate” the ten scores for each student to obtain a measure of that student’s knowledge and (2) “average” the scores, dimension by dimension, over all students at SVU to obtain ten numbers that will then be aggregated by a polynomial objective function that will capture SVU’s overall educational excellence. [Aside: When it comes to defining these aggregation formulae, I assume we will encourage USN&WR, the NCAA’s BCS committee, and Halliburton to submit bids for the contract.]

Oh thank God, we’ve finally got all of those A/A gurus off our backs. And by implementing their recommendations the world will be sooo much better. By the way, Margaret Spellings, my consulting fees are clearly within the Department of Education’ feasible set ...

http://www.chinadaily.com.cn/engl...11/22/xin_5211012213084351348816.jpg

Frizbane Manley, at 2:20 pm EDT on August 7, 2008

re: Responsio

Missing the point"There are two reasons why taxpayers can’t just use the usual “consumer reports” method for knowing whether their education dollars are being well spent:

1. Most taxpayers aren’t the consumers. The people making the decision to use the “product” (ie, the students) are not always the same as the people paying most of the cost (ie, the taxpayers).

2. Information is lacking. That’s the whole point of the assessment debate. The quality of the product either can’t be measured or isn’t being measured.”

Please be serious. There is no end of information available on how various colleges contribute to the economy in various ways, what gets taught, how it gets taught, how large the classes are, the size of the library, the graduation rate, what the college’s graduates do afterward, etc. All it takes is a modicum of effort on the part of whoever pays the tuition, and if something is not available, doubtless the information is a phone call away. And if the university chooses not to divulge whatever it is you want to know, perhaps that is an indicator that your tuition dollars might be spent elsewhere.

However, this detract from my main point: students, not professors, are at least fifty percent responsible for their education, and the “shoemaker” analogy elides that fact by reducing students to passive objects.

Peter C. Herman, Prof. at SDSU, at 3:25 pm EDT on August 7, 2008

To Lamont Lindstrom ...

I can’t tell you much about the origins of the phrase “mission statement,” but I’ll tell you where to look ... and please understand that this is my “quick-and-dirty” analysis.

W. Edwards Deming was born in 1900, and he received his Ph.D. in mathematical physics in 1928. He was, first and foremost, a statistician, especially well known for his work in sampling. In the late 40s the U.S. Army “sent” him to Japan to help the Japanese with their 1951 Census. I think it’s safe to say that he knew quite a bit about statistical process control at the time, but had, at best, a very rudimentary “theory” of continuous quality improvement.

In 1950 Deming was invited to address the newly formed Union of Japanese Scientists and Engineers (JUSE) about statistical process control ... and the rest is history (remember when “Made in Japan” was a slur?). Deming did not invent quality, and he was not responsible for the quality revolution in Japan; but no one contributed more than he to the Japanese quality transformation from what it was then to what it is today. I know for a fact that in the early 60s he was using mission (vision), values, and guiding principles precisely as they are used all over the business and manufacturing world today.

Difficult as this is to believe, it was 1980, thirty years after he began influencing the continuous quality improvement of processes and products in Japan, when the NBC documentary (created collaboratively by Deming and Clare Crawford-Mason), “If Japan Can, Why Can’t We?” came to the attention of Don Petersen and other executives at Ford Motor Company. They were captivated, and, thereafter, Deming spent most of his time and energy helping American companies and influencing American academics.

I recall his asking a huge collection of the highest level of executives at Ford what their mission was. “What are you doing? What’s the point?” As it turned out, they thought their mission was to make cars and trucks. I know you’re eagerly awaiting my prejudice that the Deming message never took at Ford. It was there for awhile – maybe ten or fifteen years — but it was never seen to be essential ... and that’s the primary explanation for why the former big three have become the pathetic shadows of the companies they once were.

Not that I ever believed it, but when Billy Ford was CEO of his great grandfather’s company, he used to boast – even in commercials – that gasoline surged through his veins. Deming would have cringed at that. It should have been customer satisfaction that was surging through his veins.

Anyway, watch ...

http://www.youtube.com/watch?v=GHvnIm9UEoQ

http://www.youtube.com/watch?v=mKFGj8sK5R8&feature=related

And, if you don’t mind, please don’t mention six-sigma quality (ugh!) in the context of this discussion.

Frizbane Manley, at 5:05 pm EDT on August 7, 2008

To R.J. O’Hara ...

Needless to say, I’m willing to live with the very imperfect evaluation/assessment of student learning that all of us do on a day-to-day basis. And I assume we all use our own personal continuous improvement strategies to revise our teaching in a manner that enhances our students’ learning (not to mention enhancing a great many other student characteristics besides learning).

On the other hand, since the A/A folks seem to be Hell-bent on our pursuing measures of institutional excellence, I suppose you can tell (from my first post) that I think we must do that within the context of our college’s or university’s mission statements. That said, here is one of my favorite mission statements.

Matthew 28 (Jesus to his disciples) ...

“Then Jesus came to them and said, ‘All authority in heaven and on earth has been given to me. Therefore go and make disciples of all nations, baptizing them in the name of the Father and of the Son and of the Holy Spirit ... and teaching them to obey everything I have commanded you.”

Now that’s a mission statement we can get our teeth into! Can’t you just see those Alverno College institutional research folks running around asking, “been baptized?” ... “you a disciple?” ... “you a disciple” ... “been baptized” ... “you obeying?” ...

Frizbane Manley, at 8:25 pm EDT on August 7, 2008

Speech and PowerPoint

I assure you. My tongue was entirely in my cheek. As a “rater,” I’ve had to “assess” many a Capstone Course project in General Ed.

Malvern Hill, at 9:55 pm EDT on August 7, 2008

Response To A.D.

First, Deming didn’t say anything about young Billy Ford. He had passed to his everlasting reward by the time Billy elbowed his way into the CEO position at Ford. I only conjectured that that’s what would have popped into Deming’s mind if he ever saw those silly commercials of Billy boasting about having gasoline coursing though his veins. I think you’d have to go back to Bob Lutz and Lee Iacocca to find a recent car guy whose heart actually pumped gasoline.

Second, Deming did not advocate interrupting a process (e.g., an assembly line) in conjunction with employee satisfaction (at least not directly). He advocated interrupting and repairing a process under certain circumstances when the process was “broken” and producing defective products or services; i.e., products or services that would not satisfy CUSTOMERS’ needs and expectations.

Third, to answer your question about mission statements — and the writing of mission statements has evolved into creation of paragraphs that are much more half-baked advertisement for the “firm” than they are statements of what the “firm” is all about – I must oversimplify ... and by a long shot. The essence of any “firm’s” mission statement must emphasize what it does to insure customer satisfaction.

Here’s an example: “The mission of AACSB International is to advance quality management education worldwide through accreditation and thought leadership.”

You’d have to go some to find a mission statement more nonsensical than that.

Fourth, Deming fully believed the principles of continuous quality improvement could be applied – and applied very effectively – to academic institutions.

Fifth, I don’t know how many of Deming’s fourteen points actually make sense – maybe between 5 and 8 — but he advocated enough operational nonsense to (1) enable more than a few upper-level executives to roll their eyes and scratch their heads when they considered his recommendations and (2) constantly keep his disciples on the defensive. Whatever you do, don’t waste your time reading his classic, “Out of the Crisis.”

Sixth, I like your George Bernard Shaw quotation. I’ll definitely use it myself.

Seventh – and desperately trying to get back on subject – here are a couple of W. Edwards Deming quotations ...

“The most important things cannot be measured.”

“Experience by itself [without theory] teaches nothing.”

“The problem is at the top; management is the problem.”

http://www.youtube.com/watch?v=7UmigSCSico

Frizbane Manley, at 8:20 am EDT on August 9, 2008

Missing Dimension

I’d be glad to volunteer to help Frizbane vizualize his multi-dimensional space, but it needs to have 11 dimensions if it is to be nearly as hip as String Theory.

The missing dimension is a genetic and/or cultural predisposition toward alcoholism, binge drinking, use of controlled substances, etc.

If schools will be held to account for their students learning more than beer pong (which can now be done on a Wii), they might want to have a Alcohol Aptitude Test as well as an SAT or ACT score as part of the admissions package.

CCPhysicist, at 7:00 am EDT on August 10, 2008

Good Questions A.D. ...

First, Deming did not fall back on the standard economists’ cop-out when he tried to characterize customer satisfaction. When economists don’t understand customers’ decision mechanisms – which is most of the time – they always attribute it to the customer maximizing hir “utility” ... whatever that means. I suppose when I purchase a blue (not a white) Prius (not an Expedition) with leather (not fabric), a hands-free phone capability, nine speakers, wireless technology, and a voice-activated DVD navigation system, my decision must have been based on nothing more than my eagerness to maximize my utility. What else could it be?

Utility maximization is so powerful I’m certain it also explains why I’d prefer to hike on the Appalachian Trail this afternoon instead of stay home, bubble some Johnny Walker Red, and watch Michael Phelps win more gold medals; have sex with redheads instead of with blondes; attend Harvard instead of Notre Dame; major in mathematics instead of political science; vote Democrat instead of Republican; etc. I’m just a glutton for those utils, aren’t I? Sadly, I can’t will them to my son, but, fortunately, I don’t have to pay taxes on them either.

In Deming’s view, no process made the least bit of sense unless it satisfied customers’ needs and expectations. Damn, are those words just synonyms for utility? If it’s a system (network of processes) we’re talking about and if the output of the system is a nice shiny automobile; then the customer’s satisfaction is probably related to cost, style, fit-and-finish, safety, performance, durability, longevity, prestige ... lots of additional dimensions ... and don’t you just hate it when we’re forced to view things in a multidimensional space?

And to make matters worse, it’s even difficult to tell who the customer is. I suppose in education, the primary customer is the student. But what about the student’s parents (they’re the ones with the big aspirations, and they’re usually footing a significant part of the bill)? But what about the student’s subsequent employer(s)? But what about the society in which the student will be a citizen (it’s also footing a big part of the bill)? But what about the discipline to which the student will possible make practical or intellectual contributions? And I’ve just gotten started ...it’s all just too complicated for me. And, by the way, if we’re going to do anything at all – like advocate assessment and accreditation (A/A) we had damned sure better be prepared to specify our domain of definition, our objectives, our customers, our system (in terms of a network of processes), and then go on from there.

Granted, I have done little more than say, “This is an exceptionally difficult issue.” But when you read the comments in blogs like IHE’s – and I’m a frequent contributor myself – you would get the impression most academics think these challenges are trivial (or unnecessary) to define and are even easier to resolve. Think about it for a moment and I’ll bet you’ll agree with me that more than 90% of the disputes perpetrated on these pages are a result of (1) differences in barely defensible political prejudices or (2) failure of the discussants to converse within the context of a common domain of definition (the curse of the social sciences).

Second, remember the good old days when the former Big Three (G.M., Ford, and Chrysler) couldn’t care less about customer needs and expectations. Each manufacturer gave us a different assortment of cars, year after year after year (planned obsolescence) ... because they thought they knew better than their customers what they wanted and needed. Then along came the new Big Three (Toyota, Honda, and Nissan) to turn that model upside down. Hmmm, and how has that turned out?

What strikes me as being interesting – and make your “customer” some amalgam of the customers I mentioned above – American higher education is very much like the former Big Three. “Heavens, we’re certain we know much, much more than even our customers what they want and need ... it is in no one’s best interest for us to create a dialog for the purpose of getting us (the suppliers) and them (the “customers”) on the same page. We know what’s good for them. We’re delivering it. Take it or leave it!

Got to love higher education in these United States.

P.S. Am I the one who apologized above for my blah, blah; blah, blah, blah on this subject? And defending Deming? ... who would have thought?

Frizbane Manley, at 4:05 pm EDT on August 10, 2008

Oh, and Citizenship!

That cubes it.

A.D., at 5:00 am EDT on August 11, 2008

Dear Colleague:

I am hurt. You accuse me of having “a diminishing nickname for everyone.” I don’t think that’s at all true; but if it is, I wish I had some examples so I could try to avoid that in the future.

Now about the Bethel University mission statement. I’m quite certain you completely misinterpreted my intentions. First, I’m virtually certain I did not distort the BU statement of its mission at all (are you apologizing for it because it’s so expansive?). As mission statements go, I think BU’s is close to excellent.

Second, two of the things I dislike about A/A are (1) it generally proceeds in the absence of knowledge of the university’s mission — what I called Bethel’s raison d’état — and (2) it generally focuses attention on lower-level objectives to the exclusion of what most of us think are the important objectives of higher education (read what I’ve written above). I chose the Bethel mission statement because (1) I think it is very unusual, (2) I believe for assessment to proceed in the absence of the college’s or university’s mission is absurd (the school’s mission must be central to the A/A process), and (3) I wanted to challenge any advocate of universal A/A to tell me what their constructs, indicators, and variables would be if Bethel were the object of hir analysis.

P.S. This is the third time I have used Bethel’s mission statement to make a point only to have someone from BU criticize my perspective. One does not have to embrace an entity’s mission in order to respect it. I am an atheist; so, according to item 12 in the Bethel University Statement of Faith, I will spend the hereafter in endless suffering (I’m classified as one of the “wicked”) – and “endless” is a lot longer than a trillion trillion years. Even so, I can appreciate and respect the BU mission. Is it possible that you guys are overly sensitive (and defensive) when it comes to an external (to the University) discussion of the statement of that mission?

Frizbane Manley, at 10:00 am EDT on August 11, 2008

Oops ...

By the way, Colleague, here is where I got the information about Bethel’s mission ...

http://www.bethel.edu/about-bu/mission-vision.html

Frizbane Manley, at 1:15 pm EDT on August 11, 2008

History of Mission statements

The history of mission statements needs to be described for context.

Institutional mission statements came into existence as an innovation started by the North Central Association in the shadow of the Great Depression (1934) as an effort to help ailing institutions struggling to keep their doors open.

Faced with an impossible situation, the NCA regional accreditor devised a total institution pattern map that would visually justify keeping subpar institutions open.

This “total institution pattern” soon became the mission as we know it today.

It is something that is intentionally difficult, if not impossible to quantify, as was originally intended. The other reason we should not be surprised at how difficult it is to quantify is that it is primarily a rhetorical construct used to persuade and legitimate.

Glen S. McGhee, Dir., at FHEAP, at 5:05 pm EDT on August 11, 2008