Comments on Questioning College-Wide Assessments

• The Ideal Graduate
• Posted by JDW , President at American Academy for Liberal Education on June 26, 2007 at 11:25am EDT

• Dave,

  The disciplinary interests and performance of students is not what I was speaking of. One may become a biologist or lawyer without neglecting that part of education intended to teach the person as person, so that the biologist or lawyer doesn't become someone who can't write a decent paragraph (this is exactly what employers are complaining about). I was amazed to find one of my own sons (undergraduate Whittier, graduate USC)clueless about what The Renaissance was.

  As for the interdisciplinary college that lacks core requirements, it sounds like a special case, like St. Johns College, Annapolis(all four years made up of required courses). I certainly have no problem with them, or with interesting possibilities along the lines of Colorado College. Questions of public policy can only deal with generalities - whether hospitals or schools or armies or navys - not with special cases. Here we must be careful to respect the autonomy without which such institutions could not exist.

  The "ideal graduate" must, of course, be proficient in whatever he may have specialized in. But specialization has nothing to do with the core curriculum.

  Cheers,
  JDW

• Posted by George D. Kuh , Chancellor's Professor and Director at IU Center for Postsecondary Research on June 21, 2007 at 7:25am EDT

• To clarify, NSSE routinely surveys random samples of first-year and senior students. It makes sense to whenever possible disaggregate student responses by different characteristics (sex, age, race and ethnicity, place of residence and so forth) in addition to major for seniors to obtain more nuanced understandings of the student experience. Indeed, since its inception NSSE has encouraged institutions to either include as many students as prudent in the sample or target certain groups in order to learn more about them. Such approaches are especially important for guiding improvement efforts inasmuch as the variance in student engagement and many other activities and learning outcomes typically is much greater within a given institution than between colleges and universities.

• What do students talk about learning?
• Posted by Clifford Adelman , Senior Associate at Institute for Higher Education Policy on June 21, 2007 at 7:50am EDT

• . . .at the Thanksgiving family dinner table, your son majoring in anthropology is going to talk about kinship, ritual, and migration, among other topics; your daughter majoring in chemistry is going to tell you something about ferro-liquids; your brother-in-law at the local community college will talk about management information systems. This is all stuff that is directly taught, what students go to college to learn, what they are proud of learning, and the fundamental reason that institutions of higher education exist in all societies and economies: the distribution of knowledge. I'll put good money on the same tables that they don't talk about "critical thinking skills," or presentation skills, or other cognitive and symbolic fluencies that are not directly taught. So, you want to know what they learn? As I argued in the spring issue of 'Connections' (the quarterly of the New England Board of Higher Education), reinstitute the comprehensive exam in the major, and put the test you gave last year up on the Web so that anyone can see what you expect to hear at the family dinner table. Try it! You'll like it!

• Apples and Apples: Apples and Oranges
• Posted by Steve Ehrmann , Dir., The Flashlight Program at The TLT Group on June 21, 2007 at 8:35am EDT

• In calling for public display of exam content, Cliff Adelman makes an important point, and you don't need to embrace universal comprehensive exams to appreciate it.

  The problem he alludes to is this: how do you compare apples and oranges - different programs with different goals, strategies and constituencies. It's a familiar problem for those of us concerned with educational uses of technology, because ultimately new technology usually changes goals, methods and even the kinds of students attracted. So how do you decide whether the new program is 'better' than the old one. Suppose the new program is also more expensive than the old one? How do you decide whether the investment was justified?

  This same question can be posed about liberal education. After all, it's about liberation, about empowerment, about choice and individual differences. So any two departments or colleges will be qualitatively different in some ways - they'll be pursuing somewhat different goals and succeeding in differing degrees. So how can you compare them?

  One essential element, as Cliff Adelman points out, is that however you do it, the evidence, and the methods by which it was assessed, need to be public. Cliff's logic applies to eportfolios, as well: they too display the 'test.' If an institution, an accreditor, or a government agency compares institutions by studying such comprehensive exams or tests, the regulators need to be public about who the assessors were and how they analyzed the evidence. The process is not unlike the record of a murder trial, the highest stakes test of all.

  The larger point, however, is that we need both apples-oranges and also apples-apples comparisons because there are a few important ways in which colleges are all pursuing the same goals (no matter what college, no matter what major) and many ways in which the goals are different.

• What Empowers Faculty to Enhance Student Learning?
• Posted by Sean McKitrick , Assistant Provost for Curriculum, Instruction, & Assessment at Binghamton University on June 21, 2007 at 8:35am EDT

• I agree with the assertion that college-wide assessments can be misleading, if compared head-to-head, because institutional missions differ--but this is what happens when we use just one measure (or survey) to assess student learning.

  In my view, a more important aspect of this issue has to do with what universities--and most especially faculty--do with the information. The NSSE, as well as direct assessments such as portfolios and the CLA, can be used to assess to what extent different institutions are achieving their stated student learning goals and objectives, but a central question must turn to this: Are faculty and administrators discussing what various assessments have to say about the extent to which students are learning.

  We can argue ad nausiam about validity issues, but in the end--assuming we use various assessments--what matters is what we do with the information--do we enhance student learning, teaching, curriculum, etc., as a result of the process?

  Comparing institutions one to another is always going to be dicey--but insisting that institutions use whatever information they desire is more important, in my opinion, in leaps and bounds.

• Counting students with disabilities
• Posted by Nehansen , Special Education Advocate, docotoral student at The George Washington Univiversity on June 21, 2007 at 8:35am EDT

• When the data is gathered, in whatever manner, and by whomever, and for whatever ultimate purpose, are students with disabilities (SWD) figured in the count? If so how?

  Recognizing SWD first must self-identify, and then request their ADA/504 rights to equal access to evaluative/summative assessments of their collegiate experiences, are these assessments indeed accessible/universally designed to accommodate all students with all disabilities?

  Is there a means with which to disaggregate the data to see how the category of “students with a disability” fair?

  Then added to the mix—IHEs individually interpret ADA/504 in its policies/procedures and offer differing services through their Disability Support Services Offices. How can the researchers and their surveys say for certain that the academic experience is the same for all SWDs?

• Posted by Jim on June 21, 2007 at 9:10am EDT

• The title should read, "Questioning college-wide reporting." Collecting data across the institution is not the problem, it is how we decide to analyze and report the data. Institutions can certainly analyze their NSSE data by major if they so choose (and I'm sure many of them already are doing so).

• We're only looking at half of the issue...
• Posted by Aaron Brower , Vice Provost for Teaching & Learning at University of Wisconsin - Madison on June 21, 2007 at 10:00am EDT

• ...the other half being clarity in what our institutions are hoping to accomplish by educating our students. All of what's learned through these various assessment instruments and methods--higher-order or content-specific--have their purpose and value, but can only be judged by first knowing where we hope students will "be" when they graduate (and in fact, more often, only after several years post graduation once they hit their stride as citizens of the world). Without the institution's hard work to clarify its goals and aspirations for its students--and then stating these clearly and publicly--we're left arguing in the abstract over who's test is better.

• What's the bottom line?
• Posted by Doris Overton-Venable , Doctoral Candidate at George Washington University on June 21, 2007 at 10:40am EDT

• My motivation to pursue an advanced degree in higher education is directly linked to my experience managing college interns when I worked at a major corporation in the late 1980's. This Fortune 10 corporation accepted only the brightest, most promising students from many prestigous institutions across the country.

  Some of the students were indeed very bright, but sadly lacked the "connect the dots" (critical thinking) skills that employers expect of college-trained employees. A few of the students in my department had good communication skills but had difficulty going to the next level with appropriate quantitative concepts. On the other hand, some of the students with great quantitative skills demonstrated poor writing skills.

  After returning from a meeting, to my horror, I discovered an intern sitting on the floor doing needlework. Another took a three-hour lunch while her team was in the middle of a "crisis" and needed her help.

  Clearly, these are anecdotal, isolated incidents specific to one individual's experience. However, I believe that these incidents support the concerns from the proponents of standardized testing in higher education. All college graduates, regardless their institution's mission or their major should be proficient in writing in the English language, applying basic quantitative concepts, and critical thinking skills. Throw in some work ethic amd workplace etiquette and I think employers would have something to work with.

  That's the bottom line I think we expect of all of our college graduates.

• Posted by José Ricardo at Shippensburg University of Pennsylvania on June 21, 2007 at 11:00am EDT

• This is in reponse to Mr. Adelman's comments. I don't think that "the Thanksgiving family dinner" is the ideal scenario to put student learning outcomes to the test . Such simplistic views of assessment of learning are the ones that create "a fog curtain" around the true purpose of assessment. Mr. Adelman, irrespective of what a student majored in, that student should be able to think critically, apply analytic reasoning, and communicate accurately his/her thoughts both in writing and orally. These are the outcomes that assessments such as CLA intend to measure. I invite you to take a look at http://www.cae.org/content/pro_collegiate.htm. That being said, I think that parents paying big bucks for their children's education would expect their children to carry out a conversation (on their field of studies or on any topic)with professional/informed depth and breadth.
  Last but not least, conclusions on student learning outcomes must be based on triangulation from an array of assessment tools. CLA could be one of those.

• Enough already!
• Posted by Bob at State U. on June 21, 2007 at 11:05am EDT

• Testing mayhem now sweeps over higher education! And the camps rally 'round.
  Spellings/Gov't: Accountability means give us the numbers (or else).
  Testing industry: Accountability means give us the money and we'll get you the numnbers!
  U. Adminstration: Accountability means we'll make up our own measures - and look! We're doing a great job!
  Faculty: Accountability means you can't measure what we do so we'll tell you.
  Students: Accountability means we need jobs!

  If you insist on numbers, just give everyone the GRE general exam and (if you like) correlate it with SAT/ACT scores that every college has available. Not that I want to fill the imperial coffers of ETS any higher, but everyone knows the GRE already and if there are "drawbacks" with this approach, that's even better because such scores should only be a very rough indicator anyway.

  What would be useful: Uniform subject area exams (e.g. drawn up by professional societies) whose content is publicly available. These can cover multiple areas, from among which colleges can choose their own subset in which to examine - these subsets and their results should be public as well. Then we might learn something.

• Posted by ea caldwell , Director of First-Year Writing at The Ohio State University Newark on June 21, 2007 at 11:10am EDT

• The complex perspectives and the nuanced understandings of the nature and uses of information that are evident in most of the comments above, made not by legislators but by educators, illustrate quite powerfully why the study and regulation of higher education should not reside with governmental bodies, however sincere they may be. Compare the orientations to learning expressed here to the assumptions on which the Spellings report is based and it becomes even harder to imagine that any branch of government could effectively revise college education in any meaningful way.

• Posted by S. Cavote , Assoc. Dir., Assessment at U. of Nevada, Reno on June 21, 2007 at 12:20pm EDT

• Taking those first steps to achieve "scientific comparisons" [between institution or states] as Roger Benjamin assumes to be of some value will lead legislatures, state boards, and sloganizing politicians down the dangerous road to performance based funding. Is this anyplace any in higher education wants to go? Valuing what we measure instead of measuring what we value, a la teaching to the test(s) and other nasty alternatives.

  Our institution has administered NSSE and FSSE twice since 2003. We've published and presented these data to faculty, deans, & student services. As yet we've not seen either faculty or staff view the results as more than a curiosity. Without a strong commitment and interest by each institution's policy establishment in marginal activities such as assessment and program improvement, should we expect program's to do much else.

  My money is on the same table as Cliff Adelman's. Yes, employers want the newly hired college grads to possess higher order skills, but they want those skills in addition (beyond) the fundamentals students are expected to have learned via their major program. And they want those higher order skills to be discipline specific. Get real Gen Ed advocates.

• Doris
• Posted by Bob on June 21, 2007 at 3:20pm EDT

• I understand your concerns about an intern doing needlework...but I don't necessarily conclude that this is evidence of institutional failings to teach critical thinking. Perhaps, she just was not a good fit, or maybe she was bored because it did not challenge her thinking skills sufficiently, but after all the operative word is "intern"... Not enough information to come to a conclusion I am afraid.

• Testing service for employer
• Posted by Duncan on June 22, 2007 at 10:40am EDT

• I am in the opinion that the testing service companies should work harder to have employers back them up in using their tests. After all, it is employers that fulfill the value bill - in a realistic way.

  Industries had complained the quality of institution graduates with vague (a.k.a critical thinking) specifications without following through with specific skill requirement in terms of how they think it can be evaluated. The reason, of cause, can well be that evaluation of these high level skills is not their expertise. On the other hand, evaluating skill is what testing companies do.

  I think if industries is really serious about their concern, they should work with testing companies to further define their needs. Without well defined goals, no one can efficiently reach the goal.

  *If industries don't really know what they mean by 'critical thinking', I think all these are just FUD.

• Bob
• Posted by Dee from GW on June 22, 2007 at 11:05am EDT

• I am sorry you missed my point.

  There is so much at stake when institutions send their best and brightest students to show-case 'learning outcomes' to major corporations.

  Unfair as it may be, one bad fit or the student with poor writing, quantitative or reasoning skills, unknowingly tarnishes an institution's reputation. Some decision-makers use experiences from (yes) interns, especially seniors, to determine which institutions to include on the recruitment list for the next intern hiring season.

  Regardless of the current debate about learning assessment, employers still expect college-trained individuals to possess basic competencies that add value to their bottom line. Surely, academia can figure this one out.

• Defining the Problem
• Posted by JDW , President at American Academy for Liberal Education on June 22, 2007 at 2:20pm EDT

• However disconcerting it may be to witness the assessment tail wagging the education dog these days, it is nevertheless true that public accountability must follow public expenditures. But accountability for what, and to whom?

  To begin with, if there is a failure of accountability, it is NOT a subject-specific problem. Employers know full well which schools offer the best, or even competent, training in their fields. They are not, after all, complaining about the quality of the engineers, nurses, or physicists they hire.

  The problem is with general education. And it is not just because both general and liberal education are elusive and difficult to measure, though this is certainly true. Nor is it merely that an outcomes orientation tends to value testing more than learning, although that, too, is true. Measuring student success at the undergraduate level is complicated, if not made impossible, by the variety of what is being tested for, both between institutions and at the same institution. What is to be assessed, and how is it to be done, when faculty are no longer required to teach, and students are no longer required to learn, anything in particular?

  The problem we face is relatively clear in its origins: it began when American colleges abandoned their core curricula (these days, the number of required courses at our best liberal arts colleges can be counted on one hand). I’m well aware that, like everything else, the quality of core curricula can and did vary widely (I can hardly remember my own required Western Civ. course without falling asleep again), but, at least early on, the core provided common education goals. Without a common understanding of what one is trying to accomplish, measuring success or failure with regard to it is impossible. To the degree that it could be done at all, it would require defining success down to the lowest common denominator, rendering the exercise useless, or even counterproductive.

  If one can define what an educated graduate of any particular institution should “look” like, one can work backwards to establish the kind of education required to produce them. Once the institution’s pedagogy and curriculum have been brought into alignment with its mission it should not be difficult to assess how well it is working (particularly if, as suggested elsewhere here, both the goals and the means of assessment are made public). The fact that few faculties are even willing to raise, let alone answer, the question, “What is an educated man or woman?” is an indication of the practical obstacles standing in the way of reestablishing serious undergraduate education in this country.

  Jeffrey Wallin
  AALE

• the ideal greaduate
• Posted by Dave at USC on June 22, 2007 at 3:15pm EDT

• JDW: I cannot imagine what the ideal graduate should look like because they come in more flavors than Baskin Robbins. for example, my elder son decided in AP biology that he wanted to study genetics; he studied biology as an undergrad, a grad student and a postdoc, and he is now a prof. of molecular biology. My younger son is the opposite. He wants to study everything (as did and do I), so he is going to a small college that has no majors, is fully interdisciplinary, and one creates one's own program of study (with guidance, of course). He'll decide on a career after grad school.
  I couldn't possibly devise a curriculum to satisfy them both, but we have searched out colleges that let and help them do the various things that they want to do. Of course, they do come from an academic family and want to study. That helps.