June 20, 2006

Digital Assessments

Assessment is quickly becoming the new black. It’s one of the themes of the Secretary of Education’s Commission on the Future of Higher Education. More and more institutions, some prodded by accreditors, are looking for rigorous ways — often online — to compile course data.

Looking for a job?

See all 298 new postings

Browse all job listings:

Faculty: 3,608
Administrative: 2,284
Executive: 200
FEATURED EMPLOYERS

The prospect of online evaluations and assessments, for many faculty members, conjures images of RateMyProfessors.com, the unrestricted free-for-all where over 700,000 professors are rated — often to their dismay — by anonymous reviewers. Blackboard — and some others are looking to enter the evaluation field — are planning very different and more educationally oriented models. Blackboard’s approach is more oriented on evaluating the course than the professor.

Blackboard has generally enjoyed a good reputation among faculty members, dating to its beginnings as a small startup. One of the things that has endeared Blackboard to academics is the ability they have had to customize the company’s products, and Blackboard, though it’s no longer small, will seek to keep important controls in the hands of institutions.

With institutions looking to do evaluations and assessment online, Debra Humphreys, a spokeswoman with the Association of American Colleges and Universities, said that Blackboard’s outcomes assessment program “could make trends that are already underway easier for schools.”

David Yaskin, vice president for product marketing at Blackboard, said that a key component of Blackboard’s system — which is in development — will likely be online portfolios that can be tracked in accordance with learning outcomes that are determined by faculty members, departments or institutions.

Yaskin said he’d like to see a system with “established outcomes, and a student has to provide evidence” of progress toward those outcomes, whether in the form of papers, photography collections or other relevant measures. Yaskin added that faculty members could create test questions as well, if they are so inclined, but that, for Blackboard’s part, the “current plan is not to use centralized testing in version 1.0, because higher ed is focused on higher orders of learning.”

One of the most powerful aspects of the program, Yaskin said, will likely be its ability to compile data and slice it in different ways. Institutions can create core sets of questions they want, for a course evaluation, for example, but individual departments and instructors can tailor other questions, and each level of the hierarchy can look at its own data. Yaskin said that it’s important to allow each level of that hierarchy to remain autonomous. He added that there should be a way for “faculty members to opt out” of providing the data they got from tailored questions to their superiors if they want. Otherwise, he said, faculty members might be reticent to make full use of the system to find out how courses can be improved.

Yaskin added that, if certain core outcomes are defined by a department, the department can use the system to track the progress of students as they move from lower to upper level courses.

Because Blackboard, which bought WebCT, has 3,650 clients, any service it can sell to its base could spread very quickly. While details on pricing aren’t available, the assessment services will be sold individually from course management software.

The idea of online evaluation is not new. Blackboard has been looking to colleges already using online course evaluations and assessments for ideas.

Washington University in St. Louis — which wasn’t one of the consulted institutions named by Blackboard — took over five years to develop an internal online course evaluation system. A faculty member in the anthropology department developed templates, and other faculty members can add specific questions. Students then have access to loads of numerical data, including average scores by department, but the comments are reserved for professors. Henry Biggs, associate dean of Washington University’s College of Arts and Sciences, was involved with the creation of the system, and said that too much flexibility can take away from the reliability of an evaluation or assessment system.

Washington University professors have to petition if they want their ratings withheld. “If faculty members can decide what to make public, there can be credibility issues,” Biggs said. “It’s great for faculty members to have a lot of options, but, essentially, by giving a lot of options you can create a very un-level playing field.”

Biggs said that the Blackboard system could be great for institutions that don’t have the resources to create their own system, but that a lot of time is required of faculty members and administrators to manage an assessment system even if the fundamental technology is in place. “The only way it can really work is if there are staff that are either hired, or redirected to focus entirely on getting that set up,” Biggs said. “I don’t think you will find professors with time to do that.”

Humphreys added that “the real time is the labor” from faculty members, and that technology often doesn’t make things so much easier, but may make something like assessments better. “People think of technology as saving time and money,” Humphries said. “It rarely is that, but it usually adds value,” like the ability to manipulate data extensively.

Some third-party course evaluation systems already offer tons of data services. OnlineCourseEvaluations.com has been working with institutions — about two dozen clients currently — for around three years doing online evaluations.

Online Course Evaluations, according to president Larry Piegza, also allows an institution to develop follow-up questions to evaluation questions. If an evaluation asks, for example, if an instructor spoke audibly and clearly, Piegza said, a follow-up question asking what could be done – use a microphone; face the students – to improve the situation can be set to pop up automatically. Additionally, faculty members can sort data by ratings, so they can see comments from all the students who ripped them, or who praised them, and check for a theme. “We want teachers to be able to answer the question, ‘how can I teach better tomorrow?’” Piegza said.

Daily Jolt, a site that has a different student-run information and networking page for each of about 100 institutions that host a page, is getting into the evaluation game, but the student-run evaluation game.

Mark Miller and Steve Bayle, the president and chief operating officer of Daily Jolt, hope to provide a more credible alternative to RateMyProfessors.com. Like RMP, Daily Jolt’s evaluations, which should be fully unveiled next fall, do not verify.edu e-mail addresses, but they do allow users to rate commentors, similarly to what eBay does with buyers and sellers, and readers can see all of the posts by a particular reviewer to get a sense of that reviewer.

Biggs acknowledged that student-run evaluation sites are here to stay, but said that, given the limited number of courses any single student evaluates, it’s unlikely that reviewing commentors will add a lot of credibility. Miller said that faculty members will be able to pose questions in forums that students can respond to.

“A lot of faculty members want to put this concept [of student run evaluations] in a box and make it go away,” Miller said. “That’s not going to happen, so we might as well see if we can do it in a respectful way.”

— David Epstein

Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Comments

Why are professors so scared of evaluations? Their argument that students aren’t in a position to evaluate how much they have learned would hold water under normal circumstances, but in the current context of higher education professors are reluctant to share how much students learn in their class. They really shouldn’t be surprised when people assert that little to no learning occurs in the typical college classroom. Until the day comes when instructors can should their students learn something — anything! — in the classroom, student evaluations will be all that we have.

RLS, at 10:10 am EDT on June 20, 2006

Why PRofessors Don’t Like Evaluations

Faculty are generally leery of evaluations for two reasons. First, students too often evaluate the professor based upon their grade. Consequently, faculty who engage in grade inflation tend to do very, very well in student evaluation scores. Second,assessment ignores the role that students play in their education. It assumes that all students have the same ability to learn, and that if learning—however defined—does not occur, then it must be the fault of the professor. Neither assumption works in reality.

Anonymous, Professor at Large 4th-Tier University, at 10:55 am EDT on June 20, 2006

Free Advertising for Vaporware

Not sure why IHE chose to give so much space to a Blackboard product that does not exist. Vaporware is vaporware.

SkeptiK, at 12:25 pm EDT on June 20, 2006

assessment & Blackboard

Remember not to confuse “course evaluations” with “assessment of learning outcomes.” While accreditors do care that institutions have a system in place for student input into their instructional experiences, their current emphasis is on whether institutions have any clue about what and to what degree students have actually LEARNED what the institutions (i.e., faculty, staff, board, constituents) have set out for them to learn. That’s a completely different matter from course evaluations, and is in fact part of the core of the work of faculty.

M. Luebke, at 12:45 pm EDT on June 20, 2006

Then Faculty and Anon. misses the point – it’s not about fault it’s about results, students must learn “something!” Education should not be a one-way street. Students are not vessels to be filled but partners in a process that holds students and faculty accountable (there is that word) for the results whatever they are. The idea that students “often?” evaluate the professor based upon on their grade is not totally born out in my experience. Being liberal with high marks is no guarantee for a positive evaluation. To my mind, a professor with a high flunk-out-rate is as poor a performer as one who engages in grade inflation.

goodfeat, Admin at 4th tier, at 4:10 pm EDT on June 20, 2006

Digital Assessment

This assessment business is nothing more than busy work proposed, planned, and imposed by policy wonks, legislators, accreditation bodies, deans, etc., etc. After going through so called learning assurance (assessment) for accreditation purpose recently, I am hanging my shingles and quitting academia after 30 years of teaching. It is not worth it. It was the most useless make believe exercise one can concoct. I was trained to think, teach, and do original research not be a paper/form pusher. ..Garbage in garbage out is how I see the status of many of these so called assessments.

Anon Prof, at 4:10 pm EDT on June 20, 2006

“To my mind, a professor with a high flunk-out-rate is as poor a performer as one who engages in grade inflation.”

And to my mind, an administrator that is hellbent against holding standards of any kind is not only a needless (and discriminatory) roadblock to tenure, but also a blazing fire of misplaced fury that consumes faculty morale as easily as dried hay.

Upper Midwest Asst. Prof., at 5:05 pm EDT on June 20, 2006

It seems as if there’s some confusion between assessing learning and evaluating instruction, and I’m not clear on what Blackboard is proposing. There are a number of qualitative and quantitative (e.g., pre/post tests, gain scores, portfolios, interviews, standardized tests) that can reflect how a student’s understanding of a subject has changed; the primary caution there is to be sure that the assessment captures what is taught (no point in assessing a student’s knowledge of dates/places of the Civil War if the course focus has been on the economics and politics of the reconstruction). Evaluating how the professor taught that information is another matter: we’ve all experienced learning under a poor guide, and loved guides who led us nowhere.

JLN, at 9:15 am EDT on June 21, 2006

The Student Perspective

As a student, I have found course evaluations to be helpful in some instances. I attend Wash U, which has a relatively successful course evaluation system. The evaluations have two types of grading that the students give the professor and the course itself. The first type is a number grade (a number scale generally indicating magnitude (i.e. How prepared was the professor for class? 1: ill-prepared, 7: always very prepared)). I have my doubts as to how these questions improve our learning environment. However, there are also questions like, “What could the professor do to improve the class?” or “How available was the professor for help outside of class?” These questions are little nuggets of gold for the professors to hone their teaching skills. I’ve come across a number of professors that have used the suggestions of their students to become better teachers. Of course some students will blow off these questions or reply angrily if they received a bad grade, but sometimes a class can be greatly improved if a professor just looks up to see if there are questions every once in a while, or if they take time to give real-world examples of applied theory. Our evaluations contain a lot of useless information, but occasionally they contain some usefull stuff too. Many of my professors have been teaching for years. If they can get one usefull piece of advice/year, that can go a long way. I’m sure that these evaluations can be a scary thing for professors, but from a student’s perspective, I like to see that my university cares about my opinions on my learning environment and cares to make the effort to improve it.

Daniel, student at Wash U, at 11:25 am EDT on June 21, 2006

top down assessment

I recently had the honor to take part in an ePortfolio selection process for Washington state’s community and technical colleges, and rather than these tools, and we didn’t chose Blackboard, being for teaching and learning, their strength is top down assessment. This calls for each student artifact (be it an essay, a painting, a musical composition or what have you) to be rated based on some sort of rubric that all would have to adhere to. Yes, there is the availability to provide narrative comments, but since the learning is at an end stage, this is summative feedback, not formative that will go into future learning. I’m not sure the complexities of teaching and learning can all be boiled down to one stage or another on a four or five-point scale.

Frankly, it’s legislatures, business groups and administrators that this sort of thing serves, not faculty and students, and certainly not teaching and learning. It’s also damned expensive (washington state will spend about $2.5 million for a statewide Blackboard license in the coming year) to implement this sort of thing and it gets more expensive when colleges and higher ed systems get locked into this technology long term. There are better places to spend the money this sort of thing will cost to implement, particuarly when the spending of this money won’t have a positive affect on teaching and learning—the ultimate outcome of what we do in the classroom.

bradley bleck, instructor at Spokane Falls CC, at 12:10 pm EDT on June 21, 2006

While grades and the perception of an easy course do have some influence on evaluations, I believe that influence is much less than most faculty realize. As a department chair I carefully read all of the evaluations for faculty in my department. The strongest indicators for student satisfaction are perceived fairness and open communication. For example, my molecular biology instructor is very tough, but he gets exceptional evaluations. Nightmares about grades not withstanding, the students recognize the effort he puts into the course on their behalf.

Michael Rossi, Chair, Biology & Env. Sci. at University of New Haven, at 4:00 pm EDT on June 21, 2006

Why wait for Blackboard?

Drexel University has been using a powerful tool called Waypoint for the past two years to develop rich outcomes data. This web-based tool (www.gowaypoint.com) is extremely flexible and powerful, and compatible wih Blackboard. It was created by teachers who focused on the workflow of evaluating student writing and tracking their development over the year. The outcomes data is just a byproduct of giving students better feedback on their work, something students appreciate and use for peer review. Thus,Waypoiint can be used for formative as well as summative data. For example, a change in how we handled the research component on a major paper resulted in significant improvement in the type and quantity of sources students used. Such data has helped us gain administrative support for reinventing our curriculum. Our new pilot “English Alive” will rely on waypoint and Webct to provide assessment and guide revision of the curriculum.

Valarie Arms, Professor, Associate Director of Freshman Writing at Deexel University, at 11:35 am EDT on June 30, 2006

Jobs Related to Digital Assessments

or search for jobs directly.

Assistant Professor
Medical University of South Carolina

In the historic, coastal city of Charleston, the Medical University of South Carolina (MUSC) offers a wide range of ... see job

Professor, Rhetoric and Composition
University of California, Irvine

School of Humanities Department of English Endowed Chair in Rhetoric and Communication Position: Associate or Beginning Full ... see job

CE Career and Safety Coordinator
Harper College

Job Description: Coordinates the CE Driver Safety Training Program, which includes curriculum and schedule ... see job

High Performance Computing Storage Specialist
Princeton University

Position Summary: *THIS IS A 1 YEAR TERM POSITION WITH THE POSSIBILITY OF RENEWAL* The Academic Services ... see job

Human Resources Coordinator/HRIS
Hofstra University

Hofstra University seeks a Human Resources Coordinator to provide Human Resource Information Systems (HRIS) and ... see job

Veterans Affairs Counselor
University of Maryland University College

Oversee UMUC’s inclusion on the GSA schedule and establish relationships with current and potential partners/government ... see job

Staff Physician
Clemson University

Clemson University’s Redfern Health Center seeks a Staff Physician. see job

Assistant Professor of Economics
Reed College

The Economics Department of Reed College invites applications from economists who are strongly committed to undergraduate ... see job

Assistant Professor of Forest Ecology #058600
University of Montana

The College of Forestry and Conservation was founded in 1913, one of the first programs accredited by the Society of American ... see job

Professor, Department of Radiological Sciences
University of California, Irvine

Position: Professor in Residence Professor of Clinical X Health Sciences Clinical Professor Full-time professorial position ... see job

Digital Assessments

Related stories

Comments

Why PRofessors Don’t Like Evaluations

Free Advertising for Vaporware

assessment & Blackboard

Digital Assessment

The Student Perspective

top down assessment

Why wait for Blackboard?

Jobs Related to Digital Assessments

Assistant Professor Medical University of South Carolina

Professor, Rhetoric and Composition University of California, Irvine

CE Career and Safety Coordinator Harper College

High Performance Computing Storage Specialist Princeton University

Human Resources Coordinator/HRIS Hofstra University

Veterans Affairs Counselor University of Maryland University College

Staff Physician Clemson University

Assistant Professor of Economics Reed College

Assistant Professor of Forest Ecology #058600 University of Montana

Professor, Department of Radiological Sciences University of California, Irvine