The New Assessment Market

The Spellings commission may not have won the hearts of everyone in academe, but one of the most successful software businesses in higher education was clearly paying attention. Blackboard on Tuesday announced officially (as it had been suggesting less officially for several months) that it was offering products in a new line -- assessment -- from individual courses to entire institutions. The announcement cited the commission's interest in assessment as one bit of evidence of the increased pressure on colleges to document the learning that takes place on campus.

Blackboard -- which dominates the course-management industry, especially after its absorption of WebCT -- has the resources to make a big push in the industry and it has contracts with thousands of institutions for other services and high visibility on campuses, where it has many fans and also critics. Company officials say that they have unique advantages to help colleges improve education and satisfy accreditors, government officials and others who want more evidence of colleges' performance.

A few young companies that beat Blackboard into the business say that they already offer the services Blackboard is now starting. They said that they took the arrival of "the gorilla" -- as one competitor termed Blackboard -- as evidence that their industry was succeeding and would grow. Other observers of the assessment scene had a range of views, with some hopeful that the new competition could ease the data collection burdens that colleges face and others fearful that systems like the ones prevalent in this new industry will inevitably restrict the autonomy of faculty members.

David Yaskin, Blackboard's vice president of product marketing, said that the new services represented a significant shift for the company. "We've given colleges tools for the delivery of courses, but not for the management," he said. The new tools will allow colleges to measure course effectiveness and to act on those findings, to figure out if there are gender or ethnic patterns in course performance, to track alumni or employer satisfaction, and to do all of this on both micro- and macro- levels, so that a department head might have better planning information and a provost might have better information to submit to an accreditor.

The "Blackboard Outcomes System" is actually a series of systems. One tool allows for the collection of electronic portfolios, so student work in a variety of formats can be gathered and analyzed. Other tools allow colleges to set standards for course performance to see -- based on data gathered by faculty members -- whether students are meeting those standards. Options are provided for course evaluations. Other tools manage surveys. Still others group together all the mandates a college might be facing and help track performance on meeting various goals.

"In putting this together, we were trying to understand what does higher education do in terms of assessment. What are the hard things about it? What stops faculty and administrators from embracing a culture of assessment?" Yaskin said.

"We're trying to get at the heart of learning outcomes," he said, using a phrase much favored by the Spellings commission.

Yaskin stressed that there was no single formula offered in the new services. Colleges and Blackboard would customize the various programs to reflect institutional priorities and goals.

Many professors have viewed assessment requirements with some fear that they would amount to more work for them, and less autonomy over courses, and Yaskin said that "a huge concern" of the company was how to set up its system in a way that would build faculty confidence.

One way Blackboard has done this, he said, was with a "flexible permissions structure," such that a college could set up the system so that individual faculty members could gather information on their courses, but do so privately, without giving access to administrators or anyone else. But Yaskin acknowledged that it would be the college's decision on whether or not to give professors that right to keep information confidential, and that some, particularly in for-profit higher education, probably would not do so.

Blackboard does not like to reveal its pricing generally and Yaskin said he couldn't say anything about how much these services would cost, except that the formulas would be enrollment-based. He indicated that the costs aren't small, however: When asked if colleges could use some, but not all, of the services offered, he said that they could do so, but that "the value that justifies the expense is when you integrate all the work that is done."

One of the first two institutions to sign on was the University of Texas at Brownsville. Eli Peña, director of institutional effectiveness, said that assessment is complicated there, as it is both a community college and a full university, offering certificate through graduate programs. At the same time, it has goals to meet that it has set itself, and that are set for it by the university system, the state, and its accreditor. "We need things to be aligned. We need to better manage all of the processes we have in place," Peña said.

His hope for the system is that it won't just allow for more efficient production of reports and data, but will encourage faculty and administrators to evaluate how to improve. "Obviously we have to demonstrate things to others, but it's what we demonstrate to ourselves," he said.

While Blackboard is hailing its services as revolutionary, at least two young companies already offer services that they say are similar (and that Blackboard says are not as comprehensive). Nuventive, founded in 1998, offers an e-portfolio system and also an assessment program, known as TracDat. Nuventive has licensed TracDat to 165 colleges, most of them in the last few years. Malcolm Hobbs, vice president of marketing, said that the services could be purchased either by departments or entire colleges.

"Blackboard is certainly interested in pursuing additional revenue opportunities and we welcome new competitors into the space," Hobbs said. "But we've been doing this since 1998 and we have a very good insight into what institutions are trying to do." (In common with Blackboard, Hobbs declined to discuss pricing.)

Another company -- WEAVEonline -- officially launched in May, as a spinoff from an assessment program created at Virginia Commonwealth University. It is about to sign its 40th client -- most of which approached Virginia Commonwealth after hearing presentations about its assessment tools. The growth reached a level, officials said, where it made sense for the company to become independent, although Virginia Commonwealth is among the owners.

Barbara Fuhrman, director of institutional effectiveness at WEAVEonline, said that the company's strength was its close ties to academe. "We're not corporate folks. We're not technology folks. We understand how faculty work," said Furhman, a former education dean at Louisiana State University.

At the same time, Fuhrman acknowledged that assessment may involve pushing professors a bit. She gave as an example a political science department that said it wanted students to learn to think critically. That's not enough, she said. "Faculty would be prompted to say what the means in a measurable way."

WEAVEonline's costs vary by institutional size, but the initial set up fee could be $10,000 to $40,000. Annual license fees could run a little less than the set-up fee for smaller institutions and one-third of the set-up fee for larger institutions. Fuhrmann said she wasn't surprised by Blackboard's announcement because it has had employees showing up at WEAVEonline demonstrations. "Do we need to keep our eye on them? Certainly," she said, adding that she felt very confident that her company would be growing.

Judith S. Eaton, president of the Council for Higher Education Accreditation, said that there has been periodic interest from companies in helping colleges gather the materials they need in accreditation reviews and promoting better assessment. A concern she hears from many colleges is over how much data they need to gather, how to present and analyze it, and so forth.

Eaton said that if the new Blackboard services "are positioned to help colleges deal with the challenge of everything they need for assessment," the company could find a good market. If colleges view this "as one additional thing," then it will be a tougher sell, she said.

Cary Nelson, president of the American Association of University Professors and a professor of English at the University of Illinois at Urbana-Champaign, offered a different perspective. He said he had "no problem with faculty members independently choosing to evaluate their courses in any way that they want, and it's certainly possible that some faculty would find a tool like this useful in assessing whether their course met the goals that they had set for it."

But he added that there are many courses for which an assessment tool "wouldn't be helpful at all, and I'm entirely against imposing additional assessment tools on the faculty."

In his courses, students have weekly writing assignments and are constantly judged on their classroom comments, Nelson said, giving him plenty of information to assess what students are learning. He said he worried that faculty members were being encouraged to accept "half-witted, simple-minded assessment instead of a feedback system that's already pretty sophisticated."