Where's the Evidence?

As widely reported, five massive open online courses have been deemed worthy of an academic credit recommendation by the American Council on Education. The announcement by ACE has been greeted with equal parts acclaim and concern. Many see this step as a new day for increased access and reduced cost for those working toward a degree. Content costing thousands of dollars on campus may now be available for free and in the comfort of one’s own home. Others, especially in academe, continue to express concern about the nature of MOOCs and their ability to produce learning.

While no one questions the quality of instruction, it remains unclear as to how student learning by MOOC is to be measured.  Yes, there has been discussion in the media about the use of proctored exams. Yet, a poor assessment securely administered is still a problem.  Are examinations being developed from publisher test banks, or are they akin to the psychometrically validated, field tested exams that have been long used to measure other forms of non-collegiate learning? Are question banks sufficiently deep to ensure assessment variation and integrity? And, are security measures in place to minimize the likelihood of the type of cheating that has compromised other large scale testing programs (the TOEFL experience in Asia a few years ago comes to mind)?

At a time when evidence of learning is increasingly demanded by accreditors and the federal government, a determination of equivalency in instruction alone is no longer sufficient. Valid, secure learning outcome assessment must now be part of the equation as well.

Yes, the MOOCs produced by Udacity and Coursera feature some of our finest instructors from prestigious institutions. We must remember, however, that this instruction comes from outside the academy. MOOCs are not subjected to the same reviews that are part of the accreditation process for other instruction at colleges and universities. And, as we have seen, more thought must be given to the assessment process and its validity.

Would this also be the case for a course coming from within the academy? One would hope not. A higher standard is mandated when measuring the ability of non-collegiate instruction to produce meaningful outcomes. Consider that it has been standard practice for individual military students, independent study students, and those taking courses through open educational resources, as well as other forms of prior learning, to prove their mastery of specific subject matter through psychometrically validated, ACE reviewed and approved examinations. These exams are prepared by Ph.D. psychometricians, in tandem with panels of nationally prominent content specialists. They are field tested for reliability and predictability.  In the case of exams at my institution, Excelsior College,, they are also nationally normed to establish "cut scores" for the assignment of letter grades. The academic community broadly has been accepting of this type of learning validation because of the understood rigor and credibility of the methodology.

No similar attention to rigor, validity  and security is seen in the MOOCs offered to date. This is especially important as we struggle to understand why only a small percentage of students (less than 10 percent by most accounts and usually from abroad), actually complete the MOOC experience.

And yet, massive open online "courses" (doubt still exists as to whether this is what they truly are) do appear to have real value.  It can be argued that they are well-suited (and likely sustainable over the long term) as vehicles for:

1. Delivering cutting-edge continuing education, where credit is secondary to relevance for technical professionals striving to remain current and competitive.

2. Building faculty reputations through what some see as "21st-century interactive, textbooks." It is a rare professor who can boast of hundreds of thousands of "students" when preparing for a promotion or tenure review, especially at a time when traditional texts typically sell fewer than a thousand copies.

3. Creating greater institutional brand awareness. By being seen in the company of "giants," other research universities enhance their visibility overseas, which is where the majority of MOOC participants are located. (This is inexpensive PR in the eyes of many and explains the unseemly rush by some institutions to get on board).

4. Offering "free samples" of a program that will later involve real dollars. Berklee School of Music, for example, is offering an introductory guitar class to tens of thousands of interested individuals. In so doing, the prospective student can try before buying. Even if only a small number complete the course per usual, Berklee will have a greater number of potential enrollees than is typically the case with other forms of student recruitment.

With all of these promising applications, where MOOCs have yet to prove themselves remains in the area of degree completion, or even completion of enough learning to justify the awarding of credit. Traditional students (ages 18-24) and those adults who question their ability to succeed as returning students often need to interact with an instructor. Mentoring, teaching and building self-confidence are not attributes of the current MOOC. Until there is evidence of real learning here, institutions may wish to move with caution in the extension of credit toward degree requirements As the executive director of a regional accrediting body recently stated, "We will be very interested [in our visits] to learn of the basis for such decisions [by the accepting institution]."