Learning From One Another

WASHINGTON, D.C. -- “A little bit loopy and elliptical, but interesting.”

That is how J.R. Reddig, a 61-year-old program director for a Virginia-based defense software contractor, described his classmates’ essays in Internet History, Technology and Security, a massive open online course (MOOC) the University of Michigan is offering through Coursera.

The course, which largely focuses on the history of cyber-infrastructure, is one of the first humanities courses run by Coursera, the largest MOOC provider. That means it is an early proving ground for Coursera’s peer-grading system -- the company’s answer to the challenge of running a course with tens of thousands of students and only one professor. For every essay they submit, students in the course have to read and evaluate four others written by their classmates.

“Did you learn from them?” asked Charles Severance, an associate professor at the University of Michigan, who is teaching the course. Severance is sitting with Reddig and eight other students in the basement of a modish downtown coffee shop here, where he has arranged an “office hours” meet-up -- one of several he has held in cities around the country. A miniature fountain babbled nearby. Ambient jazz trickled out of unseen speakers.

Reddig shook his head: “Umm, no… no.” Not about the essay topics, anyway. Mainly, Reddig said, he learned how to read past the spelling and grammar hiccups of non-English speakers and try to grade them based on their ideas. “I said, Well, O.K., you can’t apply an empiric standard to them,” said Reddig. “These people attempted to follow a thought, and so give them a 10.”

John Norman, a middle-aged tech contractor, shared a similar experience with a fellow student. “Technically, [he] did everything perfect, but terrible writing -- it didn’t flow,” says Norman, recalling a particular classmate and essay. “But I think I ended up giving him a nine, because it was somewhat interesting.”

Severance seemed pleased that his students were making an effort to look past language barriers when assessing their peers. But his students’ testimonies highlighted the challenge of teaching a MOOC that deals heavily in writing assignments to an open classroom that includes a high rate of non-native English speakers. Coursera estimates that about 62 percent of its 1 million registered students live outside the United States.

And while many of the courses the company and its partners have offered thus far rely on the universal language of math and scalable, automated grading systems, Coursera and its partners are preparing to roll out a dozen courses in the humanities and social sciences that will probably rely on peer grading.

Severance ran into the language barrier in week two of his seven-week course. After 45,000 people signed up, 23,000 actually logged in when the course began in June. By the end of the first week and the first quiz, 11,000 had stuck around. Not the kind of retention rate Severance was accustomed to at Michigan, but par for the course in a MOOC.

That’s when Severance made what was, in retrospect, a tactical error. “I wanted to try the peer assessment as fast as possible,” he says. So he assigned a short essay -- 400 words or so -- with the following prompt:

“In many ways, the Internet is the result of experts exploring how people, information, and technology connect. Describe one example of these areas (people, information, and technology) intersecting, and how that connection ultimately helped form the Internet. Your example should be taken from the time periods we covered in the first two weeks of course.”

That’s when things started to go awry.

“Probably a third of the students do not have English as a primary language,” says Severance. “A problem I have never had before is … suddenly I’ve got people in 15 languages. I’m trying to be subtle and draw out some insight, [and] they want something they can really translate and understand.”

After that assignment, the active enrollment in his course dropped to 6,000.

Now Severance has changed the assessment criteria for the course to exclude participation in the peer-grading system as a requirement. Instead, there will be a certain number of points students will have to accumulate in order to pass, all of which can be earned via multiple-choice quizzes. “So if you don’t speak English, and you don’t feel comfortable writing or reviewing, [you can] opt out -- you’ll be fine,” says Severance. “You’ll get your certificate. And I’m not going to mark the certificates as with peer grading or without.”

That might be a viable workaround for Severance’s course, which is oriented to history with shadings of technical training. But for Severance’s colleague Eric Rabkin, essays are impossible to avoid.

Rabkin, an English professor at Michigan, is teaching Coursera’s first literature course, Fantasy and Science Fiction. Rabkin says he does not know how many of the 39,000 people who originally registered for the course, or the 8,000 who are still submitting and grading essays, are non-native English speakers -- nor whether English-speaking students in the course worry that the subtleties of their own essays are being lost on ESL classmates who are grading their work.

Rabkin himself is not worried. While he takes care that his lecture videos and syllabus reflect the rigor of a University of Michigan course, he has no illusions that a peer-grading system will accord with his own standards for academic achievement. “I believe that we are going to get a standard of passing work that is appropriate for this community,” he says. That community includes students who might not have full command of the English language.

And here is where the philosophy of MOOCs collides with the idea of certifiable achievement in a literature course: “If we’re going to keep this completely open,” says Rabkin, “then no credential can have a well-understood meaning.”

Daphne Koller, one of the co-founders of Coursera, says that the peer-grading experiment is still very much a work-in-progress. "We will undoubtedly learn a lot from the experiences of our instructors as they encounter this phenomenon, and then have a better sense of where exactly the tensions lie and how one might deal with them," she says. "We also have some ideas of our own that we'll throw in the mix and evaluate as we plan the next phase of this experiment."

Severance feels he can afford to give his students a little more wiggle room to duck writing assignments without compromising their ability to demonstrate learning. But, at his “office hours” in the basement of the coffee shop here, he is still trawling for suggestions about how to make the peer-grading system better.

Dezzie Garcia, a 26-year-old Web developer in dark-rimmed glasses and a blue blazer, has an idea.

“I think when someone tries to write in a language that they’re not fluent in, the reader -- particularly us, who are English speakers -- we have this latent bias about the content of what they’re writing,” she says.

But what if foreign students were allowed to write in their native language? “At the beginning of the course, you could [administer] a survey where you ask the person if they fluently speak another language, and then perhaps you could route assignments when you do peer review,” Garcia continues. “For example, I speak Spanish and French, so you could route those [essays] to me.”

There are murmurs of agreement from around the table. “That’s a great idea,” says Reddig. Other students join the conversation, building on Garcia’s suggestion. Severance takes notes.  

For the latest technology news and opinion from Inside Higher Ed, follow @IHEtech on Twitter.