Coursera: A Beginner's Guide to Irrational Behavior

I recently participated in a Behavioral Economics MOOC on Coursera.org taught by Dan Ariely.  I used to love listening to Dan's conversations with Kai Ryssdal on NPR/Marketplace on my drive home several years ago. And I had already been through each of his books, Predictably Irrational, The Upside of Irrationality, and The Truth About Dishonesty.  So, I thought the class would be a lot of fun.



A Beginner's Guide to Irrational Behavior

Most of the topics, research and experiments that were covered in the class were ones straight out of the books, but there were also interesting references to research papers to see more details about how the experiments were carried out and the exact findings of the research.  It is evident that Dan and his team put a lot of effort into building and presenting a very high quality MOOC.

Impressions of Coursera

I have to say I'm a fan of Coursera.  I think it is very cool that they have worked out a deal with so many top colleges and universities to offer outstanding content to the masses for free!  The Coursera website is easy to navigate.  Access to course materials, video lectures, and discussions is fantastic.  The quizzes associated with the lectures and reading assignments were straight forward and engaging.

The one area that I find to be rather disheartening is the peer grading process.  I understand that there is no way for a professor and a couple of graduate students to grade tens of thousands of essays and writing assignments, but there seems to be a rather large flaw in the the grading criteria for these non-multiple choice assignments.  The problem is subjectivity of the grader on the written assignment.

xkcd.com

Peer Grading

After submitting the written assignment, each student was asked to grade three other assignments based on the following criteria:

1. Did the student identify and describe a behavioral problem?
2. Did the student correctly identify and describe research that is relevant to the problem?
3. Did the student propose a research-based solution?

I find it odd that someone as fluent in measuring the results of research and experiments as professor Ariely (which are often dependent upon how questions are phrased, the order in which the questions appear and the types of choices presented), would purposefully introduce subjectivity into this grading process by allowing the grader a three point scale for each of these criteria.  Unless I'm mistaken, these appear to be simple "Yes/No" questions, yet the grader was given a range of choices from 1 (didn't meet the criteria) to 3 (met the criteria) thereby adding unnecessary subjectivity.

For the first criteria, either the student described a behavioral problem or they didn't.  Answering this question should not be based upon the grader's bias as to whether they perceive the issue to truly be a problem or not.  If the student articulated a topic in the context of human behavior, how can anyone honestly give them less than full credit for this criteria?

The second criteria possibly could have been worded more precisely as to convey the intent of the question, such as, "Did the student utilize material from the course reading assignments that supports the claim of the behavioral problem?"

The third criteria, again, is not asking for the grader's biased opinion as to whether or not they agree with the proposed solution.  It is simply asking if the student proposed a solution within the context of the research covered by the course reading assignments.  It is not the grader's responsibility to assess the viability of the proposed solution.  It shouldn't be the grader's prerogative to judge the effectiveness of the proposed solution.  I find it hard to believe that anyone would submit their written assignment if it were totally void of a proposed solution.

Proposed Solution for Peer Grading
 

Dear Coursera and prof. Ariely, to improve the quality of this course for the students, I believe an effort should be made to remove as much subjectivity from the peer grading process as possible.  If you are going to ask "Yes/No" questions, then grade accordingly with "Yes/No" answers.  If you want the written assignment to be based on a total of 9 points, then ask 9 "Yes/No" questions that are specific to the quality of the written assignment, such as:

1. Did the student use the correct name for the problem (if the problem has a name we discussed in this course)? 
2. Did the student give a clear indication of why the behavior is problematic?
3. Did the student tell us what the scale of the problem was?
4. Did the student summarize the experiments and findings about this behavior? This should include only relevant experiments and findings. 
5. Did the student refer to experiments from the assigned readings and/or lectures? 
6. Did the student cite his or her sources? 
7. Did the student propose a solution?
8. Did the student show the solution was based on existing behavioral research? 
9. Was the solution original? That is, did the student come up with plan that was not exactly like another we have studied?

By the way, these questions were provided as guidance for answering the original three criteria, but were left up to the subjectivity of the grader to incorporate within the faulty three point scale.

I also realize that grading any written assignment will inherently include some subjectivity of the grader, but the need to limit this subjectivity is only magnified by the fact that the graders in this case have no credibility on which to base their opinions (i.e. the students of a free online introduction class are most likely not "experts" on the subject of behavioral economics).