Tag Archive | assessment

Collabowriting & Self- and Peer-Assessment

To work in teams and being evaluated by your peers are two practices with an enormous potential and relevance for individual motivation as well as for being able to contribute value to an organisation. Both practices are being used in e-learning and the following paragraphs will introduce some approaches and cases. Stanford considered collaborative writing (here refered to as collabowriting) so important they ran the Collaborate project in order “to encourage institutional change at the level of the department, college, university, and profession–change that will foster the development of collaborative writing and research.

Why consider collaborative writing assignments?

Kate Kiefer from Colorado State University’s suggests several reasons:

  • Collaborative groups draw upon the strengths of all their members. Although one student may be stronger in critical thinking skills, another may excel in organizing. By working in groups, students learn from each other while they complete assigned tasks.
  • More and more workplace activities involve project teams. Giving students opportunities to work collaboratively on academic projects can help prepare them for the advantages and pitfalls of collaborative work on the job.
  • Students working in collaborative groups can take advantage of group members for built-in peer review as they complete writing projects.
  • Not least important, collaborative writing assignments usually entail much less grading time for the instructor.

How to Collabowrite?

People have been writing texts together for a very long time, but the computers, the internet and wiki technology in particular allows for a more efficient practice Most collabowriting today is probably done by sending documents back and forward by email (and if you endulge in this practice please take two minutes to learn how to Track Changes), wikis are really much more efficient because of version-control, online availability and much more technical reasons. But wikis are also the weapon of choice because they allow for a more transparent practice when it comes to showing who contributed how much and what to the writing process. However collaborating is a very complex effort that has to be learned and trained. Hence we recommend some work on the subject:

Bruce Speck has done very interesting research in the field of collaborative writing. His book
Facilitating Students’ Collaborative Writing (2002) is an academic publication that introduces the topic and discusses the whole range of collaborative writing opportunities that can be offered, ways to form groups, and approaches to integrate technology. A final topic is grading, which is particularly difficult for collaborative writing projects.

Also
Connery & Vohs @ UC Davis have developed a comprehensive guide to online collabowriting, and they find: “when students are asked to list problems they have encountered in previous project groups, the most commonly cited difficulties can be traced to a lack of leadership and coordination: defining/assigning tasks; enforcing participation, scheduling meetings, setting and agreeing upon group goals”. They suggest to assign the roles (Initiator, Opinion&Information seeker&giver, Clarifier, Elaborator, Summarizer) to the different team members.

Virginia Montecino and Ashley Williams from the New College have developed a guide particularly targeted at supporting students in their collaborwriting projects. Especially the section on collaborative strategies as well as the section on collaborative roles are useful.

Last but not least an example from the UPC. Professor Marc Alier (who is also responsible for the development of the new Moodle Wiki) is using wiki based collabowriting for several iterations of the History of Informatics course. He allowed students to organize their team work themselves, which allows for a much more intense learning curve. The fact that his class is semi-precential and that the students have it much more easy to meet to coordinate efforts makes this approach less attractive for pure e-learning.

What Tools can be used for Collabowriting and How to get started @ UOC

The most common tool for collabowriting is a wiki. By far the most widely used wiki is MediaWiki but there are good alternatives such as SocialText or PBwiki. Kolabora has compiled a pretty useful list of collaborative writing tools, which covers many non-wiki services available outside of an educational institutions virtual campus. Most of today’s universities offer a wiki as part of their online learning environments (to request and explore the use of a wiki @ UOC please write to Pepe Mora @ jmora@uoc.edu)

As raised in the Speck’s book, evaluation of collaborative work is particularly challenging. One of the best methods to “crowdsource” the problem is to have students engage in self- & peer-evaluation, which is covered in the following section:

Self- & Peer Assessment

Even though self- and peer-assessment are around for many many years, no standard has developed for e-learning environments to integrate the practice. Nevertheless, there has been quite some interest in the subject resulting in extensive publishing () and some Learning Management Systems (LMS) such as Blackboard have a self- and peer-evaluation feature (see video) integrated. However most solutions are rather rough homegrown (iPeer or peerScholar) or more slick and advanced comercial add-ons such as Turn-It-In. But let’s start off with two hands-on examples:

At the University of Florida the online MBA uses a very simple word-doc based peer-review system in a course designed to train students in “acquiring and practicing traditional business writing skills”. This is how it works: “Students actually download the report from the discussion board, mark it up with their comments, and upload it back to the board…..only faculty can see the actual names of the reviewers…the blind answer format thus enforces original critiquing…The professor grades the original writing and grades the efforts of the peer reviewers.” The report that “students indicate their enthusiasm for this kind of activity, and the ensuing class discussions related to these peer evaluation activities reflect the students’ increased engagement with each other and with the course material” (Fisher, 2003)

Here is another story of a Professor from the University of Toronto who had the custom system peerScholar developed in which the students review and evaluate their peers answers in exams: “There is no way I could mark the written work of even 100 students, let alone 1500…The original reason for seriously considering a peer-to-peer evaluation process was financial. We cannot afford to pay a large team of TAs to mark written answers for large classes…[But there are substantial benefits as well:] peerScholar gives you feedback in a matter of days, while the papers are still potentially on your mind….With the peer-to-peer evaluation, you think about good writing both when you write your answers AND when you are evaluating the answers of others. Moreover, because the answers are not your own, you will be better able to look at them critically and think about what they are doing well, and what they are not doing so well. In doing so you may actually be learning more about good versus poor writing that you do when you write your own answers. Thus, when you are evaluating others, you are also learning yourself…..[The system has some build in functionality to assess whether the peer is working fairly:] By examining how long they actually spend on marking, the variability of the marks they give,and the correlation of the marks they give with the marks that others give, we can detect “dud-like” behaviour and, when it is detected, the person will be warned and the marks they gave not included” (Joordens, 2006)

Exploring the Practice

A) Self-Evaluation

Overall Learning Experience: When students apply for the class have them define their personal learning goals (depending on the subject you might want to consider having them refine their goals after the introductory part of the course). Hence when the class is over they should review their original objectives and comment on how the class allowed them to learn what they wanted (and how they performed).

Learning-Journal: Have students continually write and reflect about their learning experiences. A private individual blog is commonly used as tool to have students record interesting insights and reflect upon their learning in general.

These practices also produces very relevant information for the improvement of the course content and teaching method.

B) Evaluation of Team Dynamics

Have teams assess their own dynamics and performance. See e.g. fig. 1 http://www.educ.dab.uts.edu.au/darrall/sparksite/Screen2.html for an example for “teacher-like” grading. Another solution would be to have the professor grade the overall performance of the team in form of reached points, while the team members themselves distribute the points (Winchester-Seeto, 2002). E.g. the team is assigned a percentage of the maximum score (= number of team members x 100) which the team then decides to divide amongst themselves. A robotics course at the University of Calgery uses the first version of the Team Dynamics Evaluation with a particular twist. All students have to hand in a 1/3 page evaluation of each of their team mates and while doing the peer-assessment only contributes 1% to the overall course grade, failing to hand in the team mate evaluation results in failing the course (Kermer 2008)

C) Peer Evaluation

A discussion in an online forum can be seen as a constant qualitative peer evaluation, and many online environments have the functionality to leave comments about content items. These tools can be used as qualitative non-explicit evaluations from peers. A more quantitative and explicit peer evaluation technique that is spread widely is the star rating system, allowing peers to grade the quality of a contribution e.g. from 1 to 5 stars. In order to deploy these tools pedagogically their use has to be explained and they have to be made mandatory. One interesting use of peer-evaluation can be observed on www.plastic.com (or slashdot.org) where (expert)users on the one hand peer-review & evaluate the submitted articles, but on the other hand all readers evaluate the quality of contributions to the discussion and thereby allow the most valid arguments to prevail.

Peer evaluation