Step 5 – Manage the peer assessment survey
See how we create better teams through better feedback
Manage the peer assessment
As our survey progresses, we might request resubmissions from students who appears to have rated others unfairly or assess their own contribution conspicuously different to the assessments made by their teammates. We might need to adjust the composition of a team by adding or dropping a teammate.
Checklist
- Identify symptoms of unacceptable rating behaviour by teams or individuals. Request resubmissions when appropriate.
- Proactively identify students at risk of failure or expulsion from their team. Prepare to counsel them
- Calibrate the spread of personal grades across each team to ensure ‘motivationally fair’ results among high peer assessed and low peer assessed teammates.
- Correct promptly a team’s composition when notified by a student.
- Remind laggards to submit their peer assessment responses.
- Publish provisional results for view by students in valid assessed teams, then remedy issues identified by students.
Unacceptable peer assessment rating behavior
Several symptoms of unacceptable peer assessment behavior will become evident as the survey progresses, including
- Conspiratorial teams who give everyone the same, typically a high rating across all factors of the peer assessment rubric
- Outlier raters who rate a team member conspicuously different to the rating given by other members of the team. For example they may rate a friend abnormally high and/or a disliked person abnormally low
- Lazy raters who give everyone the same, typically a high rating across all factors of the peer assessment rubric
Teams conspiring to rate themselves high
Figure 7.1 illustrates a graphical approach to identifying conspiratorial teams. The data is the first, formative peer assessment from a large first year university class of 848 students. Viewing from the left-hand side of the chart, observe that 24 teams of the 84 valid teams are highlighted as having a low quality team rating. The chart displays the data sorted so that the teams with lowest range of peer assessed scores are positioned from the left, identified by the red bars. The corresponding average (mean) peer assessed score for the same team is shown by the adjacent blue bar. The 24 teams present a low range of peer assessed scores across each team, and average team peer assessed score above 90. For instance, at the extreme left is one team with an average peer assessed score of 100, and a range of zero!
In contrast, on the right side of the x-axis, the teams with the higher intra-team range of peer assessed scores generally have a team average peer assessed score closer to the ‘expected’ value of 50/100. You may conclude these teams are conducting a more honest, valid job of rating each other.
The data presented in Figure 7.1 presents a case where Academic Policy 1 – Individual contributions to group work must be identified is unlikely to be achieved. The results are unlikely to be fair and valid. As part of your subsequent feedback step, consider retraining the class of students towards more constructive and discriminating peer assessment before they undertake the final, summative peer assessment. For example, remind students of the purpose of peer feedback, and how to provide useful feedback. Consider the training exercises presented in Step 3 – Train your students, Chapter 5.
Figure 7.1 Identification of teams conspiring to rate everyone high in a large class
Conspiratorially high rating, or high performing?
In general, poorly performing and/or poorly trained teams will rate themselves at the uppermost levels of the peer assessment rubric scale, and/or over a narrow range. You will often find superficial or no qualitative feedback to support the high ratings they provided. The team’s poor performance becomes evident from the team results you later award for their delivered outputs.
However, a related, contrary effect is sometimes found. A team likely to gain superior teacher-awarded team results for their delivered outputs may also rate over a narrow range. The team ‘got their act together’, achieved effective harmony and integration, achieved high psychological safety, and are competent at giving and receiving feedback. In contrast to the superficial feedback provided by teams likely to fail (that is, awarding a high average team peer assessed score), you will find typically detailed, thoughtful, constructive, developmental feedback provided by those teams committed towards a path to greatness.
For instance, in Figure 7.2, consider team Pukekos that has rated over the narrowest range of peer assessed scores in this small class of six teams. Such a narrow range might initially raise concern. However, Pukekos’ team average peer assessed score is about 55 confirming that the team is rating most members ‘average, normal’ according to the peer assessment rubrics such as those in Gallery 3.1. Consequently, you need not be concerned about the validity of this team’s ratings. In fact, this low range of scores and modest team average peer assessed score may be a leading indicator that Team Pukekos may prove subsequently to be a high performing team. However, confirm that the qualitative feedback given by teammates corroborates the quantitative ratings.
Figure 7.2 is an alternative graphical approach to identifying potentially conspiratorial teams who rate over a low range and relatively high team average peer assessed score. The teams are shown with increasing median team peer assessed score. The vertical bars show the range of peer assessed scores within each team. You should suspect these results arise from a well-trained class in comparison with those present in Figure 7.1. The team Red Ruru, with highest median team peer assessed score still presents with a reasonably valid-looking range of peer assessed scores, as presented in the accompanying table, ranging from 59.2 through 93.4.
Figure 7.2 Examination of peer assessed scores spread around the team median
Students at risk of failure
A formative peer assessment, conducted around one-quarter of the duration through the students’ progress on their group project is helpful in identifying teams and individual students at risk of failure. In Figure 7.2, we can readily view the students who have been rated the lowest in their teams, for example Alberto UNDERWOOD in team Red Ruru. Evidently, there are also at-risk students in teams Sparkling Violator and Grey Warblers we should investigate closely.
Another graphical approach highlights those students in the entire class rated with a relatively low peer assessed scores, as shown in the histogram Figure 7.3. Those students falling in the left-most buckets, 0 to 15 and 15 to 30 are candidates at risk of failure: for example Joslyn HOOVER and Pat DODDSTER who lie in the bucket range 0 – 15.
Figure 7.3 Histogram of raw peer assessed scores for identifying at risk students
However, a set of peer assessed scores from one team is not directly comparable with the set of scores from another team, especially when the team contains lazy assessors, outlier assessors, and/or conspiratorial teams. Consequently, the peer assessed scores require transformation to enable comparisons between teams, and, therefore, to identify a more reliable identification of at-risk students.
Superior approaches to identifying at-risk students are
- Rank order the teammates in each team according to their peer assessed score. Examine the lowest ranked students in each team
- Calculate a peer assessed index (PA Index) that awards a score of 100 to the top rated student in each team, with all other teammates scaled down in proportion to the peer assessed score.
These two approaches were earlier compared in Gallery 3.2, where team results were available. In contrast, Gallery 7.1 illustrates the case when no provisional team results are available, such as a formative assessment. Consequently, only peer assessed scores are used to develop the graphic and tabular visualizations. For instance, note how the histogram bucket 15 to 30 for the Rank-Based Personal Result (RPR) in Gallery 7.1 identifies the six students who are the lowest ranked in each of the six teams. In contrast, the Peer Assessed Index (PA Index) method identifies just two extreme-rated students.
Gallery 7.1 The hunt for ‘at risk’ students using several standardized peer assessed scores
Outlier self-assessments
It’s helpful to be forewarned of a possible complaint from students who assess the quality of their contribution to their team’s work conspicuously different to the rating by their teammates. For instance, complaints are more likely to arise from students with over-confident self-assessments. However, there is merit for the teacher in addressing students with under-confident self-assessments.
Over-confident self-assessments
Those students with an inflated sense of their contribution are likely to be shocked by the relatively poor feedback rating they find in their personal feedback report, and with less than praiseworthy qualitative remarks. Their shock may propel these students to visit your office, and/or grumble to their teammates. These students are candidates for your counseling. In advance of their complaining to you, review their personal feedback reports.
Some questions to consider as you prepare
- Are the peer assessed sub-scores awarded by most teammates consistent in rating the student lower than the student’s self-rating?
- Does the qualitative feedback provide evidence in support of the quantitative ratings?
- Does the developmental guidance provided by the teammates provide a clear basis for the student to adjust their behaviour and improve their contribution?
- Does a deeper probe of the sub-scores on the rubric factors provide any insight?
Considering these questions will give you a sound basis for a constructive, coaching-based conversation with the student.
One simple measure to detect an inflated sense of contribution is to chart each student’s peer assessed score (x) against their self-assessed score, as illustrated in Figure 7.4. On the diagonal x=y we see those students whose self-assessment matches the peer assessed score, highlighted in green. These students have an Exceptionally Realistic Self-Image. That is a ‘good thing’ as we’ll discuss next, in Step 6 – Courageous conversations, Chapter 8.
In contrast, those student in the top left quadrant, highlighted in red such as Patrick NONAME, have high self-assessment but low peer assessed scores from their teammates. These students are either overconfident or just trying to ‘game’ the peer assessment system.
Note the percent ratio of the self-assessed score divided by the peer assessed score gives an Index of Realistic Self Assessment, IRSA, shown in the table in Figure 7.4. For example, Greg GNOME has an IRSA of 63 = 100 x 51.7 / 82.5, and Patrick NONAME an IRSA of 27.
Figure 7.4 Identifying overconfident and under-confident students
Understanding outlier self-assessments
There are several reasons why a student may present with an outlier self-assessment.
One reason is that the student believes they can ‘game’ the peer assessment platform into giving them a higher personal result by rating themselves higher, even when they know in their heart they have contributed poorly. This gaming behavior is relatively easy for the student to admit when they meet you. From the table in Figure 7.4, with a self score of 100 and low IRSA, the students Patrick NONAME, Pat DODDSTER and Albert UNDERWOOD are examples of students who appear to have attempted to game the peer assessment. An example of this behavior was illustrated in the training exercise for Patrick, in Table 5.4.
A second reason is that a student truly believes their over-confident self-assessment. They have never before been called to account for their inadequate contribution in previous group’s work together. They may have failed to perceive the subtle feedback signals from their teammates!
One method to explore the mismatch between the student’s self-perception and others is to probe into the component factors of the peer assessment survey rubric, illustrated in the spider chart, Figure 7.5. Here we see the student’s self perception for the factors of Professionalism, Contribution, Initiative and Attendance are matched with the (average) perception as rated by the teammates. However, there is a severe mismatch evident in factors such as Focus, Encouraging other’s contribution, Managing conflict, and Chairmanship.
Figure 7.5 Spider chart highlighting variance between self and others’ peer assessment ratings
Under-confident self-assessments
Equally important is to address students who rate themselves as having made a significantly lesser contribution than their teammates perceive. Almost by definition, these students won’t be complaining to you about a low peer assessment or low personal result. In Figure 7.4, these under-confident students are highlighted in blue.
Under-confident students are likely to possess talents that others recognize but are not recognized by the student themself. Consequently, they will greatly value your counseling to become more confident and assertive with their team and take more pride in their work.
These blindspots possessed by relatively overconfident and under-confident students are addressed productively through the process of constructive conversations introduced in Step 6 – Courageous conversations, Chapter 8.
Respond proactively to counterproductive behavior
At best, the foregoing analysis of the quantitative ratings are hints of potentially counter productive peer assessment behavior. Despite the wonders of machine learning and artificial intelligence, a mere calculation is unable to divine the underlying motivation of the student(s) who provided the ratings given. Your human judgement and intervention are now appropriate.
Consequently, you should
- Examine the qualitative feedback of all the teammates in the teams highlighted for attention. For example, sometimes, the teammate who gave an outlier rating does need to be called to account for unfriendly or counter-productive behaviour. However, in rare cases, a student may present the defence that their ‘minority report’ presents an accurate rating based on their well-trained application of the peer assessment rubric. By inference, the remaining teammates were being lazy raters or conspiratorially counterproductive.
- Invite the students to either (a) confirm their original submission is fair and valid or (b) enable them to revise and resubmit and request further qualitative justification for the ratings they have given.
TIP! Be cautious about requesting a resubmission from a student until most of their teammates have submitted the survey response. The story you sense from just half the responses in the team may become clearer when most of the team’s responses are received.
Calibrate spread of personal results
In Step 1 – Prepare the assignment, Chapter 3 we explained the alternative methods for calculating personal result for each teammate proportional to their relative contribution to the team result. Several alternative calculation methods were illustrated in Gallery 3.2. Now you can explore the effect of choosing from the alternative methods, and explore the impact of the scaling factors available. Remember your aim is to deliver a fair grade for contribution. Secondly, you aim to provide appropriate motivational ‘early warning’ signals to at risk students so they can see the consequences of continuing with counter-productive teamwork behavior.
There is no magic algorithm that can make the choice of method for you. ‘It depends’ on the potentially idiosyncratic set of peer assessed scores your class have delivered. Furthermore, I find teachers have very personal, philosophical reasons for the choice they make. Some typical reasons for choosing one method over another were presented in Table 3.5
TIP! If this is a formative assessment where you DO NOT have provisional team results, select the peer assessed index method. If this is a summative assessment, or you have provisional team results, select the Normalized Personal Result Method with a scale factor around 1.0.
Weight the team result or peer assessed score?
In normal practice, teachers typically select scale factors between 0.75 and 1.5. However, if you want to place more weight towards the team result than the peer assessed score, select a lower scale factor. For example, a scale factor of 0.0 will deliver the team result awarded to all teammates.
Gallery 7.2 shows the impact of increasing scale factors from zero through 1.5, with each team having a different team result. Gallery 7.2 shows the spread of personal results for each team calculated using the Normalized Personal Result Method (NPR). The data is presented with the left-most team having the lowest team median personal result, which approximates the team result.
Gallery 7.2 Impact of scale factor on weighting the personal result towards team result or peer assessed score
TIP! Adjust the spread of personal results within teams so that for most teams, high contributors gain a well-justified boost to their result grade. Conversely, confirm that extremely low-rated teammates are awarded near fail personal results. Confirm the grades are consistent with the qualitative feedback they received.
TIP! Take proactive steps to transfer especially poorly performing students into a new team – even a team of one – according to Academic Policies 5, 6 and 7
If you have evidence that a team is struggling, consider dissolving the team and reallocating the members amongst other teams. I contest that it is better that students have a positive learning experience with a superior performing team, rather than struggling together towards a poor team result.
Survey administration
Having addressed the symptoms of conspiratorial teams, outlier ratings, lazy rating, at risk students, and outlier self-assessments there are several administrative matters that include
- Correct a team’s composition when notified by a student.
- Prompt laggard submitters to submit their peer assessment responses
- Respond proactively to any issues raised ‘for the teacher’ in qualitative feedback.
Adjust a team’s composition
Promptly adjust the membership of teams when one or more students have
- Been assigned into the incorrect team
- Enrolled late into the course
- Unenrolled from the course
- Not presented themselves to their team, even though nominally enrolled
- Been excluded from a team consequent on a poor feedback rating by their teammates, or other reasons suggested by the Academic Policy ‘Three strikes and you are out!’
You will discover the need for a team composition adjustment from several sources:
- Advice from a team spokesperson
- An adjustment to your official class list
- Direct notification from the survey platform when students are given the option to confirm their team membership is represented correctly.
Your survey platform should accept your adjusted teamset composition with minimal impact to survey responses already submitted by teams unaffected by the teamset update, or when a teammate has been removed from a team.
If a team has a new member added then that circumstance will require the teammates to resubmit their survey responses to include a peer assessment of the additional student.
Prompt laggard submitters to submit
The platform should automatically remind students as the due date for the survey’s completion nears. In addition, the teacher may manually initiate reminders to complete the survey.
Respond to issues raised for the teacher
The survey platform may include an option to provide feedback to the teacher about issues relevant to the teammate peer assessment process, the group assignment, or the academic programme generally. It’s important for the teacher to review and take prompt action on feedback received.
Publish provisional results for view by students
Perhaps around half of your students’ responses have been received mid-way to the survey due date. Nevertheless, there is value in publishing the progress results of the survey to students for valid assessed teams. Students can advise any issues to the teacher than emerge from the provisional feedback. Perhaps the teacher has inadvertently entered an incorrect team result for the team’s delivered output? Provisional publication also provides a ‘hurry up’ notification to students yet to respond. Ideally, laggards are prevented from viewing the feedback from their teammates until they submit their own response.
TIP! Before you release ANY results to students ensure they are prepared to receive and learn from the feedback they will receive. This preparation should be conducted in a face-to-face session or virtual meeting with your class.
Review the your preparation for giving and receiving feedback presented in Step 3 – Train your students, Chapter 5. Preview Step 6 – Courageous conversations, Chapter 8.
How Peer Assess Pro helps
Table 7.1 How Peer Assess Pro helps manage a peer assessment survey
| Feature | Benefit |
|---|---|
| Active warnings | A well-structured interface that identifies and enables proactive action by the teacher on the most important matters pertinent to their management of the survey. |
| Due date reminder | The teacher is prompted to undertake survey management activities at the survey mid-point and as the due date is neared.The notification gives timely opportunity to intervene with unsatisfactory responses, and the flexibility to allow or forbid late responses. |
| At risk and unacceptable ratings identified | Easy identification of unacceptable rating behaviour by teams and individuals. Identified behaviours include lazy raters, conspiratorial teams, outlier raters, at risk students, and unrealistic self-assessments. |
| Quick probe | The teacher can quickly probe an individual team and its teammates to review the peer assessed scores given and received, and whether the qualitative feedback reflects the peer assessed ratings. |
| Resubmission request | The platform generates a pro-forma request for the teacher to notify specific students to resubmit the survey and/or extend the qualitative justification of the peer assessment ratings given. |
| Exploratory data analysis | The survey data are displayed in several interactive charts, tables and statistics with options for exploratory data analysis and sorting. |
| Results preview | Preview and publish facility enables the teacher to calibrate the choice of method for calculating the personal result from team result and peer assessed score, before making the decision to publish the results for students’ view. Only results for valid-assessed teams are made available for student view. |
| Published results update | Provisional results published to students may be updated as further survey results are received and/or the teacher adjusts team results or result methods. Students are notified only when their specific results are changed. |
| Valid assessed teams | To reduce visual clutter and avoid distracting results, the display of results is restricted to teams where at least one half of teammates have submitted the peer assessment survey. Feedback results are hidden from both the teacher and teammates until a team qualifies as valid. For teams of 5 or less at least 3 responses must be received to be valid. |
| Survey history log | Permanent track-and-trace survey history log of actions taken by the platform, and notifications communicated to students and teacher. Reduces risk of dispute by students about whether they received survey requests or results. |
| Teamset update | The teacher can conveniently update the team arrangement any time between survey creation and date of finalisation. There is minimal impact on the survey except for teams that have additional teammates added, who must submit an additional response. |
| Due date reminder - students | The students are reminded automatically to complete the survey before the due date, only if they have yet to respond. |
| Soft due date | Allows students to submit survey responses until the teacher explicitly finalizes the survey. |
| Teacher feedback | Students may provide the teacher with anonymous feedback about concerns or suggestions about the conduct of peer assessment, group work, or the class generally. |
| © Peer Assess Pro Ltd | |
PREVIOUS: STEP 4 — Create the peer assessment NEXT: STEP 6 — Courageous conversations
Return to top of page
