- Do students who participate in in-flow peer revew of test cases perform better on assignments than those who simply write tests first?
- Does receiving feedback increase performance more than seeing other students’ work and composing feedback, or vice versa?
Two undergraduate courses:
- CS173 (Introduction to Programming Languages): Sophomores to seniors and grad students
- CS019 (Programming with Data Structures and Algorithms): Freshman who opt to take a more advanced introductory course in their first semester
We expect roughly n=90 based on previous enrollments for these courses (roughly 40 in CS173 and 50 in CS019).
Programming assignments in both courses will use the protocol described below for submitting and reviewing assignments.
All assignments will be released on Friday, and will need to be completed by the following Thursday at 11:59pm. There will be intermediate deadlines, detailed below.
The first 2-3 programming assignments will have a special structure to train students in the reviewing process (see Initial Assignments below). The remaining assignments will follow the process described here.
Each programming assignment will have a test suite deadline on the Tuesday after the assignment is released, at 11:59pm. At this deadline, students must submit only the test suite for the programming assignment. They are not forbidden from writing parts of the implementation before this point, but they do not submit the solution yet. Tests will either be submitted in a separate file from solution code, or extracted via a script provided by the course staff, in order to prevent accidental submission of implementations along with test suites.
Upon submission of their test suite, each student will be (randomly, but evenly), assigned to one of three groups, and notified of which group they are in:
- No Review (NR)
- Reviewer (RR)
- Reviewee (RE)
Note that the conditions are chosen per assignment, so a student will participate in each of the conditions (likely more than once) over the course of the semester.
The No Review group does not participate in any peer review, and simply must submit their implementations by the assignment deadline, and will be graded as described below.
The Reviewer group will be given three test suites from students in the Reviewee group, and will have to submit reviews (see Rubric and Forms for the review contents below) for those test suites by Wednesday, 11:59pm. If three submissions from the Reviewee group are not available when the Reviewer-assigned student submits, they will be assigned as Reviewee students submit, and the Reviewer will be notified by email that submissions are ready to be reviewed. Note that reviews may be assigned and completed before the Wednesday or even the Tuesday deadline, if students have submitted test suites sufficiently early.
The Reviewee group will be notified that they are being reviewed, and will receive reviews before the Wednesday deadline from their peers. If all goes well, they will receive three reviews, but note that delinquent Reviewers may cause them to receive fewer (see handling classroom exceptions below).
All groups must submit their implementations and updated test suites by the final deadline of Thursday 11:59pm. Along with their final submission, they must also fill out a form describing how they updated their submission based on the reviews they received or other work they saw. The exact contents of these are described in Rubrics and Forms below.
For the first 2-3 assignments in each course (which are largely “warm-up”) assignments that don’t count for a significant portion of the course grade), all students will participate as both reviewers and reviewees. This is to train students in the reviewing process and give the course staff to give feedback on reviews.
When reviewing, we will use the following prompts. The first three prompts have two fields for responding: a 6-point Likert scale from “Strongly Disagree” to “Strongly Agree”, and an (optional) comments field.
1: These test cases correctly reflect the desired behavior.
2: These tests are representative of the possible inputs.
3: This test suite is well-structured and well-implemented.
The final prompt’s resopnse is just free-form comments. This prompt is included in part to encourage students to have something to say on good work.
Describe something you liked about this test suite:
Along with their final submission, we ask students to fill out free-response questions. Students will be informed, and reminded at submission time, that these forms have no effect on their grade. They will be asked two out of the following four questions, depending on the condition they were in:
1. What, if any, changes did you make as a result of receiving reviews on
your tests?
2. What, if any, changes did you make as a result of reading others' tests and
giving feedback?
3. What, if any, value do you think you missed out on by not getting the
chance to receive reviews on your tests?
4. What, if any, value do you think you missed out on by not getting the
chance to read others' tests and give feedback?
The questions will be assigned as follows:
- No Review: 3 and 4
- Reviewer: 2 and 3
- Reviewee: 1 and 4
- Initial assignments: 1 and 2
Students will get 25% of their grade based on the quality of their initial test suites. If they submit test suites late, they lose these points. If students in the Reviewer group submit reviews late, they also lose these points.
Students get the rest of their grade based on the quality of their final test suites and implementations.
Reviews and review effect surveys do not effect students’ grades.
Students sometimes miss deadlines or have allowances for exceptional schedules because of family or medical emergencies, as well as other professional activites like attending conferences or job interviews. When possible, we will get notice from students in advance when they have an exceptional scheduling need and put them in the “No Review” condition for that assignment.
If enough students miss the test suite deadline that there aren’t enough Reviewee students, some submissions may need to be assigned more than three times in order for all the Reviewer students to complete three reviews. In this case, we will not show reviews past the third to the Reviewee.
If a Reviewer student misses the review submission deadline, a Reviewee student may receive less than three reviews. This is unavoidable, and we will note it if it happens in our resulting data.