The data in Table 1 can be used to draw inferences about the performance and reliability of the LLMs for grading student writing in an astronomy MOOC. Bootstrap resampling of the LLM scores yields standard deviations comparable to or lower than the standard deviations of scores from the instructor. For the astronomy course, across all the four prompts, this is true for 63% (12/19) of the questions using GPT-3.5 and 75% (15/20) of the questions using GPT-4. Next, considering the astrobiology course, this is true for 58% (7/12) of the questions using GPT-3.5 and 92% (11/12) of the questions using GPT-4. Finally, for the history and philosophy class, this is true for 56% (9/16) of the questions using GPT-3.5 and 88% (14/16) of the questions using GPT-4. Overall, these differences reflect lower internal variance when using GPT-4 than when using GPT-3.5.
Language Models versus Instructor
Next, we can look at the level of agreement between instructor scores and LLM scores. A clear pattern is seen from the data in Table 1, where across all prompts and questions, GPT-3.5 often scores the assignments higher than the instructor, 50% or 24/48 times, as it scores them lower than the instructor. On the other hand, GPT-4 generally scores assignments higher than the instructor, 83% or 40/48 times. In terms of differences between instructor scores and LLM scores relative to the LLM standard deviation from bootstrapping, other patterns are apparent. Table 2 summarizes the differences between instructor and LLM grades. The instructor grades are given in the top row of Table 2, as in Table 1. The subsequent rows treat instructor grades as “perfect” and list the number of standard deviations by which the LLM differs. The two cells with missing data are the cases where bootstrap resampling failed, as described earlier. In terms of root mean square (RMS) agreement of instructor between LLM scores, as anticipated from the comments above, GPT-3.5 performs worse with the history and philosophy class than the astronomy or astrobiology classes, and the same is true for GPT-4. There is little difference in the performance of either LLM between the astronomy class and the astrobiology class. The RMS results after averaging over all questions in each class are shown in Table 3. Two cells with numbers in parentheses reflect missing data for one question in the course. GPT-4 returns answers closer to instructor grades than GPT-3.5 for 3 out of the 4 prompts for the astronomy class and 3 out of the 4 prompts for the astrobiology class, but only 2 out of the 4 prompts for the history and philosophy class.
Table 2. Differences between Instructor and LLM Grades
Model
|
Prompt
|
Courses
|
Astronomy
|
Astrobiology
|
History and Philosophy
|
Q1
|
Q2
|
Q3
|
Q4
|
Q5
|
Q1
|
Q2
|
Q3
|
Q1
|
Q2
|
Q3
|
Q4
|
|
Instructor Grades
|
3.9
|
8.2
|
7.5
|
7.4
|
5.5
|
6.8
|
6.7
|
7.9
|
3.5
|
2.4
|
2.7
|
2.2
|
GPT-3.5
|
Instructor Answer
|
+4.0
|
-2.0
|
-1.0
|
-0.8
|
+1.2
|
+0.7
|
+0.4
|
-1.9
|
-5.5
|
+2.5
|
+0.0
|
+5.8
|
Instructor Answer + Rubric
|
-1.6
|
+0.4
|
-0.3
|
-0.8
|
+1.2
|
+0.7
|
+0.4
|
-1.9
|
…
|
+2.5
|
+0.0
|
+5.8
|
AI Rubric + Instructor Answer
|
+0.3
|
+1.0
|
-1.2
|
-0.7
|
+0.8
|
-0.2
|
-1.4
|
-2.2
|
-3.3
|
+0.3
|
-2.7
|
+0.0
|
AI Rubric Only
|
+3.0
|
…
|
+0.6
|
-3.0
|
+1.2
|
+0.6
|
+1.0
|
-0.3
|
-0.7
|
+8.5
|
+5.5
|
+1.7
|
GPT-4
|
Instructor Answer
|
+2.1
|
+2.3
|
+0.1
|
+0.2
|
+0.8
|
+1.0
|
+1.7
|
+0.6
|
+0.0
|
+3.5
|
+6.1
|
+4.1
|
Instructor Answer + Rubric
|
+1.2
|
+0.3
|
-0.3
|
-0.5
|
+0.4
|
+0.3
|
+0.8
|
-0.9
|
-1.0
|
+3.3
|
+2.5
|
+2.7
|
AI Rubric + Instructor Answer
|
+1.0
|
+1.0
|
+0.1
|
-0.3
|
+0.9
|
+0.3
|
+0.4
|
-1.3
|
-1.4
|
+2.3
|
+2.7
|
+3.2
|
AI Rubric Only
|
+3.4
|
+3.6
|
+0.5
|
+0.9
|
+1.4
|
+1.3
|
+2.0
|
+3.0
|
-1.1
|
+4.1
|
+5.4
|
+4.0
|
The best performance is observed for GPT-4 when the prompt includes both the instructor's model answer and their rubric, as well as when the prompt comprises the instructor's model answer plus an AI-generated rubric. In these instances, scores for 7 out of the 8 questions in the astronomy and astrobiology courses fall within one standard deviation. Notably, the worst performance for GPT-4 is seen when the prompt is solely an AI-generated rubric, followed by the situation where the prompt includes only the instructor’s model answer. However, across all four prompts with GPT-4 and spanning 12 assignments, the dispersion in the Language Model (LLM) grades is consistently less than or comparable to the dispersion in instructor grades. The agreement is expected to be closest to an instructor when using the 'Instructor + Rubric' prompt since it aligns closely with what the instructor will be grading. Incorporation of an AI-generated rubric has the potential to create more robust rubrics by leveraging correlations from instructor answers. It's important to note, however, that relying solely on rubrics may not be sufficient, and the presence of a master answer is crucial for ensuring greater reliability in the grading process.
Table 3. RMS Differences between Instructor and LLM Grades
Model
|
Prompt
|
Astronomy
|
Astrobiology
|
History and Philosophy
|
GPT-3.5
|
Instructor Grades
|
4.8
|
2.1
|
6.3
|
GPT-3.5
|
Instructor Answer
|
2.7
|
1.7
|
(4.7)
|
GPT-3.5
|
Instructor Answer + Rubric
|
1.9
|
2.5
|
6.3
|
GPT-3.5
|
AI Rubric + Instructor Answer
|
(4.6)
|
1.2
|
10.3
|
GPT-4
|
AI Rubric Only
|
3.2
|
2.1
|
8.1
|
GPT-4
|
Instructor Answer
|
1.4
|
1.2
|
5.0
|
GPT-4
|
Instructor Answer + Rubric
|
1.7
|
1.4
|
4.7
|
GPT-4
|
AI Rubric + Instructor Answer
|
5.2
|
3.8
|
7.9
|
Language Models versus Peer Grading
Peer assessment is often used in MOOCs and a major challenge is improving the reliability and consistency of this type of evaluation (Gamage, Staubitz, and Whiting, 2021). We analyzed peer grading for the astronomy course used here (Formanek et al., 2017). In that study, using data from 2015, peer review for 300 assignments out of 4 points had a mean score of 3.39, slightly lower than the instructor mean score, with a standard deviation of 0.78. In terms of reliability, unsurprisingly, instructor grades were the most reliable, followed by trained undergraduate graders, followed by peer graders. The correlation between the median instructor grades and the median peer grades was moderate (r = 0.49).
In this work, we compare instructor grades for a subset of the students in three courses with peer review grades for the entire course. Sample sizes are 19,661 for the astronomy course, 1705 for the astrobiology course, and 113 for the history and philosophy course. The numbers reflect the fact that the astronomy course started in 2015, the astrobiology course started in 2019, and the history and philosophy course started in 2022. Table 4 compares mean grades from the instructor and peer graders for the same questions as shown in Table 1. Uncertainties are one sigma standard deviations from the sample mean. Peer graders give higher scores and are more lenient than the instructor in almost every case, and the dispersion in their grades is higher than the instructor dispersion, by a factor of two for the astronomy and astrobiology courses, and by a factor of four for the history and philosophy course. We encountered some examples of student plagiarism in analyzing this data. Plagiarism and other forms of academic dishonesty are known to be an issue with some MOOCs (Surahman and Wang, 2021), but it does not affect the statistical results we report here.
Table 4. Instructor and Peer Grades
Source of Grades
|
Courses
|
Astronomy
|
Astrobiology
|
History and Philosophy
|
Q1
|
Q2
|
Q3
|
Q4
|
Q5
|
Q1
|
Q2
|
Q3
|
Q1
|
Q2
|
Q3
|
Q4
|
Instructor Grades
|
3.90 ± 0.54
|
8.2 ± 0.37
|
7.51 ± 0.92
|
7.41 ± 0.86
|
5.51 ± 0.94
|
6.8 ± 1.09
|
6.7 ± 0.83
|
7.89 ± 0.85
|
3.50 ± 0.21
|
2.39 ± 0.29
|
2.70 ± 0.20
|
2.20 ± 0.28
|
Peer Grades
|
5.13 ± 1.28
|
7.92 ± 1.66
|
7.97 ± 1.66
|
7.83 ± 1.81
|
7.98 ± 1.65
|
8.83 ± 2.19
|
8.77 ± 2.27
|
8.88 ± 2.08
|
3.50 ± 0.91
|
3.69 ± 0.70
|
3.65 ± 0.92
|
3.49 ± 1.10
|
Summarizing the three-way comparison between instructor grades, peer review grades, and the grades assigned by the LLMs, both GPT-3.5 and GPT-4 generally produce grades that are closer to the instructor grades and have lower dispersion than peer grades. This is true for all three courses. Given the different scoring schemes for different assignments we use a score normalized to one. For the introductory course, the mean scatter in normalized scores is 15% between the instructor and GPT-4, as opposed to 27% between the instructor and peer graders. For the astrobiology course, the mean scatter in normalized scores is 17% between instructor and GPT-4, as opposed to 22% between instructor and peer graders. Last, for the history and philosophy course, the mean scatter in normalized scores is 26% between instructor and GPT-4, as opposed to 38% between instructor and peer graders. LLMs clearly have the potential to act as proxies for the instructor, avoiding some of the pitfalls and limitations of using novices (other students in the class) to grade student assignments.
Comparisons for Individual Students
The comparisons just described are averages across all ten students sampled. However, a student cares about their own grade more than the class average, so we also made a direct comparison of instructor, GPT-4, and peer graded scores on all twelve assignments for the ten students individually. In addition, we investigated the dispersion among the peer grades, where there were four for each assignment in the introductory course, and three for each assignment in the astrobiology and history and philosophy courses. Figure 1 is a scatterplot of the difference in grades between instructor and median for the peer grades on the x-axis and the difference in grades between the instructor and GPT-4 on the y-axis. Histograms are also shown.
For the introductory course, the mean instructor minus GPT-4 score is -0.01 ± 0.15, compared to the mean instructor minus peer grading score of -0.07 ± 0.26. The LLM grade is very close to the instructor grade, with a smaller dispersion than the peer grade. For the astrobiology course, the mean instructor minus GPT-4 score is -0.02 ± 0.17, compared to the mean instructor minus peer grading score of -0.09 ± 0.20. The LLM grade is very close to the instructor grade, with a slightly smaller dispersion than the peer grade. Last, for the history and philosophy course, the mean instructor minus GPT-4 score is -0.09 ± 0.24, compared to the mean instructor minus peer grading score of -0.23 ± 0.30. The LLM grade is much closer to the instructor grade, and has a slightly smaller dispersion, than the peer grade. LLM grades and peer grades are always more lenient than instructor grades. Overall, GPT-4 performs better than peer grading.
To further understand why there was a higher discrepancy amongst peer grades, we retrieved from Coursera the individual peer grades for each of the students to see the dispersion of peer grades for each question. We calculate a representative score by taking the average of the 10 students and converting them into percentages. Then we calculate the mean absolute deviation for each of the peer grades and then choose the highest and lowest out of the sample of three or four to see the spread in the percentages for each of the question. This is shown in Figure 2. The dispersion ranges from 4% to 33% for the introductory course, from 0% to 26% for the astrobiology course, and from 0% to 20% for the history and philosophy course. The dispersion among peer grades for a particular assignment is substantial and it contributes to the dispersion in peer grades for the entire sample. This analysis of grades for individual students affirms that GPT-4 is superior to Coursera’s peer grading mechanism, and it comes very close to matching the grades assigned by the instructor.