Harvard Students Outperform ChatGPT by Two Letter Grades

Graduate students at Harvard University outperformed OpenAI’s GPT-4o model by more than two letter grades in a study conducted by researchers at the university.

The research focused on students enrolled in a molecular biology course. The findings indicate a significant performance gap between doctoral-level students and the AI model when tasked with course-related assessments.

Cognitive Performance and Critical Thinking

Researchers entered the study with specific hypotheses regarding the strengths and weaknesses of the chatbot. They expected that the AI would perform similarly to doctoral students on lower cognitive levels.

The researchers hypothesized that while ChatGPT would be capable of memorizing materials sufficiently, it would struggle when faced with critical thinking problems.

The results confirmed this disparity, as the graduate students’ performance exceeded that of the AI by more than two letter grades.

Comparison to Undergraduate Performance

The results of the graduate-level study contrast with previous experiments involving undergraduate-level work. In July 2023, Harvard student Maya Bodnick conducted an experiment to determine if ChatGPT-4 could pass her freshman year.

Bodnick asked seven Harvard professors and teaching assistants to grade unedited essays written by ChatGPT-4 in response to prompts assigned in their classes.

In that study, the AI’s essays received mostly A’s and B’s, resulting in a calculated GPA of 3.34. Bodnick noted that this score was lower than the Harvard average, but demonstrated a higher level of competency in freshman essay writing than the GPT-4o model showed in the graduate molecular biology study.