Sign In

Communications of the ACM

ACM TechNews

ChatGPT Answers More than Half of Software Engineering Questions Incorrectly

View as: Print Mobile App Share:
X in speech bubble, illustration

Users often fail to identify incorrectness or underestimate the degree of error in a ChatGPT answer.

Credit: Getty Images

ChatGPT answered 52% of 517 Stack Overflow questions incorrectly, and 77% of its answers were unnecessarily wordy, according to in a study by Purdue University researchers.

The study found that ChatGPT gave comprehensive answers to software engineering questions, addressing all aspects of the question, 65% of the time.

Researchers also asked 12 individuals with different programming skill levels to assess the ChatGPT-generated answers. "Users overlook incorrect information in ChatGPT answers [39.34% of the time] due to the comprehensive, well-articulated, and humanoid insights in ChatGPT answers,” the researchers say.

View Full Article


Abstracts Copyright © 2023 SmithBucklin, Washington, D.C., USA


No entries found

Sign In for Full Access
» Forgot Password? » Create an ACM Web Account