Practically any software engineer on the planet uses some kind of combination of ChatGPT and StackOverflow as as part of their development journey.
One of the most interesting questions in this space is: “Can I use ChatGPT as a primary source of information, or do I still need StackOverflow?”.
The research conducted by Purdue University provides an answer to this question. It offers a comprehensive analysis of ChatGPT’s responses to questions from Stack Overflow, including an examination of their correctness, consistency, comprehensiveness, and conciseness.
The research conclusions
- The authors analyzed 517 questions from Stack Overflow and found that 52% of ChatGPT’s answers contained inaccuracies, while 77% were verbose.
- Despite these issues, users still preferred ChatGPT’s responses 39.34% of the time due to their comprehensiveness and articulate language style.
- Almost half of the correct debugging answers provided by ChatGPT used different logic, API, or library to solve a problem that produced the same output as human answers.
- ChatGPT’s answers were consistently comprehensive for all categories of Stack Overflow questions and did not vary with different question type, recency, or popularity.
- Answers to all questions, irrespective of the type, recency, and popularity, were consistently verbose. Specifically, answers to popular questions were not concise 84% of the time, while answers for average and not popular questions were not concise 74% and 72% of the time.
- Answers to old questions (83%) were more verbose than new questions (71%).
- Debugging answers were more concise (40%) compared to conceptual (16%) and how-to (13%) answers, which were extremely verbose.
- The authors conducted a user study and found that real programmers consider answer correctness, quality, and linguistic features when choosing between ChatGPT and Stack Overflow.
So which one should you choose?
- ChatGPT should be used as a complementary tool to Stack Overflow, rather than a replacement.
- ChatGPT’s responses should be edited for conciseness and accuracy before being posted on Stack Overflow.
- ChatGPT’s training data should be improved to include more software engineering-specific content.
- Future research should investigate the use of ChatGPT for other types of software engineering tasks, such as code generation and code summarization.
- Developers should be aware of the limitations of ChatGPT and use their own judgment when deciding whether to rely on its answers.
A bright future ahead
As you know, the AI and LLMs space is moving fast. Just recently, StackOverflow released OverflowAI, showing that everyone wants a piece of the AI pie.
Is the future lays in the hands of machine generated content, such as ChatGPT? Or hand-written content, which is constantly edited and validated by real people, as done for years in StackOverflow?
Time will tell.
Leave a Reply