Hallucination Detection on Code Generation with SelfCheckGPT

DOI Web Site 参考文献15件
  • Ito Waka
    Graduate School of Science Division of Mathematical and Physical Sciences, Japan Women's University
  • Obara Yui
    Graduate School of Science Division of Mathematical and Physical Sciences, Japan Women's University
  • Sato Miyu
    Graduate School of Science Division of Mathematical and Physical Sciences, Japan Women's University
  • Kuramitsu Kimio
    Department of Mathematics, Physics, and Computer Science, Japan Women's University, Bunkyo

書誌事項

公開日
2025
DOI
  • 10.2197/ipsjjip.33.487
公開者
一般社団法人 情報処理学会

説明

<p>Large language models (LLMs) are expected to bring automation and efficiency to software development, including programming. However, an LLM encounters a challenge known as “hallucination, ” where it produces incorrect content or outputs that deviate from input requirements. SelfCheckGPT is one of the methods designed to detect hallucinations. Its key feature lies in its ability to infer the occurrence of hallucinations without requiring reference data or test cases. Although SelfCheckGPT has been evaluated and applied in natural language processing tasks such as text summarization and question answering, its performance in code generation has not yet been explored. In this study, we applied SelfCheckGPT to the HumanEval dataset, a standard benchmark for code generation, and investigated its evaluation performance by comparing it with execution-based evaluations. The results revealed that calculating similarity using BLEU, ROUGE-L, and EditSim is adequate for predicting the correctness of code or, in other words, hallucinations.</p>

収録刊行物

参考文献 (15)*注記

もっと見る

キーワード

詳細情報 詳細情報について

問題の指摘

ページトップへ