Performance of generative pre-trained transformer-4 on the certification test for mental health management: A factorial design

DOI Web Site Web Site PubMed 3 References
  • Watanabe Kazuhiro
    Occupational Health AI Research Group for the Japan Society for Occupational Health Department of Public Health, Kitasato University School of Medicine
  • Tsutsui Yasuhiro
    Occupational Health AI Research Group for the Japan Society for Occupational Health Fukuoka Occupational Health Support Center
  • Tsutsui Takao
    Occupational Health AI Research Group for the Japan Society for Occupational Health Krosaki Harima Corporation Healthcare Plaza
  • Yamauchi Takenori
    Occupational Health AI Research Group for the Japan Society for Occupational Health Department of Hygiene, Public Health and Preventive Medicine Showa University, School of Medicine
  • Uchida Mitsuo
    Occupational Health AI Research Group for the Japan Society for Occupational Health Center for Mathematics and Data Science, Gunma University
  • Hachiya Yuriko
    Occupational Health AI Research Group for the Japan Society for Occupational Health Department of Work Systems and Health, Institute of Industrial Ecological Sciences, University of Occupational and Environmental Health, Japan
  • Kim Ilsung
    Occupational Health AI Research Group for the Japan Society for Occupational Health Occupational Medicine Group, Health Promotion Division, Toyota Motor Corporation
  • Iida Mako
    Department of Mental Health, Graduate School of Medicine, The University of Tokyo
  • Imamura Kotaro
    Department of Digital Mental Health, Graduate School of Medicine, The University of Tokyo
  • Sakuraya Asuka
    Department of Digital Mental Health, Graduate School of Medicine, The University of Tokyo
  • Kawakami Norito
    Department of Digital Mental Health, Graduate School of Medicine, The University of Tokyo

Bibliographic Information

Other Title
  • メンタルヘルス・マネジメント（R）検定試験に対するChatGPT（GPT-4）のパフォーマンス：要因計画法を用いた検討
  • メンタルヘルス・マネジメント検定試験に対するChatGPT(GPT-4)のパフォーマンス : 要因計画法を用いた検討
  • メンタル ヘルス ・ マネジメント ケンテイ シケン ニ タイスル ChatGPT(GPT-4)ノ パフォーマンス : ヨウイン ケイカクホウ オ モチイタ ケントウ

Search this article

Description

<p>Objective: This study aimed to investigate the performance of generative pre-trained transformer-4 (GPT-4) on the Certification Test for Mental Health Management and whether tuned prompts could improve its performance. Methods: This study used a 3 × 2 factorial design to examine the performance according to test difficulty (courses) and prompt conditions. We prepared 200 multiple-choice questions (600 questions overall) for each course using the Certification Test for Mental Health Management (levels I–III) and essay questions from the level I test for the previous four examinations. Two conditions were used: a simple prompt condition using the questions as prompts and tuned prompt condition using techniques to obtain better answers. GPT-4 (gpt-4-0613) was adopted and implemented using the OpenAI API. Results: The simple prompt condition scores were 74.5, 71.5, and 64.0 for levels III, II, and I, respectively. The tuned and simple prompt condition scores had no significant differences (Odds ratio = 1.03, 95% Confidence interval; 0.65–1.62, p = 0.908). Incorrect answers were observed in the simple prompt condition because of the inability to make choices, whereas no incorrect answers were observed in the tuned prompt condition. The average score for the essay questions under the simple prompt condition was 22.5 out of 50 points (45.0%). Conclusion: GPT-4 had a sufficient knowledge network for occupational mental health, surpassing the criteria for levels II and III tests. For the level I test, which required the ability to describe more advanced knowledge accurately, GPT-4 did not meet the criteria. External information may be needed when using GPT-4 at this level. Although the tuned prompts did not significantly improve the performance, they were promising in avoiding unintended outputs and organizing output formats. UMIN trial registration: UMIN-CTR ID = UMIN000053582</p>

Journal

  • SANGYO EISEIGAKU ZASSHI

    SANGYO EISEIGAKU ZASSHI 66 (6), 303-313, 2024-11-20

    Japan Society for Occupational Health

References(3)*help

See more

Keywords

Details 詳細情報について

Report a problem

Back to top