A New Era in Digital Health: Evaluation of Artificial Intelligence Supported Chatbots (ChatGPT-4, BingAI, and Gemini) Responses on Osteoporosis Dijital Sağlıkta Yeni Bir Dönem: Yapay Zeka Destekli Sohbet Robotlarının (ChatGPT-4, BingAI ve Gemini) Osteoporozla İlgili Yanıtlarının Değerlendirilmesi
Turk Osteoporoz Dergisi, cilt.31, sa.3, ss.162-168, 2025 (Scopus, TRDizin)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 31 Sayı: 3
- Basım Tarihi: 2025
- Doi Numarası: 10.4274/tod.galenos.2025.01488
- Dergi Adı: Turk Osteoporoz Dergisi
- Derginin Tarandığı İndeksler: Scopus, CINAHL, EMBASE, TR DİZİN (ULAKBİM)
- Sayfa Sayıları: ss.162-168
- Anahtar Kelimeler: Osteoporosis, artificial intelligence, chatbot, ChatGPT, readability, health communication
- Açık Arşiv Koleksiyonu: AVESİS Açık Erişim Koleksiyonu
- Sağlık Bilimleri Üniversitesi Adresli: Evet
Özet
Objective: This study aimed to evaluate and compare the quality and readability of osteoporosis-related information generated by three artificial intelligence (AI) chatbots: ChatGPT-4, BingAI, and Gemini. Materials and Methods: 25 frequently asked questions about osteoporosis (obtained via Google Trends through Gemini) were submitted to each chatbot on December 23, 2024. The first responses were evaluated for readability [Flesch-Kincaid Reading Ease (FKRE) and Flesch Kincaid Grade Level (FKGL)] and quality (EQIP tool). Two experienced clinicians assessed the accuracy and completeness using Likert scales. Results: The mean FKRE scores were 34.5±12.9 (ChatGPT-4), 33.8±14.3 (BingAI), and 36.1±10.9 (Gemini), indicating difficulty in reading the texts. The FKGL scores ranged from 11.2 to 12.5, suggesting that college-level reading ability was required. However, BingAI (EQIP: 55.4±7.9) and Gemini (54.4±8.8) outperformed ChatGPT-4 (48.6±6.3) in terms of quality (p=0.005). Accuracy and completeness were high across all models, with mean scores exceeding 4.3/5 for each. Conclusion: While all three AI chatbots delivered accurate and complete answers on osteoporosis, their content readability remained suboptimal. BingAI and Gemini provide higher-quality information, possibly due to real-time web integration. Future chatbot development should focus on enhancing readability and real-time data access to support effective health communication, particularly in conditions such as osteoporosis, where patient understanding is crucial.