Reference Hallucination in AI-Assisted Academic Writing: A Comparative Analysis of ChatGPT, Gemini, and Perplexity in Rotator Cuff Literature
Indian Journal of Orthopaedics, cilt.60, sa.8, ss.1949-1956, 2026 (SCI-Expanded, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 60 Sayı: 8
- Basım Tarihi: 2026
- Doi Numarası: 10.1007/s43465-026-01807-0
- Dergi Adı: Indian Journal of Orthopaedics
- Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, CINAHL, Academic Search Ultimate (EBSCO), Health Research Premium Collection (ProQuest)
- Sayfa Sayıları: ss.1949-1956
- Anahtar Kelimeler: Artificial intelligence, Reference hallucination, Rotator cuff
- Sağlık Bilimleri Üniversitesi Adresli: Evet
Özet
Objective: This study aimed to compare the reliability of references generated by three different AI-based conversational agents (ChatGPT, Gemini, Perplexity) within the rotator cuff literature. Methods: Designed as a cross-sectional study, 30 subtopics were posed to each bot in two formats (letter to the editor and original article), resulting in a total of 3150 references. Reference reliability was assessed using the Reference Hallucination Score (RHS), which included the following criteria: existence/verifiability, bibliographic accuracy, PMID validity, and topical relevance. Results: When all formats were analyzed together, ChatGPT had the lowest mean RHS (1.81 ± 3.40) and thus emerged as the most reliable model. Gemini scored 4.01 ± 4.89, while Perplexity had the highest score at 6.51 ± 4.89 (p < 0.001). In the letter format, ChatGPT, Gemini, and Perplexity scored 1.81 ± 3.20, 3.81 ± 4.83, and 6.43 ± 4.78, respectively. In the article format, ChatGPT scored 4.02 ± 1.70, Gemini 4.13 ± 1.57, and Perplexity 6.31 ± 1.68. ChatGPT was more reliable than Gemini and significantly superior to Perplexity. Conclusion: AI-based conversational agents may play a supportive role in academic writing; however, they exhibit critical limitations in reference accuracy. While ChatGPT was relatively the most reliable model, Perplexity had the highest hallucination rate. Researchers must refrain from using references generated by these tools without verification, as doing so poses significant concerns for scientific integrity and ethics.