AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology I: Literature Review
📄 arXiv:2607.25672 · 📥 PDF · 2026-07-28 · gr-qc
Authors: Anamaria Hell [arXiv · scholar] , Kateryna Vovk [arXiv · scholar] , Veena Krishnaraj [arXiv · scholar] , Jia Liu [arXiv · scholar] , Kosuke Aizawa [arXiv · scholar] , Adrian E. Bayer [arXiv · scholar] , Linda Blot [arXiv · scholar] , Jessica Cowell [arXiv · scholar] , Suyog Garg [arXiv · scholar] , Jonathan Grée [arXiv · scholar] , Ben Horowitz [arXiv · scholar] , Masaya Ichikawa [arXiv · scholar] , Kanyuni Iemoto [arXiv · scholar] , Keigo Kondo [arXiv · scholar] , Zacharie Lorsin [arXiv · scholar] , Kevin McCarthy [arXiv · scholar] , Jamie Robinson [arXiv · scholar] , Miguel Ruiz-Granda [arXiv · scholar] , Leander Thiele [arXiv · scholar] , Ievgen Vovk [arXiv · scholar] , Mingshen Zhou [arXiv · scholar]
🕰 Orloj analysis
Tato studie zkoumá schopnost velkých jazykových modelů (LLM) asistovat při rešerších literatury ve fyzice, astrofyzice a kosmologii, porovnávajíc výsledky AI s lidskými experty. Zjišťuje nízký překryv a vysokou míru halucinací (zejména nesoulad metadat), ačkoli budoucí modely vykazují výrazné zlepšení.
💡 Jde o včasnou a hodnotnou empirickou studii, která kvantifikuje současné limity a potenciál AI pro vědecké rešerše, s důležitými poznatky o spolehlivosti generovaných referencí.
Categories:
MET-1
✓ controlled_study, modest_claims, falsifiable
⚠ Use of future/unreleased AI models (mid-2025, 2026) without clear explanation of access or simulation methodology, Limited sample size for advanced model evaluation (single-project test for ChatGPT Pro 5.5)
📄 Abstract
We investigate how well large language models (LLMs) can assist with literature reviews for scientific research. We perform a controlled study of eight expert-conceived research projects across the areas of physics, astrophysics, and cosmology. Each project has a defined background and goal, and human experts and AI prompters are asked to perform identical literature review tasks in parallel. We compare the relevant literature selected by humans with that selected by mid-2025 LLMs (ChatGPT-4o, ChatGPT Deep Research, and Gemini). We find the overlap between human- and AI-selected references to be small ($<$6\%), indicating that AI models do not yet reproduce a competent expert search on their own, though they have the potential to complement literature searches by humans. We then assess the reliability and completeness of AI-generated candidate references, distinguishing two types of hallucination: fabrications (references to nonexistent papers) and metadata mismatches (real papers with one or more incorrect fields). We find that while fabricated references make up 3\% of the AI-generated references, 64\% are real papers with at least one incorrect field (title, author, year, journal, DOI, or link), indicating that the mid-2025 models require systematic verification. However, the performance is significantly improved for the 2026 model ChatGPT Pro 5.5, with a single-project test showing zero fabrication or metadata mismatches.