AI research
A research paper titled "Ontology--Grounded, Reasoner--Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI" (arXiv:2610.00682), published on September 30, 2026, by researchers from Siemens AG and the Technical University of Munich, introduces an automated pipeline for generating formally verified benchmarks to evaluate logical deduction in Large Language Models (LLMs). Addressing the critical issue of logical hallucinations and ungrounded inferences in domain--specific AI, the framework leverages Description Logic and OWL 2 ontologies to derive rigorous question--answering datasets.
The methodology extracts class definition axioms to ground correct answers, while generating distractors through systematic perturbations of class expressions. Crucially, the invalidity of each distractor is formally certified via automated OWL reasoners using entailment checks, eliminating manual curation bottlenecks and subjective annotation biases. Evaluated across domain ontologies—including DOID (biomedicine) and PMDco (materials science)—the pipeline produced over 17,000 verified multiple--choice problems spanning four granular semantic failure modes, from class unsatisfiability to weakened subsumptions. Benchmarking revealed that six frontier and open--source LLMs scored between 41.1% and 76.8% zero--shot accuracy, proving that surface linguistic fluency masks severe deficiencies in formal symbolic reasoning. The research is scheduled for presentation at the AIDaR Workshop during the 40th Conference on Neural Information Processing Systems (NeurIPS 2026) in Paris, taking place December 12–13, 2026.