How Much Do LLMs Hallucinate in Document Q&A Scenarios? A 172-Billion-Token Study Across Temperatures, Context Lengths, and Hardware Platforms
arxiv.org

Evaluating 35 open-weight models across three context lengths (32K, 128K, 200K), four temperatures, and three hardware platforms—consuming 172 billion tokens across more than 4,000 runs—we find that the answer is “substantially, and unavoidably.” Even under optimal conditions—best model, best temperature, temperature chosen specifically to minimize fabrication—the floor is non-zero and rises steeply with context length. At 32K, the best model (GLM 4.5) fabricates 1.19% of answers, top-tier models fabricate 5–7%, and the median model fabricates roughly 25%.

    • cmhe@lemmy.worldEnglish
      01·
      7 months ago

      Hallucinations of LLMs are just one class of errors, and the most dangerous one.

      Other stuff like garbeled or repeating output are other errors.

  • rekabis@lemmy.caEnglish
    0·
    7 months ago

    How much do large language models actually hallucinate when answering questions grounded in provided documents?

    Okay, this is looking promising, at least in terms of the most important qualifications being plainly stated in the opening line.

    Because the amount of hallucinations/inaccuracies “in the wild” - depending on the model being tested - runs about 60-80%. But then again, this would be average use on generalized data sets, not questions focusing on specific documentation. So of course the “in the wild” questions will see a higher rate.

    This also helps users, as it shows that hallucinations/inaccuracies can be reduced by as much as ⅔ by simply limiting LLMs to specific documentation that the user is certain contains the desired information, rather than letting them trawl world+dog.

    Very interesting!