Show HN: LLM Hallucination Detector – Works with GPT, Claude, and Local Models

I built a lightweight hallucination detector that works with any LLM API.

It checks for signs of hallucinated or unreliable output using a multi-method approach (overconfidence patterns, factual density, coherence, contradictions, etc).

What it does:

Works with GPT, Claude, local models (e.g., Mistral, DialoGPT)

Outputs a hallucination probability (0.0–1.0)

Flags overconfident or uncertain language

Scores factual density, coherence, and contradictions

Compares responses to context (if provided)

Fully framework-agnostic — no extra dependencies

Built for production + research workflows

Benchmarked on 1,000+ samples:

F1: 0.75

AUC-ROC: 0.81

Fast: ~0.2s per response

Comes with plug-and-play examples:

OpenAI, Anthropic, local models

Flask API

Custom scoring configs

I’m giving this away free under MIT. Would love feedback, issues, PRs — or just to know if it helps you build safer LLM apps.

GitHub: https://github.com/Mattbusel/LLM-Hallucination-Detection-Scr...

github.com

・

2 points

・

Shmungus

・

2 days ago

3 comments

Shmungus ・ 2 days ago

Hi HN!

I’m excited to share this lightweight hallucination detector I built to help identify unreliable or “hallucinated” outputs from LLMs like GPT, Claude, and various local models.

It uses multiple methods — from spotting overconfidence and contradictions to scoring factual density and coherence — to give a hallucination probability score for any generated response.

It’s framework-agnostic, fast (~0.2s per response), and designed for both research and production use. Plus, it’s completely free under the MIT license.

I’d love to hear your thoughts, feedback, and if you find it useful for your projects. Happy to answer questions or discuss how it works under the hood!

Thanks for checking it out!

akoboldfrying ・ 2 days ago

I'm impressed that you give precision and recall metrics for this -- and amazed that they are non-terrible. I'm amazed because a fully general hallucination detector is obviously a truth oracle -- it can answer any question about anything in the world, by framing the question as a statement and then asking whether that statement is a hallucination.

From among the analyses the tool makes, it makes sense to me that contradictions can be detected, since that doesn't require knowledge of the real world. I'm very interested in how you do this detection ("Logical inconsistencies") in practice. Likewise for "Logical progression".

Two questions:

1. Since "overconfidence" is treated as a red flag, won't applying your tool as a filter cause LLM response precision to drop, often unnecessarily? The safest answer an LLM can give to "When was the Eiffel Tower built?" is surely along the lines of "The Eiffel Tower may or may not have been built at some time in the past."

2. I don't see how this tool can detect the kind of hallucination that (a) involves no contradiction and (b) requires knowledge of the world. These come up often. Examples: Citing plausible-sounding but nonexistent court cases, calling plausible-sounding but nonexistent methods in an API.

  • Shmungus ・ 2 days ago

    Thanks, really appreciate the thoughtful questions and skepticism (and totally agree: a “perfect” hallucination detector would be a truth oracle).

    To your points:

    1. Overconfidence and precision You're right that filtering on overconfidence alone could tank precision. That’s why the tool doesn’t treat it as a strict red flag, it’s one of several signals, and the final hallucination score is a weighted combination of multiple metrics (confidence, density, contradictions, progression, etc.). Overconfident phrasing tends to correlate with hallucinations in aggregate, but the idea is never to penalize all confident answers, just to flag the ones where that confidence is unjustified by the context or content.

    2. Detecting hallucinations that require world knowledge Absolutely, those are the hardest cases. This tool doesn’t solve that. Instead, it acts as a proxy evaluator:

    Factual density gives a rough measure of “how many claims are being made”

    Overconfidence vs. ambiguity highlights where a model might be bluffing

    Logical coherence and contradiction checks flag when a model violates internal structure (not ground truth) But it won’t catch the subtle world-knowledge misses (like fake court cases or made-up API calls) unless you pair it with a grounded context or use external validators.

    The long-term hope is: use this tool to raise suspicion, not declare judgment. It's a cheap sanity layer, a “weak oracle” that’s fast, pluggable, and good enough to catch the dumb stuff before you escalate to expensive validators or human review.