Deterministic, no-LLM gate for LLM output - traces every claim to an evidence bank, scores quality with a fixed rubric, and flags repetition runaway, so the check is auditable and repeatable.
veritas-gate is a small, zero-dependency library that gates LLM output with deterministic rules
instead of an LLM judge. You supply an evidence bank and a set of things the subject may not claim;
it flags claims that contradict them, scores quality with a no-LLM rubric, and catches
repetition-runaway that passes both honesty checks and length floors.
It is fast, reproducible, and auditable. It is not a general hallucination detector, and this
README includes the benchmark that shows exactly where the line is.
from veritas_gate import TruthChecker, rubric_score, is_degenerate
gate = TruthChecker(
experience_evidence="Built a RAG system over a 35,000+ doc corpus that cut research time 85%.",
forbidden_skills=["kubernetes", "lora", "fine-tuning"], # things the subject has NOT done
forbidden_claims="cloud-native\nfull-stack",
)
gate.check("Built a RAG pipeline over a 35,000+ document corpus.").is_valid # True
gate.check("Expert in Kubernetes and LoRA fine-tuning.").is_valid # False (fabrication)
gate.check("I have no experience with Kubernetes.").is_valid # True (honest omission)
is_degenerate("AI futures, AI potentials, AI possibilities, " * 100) # True
Run the full demo: python -m veritas_gate.example
Measured against RAGTruth
Most tools in this space assert that rules beat an LLM judge and stop there. This one was measured
against RAGTruth (Niu et al., ACL 2024), a corpus of
human-annotated LLM responses labeled for hallucination against their source passages.
Reproduce it:
python benchmark/fetch_data.py # 36 MB, fetched to a git-ignored dir, not vendored
python benchmark/run.py
Two different things run under benchmark/, and the numbers below are only ever the first.
| | what it is | what it measures | writes results.json? |
|---|---|---|---|
| python benchmark/run.py | the external RAGTruth evaluation, against the fetched corpus | generic-grounding performance — every number in this README | yes |
| python benchmark/run.py --fixture | an offline regression run against a committed synthetic fixture | loader, harness and scoring branches only | no, never |
The fixture contains no corpus text: RAGTruth derives from CNN/DailyMail, MS MARCO and the Yelp
Open Dataset, whose terms its own MIT licence does not relicense, so nothing from it is committed
here. The fixture is generated by benchmark/fixtures/build_synthetic_fixture.py. Its tp/fp/fn are
properties of that synthetic data and say nothing about performance on RAGTruth — a test enforces
that a fixture run cannot write results.json or label itself as a corpus run. See
benchmark/DATA-PROVENANCE.md.
What was measured. Only the two checks whose decision logic contains no domain vocabulary:
unverified_metric (a %/$/multiplier absent from the evidence) and unverified_count (a
count-anchored magnitude absent from the evidence). Every résumé-specific rule was disabled, and
predictions were filtered to those two violation types so nothing else could leak into the score.
Result on the 2,700-response test split, 450 source clusters, 34.9% base rate:
† published in the RAGTruth paper, Table 5. Cited, not reproduced here.
Precision and recall intervals are Wilson 95%. The F1 interval is a bootstrap that resamples whole
source clusters, because six model responses share each source passage and a per-response interval
would assume an independence the corpus does not have.
The generic checks do not beat a trivial classifier. F1 5.0% against a 51.8% floor. That is the
honest headline and it is not going to be tuned away, because the reason is structural: both
surviving checks are digit matchers, and only 20.8% of RAGTruth's 14,289 annotated hallucination
spans contain a digit at all. The rest are fabricated names, relations and entities. Recall is
capped near 0.21 by construction.
The one number that holds up is precision: when it fires, it is right 50% of the time against a
34.9% base rate. The interval's lower bound is 36.6%, so even that is weak evidence of lift.
Speed and determinism do hold. Across three runs of the full test split: 0.288–0.298 ms per
response building a fresh checker and evidence bank each time, and 0.122–0.127 ms for check()
alone once a checker is reused. results.json is byte-identical across runs, timing excluded from
it on purpose. At a judge latency of 1–3 s that is three to four orders of magnitude cheaper, with
no drift. That part is real, and it is the honest reason to reach for rules: as a cheap
deterministic pre-filter inside a domain you have configured, not as a replacement for semantic
verification.
An aside worth its own line. The RAGTruth paper's prompt-GPT-3.5 baseline scores F1 52.9%. The
trivial always-say-hallucinated classifier scores 51.8% on the same split. A widely-cited
LLM-as-judge baseline beats "always answer yes" by 1.1 points. Whatever else this benchmark shows,
it is worth knowing what these numbers are being compared against.
What was not measured. The résumé-specific rules (forbidden skills, credentials, employer
attribution, the entity index, rubric_score) have no ground truth here and were excluded rather
than scored on a corpus they were not built for. Span-level localization was not attempted; the gate
returns a verdict, not character offsets.
Grounding checks for the other 79.2%
TC-3/TC-6 above are digit matchers, structurally blind to any hallucination span with no digit in
it. src/veritas_gate/grounding.py adds three checks aimed at that remainder, all opt-in and OFF by
default (TruthChecker(..., enable_broadened_numeric=True, enable_entity_grounding=True, enable_novelty_check=True)):
unverified_number — every digit token in the draft absent from the evidence, dropping TC-3/
TC-6's %/$/multiplier/count-noun restriction.
ungrounded_entity — a capitalized token the evidence never mentions (a cheap proxy for a
fabricated name), excluding common sentence-openers so ordinary capitalization from sentence
position isn't mistaken for a name.
novel_content_window — a sliding window of stopword-filtered content words where the whole
window is absent from the evidence. The window size and novelty threshold (10 tokens, 60% novel)
are the winner of a train-split-only grid search (python benchmark/tune.py) over window sizes
3/5/7/10 and thresholds 0.6/0.8/1.0 — every other point scored lower on train.
The F1 lower bound (54.0%) does clear the trivial floor (51.8%) — a real, if narrow, margin. It is
still scored DOES_NOT_WORK by this project's pre-registered rule (benchmark/metrics.py,
written before this number existed): WORKS and HIGH_PRECISION_FLAGGER_ONLY both require a
precision lower bound of at least 60%, and this ensemble's is 38.1%. It fires on 2,255 of 2,700
responses — 84% of the test split — which is also why recall is 96.0%: at that fire rate, missing a
real hallucination is hard, and so is being right about it.
Per task, not pooled — the pooled number hides two different stories:
| task | n | naive F1 | ensemble F1 | verdict |
|---|---|---|---|---|
| Data2txt | 900 | 78.3% | 78.3% | fires on 100% of responses — ties the floor exactly, no signal |
| QA | 900 | 30.2% | 34.5% | real margin over the floor |
| Summary | 900 | 37.0% | 41.6% | real margin over the floor |
Data2txt's recall is 100.0% and its precision (64.3%) equals that task's own base rate exactly —
the ensemble fires on every single Data2txt response, so its F1 matching the naive floor is
arithmetic, not detection. QA and Summary genuinely beat their own floors, by 4.3 and 4.6 points —
modest, real, and the only two task types where these checks are doing something a coin flip isn't.
The honest read: three cheap, zero-dependency, literal-token checks recover a real (if thin)
signal on two of three task types, at a precision too low to trust unsupervised. Useful as a
pre-filter to route to a human or a judge model, not as a standalone gate.
What it checks
TruthChecker — deterministic verification against a configured evidence bank:
Forbidden and over-claim phrases — substring-robust, whitespace- and hyphen-normalized.
Not-claimable skills — word-boundary matched, with an honest-omission whitelist so
"I have no experience with X" is a disclosure, not a claim.
Unverified impact metrics and counts — a %/$/×/large-count in the draft that is not in
the evidence is flagged. These are the two checks the benchmark above measures.
Misattribution (optional) — flags a personal-project signature under an employer block;
config-driven, inert unless you supply the markers.
Credentials and named entities — asserted but not evidenced, flagged.
Grounding checks (optional, see above) — broadened numeric, ungrounded entity, and
content-word novelty; measured DOES_NOT_WORK standalone, opt-in for a reason.
Claim registry (optional) — pass claim_rules=[...] and every row's rule is enforced at
check time; see below.
Note what this list is not: it does not parse arbitrary prose into claims and verify each one. It
enforces the constraints you configure. The benchmark exists because that distinction matters and is
easy to blur.
rubric_score — a deterministic 0–100 quality score with no LLM judge, built on keyword
alignment without stuffing, title and intent alignment, real-metric density, and parseable
structure.
is_degenerate — catches repetition-runaway output (collapsed vocabulary, or a long run of
comma-items sharing a leading word) that passes both the honesty checks and any length floor.
check_claim_rules — a declarative registry where one row is both the rule the gate enforces
and the instruction the prompt renders, so the two cannot drift apart. Callable standalone, or
passed straight to TruthChecker(claim_rules=[...]) so it runs as part of check(). See below.
The claim registry, and where declarative rules stop working
When a pipeline keeps producing the same class of wrong claim, the usual fix is another hand-written
detector. Do that seventeen times and each nuance lives in two places that drift: the prompt re-types
the fact as an instruction, the checker encodes it as a regex. A policy written in a spec file
reaches the model immediately, because the model reads prose — and stays invisible to the gate until
it leaks into production and someone writes another regex.
from veritas_gate import check_claim_rules, prompt_rules_block
rules = [{
"id": "initiative-attribution",
"violation_type": "attribution_error",
"severity": "high",
"prompt_instruction": "The retrieval platform was self-initiated; the migration was assigned.",
"subjects": {
"self_initiated": {"any_of": ["retrieval platform"]},
"assigned_work": {"any_of": ["migration workstream"]},
},
"forbid": [{
"subject": "self_initiated",
"within_sentence": ["was assigned", "tasked with"],
"message": "the retrieval platform was self-initiated, not assigned",
}],
}]
# TRUE sentence covering both subjects -> passes
check_claim_rules("I initiated the retrieval platform and was assigned to the migration "
"workstream.", rules) # []
# the term attached to the wrong subject -> fires
check_claim_rules("I was assigned to the retrieval platform.", rules) # 1 finding
prompt_rules_block(rules) # the same row, rendered for the system prompt
Nearest-marker attribution is the mechanism worth stealing. One real sentence often covers
several subjects legitimately, so banning a term whenever the sentence mentions a subject would
flag correct writing. Co-occurrence is the wrong test; ownership is the right one. Each forbidden
term attaches to whichever subject's marker sits nearest it.
Now the part most rules engines leave out. The original goal was to retire every hand-written
detector into rows here. That was tried and rejected. Rules here match literal lower-cased
substrings with no word boundaries, and at scale that is not a subtle problem — it produces both
false positives (a short term sits inside a longer, unrelated word) and false negatives (a rule
translated out of code loses the word-boundary/context logic that made the original check correct).
The cited finding, plainly labeled as such. In 2026, a prior version of this same substring-rule
approach was measured against 9,963 real generated documents (36.7M characters) from a private
corpus of real personal application data — never vendored into this public repo, and not
reproducible from anything here. That run found a two-character term (rl) fired on 3,410 of 9,963
documents as a registry row versus 107 for the equivalent word-boundary regex (a 31.9× blast
radius); done/grow/initial matched inside abandoned/outgrew/uninitialized; a
single-word company-name ban matched an unrelated company containing it as a substring; and
translating one numeric check into rows produced 112 false negatives on the incident class it
existed to catch. Cited, dated, and not verifiable by a reader of this repository — the corpus
it ran against will never be published.
What a reader CAN verify: the mechanism, not that specific number. Bare-substring-vs-no-digit
matching failure is a property of English text in general, not of that one private corpus.
benchmark/registry_blast_radius.py demonstrates it on the same public, pinned RAGTruth corpus this
whole benchmark uses:
python benchmark/fetch_data.py # once
python benchmark/registry_blast_radius.py # public mode, writes registry_blast_radius.json
Measured on RAGTruth's 2,700-response test split: rl never occurs as its own word (0
word-boundary matches) yet a bare-substring rule would still fire on 598 of 2,700 documents —
100% false positives, purely from words like airline, beverly, clearly, disorderly, earlier, girl,
nearly, orlando, world. Not the same number as the private-corpus finding (different corpus,
different domain, and rl happens to have zero legitimate uses in RAGTruth's news/QA/data-record
text where the private corpus's technical writing had 107) — but the same mechanism, reproducible by
anyone who clones this repo, with no private data involved.
A --root <path> private mode exists for running the same check against your own local documents;
it is never the default, and prints only aggregate counts — never document text, never file paths.
So a rule belongs in the registry when its terms are multi-word and cannot occur inside a larger
word. A rule belongs in code when it needs word boundaries, cross-sentence state, occurrence
counting, open-ended numeric comparison, negative lookbehind, or the surrounding document. Adding a
boundary-sensitive rule to a registry like this does not make the gate stricter — it makes it wrong
in both directions at once, over-firing on substrings while under-firing on the real pattern.
That boundary is not a limitation to fix later. It is the finding.
Design notes
Blocklist → allowlist tradeoff. Skill claims are gated by a not-claimable blocklist with an
omission whitelist; the honest "no experience with X" sentence must pass while the affirmative
"expert in X" must fail. Per-clause detection prevents an honest sentence from whitelisting an
affirmative claim elsewhere on the same line.
Calibrating anti-stuffing. Aggregate keyword density wrongly penalizes a concise skills list,
so the penalty keys on per-term over-repetition instead; only the excess is docked.
"Quantified" means a real impact metric. Counting "any digit" rewards version numbers and
years; the rubric counts only %/$/×/thousands magnitudes.
Catching runaway that passes everything else. A repetition loop is truth-valid and long, so it
needs its own detector: unique-token ratio plus a leading-word run check.
Vocabulary was not tuned on the benchmark. The count-noun list ships unmodified even though it
is résumé vocabulary that barely fires on news text. Extending it by reading RAGTruth would be
inventing detector vocabulary from the evaluation corpus, which is the failure this measurement
exists to avoid. The near-zero fire rate is the honest cost.
Install and test
pip install -e ".[dev]"
pytest -q
Zero runtime dependencies; tests are pure-Python and offline. CI runs the suite on Python 3.10–3.12.
License
MIT.
Ecosystem Role
Standard MoltPulse indexed agent.
Embed Badge
Show off your Pulse Score in your GitHub README to build trust and rank higher.
