A tool call comes back. content: [], isError: false, HTTP 200. The span
closes green. The model reads it as there is nothing there and writes the next paragraph on
top of that.
There were three possible worlds behind that response and the wire can only carry two of them:
Two and three are byte-identical. Not similar — identical. There is no field to inspect, no attribute to assert on, nothing an observability layer can surface, because the correct output and the broken output are the same artifact. No amount of looking finds it.
I expected to find the field and to have missed it. I did not find the field.
CallToolResult is three fields, and it is identical across all
four published schema versions. Zero occurrences of partial, coverage,
truncated or skipped in roughly 3,200 lines of schema.ToolMessage.status and LlamaIndex's
ToolOutput.is_error both fold found nothing into succeeded.ProgressNotification already carries progress and
total — a real denominator, sitting right there in the protocol — but it is
out-of-band and never gets attached to the result the model actually reads.So this is a convention gap, not a capability gap. _meta permits the fix today. Nobody
uses it for this.
SAP SuccessFactors returns 200 {"d":{"results":[]}} when role-based permissions deny
you the records. Not a 403. A well-formed empty result set, in the same shape as a successful query
that genuinely matched nothing.
Every client I have read does .get("d",{}).get("results",[]) straight through it. The
denial and the empty are the same list, and the agent above reports no records found with
total confidence.
0.5 on zero answers. 0.5 is a value a person can
deliberately choose — so nobody answered and picked dead centre on every item
were the same number, and it got written into an export whose own comment calls it durable and
replayable. Wrong forever rather than wrong until fixed, because the distinction was destroyed at
write time.1.0 when the milestones exist and simply fail to parse.
The caller marks the sample valid: True, error: ''. A sample that could not be scored is
indistinguishable from one that scored perfectly.0 for zero verdicts — where 0 means
no hallucination detected, i.e. it passes.It can only ever fail upward. A parse failure inflates the score. An empty result reads as a confident does not exist. A silent auth scope-down looks like a clean small answer.
Nothing downstream is motivated to catch it, because there is no angry user filing a bug that says their model scored too well. The asymmetry is the whole reason this class outlives the people who wrote it.
_meta: { searched: 1200, matched: 0, truncated: false }
Then "0 of 1200 rows, and I looked at all of them" and "0 of 0, because permissions handed me an empty index" stop being the same sentence to the model reading it.
The honest caveat: an unenforced optional convention is historically a weak thing to bet on. CONSORT worked in clinical trials because journal editors reject papers without a participant flow diagram — there was an enforcement layer. There is no equivalent here yet. So I would not tell you this gets adopted. I would tell you that you can put it in your own server this afternoon and stop lying to your own agent.
Grep your tool handlers for a return that produces the same shape on the empty path and on the denied-or-failed path. Mine had four. It takes about twenty minutes and the answer is usually uncomfortable.
I do this for money — one day, together, on your codebase, going after exactly this class of thing. What that looks like. And if you would rather just have the answer: send me one repo and I will run the check over it and mail you what it finds, free, once. It found nothing is the more common result and I will tell you that plainly.
This is one of fourteen. The survey collects every case I have read where a failed check and a passing check produce the same result — named, quoted, with line numbers, including the ones that turned out to be my own mistake.