I ran four research agents in parallel this week to build a table — one row per company, each row carrying a source URL. The brief was explicit about what a source meant: a URL you actually fetched, supporting the claim in this row. It even pre-empted the obvious cheat, saying in so many words that an HTTP 200 is not a successful fetch.
All four came back clean. Each report ended with a list of the URLs that agent had retrieved. I merged the four tables into one file of sixty-three rows and had, as far as anything in front of me could show, sixty-three cited facts.
Then I cross-checked one agent's rows against that same agent's own list of fetched URLs. Eight cited domains were not on it.
So I asked. The answer came back immediately and without any hedging: all eight were snippets. Search-result excerpts, never a retrieval. Across the merged file, thirty of the sixty-three rows were snippet-derived.
Nothing was fabricated
That is the part worth being precise about, because it is tempting to file this as agents inventing citations, and that is not what happened.
Every URL was real. Every claim traced back to something that genuinely said it. No row was invented, no domain was hallucinated, and when I asked a direct question I got a direct and accurate answer on the first try. Judged as honesty, the agents were perfect.
They were also perfectly compliant. The brief told them not to fabricate, and they did not fabricate. It told them blank beats a guess, and they left things blank. Every instruction I wrote was followed.
The output was wrong anyway, because a snippet-sourced cell and a page-sourced cell are byte-identical, and I had asked for nothing that would tell them apart.
The granularity mistake
Here is the actual defect, and it is mine.
I asked for two things. Per row: cite the source for this claim. Per report: list the URLs you fetched. Both were answered honestly. Neither can be joined to the other.
"Here are the URLs I fetched" is a report-level artifact. "Here is the source for this row" is a row-level one. An agent can satisfy both truthfully and still leave the reader unable to determine, for any specific row, which kind of source is behind it. The join is precisely where the information lives — and no instruction in my brief demanded the join.
That is not an agent failure. It is a contract failure, and it has a general form: provenance has to be recorded at the same granularity as the claim it qualifies, or it degrades into decoration. A footnote attached to the document tells you nothing about the sentence.
The second contributing factor is subtler. My anti-fabrication language policed invention. It said nothing about strength. There is a whole axis between "I made this up" and "I read the page and it says this," and everything on that axis — snippet, cached copy, abstract, secondhand summary — lands in the output looking exactly like the strong end, because the weak end has no way to announce itself.
The fix, and the trap inside it
The fix is one column. source_url gets source_type next to it, taking fetched or snippet, filled per row by the agent that produced the row.
But there is a trap in writing that instruction, and I nearly walked into it. If the brief demands a provenance tag without also saying that snippet is an acceptable value, the tag becomes a quality bar. And a quality bar applied at the moment of writing a cell creates pressure to claim the better value — which converts a labelling problem into an incentive problem and makes the data worse than having no column at all.
So the instruction has to carry its own permission: tag every row, and a snippet row is a fine row. The purpose of the column is not to raise the floor on sourcing. It is to make the floor visible. Thirty snippet-sourced rows in a company map is a perfectly reasonable result; thirty snippet-sourced rows that read as verified is a landmine sitting in a file I will trust in three months without re-deriving anything.
Why I only found it by accident
I did not run a provenance audit because I suspected anything. I ran a spot-check on one agent because I was curious about a single unfamiliar domain, and the mismatch fell out of that.
Which means the detection here was luck, and luck does not generalise. The reason this class of defect survives is that it produces output with every visible marker of rigour — real URLs, consistent formatting, an explicit method note, a tidy count. There is nothing to catch, because the thing that is missing was never displayed in the first place.
The cheapest question in the whole episode was the one I asked last. Asking cost one message and the answer was immediate and honest — the failure was never that the agent would not tell me, it was that nothing in the pipeline asked until the merge was already built.