Git Shows 0 Deletions — Your Agent Already Proved That’s Not the Whole Story

Agent reliability is the property that describes whether an AI agent reproduces a successful outcome consistently across repeated runs — distinct from average task success, which can hide wide run-to-run variance.

The number that looks like proof

On 2026-08-04, a git log command filtered for deletions across a two-hour window — --diff-filter=D, --since="2026-08-03 19:00", --until="21:00" — returned zero results for the agents path in commit ba23553. That commit carried the label “auto-commit 20:03,” covering 43 agent-generated outputs.

Zero deletions. Clean log. End of story?

Not quite. The same operation that generated those 43 outputs also involved a path-resolution problem that the deletion counter never touched. A wiring_parity.scan() run on 2026-08-31 found that response_prefix_gate was being read from a mirror path — UNIS/config/claude_home_mirror/claude_home/hooks/… — rather than from the live installation at ~/.claude/hooks/. The agent was operating against a shadow of the real environment. Only after 23 mirror_sync operations did the UNKNOWN count drop from 3 to 0 and the CODE count climb from 15 to 18.

The deletion log was accurate. The deletion log was also incomplete. Those are not contradictions — they are the same structural problem that IBM Research quantified at scale.

The 24.4-point gap IBM Research measured

On AppWorld, a ReAct agent using GPT-4.1 succeeded on 77.4% of runs averaged across five repetitions, according to results published September 15, 2026 by IBM Research. But it succeeded on all five runs for only 53.0% of tasks — a 24.4-point consistency gap.

Most benchmarks report the first number. The second number is the one that matters in production.

The gap has a name: the difference between average-case performance and consistent performance. When the IBM Research team built ALTK-Evolve — a system that converts an agent’s own past trajectories into reusable guidelines and injects them back at inference time — they found it measurably improved task success. But even that measurement initially asked only the average-case question. The consistency question is harder because it requires running the same task multiple times and checking whether the agent arrives at the same outcome, not just whether it sometimes does.

For mission-critical work — reconciling a financial transaction, checking a contract for an obligation — a 77.4% average is not a reliability number. It is a coin flip with better odds.

Why agents lose their reasoning between sessions

The session-boundary problem is the other half of this. David Corvoysier, writing for Hugging Face on September 3, 2026, describes working across several machines and switching coding agents depending on the task: every agent meets the project as a stranger. The reasoning from a previous session disappears when the session ends.

An earlier analysis — “Software Forgets: Agent Traces Are the Memory” — established that coding agents already produce a dense record of not just what changed but why: approaches tried, errors hit, documentation read, direction changed. The problem is that session logs are an archive, not a memory. Grepping ten thousand turns to reconstruct “why did we move off the streaming parser?” is not retrieval — it is archaeology.

The funes tool Corvoysier introduces addresses this directly: a durable memory layer built from existing session files, supporting Claude Code, Codex, and other agents, with local inference, no ML runtime dependency, and optional sync to a private Hugging Face dataset. The installation path is a single command. The architectural point is not that this tool exists — it is that its existence reveals the default state: every new agent session starts from zero, audit trail included.

Our own measured data shows this default state at work. On 2026-08-31, the wiring_parity scanner looked at a live path and found the wrong one. The agent that generated 43 outputs on 2026-08-04 was not operating with full context about its environment. Neither failure was a deletion. Neither appeared in the deletion log.

What a zero count can and cannot tell you

A deletion filter answers one question: were files formally removed in this window? It cannot answer whether:

• An agent read from a stale configuration path rather than the live one
• A tool’s counter reflected reality (our counter showed a value of 1 after backup copy, a post log showed 32 published entries plus one additional live post, totaling 33 — the --list classification confirmed which were published and which were not, but that reconciliation required active verification, not passive log inspection)
• An agent that succeeded five times will succeed on the sixth

The deletion count is accurate by definition. It is incomplete by design. It was designed to count deletions, not to audit whether the environment the agent operated in matched the environment the audit assumed.

This is the same structure as the IBM Research consistency gap. The average-case benchmark is accurate for what it measures. The consistency gap is invisible to it by design.

The structural interpretation

These three data points — the IBM Research 24.4-point gap, the funes session-memory architecture, and our own path-resolution incident — point to the same principle: agent outputs are reproducible only if the inputs, environment, and reasoning context are also reproducible.

A git log audits outputs. It does not audit inputs. It does not audit whether the agent’s effective configuration matched its intended configuration. It does not audit whether the agent would reproduce the same outputs tomorrow.

LLMs compound this problem. Apple ML research published in 2026 finds that large language models are not consistently Bayesian — they can report two different confidence levels for the same belief depending on how the question is framed. An agent that passed a task five times is not guaranteed to pass it because it reasoned correctly five times; it may have reasoned differently each time and arrived at the same output by different paths. Only one of those paths may be the reliable one.

What this means for teams running agents in production

The decision this knowledge supports is an audit design decision, not a technology choice.

For teams using coding agents: session memory tools like funes change the default state from “agent starts from zero” to “agent starts from indexed context.” That is not a convenience upgrade — it is the difference between an agent that can reproduce its own reasoning and one that cannot. Reproducible reasoning is auditable reasoning.

For teams evaluating agent reliability: the IBM Research framing is the right one. Report task success; also report consistency across repeated runs of the same task. A 24.4-point gap between those numbers is not a failure — it is a measurement that tells you where the reliability work needs to go.

For teams reading audit logs: a zero in a filtered log is accurate for what the filter counts. Build a second question into every audit: what does this log not count? Path mismatches, configuration shadows, and session-boundary resets will not appear in deletion filters. They require positive verification — running the scan, confirming the sync, checking the count against an independent source.

The deletion log from 2026-08-04 was correct. The agent also operated against the wrong hooks path for a period that the log could not detect. Both things are true. The audit that stops at the log stops too early.

Looking for more on ai & digital income? Visit SAVYX

Frequently Asked Questions

Can a git deletion filter confirm an agent didn’t change anything?

No. A deletion filter confirms no files were formally removed — it cannot confirm that the agent operated against the correct environment, read the right configuration paths, or produced outputs that would reproduce under the same conditions. Path mismatches and configuration shadows do not appear as deletions.

What is the consistency gap IBM Research found in AI agents?

According to IBM Research results published September 15, 2026, a ReAct agent using GPT-4.1 on AppWorld succeeded on 77.4% of runs averaged across five repetitions, but succeeded on all five runs for only 53.0% of tasks — a 24.4-point gap. Most benchmarks report only the average, which hides this variance.

Why does an agent that succeeded before sometimes fail the same task?

Agents can reach the same output by different reasoning paths, and only some of those paths are reliable. IBM Research notes that this variability is hidden by average-case benchmarks. Session-boundary resets — where an agent starts each session from zero without retained context — compound the problem.

What does ‘funes’ do that normal agent session logs don’t?

According to David Corvoysier’s September 3, 2026 post, funes adds indexing, retrieval, ranking, and provenance to existing session traces, making them queryable memory rather than a raw archive. It runs locally with no ML runtime dependency and optionally syncs to a private Hugging Face dataset the developer owns.

Are LLMs reliable enough to trust as a single source of audit truth?

Apple ML research published in 2026 finds that LLMs are not consistently Bayesian — they can report different confidence levels for the same belief depending on framing. Using an agent’s own output as the sole audit of that output’s correctness is a structural conflict; independent verification steps are necessary for mission-critical workflows.

Disclosure: As an affiliate, this post may earn a commission from qualifying purchases at no extra cost to you.