If your AI agent keeps making the same mistakes, a more expensive model may not be the first thing to reach for. NVIDIA’s ARC-AGI-3 result shows that, more than a model’s momentary intelligence, the work system that remembers, verifies, and changes course when stuck determines success on long-horizon tasks.
The star behind the score of 100 was not a ‘smarter model’
AVO (Agentic Variation Operators), released by NVIDIA, is not a new language model but an agent system for long-running autonomous work. Its primary agent investigates the situation, makes a plan, executes it, and evaluates the outcome, while persistent memory and tools carry the process forward. A separate supervisory agent redirects the search to another strategy when it stalls or repeats the same failure.
The test arena was ARC-AGI-3. It is an interactive reasoning benchmark where, in unfamiliar 2D environments that do not reveal their rules or goals, agents must observe the effects of actions, infer the game’s objective, and solve increasingly difficult levels. Rather than asking for a single correct answer, it measures the ability to form hypotheses, act on them, and revise incorrect ones.
Built on Claude Opus 5, AVO completed all 183 levels across the public set’s 25 environments and recorded an RHAE of 100.00. RHAE is not simply an accuracy rate: it combines whether a level is completed with action efficiency relative to a human on a first attempt. A score of 100 means it finished every game and level at efficiency equal to or better than the human benchmark.
The number TechCrunch highlighted was the gap between Claude Opus 5’s roughly 30% model evaluation and the AVO system’s 100%. But this should not be read as a controlled experiment showing that “a harness alone added exactly 70 percentage points.” NVIDIA explicitly states that the two results differ in reasoning setup, agent system, and evaluation configuration, so this is not a direct comparison that measures AVO’s contribution alone.
A score of 100 is not a declaration that AGI has been achieved
This result came from 25 publicly available environments, not from the semi-public or private competition sets. ARC Prize also warns against interpreting a harness score tuned to public environments as progress in general intelligence. A public score may demonstrate the economic value of harness engineering, but whether it generalizes directly to unfamiliar work requires separate validation.
A harness is not prompt wrapping; it is the operating system for work
In practice, a harness is not just a thin layer of code attached around a model. It is the execution layer that decides what context to provide, which tools to allow, how to judge results, and what to preserve for the next run. If the model handles ‘what it can think,’ the harness makes sure that capability does not disappear midway through a long task.
| Failure scenario | When only changing the model | What to fix in the harness |
|---|---|---|
| Research starts from scratch every run | Choose a model with a longer context window | Persist hypotheses, attempts, and outcomes in a structured form |
| The same error repeats | Add “be careful” to the prompt | Check failure signatures and retry limits before execution |
| It says it is done, but the result is wrong | Ask for one more self-evaluation | Connect external judges such as tests, schemas, and static analysis |
| Goals drift during long tasks | Rerun with a stronger model | Have an independent supervisor regularly check goals, progress, and cost |
The first lever is memory. OpenAI also reported that, in ARC-AGI-3, having GPT-5.6 Sol retain prior reasoning and compress rather than discard old records raised its public-set score from 13.3% to 38.3%, while reducing output tokens to about one-sixth. The explanation is that if you preserve only past actions and discard “why the agent acted that way,” it must reinterpret the problem every turn.
The second is how observations are represented. On the same public set of 25 games, MIT’s VISTA achieved RHAE 100 by giving Claude Opus 5 enlarged PNGs of the raw state and adding visual memory that can retrieve past observations in their original form when needed. AVO, by contrast, did not use images and instead supplied an exact 64×64 text grid. Their shared success does not mean there is one correct interface; it signals that the important design is one that lets the model observe without loss and retrieve the relevant past when needed.
The third is separating supervision from verification. AVO’s primary agent handles investigation, revision, and execution, while its supervisor watches for stagnation and repetition across longer search trajectories. This differs from telling a model, “Check for yourself whether you’re doing well”: it adds a supervisory layer with separate state and stopping conditions. Still, NVIDIA’s experiment was not an ablation study that separately measured the effects of memory and the supervisor, so we cannot conclude that a supervisor alone produced the score of 100.
More important than the benchmark is the failure loop in our work
The takeaway for product development is not “make every agent multi-agent.” First, break down whether failure comes from missing model knowledge, lost memory, tool errors, or the absence of a completion judgment. In long-running work, the costliest error is often not one wrong answer but incorrect state quietly propagating to the next step.
In fact, the DELEGATE-52 study of long-horizon document editing across 52 professional domains tested 19 LLMs and reported that even frontier models damaged an average of 25% of document content by the time tasks ended. Merely adding tool use did not improve performance, and damage worsened as document size, interaction length, and distracting files increased. It is a counterexample showing that connecting tools alone does not make an agent reliable.
So an agent performance dashboard needs more than success rate. Track completion rate, number of human interventions, repeat rate for the same failure, recovery rate after validation failures, and tokens, time, and cost per completed task. ARC-AGI-3’s RHAE likewise measures completion and action efficiency together, and does not count internal reasoning or tool calls that do not change the environment as actions. To assess service cost, log model-call volume and execution time separately from benchmark scores.
Four steps to apply harness design to your agents
1. Lock down a failure-reproduction set first
Collect 20–50 cases that failed in real work, and save their inputs, expected outputs, and tolerances. Attach mechanically verifiable criteria: test pass rate and change scope for coding work, source validity for research, and schemas and row counts for data work. Rerun the same set whenever you change the model, prompt, or harness so you can distinguish improvement from luck.
2. Split memory into a ‘full transcript’ and a ‘work ledger’
Keep the full conversation as a compressible reference record, while preserving goals, established facts, attempted approaches, reasons for failure, generated files, and remaining work separately in structured form, such as JSON or database records. When context fills up, do not blindly delete old messages; first check that the work ledger’s required fields remain, then summarize. Do not record sensitive information or credentials in the ledger.
3. Define supervisor intervention conditions numerically
Instead of “help when needed,” set triggers such as two instances of the same error, five tool calls without progress, reaching 80% of the expected cost, or three test failures. The supervisor should not do the work itself; have it read the current hypothesis, failure record, and remaining budget, then return one of: continue, change strategy, call a human, or stop.
4. Run harness ablation tests before changing models
Starting from a baseline, add memory retention, compression, an external validator, and a supervisor one at a time, recording changes in completion rate and cost. If you enable several features at once, you cannot tell which one made the difference. Change the model only at the end and repeat the same tests to separate model effects from system effects.
If you want to dig deeper
NVIDIA AVO Reaches 100% on ARC-AGI-3, Demonstrating a Frontier-Level General-Purpose Architecture for Long-Horizon Autonomous Agents — Learn about AVO’s memory, supervisory structure, public-set results, and the limits of comparison. developer.nvidia.com
ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence — Explains the benchmark’s design goals and why public harness scores should not be interpreted as progress toward AGI. arxiv.org
ARC-AGI-3 Scoring Methodology — Read the original methodology for RHAE, which combines completion rate and action efficiency relative to humans. docs.arcprize.org
How enabling two settings tripled our scores on the ARC-AGI-3 benchmark — An OpenAI experiment showing how retaining reasoning and compressing context changed scores and token use. openai.com
VISTA: A Visual Harness for Reasoning in an Interactive World — Compare another score-100 harness that uses visual input and lossless memory for past observations. vista-research.github.io
LLMs Corrupt Your Documents When You Delegate — Covers the accumulation of errors in long-horizon delegated work and why tool integration alone does not solve it. arxiv.org
