Caudena introduced AI Graph Summaries in summer 2023, becoming the first blockchain intelligence platform to bring this capability to crypto investigators. We have since expanded from AI-assisted reporting to agent workflows that use Caudena’s investigation tools and data. You can read about that progression here.
This benchmark continues our work. We tested 11 AI model configurations across multiple investigations and assessed the quality of their graphs and reports alongside completion time and estimated investigation cost.
Gemini 3.8 Flash at medium reasoning effort delivered the strongest balance under the tested conditions. It achieved a quality score of 100 with an average completion time of 3 minutes 40 seconds. The estimated investigation cost came in at $0.32.
Several configurations finished close to the leading quality score, but they came with varying time and cost requirements. Higher reasoning settings produced mixed results. For teams evaluating AI agents for blockchain investigations, these differences make the choice of configuration worth testing against the work it needs to perform.
A Repeatable Test for Investigation Work
This idea took inspiration from Simon Willison’s recurring pelican-riding-a-bicycle test. The test uses a recognizable task to make differences between models easier to inspect.
For blockchain investigations, a useful test needs reference evidence and explicit completion criteria. We used multiple investigation cases with reference transaction paths, network links, and required findings.
Each agent started with the analyst’s prompt and an empty or seeded Prism graph. It could query investigation data through our tools and covered transaction history, counterparties, exposure, fund tracing, cross-chain swaps, and open-source intelligence.
The evaluation measured each model working with a particular set of tools, instructions, and data. As Anthropic’s guidance on agent evaluations explains, an agent evaluation tests the model together with the system that enables it to act.
Different Cases, Different Investigation Demands
The cases covered BTC-to-ETH tracing, onward Bitcoin payments, and TRON network analysis.
Cross-chain tracing required agents to follow a starting cluster through a peel chain and THORChain swaps, including partial refunds that were subsequently swapped again. Other tasks required agents to follow specified Bitcoin receipts through subsequent transactions to service destinations.
The TRON investigation tested whether agents could distinguish observed activity from source labels and gaps in the available data. An address missing from an index leaves an information gap, and it provides no basis for treating that address as inactive.
Together, these tasks required agents to retrieve relevant data, construct an investigation graph, and explain their findings in a written report.
How Quality, Time, and Cost Were Measured
The benchmark’s quality score combined three automated measures, each scored from 0 to 100:
- Graph recall measured coverage of the expected route transactions, addresses, and destination services.
- Graph precision measured the proportion of plotted transactions that belonged to the reference route.
- Findings measured whether the final report included the required facts, with exact identifiers and amounts within defined tolerances.
The overall quality score was the mean of these measures. Additional correct findings could not compensate for the absence of a required fact.
Every test attempt had a 25-minute limit and a $3 model-usage budget. Exceeding either limit or failing to finish with a final report resulted in a score of zero under the benchmark rules.
Graph construction was also constrained. The agent selected blockchain objects, and the backend constructed the corresponding nodes and edges using Prism’s graph generators. This ties plotted objects to the underlying data while leaving the agent responsible for selecting relevant objects and explaining them accurately.
Results Within the Tested Conditions
Testing took place from September 25 to the 28th and produced 69 scored attempts. 68 completed tests and one timeout.
Configurations within one quality point of the best result were treated as a tie. A combined cost and time measure then determined their order.
Gemini, Claude Opus 5.5 at medium effort, and GLM-5.3 at high effort were tied under the quality rule. Gemini placed first because it had the lowest reported cost and shortest average runtime within that group.
The individual results also show why averages need context. All 15 Gemini tests finished in under six minutes and below $0.70, each scoring 100. Other configurations showed a wider spread in completion time and cost.
Investigation time and estimated cost
68 completed tests across multiple investigation cases
Estimated investigation cost (USD)
Investigation time (minutes)
- Investigation time
- Estimated cost
- Quality score
Each marker represents one completed test. The timeout is shown separately because its cost was not recorded.
Testing Reasoning Effort
Higher reasoning settings did not consistently improve the reported quality scores.
Claude Opus 5.5 increased from 98.5 at low effort to 99.9 at medium effort. GLM-5.3 increased from 98.0 at low effort to 99.9 at high effort. GPT-6 Sol and Grok 4.6 scored lower at their higher tested settings, while GLM-5.3-Flash at high effort reached the time limit without producing a final report.
Higher reasoning effort had mixed results
Reported quality scores for configurations tested at more than one effort setting.
Quality score / 100 · Detail view: 88–100
Reported quality score / 100
- Change
Higher reasoning settings produced mixed results in the tested configurations. Unequal test counts limit comparisons.
These observations support testing each configuration on the intended workflow. They do not establish a general rule about the value of additional reasoning.
What the Results Can Establish
The benchmark provides evidence about performance across multiple selected investigation cases. Repeated testing reveals variation on those cases, but it does not broaden the range of investigation problems represented.
Test allocation was adaptive. Configurations still competing for the lead received more attempts. This helped explore the ranking efficiently while limiting how confidently small differences could be interpreted.
A score of 100 means the output satisfied the automated checks. It does not establish that every narrative claim was supported, every uncertainty was expressed appropriately, or every possible investigation lead was exhausted. Report readability was reviewed separately by hand.
The component scores also matter. GPT-6 Sol achieved full graph recall and precision at both tested settings, while its reports omitted some required findings, including refund details. This supports checking report completeness separately from graph coverage.
Ten further attempts were excluded as provider or evaluation-system failures. Those interruptions need separate accounting when assessing operational reliability. The benchmark also provides no comparison with human investigators, so it does not establish how much investigator time the workflow saves or how much review effort its outputs require.
What This Means for AI-Assisted Investigations
For Caudena, the results support evaluating quality, completion time, and cost together. A useful configuration needs to complete the intended work within practical limits and produce findings that an investigator can verify.
Further evaluation should add unseen cases, track completion failures, and measure the time investigators spend checking and correcting outputs. It should also examine unsupported statements and how agents handle missing data.
Since introducing AI Graph Summaries in 2023, Caudena has continued to advance practical AI for crypto investigations. For us, leadership in this field means developing useful capabilities and putting their performance to the test.
This benchmark is part of that commitment. It provides a measurable basis for choosing agent configurations and identifying where further work is needed. Our focus remains on helping investigators complete demanding work efficiently, with findings they can verify and conclusions they remain responsible for reviewing.