Evaluation

Benchmarks

Both benchmark results are Catenation’s own measurements, set against published scores. On Terminal-Bench Science, GPT-6 Astra running with Catenation outscores every published comparison below. ProteinGym numbers come from a completed research experiment with a held-out comparison, confirmed by an independent evaluator replay. The runtime efficiency result compares the memory Catenation needs with Claude Code and Codex CLI, and its time to first frame with Claude Code.

Terminal-Bench Science 0.1

This benchmark tests scientific workflows using code and terminal tools. GPT-6 Astra with Catenation reaches a 72.8% task-resolution rate, 8.2 percentage points above the published GPT-6 Astra score of 64.6%.

The comparison scores appear in OpenAI’s published evaluation; the benchmark announcement also documents the Opus 5 result. The gain is the percentage-point difference between the two task-resolution rates (72.8% − 64.6%).

ProteinGym

Our research experiment selected an equal-weight rank ensemble of ProSST-2048, VenusREM, and GEMME. It scores 0.53654 across the full benchmark, above Aster’s published 0.524. Scores use mean Spearman correlation with ProteinGym’s official aggregation: assays within protein/category, proteins within category, then the five functional categories.

On the same 61 held-out proteins / 72 assays, the gain is +0.02519, with a paired protein-bootstrap 95% interval of [+0.01128, +0.03770]. The gain uses unrounded scores. Of 10,000 bootstrap samples, 9,957 contained all five functional categories and were retained.

Before evaluation, the experiment assigned complete UniProt identities to 125 development proteins / 145 assays and 61 held-out proteins / 72 assays. It selected from 63 equal-weight ensembles on development data and froze the winning configuration before evaluating the holdout. The final method averages each component’s within-assay percentile ranks.

A separate evaluator reconstructed all scores using pandas ranking, SciPy Spearman correlations, and explicit protein/category averaging. Replay passed across 217 assays and 2,465,767 variants, with all official variant counts matching and no missing predictions. Each assay agreed within 1e-11; the aggregate agreed within 1e-12. The replay record includes per-assay prediction hashes.

The full-benchmark score spans development and held-out proteins; the held-out comparison uses a fixed split by UniProt identity. The ensemble combines three models’ published predictions; the reproduced Aster formula uses ProSST. A separate evaluator implementation replays every score.

Runtime efficiency

Catenation’s runtime is built almost entirely in Rust, with virtually no latency. It uses 21× less RAM than Claude Code with one active session and about 22× less with ten: 9.5–9.6 MB against 201.6–208.5 MB, and 94.4 MB against 2,042 MB. Against Codex CLI it uses 4.4× less with one active session and 2.5× less with ten. It paints its first frame in 53–110 ms, where Claude Code takes 1.0–2.8 s. That efficiency is what lets Catenation run 1,215 agents at once.

How it was measured: a local run on Windows 11 comparing the Catenation terminal client with Claude Code 2.1.280. Each session ran idle in its own headless 160×45 pseudo console, with no model calls. RAM is the working set of each session’s process tree once it settles, summed across sessions. Time to first frame runs from launch until the session has painted a full screen, the first 2 KB of terminal output. The one-session runs were repeated three times and the ten-session runs once; ranges span every session measured. The Codex CLI multiples come from Catenation’s measurement of September 23.

Catenation’s measurements, September 24, 2026 (Codex CLI: September 23, 2026). Each multiple is the other tool’s RAM divided by Catenation’s at the same number of active sessions, so 21× means Claude Code used 21 times Catenation’s RAM. At ten sessions the exact multiple is 21.6× (2,042 MB ÷ 94.4 MB).

Expanded biology teams

30 campaign teams · 37–44 AI agents per team · 1,215 saved source audits.

Every selected screenshot reconciles with its native terminal record. Every final audit was checked byte-for-byte against the preserved transcript output.

The expanded gallery gives each biology campaign its own team of AI agents. Each agent receives the campaign brief and a distinct source-audit role. The native terminal shows the actual sessions, their work states, and their retained results.

The recording uses huihui-qwen3.8-27b-abliterated, with a shared queue of four simultaneous model calls. Campaigns run in batches. Native session counts show the agent teams; queue telemetry identifies generating and waiting requests.

The capture manifest records each campaign’s roster size, observed activity, capture time, model, and completed output count. The corresponding local evidence retains the source briefs, session bindings, outputs, request telemetry, raw terminal cells, and original frame timestamps. The video follows the recorded timing.

The recording follows real agent execution from campaign brief to saved research output. The interactive architecture diagrams show the runtime’s scientific tools, context programs, and harness refinement.

Original fleet recordings

The earlier September 21 captures show native research sessions running on an isolated NCShare CPU allocation, with Astra model calls and persistent Python kernels. These two recordings cover separate Kunz and biology views and predate the expanded local-model teams above.

Each AI agent then audited its own saved draft in the same session. The 79 follow-up self-audits preserve the checks, corrections, and research questions that shaped the next round of work.

Each count is read from the native workload in its 45-second recording.

Controlled local system checks

A disposable ledger contained one target, two claims joined by a dependency, one evidence artifact, and one strictly discharged obligation. The checks ran locally through the compiled research CLI on Windows.

Evidence withdrawal and claim retraction are separate operations. The first blocked completion while retaining the claim records. The second changed claim validity and reopened dependent work.

Each check runs against a disposable synthetic fixture, with normal process exits and a warm operating-system cache. Synthetic review identities drive the completion policy; four postconditions verify the retraction.

Method and provenance

1. Match archive bindings to transcript identities; distinguish the 79 research sessions from the runtime probe. Count Markdown files only inside campaign output directories.
2. Create a fresh local ledger with a locked target. Import one fixture artifact, add a verifier assertion, then record a premise and a dependent goal claim. Link the dependent claim to an obligation and strictly discharge it.
3. Read the dependent claim’s context in a new process. Repeat twenty times, requiring equality of the complete canonical JSON response and confirming that it contains no transcripts.
4. Add four synthetic final-review identities. Copy the closed fixture and confirm that the clone can reach completion. Withdraw the original fixture’s evidence assertion and require the completion attempt to fail for missing verified authority.
5. Retract the premise. Read a fresh snapshot and check claim validity, obligation status and discharge basis, and goal state.

Measurement date

September 21, 2026 (UTC)

Runtime

forge-research 0.4.8, existing Windows release binary

Environment

Windows 11, build 26200; Python 3.14.0 standard library

Binary SHA-256

7d3acaf1dfbd5241101d84177bc4a66d6d2bf8539e81a6519fadacb8e9017248

Archive SHA-256

8b34ba2c10f6f002baeef05be10cde2602da65350ab9f546cafe1335f9a84140

The evidence record retains command outputs, before-and-after snapshots, per-output hashes, and timing samples. The binary hash identifies the exact executable used. Download the method and aggregate results below.

Research record

The proof of Nivat’s conjecture, research manuscripts, exact computations, and the records behind them.

Nivat’s conjecture, posed in 1997, connects local pattern complexity with repetition across an infinite two-dimensional grid. It stood open for 29 years, a longstanding question in combinatorics on words, until Catenation’s research campaign proved it in 2026. The proof has been independently verified by researchers in symbolic dynamics. Historical background ↗ · Mathematical context ↗

The actual September 8 Nivat campaign preserves its original agent histories and research artifacts. Its proof dependency map contains 20 connected claims and 25 prerequisite relationships, within an archive of 46 imported claims. Every review record is pinned to the exact hash of the version it assesses, and the September 14 proof carries its own version identity. The archive connects each argument, computation, and review to its source.