01 — What we measure

Score the output. Measure the run.

Compare answer quality, response time and cost on the same workload. See which tasks each configuration completes, which errors matter and how much compute it uses.

Behavior

How the model responds

What the output says, how stable it is, and how it fails.

Prompt sensitivity: How much the answer moves when the same task is worded differently.
Consistency: Agreement across repeated runs of the same prompt.
Instruction and policy adherence: Whether the output follows the brief, format and rules it was given.
Severe errors: Mistakes that would cost money or trust if shipped, counted separately from minor ones.
Tool use: For agents: the right tool, the right arguments, and recovery when a call fails.
Human correction time: How long a person needs to fix the output, where people stay in the loop.

Performance

How fast and how much

Speed and capacity as your users and systems experience them.

Time to first token (TTFT): How long until the answer starts arriving.
Output speed: Tokens per second once it does.
End-to-end latency: From request to complete answer, including tool calls for agents.
Throughput under a plan: Requests and tokens processed within a plan’s or tier’s rate limits.
Variability: Spread across runs and times of day, so a quoted speed comes with its range.
Cost: Per task, at the measured token counts.

02 — The knobs

Test the choices your team can actually change.

A frontier provider controls the model weights and serving stack. RadianVector tests the choices exposed to your team: model, prompt, output format, context and supported request settings. Record exact model identifiers and compare changes under matched conditions.

Model & API

What answers the request

Model candidates, provider and version or dated identifier
Endpoint, region, service tier or provisioned capacity
Reasoning mode or effort
Context and output-token budgets

Prompt & output

How the task is specified

System instructions and task wording
Few-shot examples and reference material
Temperature, top-p and seed where the API supports them
Structured-output schema, stop conditions and response length

Knowledge & tools

What the model can use

Retrieval corpus, chunking, filters and number of results
Tool set, tool descriptions and argument schemas
Tool-choice policy and parallel or sequential calls
Memory, conversation history and supplied context

Workflow & oversight

What happens around the model

Routing by task, risk or confidence
Response-time budgets and failure handling
Task-completion requirements
Human approval, sampled review or autonomous execution

API support differs by provider and model. Each run records the exact model identifier and accepted settings. The recommendation includes the configurations tested, their measured trade-offs and the requirements used to select finalists.

03 — Performance controls

Timing a model that sits behind someone else’s API.

API response time includes network travel and provider processing. Comparisons record the endpoint, model version, request shape and retry policy. Repeated, interleaved runs help distinguish configuration differences from changes in network or provider conditions.

01

Length-matched prompts

For controlled timing tests, match input lengths and output limits across candidates. Record actual output lengths and cache conditions with the results.

02

Network measured on every run

Record network conditions alongside request timings where available. Report end-to-end API latency separately from any provider-reported timings.

03

Repeated and interleaved

Runs are spread over time and alternated between settings. A one-off slowdown lands in the spread, not on one side of the comparison.

04

Intervals and significance

TTFT and throughput are reported with confidence intervals and significance tests. A difference is called only when it clears the measured run-to-run spread.

04 — Behavior controls

How the model actually responds.

The task, source material, expected outcome, scorer and error-severity rules are frozen before the comparison. Prompt wording can then be held fixed or varied deliberately. Repeated runs show whether a result is stable or depends on one lucky response.

Prompt variants: the same task in several wordings, including the one your team uses today
Repeated runs: agreement across runs separates a stable behavior from a coin flip
Task results compared against the same requirements across configurations
Severe and minor errors counted separately, with examples from the runs

05 — Evaluation sets

A workload built around your decision.

Start with representative tasks, source material and criteria for a successful answer. Your domain leads define correctness; the tools compare candidates against those criteria.

Illustrative study design

1,000 questions. Models, prompts and output formats.

For a document-extraction workflow, a study could screen 10 model and reasoning-level settings with 5 prompt variations, then explore output schemas and supported sampling or reasoning settings on promising candidates. Compare correct facts, citation support, format compliance, response time and cost against the current setup.

The search is staged and bounded by an experiment budget. Final validation cases stay separate from the cases used to select candidates, with repeated runs where output variation matters.

Explore the engineering and measured examples →

Representative work

Workload suites

Support replies, claims, research, drafting, coding, extraction, reconciliation or any repeated task with a decision attached. The set is stratified by task type, difficulty and business risk.

Grounded answers

Document and retrieval suites

Questions over contracts, manuals, PDFs, knowledge bases or retrieved passages, scored for answer correctness, citation support, omission and unsupported claims.

Multi-step systems

Tool and agent suites

Tool selection, argument construction, result use, state transitions, recovery from failed calls and final-task completion—with latency and cost measured across the whole run.

Boundaries

Policy and failure suites

Instruction conflicts, missing information, out-of-scope requests, high-cost errors and escalation cases. Severe failures are defined before the run and reported separately.

A reusable instrument

Keep the workload and results for the next comparison.

Each task has an identifier, source material, expected outcome and requirements. Keep a versioned record of the workload and tested settings to compare future model, prompt or workflow changes against the same baseline.

06 — Measured

What we found running frontier models, October 2026.

We ran models from Anthropic, OpenAI and xAI through each vendor’s own command-line tool, on questions we generate and check by code. Ten findings for teams that choose a model, an effort setting and a tool: what worked, where the tokens went, and what to change.

Observed October 4–15, 2026 · Original RadianVector results

Findings 02 and 07

Max reasoning is not always the right answer.

Opus 5.5 High2/233,411 and 36,387 tokens

VS

Opus 5.5 Max0/2139,699 and 514,985 tokens

Right answers on two 300-step calculations

GPT‑5.5 Low7/14

VS

GPT‑5.5 High12/14

Right answers on 14 level-10 questions

On the most complex questions, high effort solved them; max effort over-analysed and ran out of room.

At max, Opus spent 503 to 630 tokens a step, so 300 steps need more than one reply holds (128,000).

What it means for you. Do not default to the highest setting. Test effort settings on your own questions.

Tested: Opus 5.5 max: Claude Code 2.1.292 to 2.1.301, October 8–15; high: Claude Code 2.1.301, October 15 · GPT-5.5: Codex 0.154.0, October 4, 2026

Finding 01

At the output limit, each retry started from the beginning

514,985 output tokens on one question, and no answer ($10.33 at API list price)

One reply from Claude Opus 5.5 can hold up to 128,000 output tokens, its hidden reasoning included. When a reply reached that limit, Claude Code asked the model to continue, up to three times. The reasoning written so far is not part of what the model receives when it continues, so each try started from the beginning. A question that did not fit was worked on four times and still ended without an answer. This is Claude Code’s behaviour; we did not test the API.

What it means for you. When a question needs more than one reply can hold, retrying does not help, because each try starts over. Split the work into smaller steps, or try a lower effort setting first (finding 2).

Tested: Claude Opus 5.5, max effort · Claude Code 2.1.301 · October 15, 2026

Finding 02

High effort answered what max effort could not

2 of 2 300-step questions answered right at high effort; max effort gave no answer on either

On two long calculations of 300 steps each, Claude Opus 5.5 at max effort reasoned past its output limit and gave no answer, after 514,985 and 139,699 output tokens. At high effort it answered both right in one reply, with 36,387 and 33,411 tokens. On a 50-step question max, high and medium effort were all right; max used 24,875 tokens, high 8,932 and medium 7,584.

What it means for you. The highest effort setting is not always the best one. Test effort settings on your own tasks: a lower one can be cheaper, faster and still right.

Tested: Claude Opus 5.5 · max effort: Claude Code 2.1.292 to 2.1.301, October 8–15 · high and medium: Claude Code 2.1.301, October 15, 2026

Finding 03

Claude Opus 5.5’s safeguards declined questions that ask what a program prints

24 of 24 code-tracing questions on the depth ladder declined, before any output

Through Claude Code, Claude Opus 5.5’s safeguards declined 29 of our questions as attempts to extract its reasoning (tag [reasoning_extraction]): every depth-ladder question that asks what a short Python program prints, one such question at levels 11–12, and 4 long calculations after they passed the output limit. A 50-step calculation with one added sentence asking for each step in the reply was declined the same way within 3 seconds; without that sentence it was answered right.

What it means for you. If your work asks a model to trace code or to show each step, test that on the model and tool you plan to use. A declined question needs a retry or another model.

Tested: Claude Opus 5.5, max effort · Claude Code 2.1.292 to 2.1.301 · October 8–15, 2026; the step-by-step request October 15 (2.1.301)

Finding 04

In Codex, GPT-6 Astra could run code with the shell switched off

28 of 28 right with its code tool; 26 of 28 on the same questions with the tool switched off

We ran Codex with its shell tool switched off and read its --json event stream for tool calls. GPT-6 Astra works in code mode, where its tools go through one JavaScript exec tool. That tool stayed available, and its calls are recorded in Codex’s session log, not in the event stream. With it, Astra got 28 of 28 at a median of 1,416 output tokens; with code mode switched off, 26 of 28 at 3,511. The two answers that changed were long calculations. All GPT-6 Astra results on this page come from runs with code mode switched off.

What it means for you. For exact work, a code tool can do more than extra reasoning: on these 28 questions it gave more right answers for fewer tokens. To know when it was used, read the session log as well as the event stream.

Tested: GPT-6 Astra, xhigh effort · Codex 0.154.0 (codex exec --json) · October 5, 2026

Finding 05

With an empty tool list, the Grok CLI still called tools

10 of 38 attempts called a tool with an empty tool list

On October 4 we ran Grok 4.7 through its CLI with --tools "", an empty tool list. In 10 of 38 attempts, on 5 of 28 questions, it called tools such as write (write a file) and workflow. From October 5 we removed 31 tools by name, and no tool on our machine was called again. Grok’s X search runs on xAI’s servers, and the CLI has no setting to switch it off; it ran on 6 attempts.

What it means for you. Check what “no tools” means in each CLI, and confirm it from the event stream. Run unattended jobs in an empty, throwaway directory.

Tested: Grok 4.7, low and high effort · Grok CLI 1.0.46 · October 4–5, 2026

Finding 06

On long exact calculations, only Claude Opus 5.5 kept its tokens in step with the work

12 of 12 Opus answers right, up to 200 steps

We divided each answer’s output tokens by the number of calculation steps. Claude Opus 5.5 used about the same per step at every length it answered, 375 to 771 tokens, and got all 12 right; past 200 steps the work no longer fit in one reply (finding 1). GPT-6 Astra and Grok 4.7 answered at every length but spent less per step as the questions got longer, and at 700 steps none of their answers was right.

What it means for you. On long exact work, falling tokens per step went with wrong answers. Split such work into shorter pieces, or check it with code.

Tested: 4 questions per length · GPT-6 Astra, xhigh (Codex 0.154.0, October 5) · Grok 4.7, high (Grok CLI 1.0.46, October 5) · Claude Opus 5.5, max (Claude Code 2.1.292 to 2.1.301, October 8–15)

Finding 07

More reasoning effort helped GPT-5.5 most

7 → 12 of 14 level-10 questions right, low to high effort, for about twice the tokens

We asked three everyday models the same questions at low and at high effort. At level 10, GPT-5.5 went from 7 to 12 of 14 right for about twice the output tokens. Claude Sonnet 5.5 gained 1 for 17% more tokens, and Grok 4.7 gained 2 for more than twice the tokens. At level 7, GPT-5.5 at high effort missed two questions it had solved at low.

What it means for you. Raising the effort setting can buy right answers, but the gain depends on the model. Compare question by question on your own tasks before paying for more effort; with 14 questions, even GPT-5.5’s gain could be chance (p = 0.06).

Tested: Claude Sonnet 5.5 (Claude Code 2.1.288) · GPT-5.5 (Codex 0.154.0) · Grok 4.7 (Grok CLI 1.0.46) · October 4, 2026

Finding 08

Codex and the Grok CLI add about 7,000 tokens to every question

~7,000 input tokens of the tool’s own instructions on every question

The same 56 questions went to three models through each vendor’s CLI. Through Claude Code, with its system prompt replaced by our one line, a question came to a median of 778 input tokens. Through Codex it was 8,473, and through the Grok CLI 8,022: each tool adds its own instructions. In codex exec the agent instructions cannot be replaced; in the Grok CLI they remain when the system prompt is replaced.

What it means for you. A score through a vendor’s CLI measures the model together with that tool’s prompt. Account for it when you compare models, or when you estimate API costs from CLI runs.

Tested: Low effort · Claude Sonnet 5.5 (Claude Code 2.1.288) · GPT-5.5 (Codex 0.154.0) · Grok 4.7 (Grok CLI 1.0.46) · October 4–5, 2026

Finding 09

Most of the output you pay for is reasoning you do not see

61–94% of output tokens are hidden reasoning, at the median

Frontier models reason before they answer. That reasoning is billed as output, but the CLIs show only the final reply. Where the provider reported both counts, hidden reasoning was 61% to 94% of output tokens at the median. GPT-5.5 at high effort wrote a median of 1,854 output tokens for a reply of 190 characters, about 50 tokens.

What it means for you. A short reply says little about its cost. Compare settings by output tokens, which are billed, not by the length of the reply.

Tested: Every setting on this page · October 4–15, 2026

Finding 10

Long reading held up; long exact arithmetic did not

24 of 24 log-and-lookup questions right, for each of the three strongest settings

The depth ladder makes three tasks longer, step by step. Finding who held an item in a log of up to 600 events, looking that person up in a table and running a short calculation: 24 of 24 for GPT-6 Astra, Grok 4.7 and Claude Opus 5.5. On a long exact calculation, Astra got 9 of 24 right and Grok 11 of 22; Opus answered up to 200 steps, all 12 right. Tracing a Python program of up to 130 statements: Astra 24 of 24, Grok 12 of 24; Opus declined all 24 (finding 3).

What it means for you. Long inputs were not where these models broke; long chains of exact arithmetic were. For that kind of work, check the results with code.

Tested: 24 questions per task · GPT-6 Astra, xhigh (Codex 0.154.0, October 5) · Grok 4.7, high (Grok CLI 1.0.46, October 5) · Claude Opus 5.5, max (Claude Code 2.1.292 to 2.1.301, October 8–15)

How we measured

We ran each model through its vendor’s own command-line tool — Claude Code, Codex and the Grok CLI — on a standard subscription, with every tool switched off and the session logs checked for tool calls. The questions come from our own bank, generated at 12 levels of difficulty and on a depth ladder, and every answer is checked by code. Vendors update models, tools and safeguards often, so each finding holds for the versions and dates it names.

Each setting, the tool and version it ran through, and when
Model, effort	Tool and version	Dates, 2026
Claude Sonnet 5.5, low and high	Claude Code 2.1.288	October 4
GPT-5.5, low and high	Codex 0.154.0	October 4
Grok 4.7, low and high	Grok CLI 1.0.46	October 4–5
GPT-6 Astra, xhigh	Codex 0.154.0	October 5
Claude Opus 5.5, max	Claude Code 2.1.292 to 2.1.301	October 8–15
Claude Opus 5.5, high and medium (finding 2)	Claude Code 2.1.301	October 15

The data: all results tables

Reading the tables

Level. How hard a question is, from 1 (everyday) to 12 (built for the strongest models). Each family gets harder in its own way: more unknowns, more rows, a longer program.
Depth. A second ladder, for the strongest settings only: one task made longer, step by step, with nothing else changed. A code trace grows from a 30-statement program to a 130-statement one, a long calculation from 50 steps to 700, a log lookup from 40 events to 600.
Effort. A setting each provider offers: how much the model reasons, out of sight, before it answers. More effort means more reasoning tokens, billed as output, and a longer wait. Each provider names its own steps, and the same name does not mean the same amount at two providers.
Output tokens. The provider’s own count of what the model wrote for one question, its hidden reasoning included.
Output limit. The most a model can write in one reply, its reasoning included: 128,000 tokens for Claude Opus 5.5.
Not graded. The model gave no answer of its own: the provider’s safeguards declined the question, Grok’s X search ran, or the model reached its output limit. These questions are listed beside the score, not counted in it, so read a score with the questions listed beside it.

The ladder

We test frontier models on our own question bank: 14 task families, from algebra, counting and probability to code tracing, logic puzzles, scheduling, tables and long-log state tracking, each generated at 12 levels of difficulty, plus a depth ladder.

Question-bank tiers and task counts
Levels	Questions	Built to separate
1–4	1,008	Everyday to hard problems; small open models
5–6	336	Open models, with thinking on
7–10	448	Frontier models
11–12	224	The strongest frontier settings. The six families every frontier model solved at level 10 are redesigned so that routine search no longer works
Depth 1–6	72	Where the strongest models break: the same task, made longer one step at a time

How task size grows across the ladder; these values describe the tasks, not scores
Task and difficulty measure	Level 1	Level 4	Level 6	Level 10	Level 12
Code tracing: program length (lines)	6–12	18–40	39–56	100–126	134–164
Linear equations: unknowns	2	4	6	10	14
Logic grids: people × attributes	3 × 3	5 × 4	6 × 4	7 × 6	No fixed positions
Table questions: rows	8	30	60	200	Three-table joins
Scheduling: jobs	4	13	24	70	80, with a cap on how many

Results

One question per family at each level, 14 per level. Everyday settings start at level 4, high-effort settings at the frontier levels, and the strongest settings go on to levels 11–12 and the depth ladder. A blank: not run at that level.

At a glance: right answers out of questions graded
Model, effort	Level 4	Level 6	Level 7	Level 10	Level 11	Level 12	Depth ladder	Not graded
Claude Sonnet 5.5, low	14/14	14/14	13/14	11/14
Claude Sonnet 5.5, high			14/14	12/14
GPT-5.5, low	14/14	13/14	13/14	7/14
GPT-5.5, high			12/14	12/14
Grok 4.7, low	14/14	13/14	14/14	11/14
Grok 4.7, high			14/14	13/14	10/14	11/13	47/70	3: X search ran
GPT-6 Astra, xhigh					13/14	13/14	57/72
Claude Opus 5.5, max					11/11	12/12	36/36	31 declined, 2 over the output limit, 8 not asked

The strongest settings, levels 11 and 12; tokens and seconds are medians of the graded questions
Model, effort	Level 11	Level 12	Not graded	Output tokens	Seconds
GPT-6 Astra, xhigh	13/14	13/14		3,511	127
Grok 4.7, high	10/14	11/13	1: X search ran	21,297	248
Claude Opus 5.5, max	11/11	12/12	5 declined	18,452	139

Depth ladder, code trace: statements in the program; 4 questions a depth
Model, effort	30	45	60	80	100	130
GPT-6 Astra, xhigh	4/4	4/4	4/4	4/4	4/4	4/4
Grok 4.7, high	4/4	3/4	3/4	1/4	1/4	0/4
Claude Opus 5.5, max	4 declined	4 declined	4 declined	4 declined	4 declined	4 declined

Depth ladder, long calculation: steps to compute; 4 questions a depth
Model, effort	50	100	200	300	450	700
GPT-6 Astra, xhigh	2/4	2/4	2/4	2/4	1/4	0/4
Grok 4.7, high	3/4	3/4	3/4	2/4	0/4	0/2 (2: X search ran)
Claude Opus 5.5, max	4/4	4/4	4/4	1 over the limit, 1 declined, 2 not asked	1 declined, 3 not asked	1 over the limit, 3 not asked

Depth ladder, log, table and calculation: events in the log; 4 questions a depth
Model, effort	40	80	150	250	400	600
GPT-6 Astra, xhigh	4/4	4/4	4/4	4/4	4/4	4/4
Grok 4.7, high	4/4	4/4	4/4	4/4	4/4	4/4
Claude Opus 5.5, max	4/4	4/4	4/4	4/4	4/4	4/4

Depth ladder, all 72 questions; tokens and seconds are medians of the graded questions
Model, effort	Right	Not graded	Output tokens	Seconds
GPT-6 Astra, xhigh	57/72		8,311	257
Grok 4.7, high	47/70	2: X search ran	27,148	252
Claude Opus 5.5, max	36/36	26 declined, 2 over the output limit, 8 not asked	15,173	104

For scale: open models on the RTX 4090 GPU family, thinking on, 28 questions a level
Model	Level 5	Level 6
gpt-oss-20b, high effort	25/28	22/28
Qwen3.8-27B, 4-bit	21/28	21/28
Qwen3-8B, 4-bit	18/28	10/28
GLM-4.7-Flash, 4-bit	14/28	6/28

Notes

Each frontier setting answered the first question of each family at each level: 14 a level. With 14 questions, a difference of one or two is not a ranking; the tables show where each setting stops being perfect.
Claude Opus 5.5 was not asked 8 of the 72 depth-ladder questions, long calculations at 300 to 700 steps: each of the 4 it was asked there ran past its 128,000-token output limit.
Grok CLI 1.0.46 has no setting to switch off Grok’s built-in X search. A question on which it ran is reported apart, not scored: one at level 12, two on the depth ladder.
Output tokens are each provider’s own count, hidden reasoning included. Tokens and seconds are medians, rounded half up. Measurements made with RadianVector tools.

07 — Reports

Frontier Model Reports.

Keep task-level comparisons with your baseline, the API settings used and important failure examples. Reports include response-time and cost distributions, variation across repeated runs and results from separate validation cases.

Configuration reports

Compare models, prompt variants or reasoning settings on a fixed task set. Inspect the quality, latency and cost trade-offs and the operating conditions for each candidate.

Reusable evaluation record

Keep a versioned record of the workload, scoring rules, baseline and tested settings. Use the same evaluation to check future model and workflow changes.

MODEL UPDATE — OCTOBER Revalidation summary

312 TASKS RERUN

Workflows unchanged23

Workflows improved4

Regressions detected2

Candidates for less human review Shipment statusRoutine quotations

Human review remains ClaimsContract changes

Illustrative — the shape of a revalidation report, not a measured result.

08 — What you can compare

Compare the choices.
Validate the recommendation.

Some tasks may work well at low reasoning effort; others may benefit from more computation or human review. Measure those differences on your workload, develop a routing hypothesis, and validate it on held-out tasks before changing production traffic.

Example study matrix

model candidates / fixed versions
current prompt / revised prompt
reasoning low / high
tools off / on
retrieval off / on
workflow variants
output formats
autonomous / sampled review / approval
human baseline

01

Model & Prompt Choice

Which model and prompt meet your workload requirements?

Compare supported models and prompt variants on the same tasks, then check the finalists on held-out cases.

02

Output & Context

Which output format and source context produce usable answers?

Measure schema compliance, required facts and citation checks. Compare context and retrieval variants chosen for the study.

03

Reasoning Settings

Which tasks benefit from more reasoning, and which can use less?

Compare supported reasoning settings on the same tasks. Check quality, response time and cost on separate validation cases.

04

Change Validation

Did the latest change improve the system or introduce regressions?

Rerun the saved suite after model or workflow updates and inspect newly passing and newly failing cases.

The method behind it

Published results. Company-specific evaluations.

Beyond tokens per second and Thinking on, thinking off use fixed questions, matched comparisons and recorded test conditions. We apply the same approach to the frontier-model difficulty ladder above. An evaluation on your own question bank follows the same approach. The document-workflow study and sample report elsewhere on this page are illustrative.

Start a conversation

Measure how frontier models handle your work.

Tell us how your team uses AI today, even if it is just a chat subscription. We will work out with you which tasks are worth measuring, and how.

hello@radianvector.com