Research and public work

We publish
what we measure.

RadianVector studies how open-weight models behave after deployment choices are applied. Each published study fixes the model and hardware, changes a defined part of the serving stack, and reports quality beside performance. The same work produces open-source measurement tools and tutorials on the systems being measured.

01 · Published studies

Benchmarks on
open-weight models.

Both studies below use one model, a custom question set and GPU family, changing only the serving stack. Both report accuracy alongside speed, because a configuration that is faster and answers differently needs the answers to be evaluated for their quality.

QWEN3.8-27B · CONFIGURATION STUDYPUBLISHED

Beyond tokens per second

Seventy-one controlled inference configurations on a 24 GiB card: four 4-bit checkpoint builds, KV-cache precision, CUDA graphs, speculative decoding, prefix caching and concurrency from one user to thirty-two — with answer accuracy measured on every arm.

It includes a throughput gain we measured and then qualified. MTP speculative decoding at depth 1 ran 7.4% faster at six concurrent users, and cost 6.5 accuracy points on short answers while costing nothing measurable on long-form reasoning. At depth 3 the short-answer cost was 11.9 points. A speed-only benchmark would have recorded either as a clean win.

Qwen3.8-27BRTX 4090 GPU familyvLLM 0.28380 scored tasks

QWEN3.8-27B · REASONING STUDYPUBLISHED

Thinking on, measured

What the model's native reasoning mode is worth, and where it stops paying. Four reasoning-effort settings across four checkpoints and nine concurrency levels, scored on the same fixed question set, with the throughput cost measured beside the accuracy gain.

The finding is a stopping point rather than a speed-up: almost the whole gain arrives at the lowest setting, while the template's default is the expensive end of a curve that has already flattened.

Qwen3.8-27BRTX 4090 GPU familyvLLM 0.28native thinking

02 · The instrument

RV-380.

Both published studies use RV-380, one implementation of a broader evaluation method. A company-specific evaluation can use generated tasks or a suite designed around the target workload.

What RV-380 is

380 fixed tasks spanning reasoning and calculation, long-context retrieval, instruction following and structured output, generated deterministically from a seeded script with a suite hash. Runs with different hashes are never pooled.
Scored by code, not by judgement. No model grades another model, so the scorer cannot drift between runs or favour a verbose answer.
Built to separate configurations, not to rank general intelligence. It approximates the mixed traffic of a newly onboarded customer rather than a leaderboard.
One suite, not a universal test set. Other evaluations can vary the task count, domains, difficulty, languages, modalities and failure modes. Each suite is then fixed and versioned for the comparison.

What it measures that most harnesses do not

A repeatability floor. Identical re-runs of one configuration still move 3–4 answers in 380. We measure that before claiming any difference is real, so a small effect cannot be mistaken for a finding.
Answer drift as well as score. “Same accuracy” and “same answer” are different claims; a configuration can preserve the aggregate and rewrite the output.
Refusals and coverage, so a configuration is never silently penalized for a context-window limit without that being visible.

03 · Method

How a number
earns its place.

The same six rules govern every figure we publish. They are why our studies can be checked, and every RadianVector evaluation inherits them.

Freeze the instrument

Fixed seed, deterministic task generation, a suite hash, one model and one machine. The serving stack is the only thing allowed to move.

Compare item by item

Every arm answers identical questions and comparisons are paired question by question, so only the items where two configurations disagree carry information. We do not subtract two scores.

Measure the noise before the signal

Every setting runs at least twice on separately restarted servers. Differences smaller than the measured run-to-run variation are reported as no difference, not as a small win.

Prove the treatment happened

A null result is only interpretable with a positive control. When we report that cache pressure changed nothing, we show the counters proving the cache really was starved.

Trace every figure to its run

Each published number carries the campaign directory it came from, and automated checks recompute the figures independently from the raw results before the page ships.

Turn an unexpected result into the next experiment

On Qwen3.8-27B with vLLM 0.28, MTP speculative decoding changed more answers than repeat-run variation explained. The study scopes the result to that stack; the follow-up traced it to a defect in vLLM's V1 model runner and tested a fix.

04 · Tutorials

The background
reading.

Measurement claims are only as good as the reader's model of what is being measured. We write the explanations we needed ourselves, free and without registration.

SERIES · FIVE PARTSPART 1 PUBLISHED

Inside an LLM

A visual walkthrough of what happens between a prompt and the next token: tokenization, embeddings, twenty-four transformer layers, attention and the KV cache, the LM head, temperature and sampling. One reference model throughout, Qwen2.5-0.5B, so every number lines up from the first page to the last.

Part 1 is published. Parts 2 to 5 — the decode phase, attention in detail, the LM head magnified, and an interactive column view — are drafted and are being checked for accuracy before release. The tutorials index tracks what is out.

freeno registrationinteractive diagrams

05 · Experiments planned

What we are
measuring next.

Each published study leaves a question the experiment was not designed to answer. These are the two we are running next, and the campaigns are scoped.

EXPERIMENT 01PUBLISHED

Why speculative decoding changed the answers

On this model and stack, MTP speculative decoding changed answers well beyond the measured repeat-run variation. The follow-up compared every reply token for token, with and without other traffic, and traced the change to a defect in vLLM's V1 model runner rather than to MTP.

A fix tested with the same checks before and after brings MTP back to the no-speculation baseline, on vLLM 0.28 and 0.30.

EXPERIMENT 02SCOPED

Tightening the paired statistics

Two rounds of the same question are repeated measurements of one item, not independent observations. The current paired tests count discordances over question-rounds and therefore understate that dependence.

The revised analysis clusters by question and reports bootstrap intervals. Those figures will replace the current point p-values on both study pages.

06 · Work with us

Not sure what to measure?
We work it out with you.

RadianVector applies these measurement methods to your work. You don’t need a defined question to start: we work out with you what is worth measuring, then measure it.

Work with RadianVector →

We publishwhat we measure.

Benchmarks onopen-weight models.