Home / Open Models
Open Models

More control over
your intelligence.

Open models demand a deeper level of engineering: more settings to check, more ways to run them, and more performance to unlock.

RadianVector has built tools that explore these settings and measure the outcomes, with AI agents orchestrating the experiments to find what matters.

Read how our tools found a vLLM output bug and helped test a fix →

See the experiment workflow
Find which settings make a difference
01 — Open Source

Open Source from RadianVector.

Our tools help teams inspect and improve inference systems. inferlint is one you can use today. Internally, it is one in a growing list of tools our AI agents use to measure results and search for better configurations.

Open source · Apache-2.0

inferlint shows what your inference server actually did.

For vLLM, SGLang and TensorRT-LLM. Run inferlint around an existing benchmark: it reads the server’s boot log, Prometheus metrics and GPU process state, and generates a self-contained report showing achieved concurrency, queueing, KV-cache use, preemptions and teardown.

Each engine can change the numbers without raising an error. vLLM preempts a running request and writes no log line. TensorRT-LLM pauses requests for recompute and counts none. SGLang can lower its running limit at start-up and still report the limit it was given. inferlint flags each of these when it happens, so the load a benchmark requested can be checked against the work the server actually ran.

Explore inferlint Example report GitHub PyPI
pipx install inferlint
Dark inferlint report showing concurrent requests, preemptions, throughput and GPU memory findings from a vLLM benchmark Actual inferlint report · Open the example
02 — RadianVector tools

A configuration decision.
Backed by experiments.

RadianVector tools compare task results with serving measurements. Keep tested settings, failure examples and the operating conditions for each result.

Find which settings help

Compare precision, KV cache, decoding and concurrency on the same workload. Each candidate is checked against quality and performance requirements before it reaches the final shortlist.

Build evaluations around your work

Compare configurations on company-specific tasks with the same requirements. Report task results, important failures, response time and resource use together.

Keep tested serving settings

Keep the runtime version, model build, hardware and serving settings behind each result, so every configuration can be reproduced.

Experiments run by AI agents

AI agents orchestrate the experiments: they choose the next settings to try from earlier results, run the comparisons and flag what matters.

03 — The problem

Your current settings are a starting point.

Precision, KV cache, batching, parallelism and decoding interact. A change that improves speed can also affect answer quality or memory use.

RadianVector tools compare supported settings and record their effects on your workload. Explore how configuration affects AI performance →

The space of valid deployment configurations, with the one currently running highlighted EVERY DOT IS A VALID WAY TO RUN YOUR MODEL WHAT YOU RUN TODAY Selected to meet the deployment need. Not yet measured across the wider space. SOMEWHERE IN HERE Comparable quality with much faster responses— or less hardware. Nobody has tested it yet.

Faster responses under real load

The right combination of serving runtime, precision, batching, parallelism and decoding can reduce time to first token and preserve generation speed as concurrency rises. The result is visible in response time, queueing and the number of users the deployment can serve.

More usable capacity from the same system

Well-chosen weight and KV-cache quantization can free memory for longer context windows, larger batches or more concurrent requests. We measure the capacity gain beside workload quality, because a configuration that fits more work but changes the answers is a trade-off, not a free improvement.

Uptime and GPU utilization describe whether the deployment is running. They do not show whether another compatible configuration would answer faster, fit more work or preserve quality at lower cost.

04 — Staged experiments

Compare configurations that fit your stack.

Hardware, memory and runtime compatibility define which configurations can run. Compare those candidates on your workload, then check promising combinations against quality and performance requirements. The diagrams below illustrate the search and the trade-offs to measure.

The deployment search space narrowing from thousands of combinations to one shipped build TYPICAL SEARCH DIMENSIONS precision format ×6 KV-cache format ×3 runtime & kernels ×4 tensor parallelism ×3 batching & prefill ×4 context handling ×3 speculative decoding ×2 hardware target ×3 15,552 combinations FEW worth measuring 3 on the frontier 1 you ship The first decision saves most of the work. Focus measurement on compatible candidates most likely to change the deployment decision. dimension counts are typical, not a quote — the real space depends on your model, hardware and workload
Measured trade-offs

Measure the shortlist. Keep the trade-offs that hold up.

Candidates are compared across quality, speed, capacity and cost. Your requirements define the acceptable region; the measurements reveal the frontier inside it.

Illustrative Pareto frontier of deployment configurations plotted by inference cost against quality on the customer workload 100% 98% 96% 94% QUALITY ON YOUR WORKLOAD ← cheaper more expensive → your quality floor below your floor — rejected INT4 · AWQ INT8 FP8 BF16 · reference RECOMMENDED — FP8 + FP8 KV CACHE ship here cheapest point above your floor Everything right of the line is quality you were already paying for. dominated on the frontier meets your constraints, costs least illustrative — your frontier will look different
From candidate to configuration

One experiment can span several layers of the stack.

A candidate can combine a model, quantization, compiler and serving runtime. RadianVector tests those combinations under a defined workload and records quality, speed and memory behavior. Final selection follows your acceptance criteria and held-out results.

When runtimes, kernels or formats change, earlier evidence provides the starting point for the next search.

vLLM TensorRT-LLM SGLang llama.cpp ONNX Runtime NVIDIA Model Optimizer LLM Compressor AWQ GPTQ SmoothQuant NVFP4 and whatever ships next
05 — Illustrative example

The findings our tools help you compare.

EXAMPLE CONFIGURATION · ILLUSTRATIVE VALUES

FP8 weights & activations, FP8 KV cache

runtime  vllm
precision  fp8_w8a8
kv_cache  fp8
max_num_seqs  32
99.5%
of reference quality
1.5s
p95 latency
−50%
serving cost
THE ONE WE REJECTED

INT4 saves more memory and holds up on short answers and classification, but long-context tool use falls sharply. It is rejected unless those requests can be routed elsewhere.

Illustrative — this is the shape of the output, not a measured result.

Illustrative per-capability degradation for the rejected lower-precision configuration IF YOU WENT ONE STEP FURTHER short answers −1% classification −2% summarisation −4% code −11% tool calling −14% long-context tool use −38% the average would have read −5%. the average is not the problem. illustrative — your numbers will differ
06 — Ongoing optimization

Keep improving from a measured baseline.

Start by measuring the system you have. Use that baseline to check whether each change helps.

A new checkpoint, runtime, driver, hardware target, system prompt or workload can change the result. Revalidation starts from the saved baseline and reruns the measures affected by the change.

Review a change
EXAMPLE OPTIMIZATION UPDATE Configuration review
NEW OPTION
RadianVector added New serving technique New evaluation check
Workload quality floorPASS
New capability checkPASS
Response-time policyIMPROVED
Context-capacity policyIMPROVED
Updated recommendation BETTER CONFIGURATION FOUND

Illustrative workflow — the actual result depends on your model, workload, constraints and hardware.

Protect the result

Check improvements as your system changes.

RadianVector reruns the same evaluations against the saved baseline, so regressions and performance changes show up before release.

  • Affected quality and performance measures rerun on the same representative workload
  • Your requirements remain the decision boundary, with failing cases made visible
  • A broader search starts when a change materially alters the available trade-offs
  • Regressions and better candidates become visible before the release reaches production
EXAMPLE FINDINGS

→ The runtime upgrade you're about to take improves throughput, but tool-calling accuracy drops on your workload.

→ The new checkpoint holds quality at a lower precision than the old one did. You can drop a tier and stay above your floor.

Examples of the form, not results we are reporting.

Illustrative revalidation checks before a release new build retested gate PASS HELD · regression on a slice v2.4.0v2.4.1
07 — What the evaluation delivers

A configuration your team
can inspect and apply.

Keep workload requirements, tested settings and validation results together.

01

Representative tasks

A versioned workload with the source context, scoring rules and failure categories that matter to your team.

02

A measured baseline

The current setup, repeated runs and operating conditions against which candidates are compared.

03

A configuration shortlist

Supported model and serving settings that meet your requirements, with rejected candidates and their reasons.

04

Held-out checks

Finalist results on cases reserved from the search, with repeated runs and important failure examples.

05

A reproducible setup

The model build, runtime version, hardware and launch settings behind the chosen configuration, ready to apply.

06

A saved evaluation record

Tasks, configurations, measurements and scoring for checking the next model, runtime or workload change.

See the engineering depth in a published investigation →

08 — Company

Based in Silicon Valley.

Close to the inference ecosystem

New accelerators, quantization formats, runtimes and kernels continually change the available configuration space. The evaluation process is designed to admit new candidates as they become usable.

Private by default

Client names, model weights, prompts, datasets and results are not published without written permission.

09 — Independent Evidence

Independent evidence for model and deployment claims.

RadianVector tests models and deployment configurations independently of the team making the product claim. Our tools generate reports documenting the method, configuration, operating conditions and material limitations.

RV-380 is one published example of our evaluation method: a fixed, code-scored dataset used to compare configurations question by question and measure run-to-run disagreement. For other models and workloads, we can generate or assemble larger, more varied evaluation sets around the target domain, difficulty and failure modes. In one published comparison, MTP depth 1 increased throughput by 7.4% at six concurrent users and reduced short-answer accuracy by 6.5 points. Both numbers came from the same experimental arm.

Test what will actually ship

Evaluate the model with the quantization, runtime, hardware and serving configuration intended for deployment.

Produce evidence others can examine

Use the result as a private decision record or as the basis for a public third-party report with traceable measurements.

A record for future decisions

Retain the tested settings, tasks and scoring to compare the next model, runtime or hardware change against the same baseline.

10 — Benchmarks

Published benchmarks, with the method attached.

Each study includes the deployment conditions, experimental design, raw figures, corrections and results that did not support the expected conclusion.

QWEN3.8-27B — CONFIGURATION STUDY

Beyond tokens per second

A joint performance-and-behavior comparison across quantized builds, cache formats, execution modes, speculative decoding and concurrency. Four quantized builds, speculative decoding, KV-cache precision and concurrency from one user to thirty-two, all measured on the RTX 4090 GPU family with 24 GiB of GPU memory.

On this model, MTP depth 3 reduced short-answer accuracy by 11.9 points. Under full reasoning, the measured loss was 1.2 points. The study reports speed and quality by arm.

Qwen3.8-27BRTX 4090 GPU familyvLLM 0.28380 scored tasks
Read the study
QWEN3.8-27B — REASONING STUDY

Thinking on, thinking off

The same 380 questions with the model’s native reasoning mode switched on, across every setting it offers. What each step of the effort dial buys in accuracy — and what the same step costs in wait, throughput and how many users still fit.

The short version: turn it on, and the gain has almost entirely arrived by the lowest setting. The template’s own default is the expensive end of a curve that flattened two settings earlier.

4 checkpoint builds4 effort settingsPaired tests216 load rungs
Read the study
Start a conversation

Which configuration meets your deployment requirements?

Tell us what you run and where it falls short, even if you are not sure why. We will work out with you what to measure, from answer quality to latency and capacity.

hello@radianvector.com