BenchZgxFury

HPGuide to the tests
Inference Server Evaluation

Guide to tests carried out

What each test measures, why it exists alongside the others, how to read its numbers, and what a person should check for themselves. The numerical results and their interpretation by family are in the dashboard.

human control: none — the number is enough recommended — a glance avoids a misreading required — watch, listen or play before concluding

Overview

15 tests · 12 families

One line per test. “Measurements” counts the recordings of runs included in the dashboard (excluding examples).

FamilyTestQuestion he answersHuman controlMeasurements ok/failModels measured
Machine Machine inventory On which machine, which GPU, which driver and which versions were the measurements taken? aucun 2 / 0 –
Machine GPU health Is the GPU able to produce reliable measurements? advised 1 / 0 –
LLM performance (service) LLM service performance How many tokens per second does the machine produce, and what latency does each user see, when the load increases? aucun 854 / 5 15
Goodput Goodput (capacity under service objective) How many simultaneous users does the machine serve while respecting latency objectives? aucun 3 / 3 3
llama-bench (GGUF) llama-bench (GGUF) What is the raw throughput of llama.cpp for a GGUF file, without a server or network? aucun 24 / 1 3
Model quality Quality of LLMs (lm-eval) Does quantification cause a loss of accuracy compared to the reference BF16 model? advised 4 / 5 4
Model quality Multimodal quality (lmms-eval) Does the model understand images and videos correctly? aucun 2 / 0 2
2D games (code quality) 2D games (code quality) Does the model write, alone or as an agent, a complete program that works? requis 50 / 5 4
Vision-language (throughput) Vision-language (throughput) At what rate does the model serve chat requests containing images? aucun 20 / 0 2
Voice recognition Speech recognition (ASR) How much audio is transcribed per second, and with how many errors? advised 36 / 0 3
Diarization Diarisation (who speaks when) Does the model correctly attribute each passage to the right speaker, and quickly enough? aucun 0 / 2 –
Real-time voice (Kyutai) Real-time voice (Kyutai) Can we hold a voice conversation in real time, and with how many users? requis 2 / 5 2
Embeddings Embeddings How many texts vectorized per second, and does the search find the right documents? aucun 24 / 4 2
Diffusion (image, video) Image and video generation How many images (or clips) does the machine generate per second? requis 6 / 0 2
Real-time neural rendering Real-time neural rendering (Matrix-Game 4.0) Can a video streaming model replace real-time game rendering on this machine? requis 2 / 0 1

Comparing tests: what each measures

No test alone answers “which is the best model”. The matrix shows what each covers; a solid conclusion crosses at least one speed test and one accuracy test from the same category.

TestThroughputLatencyCapacity under SLOAccuracy/qualityReal timeCostMachine
Machine inventory ●
GPU health ●
LLM service performance ●●
Goodput (capacity under service objective) ●●●
llama-bench (GGUF) ●
Quality of LLMs (lm-eval) ●
Multimodal quality (lmms-eval) ●
2D games (code quality) ●●
Vision-language (throughput) ●●
Speech recognition (ASR) ●●●
Diarisation (who speaks when) ●●●
Real-time voice (Kyutai) ●●
Embeddings ●●●
Image and video generation ●
Real-time neural rendering (Matrix-Game 4.0) ●●●

Grouped by model category

Models can only be compared within the same category. For each: the question asked, the tests that answer it, and the models in the catalog (measured = at least one valid result).

LLM text and code

Which model and engine serves the most users, fast enough, without losing quality?

Tests
Catalog models
  • dfm-mimir-1.5-gguf tier 1
  • ornith-1.5-35b-fp8 tier 1
  • qwen3-coder-next-bf16 tier 1
  • qwen3-coder-next-fp8 tier 1
  • qwen3.8-27b-bf16 tier 1
  • qwen3.8-27b-fp8 tier 1
  • qwen3.8-27b-gguf-q8 tier 1
  • veriloop-e2-bf16 tier 1
  • apertus-1.5-70b tier 2
  • apertus-1.5-8b tier 2
  • dfm-mimir-1.5-gguf-q8 tier 2
  • gpt-oss-120b tier 2
  • normistral-11b-long tier 2
  • normistral-11b-thinking tier 2
  • ornith-1.5-35b-bf16 tier 2
  • qwen3.8-27b-gguf-q4 tier 2
  • qwen3.8-flash-next tier 2
  • qwen3.8-flash-next-fp8 tier 2
  • veriloop-e2-gguf tier 2
  • qwen3-coder-480b-nvfp4 tier 3
  • qwen3.8-27b-nvfp4 tier 3

Audio (transcription, diarization, voice)

How many hours of audio per hour of calculation, with what accuracy, and in real time?

Tests
Catalog models
  • nemotron-3-diarization tier 1
  • qwen3-asr-1.7b tier 1
  • whisper-large-v3 tier 1
  • whisper-large-v3-turbo tier 1
  • qwen3-asr-0.6b tier 2
  • kyutai-stt-1b-en_fr tier 3
  • kyutai-tts-1.6b tier 3
  • moshi tier 3

Embeddings (semantic search, RAG)

How many documents indexed per second, and with what search quality?

Tests
Catalog models
  • bge-m3 tier 1
  • qwen3-embedding-8b tier 1
  • qwen3-embedding-0.6b tier 2

Test sheets

LLM performance (service) see results →

LLM service performance llm_serve

human control: none

How many tokens per second does the machine produce, and what latency does each user see, when the load increases?

What is measured
Time to first token (TTFT), time per token (TPOT), inter-token latency (ITL), aggregate throughput and per user, up to 1 to 128, for each load profile.
Protocol
vllm bench serve against engine OpenAI server (vLLM, SGLang, llama.cpp, Ollama), fixed input and output lengths, --ignore-eos, temperature 0, prefix cache disabled, 4 × concurrency queries, 3 repetitions, median.
Metrics
  • aggregate throughput (tok/s) ↑ higher = better
  • throughput per user (tok/s) ↑ higher = better
  • TTFT (ms) ↓ lower = better
  • TPOT (ms) ↓ lower = better
Interest compared to other tests
This is the measure of raw capacity: it tells how much the machine can produce. It does not say if this throughput is usable (this is the role of goodput, which adds latency objectives), nor if the responses are correct (this is the role of lm-eval). Unlike llama-bench, it goes through a real HTTP server with simultaneous requests: this is what an application will experience.
Comment lire
The aggregate throughput increases with the concurrency then plateaus: the ceiling is the capacity of the machine for this model. The throughput per user drops: below around 20 tok/s, live reading becomes painful. Comparing FP8 and BF16 of the same model gives the quantification gain.
Limits
Fixed-length synthetic queries lack the variety of real traffic; the cache prefix, which helps a lot in production (RAG, agents), is deliberately deactivated.
Coverage
854 mesures valides, 5 en échec · Apertus-v1.5-70B bf16, Apertus-v1.5-8B bf16, DFM-Mimir-v1.5-GGUF gguf-bf16, gpt-oss-120b mxfp4, normistral-11b-long bf16, normistral-11b-thinking bf16, Ornith-1.5-35B-A3B bf16, Ornith-1.5-35B-A3B-FP8 fp8, Qwen3-Coder-Next bf16, Qwen3-Coder-Next-FP8 fp8, Qwen3.8-27B bf16, Qwen3.8-27B-FP8 fp8, Qwen3.8-27B-GGUF gguf-q8_0, VeriLoop-E2 bf16, VeriLoop-E2-GGUF gguf-bf16

Goodput see results →

Goodput (capacity under service objective) goodput

human control: none

How many simultaneous users does the machine serve while respecting latency objectives?

What is measured
Maximum concurrency for which 99% of requests respect TTFT ≤ 500 ms and ITL ≤ 25 ms (default), and corresponding useful throughput.
Protocol
GuideLLM, load sweep (sweep) then goodput profile with SLO; fallback to the sweep (p99 ≤ SLO) if the profile fails.
Metrics
  • max concurrency under SLO ↑ higher = better
  • useful throughput (tok/s) ↑ higher = better
Interest compared to other tests
Where llm_serve gives the maximum throughput, goodput gives the throughput that can be sold: an engine that produces a lot but with a p99 of 3 s does not serve anyone properly. This is the figure to remember to size a service.
Comment lire
A concurrency under SLO of 0 means that even a single user exceeds the objective (often the TTFT of a long context).
Limits
The result depends entirely on the SLOs chosen; changing 25ms to 50ms can double the capacity.
Coverage
3 mesures valides, 3 en échec · Ornith-1.5-35B-A3B-FP8 fp8, Qwen3-Coder-Next-FP8 fp8, Qwen3.8-27B-FP8 fp8

llama-bench (GGUF) see results →

llama-bench (GGUF) llama_bench

human control: none

What is the raw throughput of llama.cpp for a GGUF file, without a server or network?

What is measured
Prefill (pp, tok/s) and generation (tg, tok/s) throughput at multiple context depths.
Protocol
official llama-bench binary, one query at a time, all layers on the GPU.
Metrics
  • tok/s ↑ higher = better
Interest compared to other tests
Isolates the llama.cpp engine and GGUF format from the effects of the HTTP server. Used to compare the GGUF quantifications with each other (Q8, Q4, BF16) and to identify a loss due to the server (difference with llm_serve at concurrency 1).
Limits
Single stream, no concurrency: Does not predict multi-user capacity.
Coverage
24 mesures valides, 1 en échec · DFM-Mimir-v1.5-GGUF gguf-bf16, Qwen3.8-27B-GGUF gguf-q8_0, VeriLoop-E2-GGUF gguf-bf16

Model quality see results →

Quality of LLMs (lm-eval) lm_eval

human control: recommended

Does quantification cause a loss of accuracy compared to the reference BF16 model?

What is measured
Score (%) on gsm8k (mathematical reasoning), gpqa_diamond (expert questions), mmlu_pro (knowledge, 1000 examples), humaneval (code), and the difference in points with the BF16 reference.
Protocol
lm-eval-harness on server OpenAI endpoint still hot, temperature 0, generative tasks only; the BF16 reference is evaluated in the same run.
Metrics
  • score per task (%) ↑ higher = better
  • gap vs BF16 (points) ↑ higher = better
Interest compared to other tests
The counterbalance of speed tests: an FP8 twice as fast is only of interest if its deviation remains within tolerance (2 points). It is not an absolute classification of the models among themselves (the tasks are short and sampled), it is a safeguard for quantification.
Comment lire
A negative deviation beyond 2 points is an alert; a few tenths in one direction or the other is noise.
Limits
Subsets of examples, no log-probabilities; Differences in response format may cost points without substantive errors.
Check it yourself
In case of alert, read some false responses in the raw outputs (poorly extracted response format or real error?).
Coverage
4 mesures valides, 5 en échec · Apertus-v1.5-70B bf16, Apertus-v1.5-8B bf16, normistral-11b-thinking bf16, Qwen3.8-27B-FP8 fp8

Multimodal quality (lmms-eval) lmms_eval

human control: none

Does the model understand images and videos correctly?

What is measured
Score (%) on MMMU (university questions with image) and Video-MME (questions on video).
Protocol
lmms-eval on the served endpoint, responses logged (--log_samples).
Metrics
  • score per task (%) ↑ higher = better
Interest compared to other tests
Completes the vlm test, which only measures throughput with synthetic images without judging the responses.
Limits
Subassemblies; Scores sensitive to response format.
Coverage
2 mesures valides · Qwen3.8-27B bf16, Qwen3.8-27B-FP8 fp8

2D games (code quality) see results →

2D games (code quality) codegen

human control: required

Does the model write, alone or as an agent, a complete program that works?

What is measured
Seven HTML5 games (snake, brick breaker, flappy, invaders, tetris, Doom-like, kart) by model and mode (skills, agent). Automatic proofs: loading, animation loop, images rendered in 15 s, exceptions, console errors. Rating from an LLM judge (completeness, playability, code quality, out of 10). Cost: duration, tokens, agent turns.
Protocol
Same specification and prompt for all, temperature 0; running in Chromium without a screen (Playwright); same judge for all models in a run.
Metrics
  • judge score (/10) ↑ higher = better
  • complete execution ↑ higher = better
  • generation time(s) ↓ lower = better
Interest compared to other tests
The only test that judges a job from start to finish rather than a short answer: humaneval checks a function, codegen checks a running program. Agent mode additionally measures the ability to use tools and self-correct, at the cost of more time and tokens (displayed in “Cost of a game by mode”).
Comment lire
A high rating with “failed execution” is suspicious. A score obtained by the model which judges itself is marked “self-assessment”.
Limits
The LLM judge is not perfectly reproducible and may prefer his own style; “runs for 15 seconds without error” does not mean “fun” or “compliant”.
Check it yourself
Play games from the dashboard (“play” tile, instant model change) and confirm that the judge’s score corresponds to actual playability, especially for those at the top of the rankings.
Coverage
50 mesures valides, 5 en échec · Apertus-v1.5-70B bf16, Apertus-v1.5-8B bf16, Qwen3-Coder-Next-FP8 fp8, Qwen3.8-27B-FP8 fp8

Vision-language (throughput) see results →

Vision-language (throughput) vlm

human control: none

At what rate does the model serve chat requests containing images?

What is measured
TTFT, TPOT, aggregated bitrate according to concurrency, with synthetic images of fixed size (video option).
Protocol
AIPerf, generated images, fixed output lengths.
Metrics
  • aggregate throughput (tok/s) ↑ higher = better
  • TTFT (ms) ↓ lower = better
Interest compared to other tests
Shows image cost: Compared to llm_serve on the same model, the deviation of TTFT is the visual encoding time. The accuracy of the responses is measured separately by lmms-eval.
Limits
Synthetic images without content; responses are not judged.
Coverage
20 mesures valides · Qwen3.8-27B bf16, Qwen3.8-27B-FP8 fp8

Voice recognition see results →

Speech recognition (ASR) asr

human control: recommended

How much audio is transcribed per second, and with how many errors?

What is measured
RTFx (seconds of audio per second of computing), TTFT and concurrency latency; WER on LibriSpeech test-clean (English) and FLEURS (French).
Protocol
asr_bench script (latency, throughput, RTFx) and asr_wer against the server audio endpoint; normalized text before WER calculation.
Metrics
  • RTFx ↑ higher = better
  • WER (%) ↓ lower = better
Interest compared to other tests
Speed and accuracy are measured on the same models: we can see directly if a faster “turbo” model loses in WER. FLOWERS in French is used to identify models that are only valid for English.
Comment lire
~2.7% WER on LibriSpeech is the published benchmark for Whisper large-v3; French is always higher.
Limits
The WER depends on the standardization of the text (punctuation, numbers, accents).
Check it yourself
Read some French transcriptions: part of the WER may come from normalization (figures, elisions) rather than actual errors.
Coverage
36 mesures valides · Qwen3-ASR-1.7B bf16, whisper-large-v3 fp16, whisper-large-v3-turbo fp16

Diarization see results →

Diarisation (who speaks when) diarization

human control: none

Does the model correctly attribute each passage to the right speaker, and quickly enough?

What is measured
DER (diarization error rate) broken down into missed speech, false alarm and speaker confusion; Offline RTFx (batches 1 and 8); latency per segment in streaming.
Protocol
Nemotron-3-Diarization via NeMo on VoxConverse test (50 files), pyannote.metrics, collar 0.25 s, overlaps included.
Metrics
  • DER (%) ↓ lower = better
  • RTFx ↑ higher = better
Interest compared to other tests
Completes the ASR for meetings and interviews: the transcription says what, the diarization says who.
Limits
VoxConverse is in English, from television debates; another type of record may give a very different DER.
Coverage
0 mesure valide, 2 en échec

Real-time voice (Kyutai) see results →

Real-time voice (Kyutai) kyutai

human control: required

Can we hold a voice conversation in real time, and with how many users?

What is measured
Go/no-go aarch64, latency to first audio, number of simultaneous streams, real-time factor (RTF) for Moshi (voice-to-voice), STT and TTS.
Protocol
Official Kyutai servers, synthetic clients in parallel.
Metrics
  • first audio (ms) ↓ lower = better
  • RTF ↓ lower = better
Interest compared to other tests
The only interactive real-time audio test: it is no longer a question of throughput but of maintaining the cadence of speech (RTF < 1).
Limits
Do not judge the naturalness of the voice or the relevance of the responses.
Check it yourself
Listen to exchanges (naturalness of voice, cuts, perceived delay); an RTF < 1 does not imply a pleasant conversation.
Coverage
2 mesures valides, 5 en échec · stt-1b-en_fr bf16, tts-1.6b-en_fr bf16

Embeddings see results →

Embeddings embeddings

human control: none

How many texts vectorized per second, and does the search find the right documents?

What is measured
Requests/s, tokens/s, p50/p99 latency on /v1/embeddings at multiple input sizes; Average MTEB score on a subset of FR/EN search tasks.
Protocol
AIPerf for speed, mteb_eval script on the served endpoint for quality.
Metrics
  • requests/s ↑ higher = better
  • Average MTEB ↑ higher = better
Interest compared to other tests
Dimensions the indexing of a RAG (speed) and the choice of model (quality); a small, very fast model is only useful if its MTEB remains close.
Limits
MTEB subset; does not replace an evaluation on your own documents.
Coverage
24 mesures valides, 4 en échec · bge-m3 fp16, Qwen3-Embedding-8B bf16

Diffusion (image, video) see results →

Image and video generation diffusion

human control: required

How many images (or clips) does the machine generate per second?

What is measured
Frames/sec by batch size and resolution (SDXL, Qwen-Image), seconds per clip (Wan 2.2), latency, GPU memory.
Protocol
Diffusion pipelines, fixed number of steps, fixed prompts.
Metrics
  • images/s ↑ higher = better
  • s per clip ↓ lower = better
Interest compared to other tests
Only capacity measurement on image models: it compares models with equal number of steps, not with equal quality.
Limits
The quality of the images is not measured (neither automatic scoring nor systematic preservation of images).
Check it yourself
Generate and view sample images per model with the same prompts before comparing bitrates; a faster model may produce unusable images.
Coverage
6 mesures valides · Qwen-Image bf16, stable-diffusion-xl-base-1.0 fp16

Real-time neural rendering see results →

Real-time neural rendering (Matrix-Game 4.0) matrixgame

human control: required

Can a video streaming model replace real-time game rendering on this machine?

What is measured
Images/s of the model alone, served online and encoded; delay before first image; latency per block (p50/p90/p99); VRAM at peak; lost images. 20 sec clip in dual view.
Protocol
Official repository server, synthetic client that plays 60 s actions per scene (Egypt, race) and preset (realtime, quality).
Metrics
  • images/s live ↑ higher = better
  • first image(s) ↓ lower = better
Interest compared to other tests
Measures a new load (interactive generative world) where only maintaining a cadence counts; no other test covers it.
Comment lire
Around 24 frames/s the rendering appears fluid; under 15 frames/s it is jerky. A “software rendering” session is not representative.
Limits
The action → image latency seen in a browser is not measured; visual consistency is not noted.
Check it yourself
Watch clips in dual view (low-poly | rendered) and play live (`benchzgx play`) to judge consistency and responsiveness.
Coverage
2 mesures valides · Matrix-Game-4.0 bf16

Machine

Machine inventory inventory

human control: none

On which machine, which GPU, which driver and which versions were the measurements taken?

What is measured
Hostname, GPU and Memory, Compute Capacity, Driver and CUDA, Power Limit, CPU, RAM, OS, Kernel, Docker.
Protocol
Reading nvidia-smi, /proc and docker version at the start of each run; the result is attached to the run.
Interest compared to other tests
Does not compare anything in itself: it makes the other figures comparable between two campaigns (same driver, same power limit). Two runs with different inventories should not be mixed without reporting it.
Limits
Does not check cooling or ambient temperature.
Coverage
2 mesures valides

GPU health health

human control: recommended

Is the GPU able to produce reliable measurements?

What is measured
DCGM Diagnostic (Level 2), Unfixed ECC Errors, Recent XID Errors, Free Disk, Host Bandwidth ↔ GPU (nvbandwidth).
Protocol
dcgmi diag -r 2, nvidia-smi -q, kernel log, nvbandwidth; before the campaign.
Metrics
  • DCGM failures ↓ lower = better
  • uncorrected ECC errors ↓ lower = better
  • XID errors ↓ lower = better
Interest compared to other tests
A faulty GPU produces low throughput without error messages. This test separates “the model is slow” from “the machine is sick”. To be looked at before any conclusion of performance.
Limits
A short diagnostic does not detect a thermal throttling which only appears after one hour of charging (see telemetry).
Check it yourself
Check the checklist before campaign (rest temperature, active liquid cooling, no other GPU processes).
Coverage
1 mesure valide

Human-controlled tests

7 tests

These checks are not automated and do not enter any numbers. Do them before publishing a conclusion.

LevelTestWhat to check
requis Image and video generationGenerate and view sample images per model with the same prompts before comparing bitrates; a faster model may produce unusable images.
requis 2D games (code quality)Play games from the dashboard (“play” tile, instant model change) and confirm that the judge’s score corresponds to actual playability, especially for those at the top of the rankings.
requis Real-time neural rendering (Matrix-Game 4.0)Watch clips in dual view (low-poly | rendered) and play live (`benchzgx play`) to judge consistency and responsiveness.
requis Real-time voice (Kyutai)Listen to exchanges (naturalness of voice, cuts, perceived delay); an RTF < 1 does not imply a pleasant conversation.
advised Quality of LLMs (lm-eval)In case of alert, read some false responses in the raw outputs (poorly extracted response format or real error?).
advised Speech recognition (ASR)Read some French transcriptions: part of the WER may come from normalization (figures, elisions) rather than actual errors.
advised GPU healthCheck the checklist before campaign (rest temperature, active liquid cooling, no other GPU processes).

Glossary

36 termes
Agent (mode)
Codegen mode where the model has tools (read, write, execute) and can test then correct its game in several rounds.
BF16
16-bit digital format (bfloat16), reference accuracy of model weights; serves as a basis of comparison for the quantified versions.
Concurrency
Number of requests sent at the same time to the server; simulates that many simultaneous users.
Human control
Verification that a person must do (watch, listen, play) because the automatic number is not enough to conclude.
DCGM
NVIDIA GPU diagnostic tool (Data Center GPU Manager); level 2 launches calculation and memory tests.
Decode
Phase where the model generates the response tokens one by one; limited by memory bandwidth.
Delta quality
Difference in points between the score of a quantified model and that of its BF16 reference. Alert beyond 2 points.
DER
Diarization Error Rate: portion of speaking time incorrectly allocated (missed, false alarm or wrong speaker). Lower is better.
Embedding
Digital vector representing the meaning of a text, used for semantic search and RAG.
FP8
8-bit floating point quantization: about half the memory of BF16, often faster, generally low quality loss.
GGUF
File format of llama.cpp and Ollama, which includes quantified weights (Q8_0, Q4_K_M…) and metadata.
Goodput
Useful throughput: the share of traffic served while respecting the latency objectives (SLO).
HBM
High bandwidth memory soldered to the GPU (256 GB on the GB300).
ITL
Inter-Token Latency: time between two consecutive tokens of a response.
LLM Judge
Language model which notes a result (here the games generated) according to a grid; practical but not perfectly reproducible.
KV cache
Memory where the server keeps the state of each ongoing conversation; its size limits concurrency and context length.
lm-eval
lm-evaluation-harness: standard evaluation bench for the accuracy of LLMs on sets of questions.
MTEB
Massive Text Embedding Benchmark: battery of tasks that measures the quality of embeddings.
MTP
Multi-Token Prediction: speculative decoding where the model offers several tokens at once, then verified; speeds up generation.
NVFP4
NVIDIA 4-bit quantization specific to Blackwell GPUs: four times less memory than BF16.
p50 / p99
Median / 99th percentile: p99 is the value that 99% of requests do not exceed, i.e. the unfavorable case experienced by 1 user in 100.
Prefill
Phase where the model reads the entire prompt before the first token; limited by computing power. Dominates the TTFT.
Prefix cache
Reuse of the calculation of a previously seen prompt start; disabled during measurements so as not to flatter the results.
Load profile
Inlet length / outlet length pair (cat 1024/1024, rag 8192/1024, long 32768/1024, agent 4096/2048, etc.).
Quantization
Reduction of weight precision (FP8, NVFP4, GGUF Q8/Q4) to gain memory and speed at the cost of a possible loss of quality.
RAG
Retrieval-Augmented Generation: the model responds from documents found by semantic search.
RTF / RTFx
Real time factor. RTF = calculation time / audio duration (real time if < 1); RTFx = the opposite (seconds of audio processed per second, higher is better).
Skills (mode)
Single answer codegen mode, with identical best practice instructions for all models in the system prompt.
SLO
Service Level Objective: latency objective to be respected (here TTFT p99 ≤ 500 ms and ITL p99 ≤ 25 ms by default).
tok/s
Tokens per second. Aggregated: all users combined (capacity). By user: perceived reading speed.
Token
Unit of text manipulated by the model (on average ¾ of a word in English, a little less in French).
TPOT
Time Per Output Token: average time per token generated after the first; 1000 / TPOT = tok/s seen by a user.
TTFT
Time To First Token: delay between sending the request and the first token received; it’s perceived responsiveness.
Telemetry
Second-by-second reading of GPU (and CPU) power, temperatures, clocks and memory during a measurement.
VLM
Vision-Language Model: model that accepts images (or videos) as input in addition to text.
WER
Word Error Rate: share of incorrectly transcribed words (substituted, forgotten, added). Lower is better.