Inference
Running a trained model to produce an output: an answer, a summary, a transcript. Every model API call is an inference request.
Learn the language of AI inference
Plain definitions for the words you meet when you buy, run and compare models.
Core concepts
The building blocks: what a model is, what a token is, and what happens when you call one.
Training
Building a model on huge datasets. Expensive, rare, and done mostly by model labs.
Fine-tuning
Extra training on a customer's own data to specialize an existing model for a task, tone or domain.
Model
The trained AI system, e.g. Llama, Mistral, Qwen, DeepSeek, GPT or Claude.
Weights
The files that make up a trained model. Whoever has the weights can run the model.
Parameters
The number of learned values in a model, written like 8B or 70B (billions). A rough signal of capability and of cost to run.
Token
The unit models read and write. In English, one token is roughly three quarters of a word.
Input vs output tokens
Input (prompt) tokens are sent to the model; output (completion) tokens are generated by it. Output is slower to produce and priced higher.
Context window
The most tokens a model can handle in one request, input and output combined, e.g. 128K.
Prompt / system prompt
The instructions and content sent to a model. The system prompt is the standing instruction an app sends with every request.
Embeddings
Numeric representations of text or images, used for search, matching and recommendations rather than for writing text.
RAGretrieval-augmented generation
Pulling relevant passages from the customer's own data into the prompt so the model answers from that data.
Agent / agentic workload
An application where a model plans and takes several steps, calling tools and other models along the way.
Tool callingfunction calling
A model's ability to output structured requests to use external tools or APIs.
Structured outputsJSON mode
Forcing a model to reply in a fixed format such as JSON.
MCPModel Context Protocol
An open standard for connecting models and agents to tools and data sources.
Multimodal
Models that handle more than text: images, audio, video, speech-to-text or text-to-speech.
Models and licences
Open or closed, big or small: how models differ and what their licences allow.
Frontier model
The most capable models at any given time, usually from big labs such as OpenAI, Anthropic and Google. Almost always closed.
Closed
Available only through its creator's API or approved cloud partners. Weights are never released.
Open-weight model
A model whose weights are published for anyone to run under a license, e.g. Llama, Mistral, Qwen, DeepSeek, gpt-oss.
Open source vs open weight
Strictly, open-source AI means weights plus code and enough training detail to rebuild the model, under terms that allow any use. Most "open" models only publish weights.
Model license
The legal terms for using a model. Apache 2.0 and MIT are permissive; custom licenses can cap company size, restrict uses or require attribution. Meta's license for its multimodal Llama models, which covers all of Llama 4, grants no rights to EU-based companies (Llama 4 policy).
Dense vs Mixture of ExpertsMoE
A dense model uses all its parameters for every token. An MoE model uses only some "experts" per token, written as total vs active parameters (e.g. 235B total, 22B active).
Reasoning
A model that writes out internal reasoning before answering, which improves results on hard tasks. Reasoning tokens are billed, usually as output.
SLMsmall language model
A compact model, typically under about 10B parameters, cheap and fast to run.
Distillation
Training a smaller model to imitate a larger one, keeping much of the quality at lower cost.
LoRA / adapters
A light fine-tuning method that produces a small add-on file instead of a full new model. Many LoRAs can share one base model on the same GPUs.
Model provenance
Who built a model and where. Some buyers restrict models from certain countries, for example Chinese labs.
Hugging Face
The main public hub where open models are published and downloaded. "Bring your own Hugging Face model" means deploying any model hosted there.
Speed and performance
What "fast" means. The numbers that describe how quickly answers arrive.
Latency
How long a request takes. For generative models it splits into time to first token and per-token speed.
TTFTtime to first token
Time from sending a request to receiving the first token back.
ITL / TPOTinter-token latency / time per output token
Average gap between generated tokens after the first one.
Output speedtokens per second
Tokens a single request receives per second while generating.
Throughput
Total tokens or requests a deployment processes per second across all users.
End-to-end latency
Total time from request to complete response, including the network.
P50 / P95 / P99
Percentile latency. P95 means 95% of requests were faster than this value.
Concurrency
Number of requests processed at the same time.
Goodput
Throughput that actually meets the latency target.
Cold start
Delay while a model loads onto a GPU before it can serve, e.g. after scaling up from zero. Ranges from seconds to minutes.
Rate limitsRPM / TPM
Caps on requests per minute and tokens per minute for an account or API key.
Benchmark
A standard test. Quality benchmarks (e.g. MMLU, GPQA, SWE-bench) rate the model; serving benchmarks (e.g. Artificial Analysis, MLPerf Inference) rate speed, and sometimes price, across providers and hardware.
Pricing
How inference is priced, and how to compare one bill with another.
Price per million tokens$/1M, "per MTok"
The standard serverless pricing unit, usually quoted separately for input and output tokens.
Blended price
One per-token price assuming a typical input-to-output ratio. Benchmark sites commonly use 3:1.
Cached input price
A discounted price for input tokens the provider has recently processed (see prompt caching).
Batch pricing
A discount, often around 50%, for requests processed asynchronously within a window such as 24 hours.
Serverlesspay per token
Shared endpoints billed only for tokens used, with no capacity commitment. Also called model-as-a-service (MaaS).
GPU-hour
The price of one GPU for one hour; the unit for dedicated capacity.
Dedicated deployment
A model running on GPUs reserved for one customer, billed by time rather than tokens.
Prepaid credits
A balance bought in advance that usage draws down.
Reserved capacity / committed spend
A contract to buy a set amount of capacity or spend over a term in exchange for a lower rate.
Provisioned throughput
Capacity reserved in units of throughput rather than GPUs. Azure calls these PTUs; Amazon Bedrock and Google Vertex AI have equivalents.
Minimum commit / overage
The minimum spend agreed in a contract; overage is usage above it, billed at an agreed rate.
Spot
Spare capacity sold at a discount that can be taken back at short notice.
Service tierspriority, standard, flex
Different prices for the same model based on speed or reliability. Priority costs more and is served first; flex is cheaper and slower.
Egress fees
Charges for moving data out of a cloud.
TCOtotal cost of ownership
Full cost of a solution: compute, engineering time, operations, compliance, support and switching.
Cost per task
The cost of one business outcome: a ticket resolved, a document processed, a call handled.
Tokenizer differences
Models split text into tokens differently, so the same text can be a different number of tokens on each model.
Cloud marketplace / commit drawdown
Buying through the AWS, Azure or Google Cloud marketplaces, often counted against an existing cloud spend commitment (e.g. AWS EDP, Microsoft MACC).
Price compression
Prices for a given level of model capability keep falling fast, year after year.
Deployment
Where a model runs and who shares the machine it runs on.
Serverless vs dedicated
Serverless: shared endpoints, pay per token. Dedicated: GPUs reserved for one customer, pay per hour.
Multi-tenant vs single-tenant
Multi-tenant: customers share infrastructure. Single-tenant: hardware dedicated to one customer.
Deployment template
A pre-configured model, GPU and serving setup tuned for a good default.
Playground
A web interface for trying models with prompts before writing any code.
Region / availability zone
A region is a geographic location; zones are separate data centers within it.
Multi-region / geo-routing
Running in several regions and sending each request to the nearest or healthiest one.
Autoscaling / scale-to-zero
Adding or removing GPUs automatically with demand. Scale-to-zero turns everything off when idle.
VPC deployment
The model runs inside the customer's private cloud network, so traffic never crosses the public internet.
BYOCbring your own cloud
The provider's software runs in the customer's own cloud account, under their billing and controls.
On-premiseson-prem
Inference on hardware in the customer's own data center.
Air-gapped
A deployment with no internet connection at all.
Hybrid
A mix of cloud and on-prem, or of serverless and dedicated.
Edge / on-device inference
Running models close to the user or on their device.
OpenAI-compatible API
An API that accepts OpenAI's request format, so existing code works by changing the base URL and key.
Custom model hostingBYOM
Hosting a customer's own fine-tuned or proprietary weights.
Faster and cheaper
The techniques behind quicker, cheaper answers from the same model.
Quantization / precisionFP16, BF16, FP8, INT8, INT4, FP4
Storing a model's numbers in fewer bits. Lower precision means less memory and faster, cheaper inference, with some risk to quality at the lowest levels.
Batching / continuous batching
Processing many requests together on one GPU. Continuous batching adds and removes requests on the fly.
KV cache
Memory a model keeps of tokens already processed in a request, so it doesn't recompute them. Grows with context length.
Prompt cachingprefix caching
Reusing the processed state of a repeated prompt prefix, such as a long system prompt or document.
Speculative decoding
A small draft model guesses several tokens ahead and the large model checks them in one step. The output is unchanged.
Prefill vs decode
Prefill processes the input prompt (compute-heavy, sets time to first token). Decode generates output one token at a time (memory-heavy, sets tokens per second).
Disaggregated serving
Running prefill and decode on separate GPU pools so each can be tuned on its own.
Parallelismtensor, pipeline, expert
Splitting one model across several GPUs because it doesn't fit on one, or to make it faster.
Inference engine
Software that runs models on GPUs and handles batching and caching, e.g. vLLM, SGLang, TensorRT- LLM, with NVIDIA Dynamo coordinating them at scale. llama.cpp and Ollama are popular for local use.
Routing rules / profiles
Policies that decide where each request runs based on a goal such as lowest cost, best performance or best quality.
Model routing / cascading
Sending each request to the most suitable model, e.g. easy requests to a small model and hard ones to a large one.
Semantic caching
Returning a stored answer when a new request means the same as an earlier one, skipping the model.
Utilization
The share of GPU time and capacity doing useful work.
Hardware
The chips and machines that run models, and why capacity is tight.
GPU / accelerator
The chips that run models. NVIDIA GPUs dominate; "accelerator" also covers AMD GPUs and custom AI chips.
NVIDIA HopperH100, H200
NVIDIA's previous data center generation. The H100 is still the market's reference GPU; the H200 adds more memory.
NVIDIA BlackwellB200, GB200, B300, GB300
NVIDIA's current volume generation, much faster than Hopper and built for low-precision (FP4) inference. B300 and GB300 are the upgraded "Blackwell Ultra".
NVIDIA Vera Rubin
The generation after Blackwell, pairing Rubin GPUs with Vera CPUs. First shipments began in Q3 2026, with volume ramping in Q4 (report).
AMD InstinctMI300X, MI325X, MI355X
AMD's data center GPUs. Large memory makes them strong for big-model inference.
TPU, Trainium, Inferentia
Custom AI chips from Google (TPU) and AWS (Trainium, Inferentia), mostly available only inside those clouds.
Specialized inference chips
Non-GPU chips built for very fast token generation, e.g. Cerebras, Groq's LPU, SambaNova.
GPU memoryVRAM, HBM
High-speed memory on the GPU. Its size decides which models fit; its bandwidth largely sets how fast tokens are generated.
FLOPS
Floating-point operations per second, a measure of raw compute.
Node / cluster
A node is one server, usually with 8 GPUs. A cluster is many nodes networked together.
NVLink / InfiniBand
Fast links between GPUs inside a server or rack (NVLink) and between servers (InfiniBand or high-speed Ethernet).
Rack-scale systeme.g. GB200 NVL72
A full rack of GPUs (72 in the NVL72) wired to act like one giant GPU.
Power
Data center capacity is now measured in megawatts. PUE is total facility energy divided by IT energy; closer to 1.0 is more efficient.
Capacity / lead time
How many GPUs are available, and how long it takes to add more. Demand has even pushed up rental prices for older GPUs like the H100 in 2026 (report).
Data and sovereignty
Where your data sits, whose law applies, and how it is protected.
Data residency
Where data is physically stored and processed, e.g. "EU only".
Residency proofrequest logs
Per-request records showing where each request was processed, with any exception flagged.
Data sovereignty
Data falls only under the laws and control of a chosen jurisdiction, not just stored there.
Sovereign cloud / sovereign AI
Infrastructure and models run under the legal and operational control of one jurisdiction, by entities based there.
US CLOUD Act
A 2018 US law that lets US authorities require US-based companies to hand over data they control, even when it's stored abroad.
Schrems II
A 2020 ruling by the EU Court of Justice that struck down the EU-US Privacy Shield over US surveillance concerns.
EU-US Data Privacy FrameworkDPF
The 2023 adequacy decision allowing personal data transfers to certified US companies. The EU General Court upheld it in September 2025; an appeal to the Court of Justice is pending (tracker).
SCCs and TIAs
Standard Contractual Clauses are EU-approved contract terms for transfers outside the EU; a transfer impact assessment checks the risks in the destination country.
Controller, processor, sub-processorGDPR
The controller decides why and how personal data is processed (the customer). The processor handles it on their behalf (us). Sub-processors are the processor's own suppliers.
DPAdata processing agreement
The GDPR-required contract between controller and processor setting how personal data is handled.
Zero data retentionZDR
The provider stores no prompts or outputs once a request has been served.
No training on customer data
A commitment never to use customer inputs or outputs to train or improve models.
Logging and retention
What request data is recorded (content or only metadata), for how long, and where.
Audit logs
Records of who did what and when: logins, key creation, config changes, deployments.
RBACrole-based access control
Permissions assigned by role, e.g. admin, member, read-only.
SSO, SAML, SCIM
Single sign-on lets staff log in with company accounts, often via the SAML standard. SCIM adds and removes users automatically.
Encryption in transit / at rest
Encryption while data moves over networks (TLS) and while it is stored.
BYOK / CMKcustomer-managed keys
Customers control the encryption keys for their stored data and can revoke them. In AI gateways, BYOK can also mean bringing your own model-provider API key.
Confidential computingTEE
Hardware protection that keeps data encrypted even while it is processed, so not even the operator can see it. Recent NVIDIA GPUs support it.
Tenant isolation
How well one customer's data and workloads are separated from others on shared infrastructure.
PII redaction
Automatically detecting and removing personal data before or after model processing.
Guardrails / content moderation
Filters that block harmful, off-policy or sensitive inputs and outputs.
Certifications and law
The standards and EU rules buyers ask about.
SOC 2Type I / Type II
A US-origin security audit. Type I checks that controls are designed properly at one point in time; Type II checks they actually worked over a period, usually 6 to 12 months.
ISO/IEC 27001
International certification for information security management, audited externally.
ISO/IEC 27701
Privacy extension to ISO 27001, covering how personal data is handled.
ISO/IEC 42001
International standard for AI management systems: how an organization governs the AI it builds or uses.
National cloud schemes: C5, SecNumCloud, ENS
Government-backed cloud security frameworks in Germany (C5), France (SecNumCloud) and Spain (ENS). SecNumCloud also requires protection from non-EU laws.
Cloud Sovereignty FrameworkSEAL levels
The European Commission's method for scoring cloud providers on eight sovereignty objectives, from SEAL-0 (no sovereignty) to SEAL-4 (full EU supply chain, chips to software). Used in its EUR 180 million sovereign cloud tender awarded in April 2026 (Commission).
NIS2
EU cybersecurity directive for essential and important entities across many sectors, including duties on supply-chain security.
DORADigital Operational Resilience Act
EU regulation, applying since 17 January 2025, on how financial firms manage ICT risk, including risk from third-party providers (DNB).
EU AI Act
EU law regulating AI by risk level: banned practices, high-risk systems, transparency duties and rules for general-purpose models. The 2026 AI Omnibus moved high-risk obligations to 2 December 2027, or 2 August 2028 for AI inside regulated products (K&L Gates).
GPAI
AI Act term for models that can do a wide range of tasks, such as large language models. Obligations on their providers already apply; the Omnibus did not delay them.
Provider vs deployerAI Act
The provider develops an AI system or model and puts it on the market; the deployer uses it in its own operations.
EU Data Act
EU regulation, applying since 12 September 2025, that gives customers the right to switch cloud providers. Switching charges are banned from 12 January 2027 (Alston & Bird).
HIPAA / BAA
US health-data law. A BAA (business associate agreement) is the contract a vendor signs to handle US health data.
Reliability
What keeps a model API dependable once it is in production.
SLA / SLO / SLI
SLA: the contractual promise (e.g. 99.9% uptime) with a remedy, usually service credits. SLO: the internal target. SLI: what is actually measured.
Uptime "nines"
Availability as a percentage. 99.9% allows about 43 minutes of downtime a month; 99.99% about 4 minutes.
Failover / fallback
Switching automatically to a backup region, deployment or provider when one fails.
AI gateway / LLM router
A layer in front of models that handles routing, failover, rate limits, caching, logging and access control across one or many providers.
Throttling429 errors
A provider rejecting requests because limits or capacity are exceeded. 429 is the HTTP error code.
Observability
Metrics, logs and traces showing latency, errors, token usage and cost per request.
Model versioning / pinning
Locking an application to one model version so its behavior doesn't change unexpectedly.
Deprecation / end of lifeEOL
When a provider retires a model version and customers must migrate.
Evalsevaluations
Tests a customer runs on their own data to check a model is good enough for their use case.
Hallucination
A model producing confident but false information.
Status page / incident
A public page showing service health; an incident is any outage or degradation.
The market
Who provides what, and the words used to describe the market.
Hyperscaler
The largest clouds: AWS, Microsoft Azure and Google Cloud, sometimes Oracle. They sell models through platforms such as Amazon Bedrock and Google Vertex AI.
Hyperscaler-backed sovereign offering
A cloud built on US technology but operated by EU companies, e.g. S3NS, the Thales and Google Cloud joint venture.
European cloud providers
EU-headquartered clouds and AI platforms, e.g. OVHcloud, Scaleway, STACKIT and IONOS.
Neocloud
GPU-focused clouds that rent compute mainly to AI companies, e.g. CoreWeave, Nebius, Lambda, Crusoe.
Inference provider
Companies specializing in serving models through APIs and dedicated deployments, e.g. Together AI, Fireworks AI, DeepInfra.
Model lab
Companies that build models, e.g. OpenAI, Anthropic, Google DeepMind, Meta, Mistral AI, DeepSeek, Alibaba (Qwen). Most also sell inference on their own models.
Aggregator / marketplace
Services offering one API or catalog across many models and providers, e.g. OpenRouter, Hugging Face.
AI factory / gigafactory
Data centers purpose-built for AI. In the EU, AI Factories and AI Gigafactories are publicly backed programs to expand European compute.
Lock-in / switching cost
How hard it is to move to another provider: proprietary APIs, custom formats, contracts, egress.
Multi-provider strategy
Using several models and providers for resilience, cost and the best fit per task.
Test-time computeinference-time scaling
Getting better answers by spending more compute at inference, e.g. with reasoning models, rather than only by training bigger models.
Jevons paradox
When something gets cheaper, usage can grow so much that total spend rises.
Move your AI to Europe.
One key, one endpoint. Your first call takes minutes.