EXPLAINER

ONNX Runtime

runtimefreshness: volatilesources not yet verified

ONNX Runtime is a cross-platform engine for running machine-learning models exported to the ONNX format across CPUs, GPUs and edge devices.

Short answer: Export a model to ONNX, then run it with ONNX Runtime and the right execution provider (CPU, CUDA, DirectML, CoreML…) for portable, optimised inference, including in browsers and mobile apps. LLM support exists via dedicated generation tooling; check compatibility.

What is it?

ONNX is a common file format for AI models; ONNX Runtime is a fast, portable engine that runs those files on many kinds of hardware and platforms, including phones and web browsers. It is popular for shipping smaller models inside applications.

Also known as: ONNX, ORT, onnxruntime, ONNX export, ONNX runtime web, DirectML, execution provider, transformers.js.

Technical definition

ONNX defines a model graph and operator set; ONNX Runtime applies graph optimisations and dispatches to execution providers. Export from PyTorch/TensorFlow uses exporters that may need operator-set/version care. Browser deployment uses WebAssembly/WebGPU backends (ONNX Runtime Web) and libraries such as Transformers.js build on it. Generative-model support, quantisation tooling and provider coverage vary by version.

Entity type: runtime · Freshness class: volatile.

When should I use it?

Use for portable, on-device and browser deployment of small to mid-size models.

Implementation patterns

  • Export, validate numerical parity, quantise, deploy.
  • Pick execution providers per platform.
  • In-browser inference for privacy-preserving demos.

Technologies and frameworks

Commonly associated: ONNX Runtime (+Web, Mobile), Optimum (Hugging Face), Transformers.js, DirectML. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Export with a recent opset.
  2. Validate outputs against the original.
  3. Optimise/quantise.
  4. Test per target platform.
  5. Mind download size for web.

Common failure modes

  • Unsupported operators during export.
  • Numerical drift after export or quantisation.
  • Large model downloads in browsers.
  • Provider-specific behaviour differences.

Troubleshooting

  • Export fails on an operator — Change opset version or rewrite the op.
  • Output differs from PyTorch — Compare layer outputs; check dtype and preprocessing.

Contrasted with

  • llama.cpp — llama.cpp is specialised for LLMs/GGUF; ONNX Runtime is a general model runtime.
  • PyTorch — PyTorch is a training/research framework; ONNX Runtime focuses on deployment.

Questions this page answers

  • What is ONNX Runtime?
  • How do I export a PyTorch model to ONNX?
  • Can I run AI models in the browser?
  • ONNX Runtime vs PyTorch for inference?

Typical goals: deploy a model across platforms; run models in the browser or on device.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: volatile — Products, versions, protocols, benchmarks, leaderboards, regulations and dates. Verify against the primary source before relying on specifics; re-review every 3 months.

Time-sensitive claims to verify before relying on them:

  • Generative-AI support, execution-provider coverage and versions.

Canonical sources (unverified leads):

  • ONNX Runtime documentation — onnxruntime.ai
  • ONNX specification — onnx.ai