EXPLAINER

Mechanistic Interpretability

techniquefreshness: evolvingsources not yet verified

Mechanistic interpretability tries to reverse-engineer what is happening inside neural networks — identifying features and circuits — rather than only observing inputs and outputs.

Short answer: It aims to explain model behaviour by analysing internal components (neurons, attention heads, features found with sparse autoencoders). Findings so far are partial and mostly on small or specific behaviours; it is a research area, not a turnkey audit tool.

What is it?

Instead of treating an AI as a black box, interpretability researchers open it up and try to find which internal parts do what — like tracing wires in a circuit. The aim is to understand, and eventually verify, why a model does what it does.

Also known as: interpretability, mech interp, explainable AI, XAI, sparse autoencoders, circuits, neuron analysis, activation patching.

Technical definition

Techniques include circuit analysis (Olah et al., "Zoom In", 2020; Elhage et al., 2021), probing classifiers, activation patching/causal interventions, and dictionary learning with sparse autoencoders to extract more interpretable features from activations (Cunningham et al., 2023; Anthropic "Towards Monosemanticity", 2023). Limitations: polysemantic neurons, scaling to large models, faithfulness of explanations, and gaps between interpreting a feature and guaranteeing safety. Generated chain-of-thought text is not a substitute for internal analysis (Reasoning Transparency and Hidden Reasoning).

Entity type: technique · Freshness class: evolving.

When should I use it?

Research and high-stakes auditing contexts; for most applications, rely on behavioural evaluation.

Implementation patterns

  • Probe layers for encoded concepts.
  • Activation patching to find causally important components.
  • SAE feature dashboards for exploration.
  • Use SHAP/LIME-style attributions for tabular/classical models.

Technologies and frameworks

Commonly associated: TransformerLens, sparse autoencoder libraries, SHAP, LIME, Captum. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. State the behaviour to explain.
  2. Pick a method suited to the model type.
  3. Validate with interventions.
  4. Report limits.

Common failure modes

  • Over-interpreting correlational probes.
  • Cherry-picked examples.
  • Assuming interpretability results scale to frontier models.
  • Treating post-hoc attributions as ground truth.

Troubleshooting

  • Explanation does not predict behaviour — Test with interventions rather than correlations.
  • Attributions unstable — Use multiple methods and sanity checks.

Contrasted with

Questions this page answers

  • What is mechanistic interpretability?
  • What are sparse autoencoders?
  • Can we understand how LLMs work inside?
  • What is explainable AI?
  • How do SHAP and LIME work?

Typical goals: understand what interpretability can and cannot do; explain model predictions.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Elhage et al., A Mathematical Framework for Transformer Circuits (2021) — transformer-circuits.pub
  • Cunningham et al., Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023) — arXiv:2309.08600
  • Olah et al., Zoom In: An Introduction to Circuits, Distill (2020) — distill.pub/2020/circuits/zoom-in