EXPLAINER

Recurrent Neural Networks and LSTMs

conceptfreshness: stablesources not yet verified

RNNs process sequences one step at a time while carrying a hidden state; LSTMs and GRUs add gates to remember longer. Transformers have largely replaced them for language.

Short answer: RNNs keep a hidden state that is updated step by step. LSTM/GRU gates ease the vanishing-gradient problem. They are sequential (hard to parallelise) and have been largely superseded by transformers and, for some uses, state-space models.

What is it?

A recurrent network reads a sequence one item at a time, carrying a "memory" forward. LSTMs add control gates that decide what to keep or forget, so they remember longer. Today most language models use transformers instead, because they train much faster in parallel.

Also known as: RNN, LSTM, GRU, long short-term memory, gated recurrent unit, sequence model, vanishing gradient, seq2seq.

Technical definition

Vanilla RNN: h_t = f(W h_{t-1} + U x_t). Backpropagation through time suffers vanishing/exploding gradients. LSTM (Hochreiter & Schmidhuber, 1997) uses input/forget/output gates and a cell state; GRU (Cho et al., 2014) simplifies the gating. Sequential dependence limits training parallelism; attention (Bahdanau et al., 2014) and then transformers addressed both long-range dependence and parallelism. Recent State Space Models (Mamba and S4) and linear-recurrent designs revisit recurrence with parallel training.

Entity type: concept · Freshness class: stable.

When should I use it?

Still reasonable for small streaming/time-series tasks; for language prefer transformers.

Implementation patterns

  • Gradient clipping.
  • Bidirectional encoders for classification.
  • Small LSTMs for on-device time-series with tight memory.
  • Encoder–decoder with attention for legacy seq2seq.

Technologies and frameworks

Commonly associated: PyTorch nn.LSTM / nn.GRU, TensorFlow / Keras. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Ask whether a pre-trained transformer is available.
  2. If recurrent, use LSTM/GRU, clip gradients.
  3. Mask padding.
  4. Benchmark against a simple baseline.

Common failure modes

  • Vanishing/exploding gradients on long sequences.
  • Slow training due to sequential computation.
  • Limited long-range memory despite gating.
  • Padding/masking bugs.

Troubleshooting

  • Cannot learn long dependencies — Use LSTM/GRU with clipping, or move to attention-based models.
  • Training slow — Use larger batches and cuDNN kernels, or switch architecture.

Contrasted with

Questions this page answers

  • What is an RNN?
  • What is an LSTM?
  • Why did transformers replace RNNs?
  • What is the vanishing gradient problem?

Typical goals: understand pre-transformer sequence models; decide if an RNN is still appropriate.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.

Canonical sources (unverified leads):

  • Hochreiter & Schmidhuber, Long Short-Term Memory, Neural Computation (1997) — locator to be identified
  • Cho et al., Learning Phrase Representations using RNN Encoder–Decoder (2014) — arXiv:1406.1078
  • Bahdanau et al., Neural Machine Translation by Jointly Learning to Align and Translate (2014) — arXiv:1409.0473