What is it?
Picture layers of simple calculators. Each takes numbers in, multiplies them by learned weights, adds them up and passes the result through a simple curve. Stack many layers and train the weights on examples, and the network can learn complex patterns like recognising faces or translating text.
Also known as: neural net, artificial neural network, ANN, multilayer perceptron, MLP, feedforward network, perceptron, activation function.
Technical definition
A feed-forward network computes h_{l+1} = σ(W_l h_l + b_l) for layers l, with non-linear activations σ (ReLU, GELU, etc.). Parameters are fitted by minimising a loss with gradients from Backpropagation and Gradient Descent. Universal-approximation results state that sufficiently wide networks can approximate broad function classes, but say nothing about learnability from finite data. Architectures specialise this template: Convolutional Neural Networks (CNNs), Recurrent Neural Networks and LSTMs, What is a Transformer? (Attention Explained), Graph Neural Networks (GNNs).
Entity type: concept · Freshness class: stable.
When should I use it?
Use for perception, language and other high-dimensional data; consider simpler models for small tabular datasets.
Implementation patterns
- Start from a small MLP baseline for tabular data.
- Normalise inputs; use standard initialisation and optimiser defaults.
- Choose architecture by data type: images, sequences, graphs.
- Use pre-trained networks when possible.
Technologies and frameworks
Commonly associated: PyTorch, JAX, TensorFlow / Keras. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Overfit a tiny batch to verify the pipeline.
- Scale inputs.
- Pick a standard optimiser (Adam/AdamW) and learning rate sweep.
- Monitor training vs validation loss.
- Add regularisation as needed.
Common failure modes
- Overfitting small datasets.
- Poor scaling or initialisation causing vanishing/exploding gradients.
- Using a neural network where gradient-boosted trees would beat it on tabular data.
- No baseline for comparison.
Troubleshooting
- Loss is NaN — Lower learning rate; check input scaling and data for NaNs.
- Training loss does not decrease — Check labels, learning rate and that gradients flow; overfit a tiny batch first.
Contrasted with
- Deep Learning — Deep learning means neural networks with many layers.
- Supervised Learning — A learning setup, not a model type; neural nets are one model family used within it.
Questions this page answers
- What is a neural network?
- How does a neural network learn?
- What is an activation function?
- What is a multilayer perceptron?
Typical goals: learn the basics of neural networks; understand layers, weights and activations.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.
Canonical sources (unverified leads):
- Goodfellow, Bengio & Courville, Deep Learning (MIT Press, 2016) — deeplearningbook.org
- Zhang et al., Dive into Deep Learning — d2l.ai
- Cybenko, Approximation by superpositions of a sigmoidal function (1989) — locator to be identified