What is it?
Learn by doing: an agent tries actions, receives rewards or penalties, and gradually learns which actions lead to the best long-term outcome — like training a dog with treats, or a program learning chess by playing itself.
Also known as: RL, reinforcement learning basics, policy, reward function, agent environment reward, exploration vs exploitation, deep reinforcement learning, deep RL.
Technical definition
Formalised as a Markov decision process (Markov Decision Processes): states, actions, transition dynamics, rewards and discount factor. Goals: learn a policy π(a|s) maximising expected return. Families: value-based (Q-learning, Deep Q-Networks (DQN) and Q-Learning), policy-gradient and actor-critic (Proximal Policy Optimization (PPO)), model-based RL (World Models), offline RL, imitation/inverse RL. Key difficulties: exploration versus exploitation, credit assignment over long horizons, sample efficiency, reward specification (Reward Hacking and Over-Optimization) and stability. Methods that optimise directly on preference pairs without a sampled-reward RL loop, such as Direct Preference Optimization (DPO), belong to Preference Optimization (Aligning Models from Human or AI Preferences) rather than ordinary RL.
Entity type: concept · Freshness class: stable.
When should I use it?
Use when decisions are sequential and feedback is delayed and a simulator or safe trial-and-error exists.
Implementation patterns
- Simulate first: train in simulation before real systems.
- Reward shaping with care and audits.
- Use proven algorithms (PPO, SAC, DQN) before inventing new ones.
- Evaluate across many seeds.
Technologies and frameworks
Commonly associated: Gymnasium, Stable-Baselines3, RLlib (Ray), TRL. Names are examples of what exists, not recommendations, and projects change quickly.
Practical guide
- Define states, actions and reward precisely.
- Build or pick a simulator.
- Start with a standard algorithm.
- Log returns over many seeds.
- Test on perturbed environments before deployment.
Common failure modes
- Reward hacking and unintended behaviour.
- High variance across random seeds.
- Poor sample efficiency making real-world training impractical.
- Training/evaluation environment mismatch.
Troubleshooting
- Agent finds a loophole — Audit the reward; add constraints and held-out evaluation.
- Learning unstable — Use established implementations, tune learning rate/advantage normalisation, check seeds.
Contrasted with
- Supervised Learning — Supervised learns from labelled answers; RL learns from delayed reward after actions.
- Direct Preference Optimization (DPO) — DPO optimises a policy on preference pairs with a supervised-style loss; it is preference optimisation, not an RL sampling loop.
- Imitation Learning (Behaviour Cloning and DAgger) — Imitation learns from demonstrations rather than rewards.
Questions this page answers
- What is reinforcement learning?
- How is RL different from supervised learning?
- What is exploration vs exploitation?
- How does reinforcement learning differ from preference optimization?
Typical goals: understand RL basics; distinguish RL from supervised learning and from preference optimisation.
Sources and verification
Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.
Freshness class: stable — Core ideas that change slowly (maths, classic algorithms). Re-review yearly.
Canonical sources (unverified leads):
- Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — incompleteideas.net/book
- Mnih et al., Human-level control through deep reinforcement learning, Nature (2015) — doi:10.1038/nature14236