What is it?
A modality is a type of data: text, images, audio, video, sensor readings, tables. Early AI systems were unimodal — one model for text, a different one for images. A multimodal model handles several modalities together, so one request can mix them: "Here is a screenshot of an error; explain it", or "Describe this chart and extract the numbers as a table".
Multimodal systems may accept multiple input types, produce multiple output types, or both. What a specific model supports varies — check its documentation instead of assuming.
Why does it matter?
Much real information is not plain text: invoices, whiteboards, slides, diagrams, handwritten notes, screenshots, voice recordings. Multimodal models let you work with these directly, which enables accessibility features (describing images), document processing, visual debugging, and voice interfaces.
How does it work?
Different media are converted into a common internal representation so the same network can relate them:
- Each modality has an encoder that turns raw input (image patches, audio frames, text tokens) into vectors (What are Embeddings in AI?).
- Those vectors are placed in a shared space, so the model can connect the word "dog" with images of dogs.
- A transformer processes the combined sequence, and a decoder produces text (or another modality) as output.
Training typically uses paired data such as images with captions. Some systems are built as a single model trained on all modalities; others chain specialised components (for instance speech recognition → language model → speech synthesis). Both approaches exist.
Example
Workflow: you photograph a receipt and ask a multimodal model to return the merchant, date and total as JSON. The model reads the image, extracts the text, and structures it. You still need to validate the result — misread digits are a classic failure — for instance by checking that line items sum to the total.
When should I use it?
Use it when the information you need is visual or audible and manual transcription would be slow: document and form extraction, describing images, analysing charts or screenshots, voice assistants. Use traditional tools when you need exact, auditable extraction (dedicated OCR with validation) or when the images contain sensitive material you cannot share with a hosted service.
Common mistakes
- Assuming perfect visual accuracy. Models can misread small text, dense charts and handwriting, and can describe things that are not in the image.
- Forgetting privacy. Images often contain faces, addresses, badges or screen contents. See AI Privacy and Security: What to Know Before You Paste Data.
- Assuming every model does every modality. Verify input and output support.
- Not validating extracted numbers.