EXPLAINER

Speech AI: Speech Recognition and Text-to-Speech

conceptfreshness: evolvingsources not yet verified

Speech AI turns audio into text (automatic speech recognition) and text into audio (text-to-speech); modern systems use large neural models and can power voice assistants.

Short answer: ASR converts speech to text; TTS converts text to speech; voice assistants chain ASR → LLM → TTS or use end-to-end speech models. Accuracy varies with accent, noise and domain terms; voice cloning raises consent and fraud concerns.

What is it?

Speech AI covers two directions: listening (turning spoken words into text) and talking (turning text into natural-sounding speech). Put them around a language model and you get a voice assistant.

Also known as: ASR, speech to text, text to speech, TTS, Whisper, voice assistant, speech recognition, voice cloning.

Technical definition

ASR models such as Whisper (Radford et al., 2022; large-scale weak supervision) and wav2vec 2.0 (Baevski et al., 2020, self-supervised) map audio to text, often with timestamps and optional diarisation by separate components. TTS uses neural vocoders and sequence models; some systems clone voices from short samples. Voice agents trade latency (streaming, endpointing, interruption handling) against quality. Evaluation uses word error rate (WER) on representative audio.

Entity type: concept · Freshness class: evolving.

When should I use it?

Use for transcription, captions, accessibility, voice interfaces.

Implementation patterns

  • Streaming ASR with endpoint detection feeding an LLM, then streaming TTS.
  • Custom vocabulary lists for domain terms.
  • Diarisation for meeting transcripts.
  • Consent capture and disclosure for synthetic voices.

Technologies and frameworks

Commonly associated: Whisper, wav2vec 2.0, streaming ASR services, neural TTS engines. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Collect representative audio.
  2. Measure WER per condition.
  3. Pick streaming vs batch.
  4. Add consent and disclosure.
  5. Protect recordings (AI Privacy and Security: What to Know Before You Paste Data).

Common failure modes

  • High WER on noisy audio, accents or jargon.
  • Latency making conversation feel unnatural.
  • Hallucinated text in silent segments.
  • Voice-clone fraud and consent violations.
  • Recording-consent law non-compliance.

Troubleshooting

  • Domain terms mistranscribed — Add vocabulary biasing or post-correct with a glossary.
  • Laggy voice agent — Stream each stage, tune endpointing, shorten responses.

Contrasted with

Questions this page answers

  • What is speech recognition?
  • How does Whisper work?
  • How do I build a voice assistant with an LLM?
  • What is word error rate?
  • Is voice cloning legal?

Typical goals: transcribe audio; build a voice assistant; assess voice cloning risk.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper) (2022) — arXiv:2212.04356
  • Baevski et al., wav2vec 2.0 (2020) — arXiv:2006.11477