Long Short-Term Memory
- Definition
- LSTM networks are a type of RNN designed to remember information over longer sequences using gated mechanisms that control information flow. Think of gates that decide what to keep, forget, or output, so the model can learn long-term dependencies. LSTMs differ from vanilla RNNs by addressing vanishing gradients and from transformers by being sequential rather than relying on global attention.
How does it work?
Long Short-Term Memory architectures are specialised neural designs; they define layer types and connectivity that bias the model to particular data structures. Implementations balance depth, width, and compute considerations for practical training.
Examples
- Speech synthesis — LSTM-based sequence models for early TTS pipelines.
- Machine translation (early systems) — Sequence-to-sequence LSTM encoder-decoders for translation tasks.
- Anomaly detection in sequences — Model normal sequential patterns and flag deviations in operational logs.
Problems
- More parameters and compute cost than simple RNNs
- Still struggles with extremely long sequences
- Sequential computation limits parallelization/training speed
- Many gates make hyperparameter tuning and debugging harder