Recurrent Neural Network
- Definition
- Recurrent Neural Networks (RNNs) process sequences by passing information from one step to the next, allowing the network to use past inputs when predicting future ones. Imagine a loop that carries a hidden memory forward through time. RNNs differ from feedforward networks by their temporal dynamics; compared to transformers, basic RNNs struggle with long-range dependencies and training stability.
How does it work?
Recurrent Neural Network architectures are specialised neural designs; they define layer types and connectivity that bias the model to particular data structures. Implementations balance depth, width, and compute considerations for practical training.
Examples
- Language modelling (RNNs) — Sequence models that predict next tokens for early text generators.
- Time-series forecasting — RNNs for short-term prediction of sensor or financial series.
- Sequence labelling — Use RNNs for tagging tasks like named-entity recognition when data is sequential.
Problems
- Vanishing/exploding gradients over long sequences
- Difficult to parallelize training due to sequential dependency
- Struggles to capture very long-range dependencies
- Slower inference on long sequences compared to attention-based models