Transformer
- Definition
- Transformers use attention mechanisms to let every part of the input interact directly with every other part, enabling parallel computation and effective handling of long-range dependencies. Imagine each token in a sequence asking "how much should I listen to each other token?" Transformers differ from RNNs by avoiding recurrence and from CNNs by using attention instead of local filters; this design underpins modern large language models.
How does it work?
Transformers compute self-attention between all positions so each token builds a context-aware representation; stacking attention and feed-forward blocks and training with large batches leads to strong sequence models. Implementation focuses on attention masks, positional encoding, and efficient batched matrix operations.
Examples
- Large language models — Pretrain transformer decoders/encoder-decoder stacks for text generation and understanding.
- Machine translation — State-of-the-art encoder-decoder transformers for high-quality translation.
- Protein folding embeddings — Apply transformer architectures to model amino-acid sequences and structural properties.
Problems
- Quadratic compute/memory scaling with sequence length
- Requires large datasets and compute to train from scratch
- Positional information must be added explicitly (no inherent order sense)
- Prone to hallucination and biases learned from training data
- High inference cost/latency for large models