Tokenization
Byte Pair Encoding
Intuition: common sequences of bytes are represented by a single token, rare sequences are represented by many tokens.
不断合并最高频率的pair
Summary:
- Tokenizer: strings ↔ tokens (indices)
- Character-based, byte-based, word-based tokenization are highly suboptimal
- BPE is an effective heuristic that is data-driven
- Tokenization is a separate step, maybe one day do it end-to-end from bytes…