The difference between tokens and words
SMRTR summary
Language models use tokens instead of words for processing. Tokens roughly correspond to words but are more flexible, allowing for capitalization, punctuation, and multi-token words. Common words like "Hello" are single tokens, while uncommon words like "flugelhorn" are broken into multiple tokens.
SMRTR provides this summary for quick context. The original article belongs to John D. Cook.
Read the original article