Tokenization in NLP: Words, Subwords, and Tradeoffs

Tokenization converts text or bytes into units a model can index. A token may be a word, character, byte, punctuation mark, or learned subword. Tokens are engineering units—not guaranteed words, morphemes, concepts, or meanings.

Main approaches

Approach	Strength	Tradeoff
Word	Readable units for some languages	Large vocabulary and poor handling of unseen forms or languages without spaces
Character	Small vocabulary and open vocabulary	Long sequences; Unicode characters do not always equal user-perceived characters
Byte	Can represent arbitrary encoded text	Longer sequences and units may be hard to interpret
Subword	Balances vocabulary size and sequence length	Splits depend on corpus, algorithm, normalization, and vocabulary size

BPE, WordPiece, and SentencePiece

Byte-pair encoding iteratively learns frequent merges under a defined implementation. WordPiece uses a likelihood-inspired vocabulary-selection procedure in its common NLP form. SentencePiece is a language-independent tokenizer trainer that can implement unigram or BPE models directly from raw text. These names do not describe one identical algorithm.

Normalization changes inputs

Case folding, Unicode normalization, whitespace handling, accent removal, and pre-tokenization can merge distinctions or change offsets. Record the tokenizer artifact, vocabulary, normalization, special tokens, library version, and model revision. Never substitute a tokenizer merely because vocabulary size matches.

Evaluate across real text

Measure sequence length, unknown/fallback behavior, truncation, round-trip behavior where expected, and splits across languages, scripts, dialects, emoji, code, identifiers, numbers, names, and accessibility text. Token counts affect context limits and cost, but model providers can count differently.

See NLP applications, NLP and LLM relationships, evaluation metrics, and neural-network basics.

Security and privacy

Tokenization does not anonymize text. Tokens can preserve personal data, secrets, or copyrighted material. Validate offsets before redaction, because normalization and splitting can break naive character mappings. Treat tokenizer inputs and logs under the same data controls as the original text.

Reviewed and substantially updated September 4, 2026. Original publication date preserved.