Tokenisation

Definition

Tokenisation

Tokenisation is the process of converting an input string into an ordered sequence of tokens by identifying their boundaries. Tokens may be words, parts of words or individual symbols; their boundaries need not coincide with spaces.

Issues

Token boundaries depend on the language and the task. Spaces and punctuation alone do not determine which units are useful.

• Apostrophes: should Finland’s remain one token, become Finland and s, or lose the apostrophe to become Finlands? Removing punctuation also changes the token’s form, rather than merely choosing boundaries.
• Hyphens: should Hewlett-Packard remain one token or split into Hewlett and Packard? Similar questions arise for state-of-the-art and co-education. Variant spellings such as lowercase, lower-case and lower case may otherwise produce different token sequences.
• Multiword expressions: should San Francisco be one token or two? Treating it as one requires recognising that the two words form a name, rather than splitting at every space.
• Numbers and identifiers: meaningful units can mix digits, letters, punctuation and spaces.
  • Dates have different formats, such as 3/20/91, 20/3/91 and Mar. 12, 1991. Recognising their boundaries is distinct from interpreting or normalising them.
  • Expressions such as 55 B.C., B-52, a key identifier like 324a3df234cb23e, or the phone number (800) 234-2333 should not necessarily be split at every punctuation mark or space.
  • Omitting numbers from an index loses useful search targets, such as error codes. Structured values, such as a document’s creation date, may also be indexed separately as metadata.
• Language-specific boundaries: useful segmentation rules differ between languages.
  • In French, should l’ensemble remain one token or split into an article and ensemble? The article could be represented as l’, l, or its expanded form le. Separating ensemble allows it to match the same word in un ensemble; expanding the article additionally changes its form.
  • German compounds such as Lebensversicherungsgesellschaftsangestellter (‘life insurance company employee’) contain several meaningful components without spaces. A compound splitter can expose these components for retrieval, allowing a query for Versicherung to match rather than requiring the whole compound.
  • Chinese and Japanese generally do not separate words with spaces. A tokeniser must infer boundaries, and several segmentations may be plausible for the same input.
  • Japanese can mix hiragana, katakana, kanji, Latin letters and digits within one sentence. A change of script does not necessarily mark a word boundary, and dates or monetary amounts may appear in different formats. A query written entirely in hiragana may need to match words written in other scripts, requiring reading-based matching or normalisation beyond boundary detection.
  • Arabic and Hebrew are written mainly from right to left, but numbers are written from left to right. Unicode stores characters in logical order; bidirectional rendering determines their visual placement. Tokenisation should follow the logical sequence, not the displayed order.
  • In Arabic, letter shapes depend on their position and neighbouring letters, and several characters can form one ligature. A displayed glyph is therefore not necessarily one character or token. These visual joins do not remove the spaces separating words.
    ]