Definition
Normalisation Function (Information Retrieval)
Query and Document Consistency
Apply the same normalisation convention to indexed text and query words, so intended variants map to the same term. Word tokens with the same output share a normalised word type.
Choose the convention according to how users are likely to write queries, including omitted accents or alternative spellings.
Language Dependence
Both tokenisation and normalisation may require language-specific rules. Language detection can help select the appropriate rules, including how to interpret dates or match different writing systems.
Asymmetric Expansion
An alternative is asymmetric query expansion, which searches a query-dependent set of forms rather than matching only by equality of normalised forms.
Semantic and Phonetic Grouping
A thesaurus can supply equivalences such as car and automobile. Soundex instead groups names using phonetic heuristics; a shared code does not imply shared meaning.
Examples
Each example uses a different normalisation function
Definition
Link to originalCase Folding
Case folding maps strings to common forms for case-insensitive matching. Under a case-folding function
, strings match without regard to letter case when
Removing periods
Deleting periods makes punctuated and unpunctuated forms match:
Removing hyphens
Deleting hyphens joins the parts rather than splitting them into separate tokens:
French accents
Removing accents allows an unaccented query to match an accented word:
German umlauts
Matching Tübingen may require recognising both the alternative spelling Tuebingen and the accent-free form Tubingen:
Removing the umlaut alone does not handle the extra e in Tuebingen.
Date formats
When 7/30 is interpreted as month/day, it denotes the same date as 7月30日 (‘30 July’). A chosen month–day representation can make them match:
Japanese kana and kanji
A language-aware rule can match kana and kanji spellings of the same word. For the Japanese word meaning ‘cat’, a reading-based rule can give