Lukas' Notes

Search titles, aliases and note text. Use # for tags.

Tokeniser

1 min read

Definition

Tokeniser

A tokeniser is an algorithm that performs tokenisation, using specified rules or a learnt model to determine token boundaries.

Backlinks