Tokeniser

Tokeniser: The component that splits text into tokens for a model to process, and reassembles tokens back into readable text on output.

Browse the full glossary →