SentencePiece

SentencePiece: An open-source tokenisation toolkit that treats text as a raw stream, including spaces, so it works across languages without needing pre-split words.

Browse the full glossary →