Subword tokenisation
AISubword tokenisation: Splitting text into pieces smaller than words, so rare or unseen words are represented by known fragments rather than treated as unknown.
Subword tokenisation: Splitting text into pieces smaller than words, so rare or unseen words are represented by known fragments rather than treated as unknown.