Tokenised dataset
AITokenised dataset: A dataset converted into sequences of tokens, the numerical units a language model reads, prepared in advance so training runs efficiently.
Tokenised dataset: A dataset converted into sequences of tokens, the numerical units a language model reads, prepared in advance so training runs efficiently.