tokenizer

TokenizerLike

Bases: Protocol

A protocol that all tokenizers should follow.

encode

encode(*args, **kwargs) -> list[int]

Tokenize the input text into a list of integers.

Returns:

• list[int] –

  Tokenized input.