Tokenizers documentation
Normalizers
Normalizers
TODO — coming soon. Reference documentation on this page is not generated yet: the rc0 bindings do not expose all of these APIs. It comes back with them, along with the other bindings. See
REQUIRED_FOR_V1.mdfor the full list.
Python
Rust
Node
ByteLevel
tokenizers.normalizers.ByteLevel Lowercase
tokenizers.normalizers.Lowercase NFC
tokenizers.normalizers.NFC NFD
tokenizers.normalizers.NFD NFKC
tokenizers.normalizers.NFKC NFKD
tokenizers.normalizers.NFKD Nmt
tokenizers.normalizers.Nmt Normalizer
tokenizers.normalizers.Normalizer Precompiled
tokenizers.normalizers.Precompiled Replace
tokenizers.normalizers.Replace Sequence
tokenizers.normalizers.Sequence Strip
tokenizers.normalizers.Strip StripAccents
tokenizers.normalizers.StripAccents BertNormalizer
tokenizers.normalizers.BertNormalizer