Function: registerCoreTokenizers()

registerCoreTokenizers(registry, options?): () => void

Defined in: provider-sdk/src/tokenizer.ts:63

Register exact, offline Tiktoken profiles for known OpenAI model families plus the DeepSeek family tokenizer. Unknown and local model names deliberately continue to use the explicit approximate fallback unless a provider plugin supplies their tokenizer.

options.tokenizerCacheDir / options.fetchImpl override where the DeepSeek tokenizer files are cached and how they are fetched (tests); both default to the data-directory convention from config.ts.

Parameters

registry

TokenizerRegistry

options?

TokenizerModelOptions

Returns

() => void