Function: registerCoreTokenizers()
registerCoreTokenizers(
registry,options?): () =>void
Defined in: provider-sdk/src/tokenizer.ts:63
Register exact, offline Tiktoken profiles for known OpenAI model families plus the DeepSeek family tokenizer. Unknown and local model names deliberately continue to use the explicit approximate fallback unless a provider plugin supplies their tokenizer.
options.tokenizerCacheDir / options.fetchImpl override where the
DeepSeek tokenizer files are cached and how they are fetched (tests);
both default to the data-directory convention from config.ts.
Parameters
registry
options?
TokenizerModelOptions
Returns
() => void