Zum Hauptinhalt springen

Class: TokenizerRegistry

Defined in: provider-sdk/src/tokenizer.ts:103

Local model-to-tokenizer registry.

Provider plugins can register Tiktoken, SentencePiece, Hugging Face JSON or model-specific profiles. Registration returns cleanup and the registry always has an explicit approximate fallback.

Constructors​

Constructor​

new TokenizerRegistry(): TokenizerRegistry

Returns​

TokenizerRegistry

Methods​

profilesList()​

profilesList(): string[]

Defined in: provider-sdk/src/tokenizer.ts:168

Returns​

string[]

register()​

register(profile): () => void

Defined in: provider-sdk/src/tokenizer.ts:106

Parameters​

profile​

TokenizerProfile

Returns​

() => void

resolve()​

resolve(model): Promise<ResolvedTokenizer>

Defined in: provider-sdk/src/tokenizer.ts:118

Parameters​

model​

string

Returns​

Promise<ResolvedTokenizer>