Skip to main content

Interface: TokenizerProfile

Defined in: provider-sdk/src/tokenizer.ts:32

Local tokenizer profile registered by core or a process-isolated provider plugin.

Properties​

approximate​

readonly approximate: boolean

Defined in: provider-sdk/src/tokenizer.ts:38

False only for a deterministic exact tokenizer implementation.

id​

readonly id: string

Defined in: provider-sdk/src/tokenizer.ts:34

Stable diagnostic id, for example openai:o200k_base.

priority?​

readonly optional priority?: number

Defined in: provider-sdk/src/tokenizer.ts:36

Higher-priority profiles win when multiple profiles match a model.

Methods​

count()​

count(text): number | Promise<number>

Defined in: provider-sdk/src/tokenizer.ts:40

Parameters​

text​

string

Returns​

number | Promise<number>

matches()​

matches(model): boolean | Promise<boolean>

Defined in: provider-sdk/src/tokenizer.ts:39

Parameters​

model​

string

Returns​

boolean | Promise<boolean>