A Specialized Vocabulary Can Cut Inference Cost. Keep the Decision Tokens.

An essay grader has two different jobs when it returns a JSON assessment. It must choose a grade from the essay's content, and it must write that grade inside a fixed structure. The braces, field names and repeated evaluation phrases help the application read the answer. The grade itself is the answer the application needs.

A vocabulary optimization can blur that distinction. If a single token represents both a field's structure and its grade, the model chooses the packaging and the judgment in one step. That saves generation steps. It also changes what the model has to learn in order to express the decision.

Dharma-AI ran into this while specializing the vocabulary of its production grader for Portuguese essays, built on Qwen3-4B. Its first compressed vocabulary folded each grade into a larger token and lowered the grader's task score. Separating the grades again added two tokens per response, yet the separated version scored higher and also ran cheaper in the team's measurement. The team's October 21 VocabX report makes a useful engineering case: compress the repeated language around a decision, then check whether the representation of the decision still works.

The tokenizer changes what one generation step can say

A tokenizer maps text to IDs from a finite vocabulary. The model reads its input and predicts its output as those IDs. One ID might stand for a character, part of a word, a whole word or a longer span. Once the IDs are decoded back into text, the application cannot see which design produced it, as long as the text is correct.

The difference stays visible to the model. Consider the response fragment {"grade": 4}. One vocabulary could give the fixed prefix its own token and leave the value as a separate token. Another could represent the whole fragment with one token, with separate entries for {"grade": 3} and {"grade": 5}. These are schematic examples, not the actual VocabX tokens or the grader's real scale.

In the first design, the model emits the prefix and then selects a value. In the second, selecting one token commits to both the structure and the value. The same visible JSON can therefore come from two different prediction problems. A response that parses correctly tells you nothing about whether the model has become better or worse at choosing the grade.

This is also why vocabulary specialization is a different kind of change from speeding up a tokenizer implementation. A faster implementation encodes the same text into exactly the same IDs. A new vocabulary changes the IDs and the prediction units the model must learn. It belongs with model adaptation and task evaluation, not with a faster text-processing library.

Why repeated output is an attractive target

The grader's responses contain repeated JSON structure and recurring assessment language. A general multilingual vocabulary, built for many languages and text types, splits those spans into many pieces. A specialized vocabulary can assign useful combinations their own tokens, so the model produces the same text in fewer steps.

Input and output savings are not interchangeable, because the model handles the two differently. It reads the prompt in a prefill phase, processing all of the prompt's known tokens together in one pass. It writes the answer by ordinary autoregressive decoding, where each new token depends on the output so far, so the server runs the model again for every token it adds. Those repeated steps make generation a different serving workload from reading the prompt; the cost depends on batching, context length and the hardware. A token count alone does not capture that workload, which is why the VocabX authors frame the problem around cost rather than count.

VocabX weights each candidate token by the measured cost of the field it appears in: system prompt, user message or model response. That replaces ranking candidates only by frequency across all text. It changes subword segmentation for domain words and also creates tokens for longer repeated spans, which the authors call macro tokens. This lets one criterion address both fragmented domain terms and repeated output structures such as the JSON envelope.

There is a second constraint: the number of choices available at each step. The model's output projection, usually called the language-model head, turns its internal state into a score for every vocabulary entry. Adding a token adds another row to score at every generated token. Shortening the sequence by expanding the vocabulary can therefore reduce one source of work while increasing another.

VocabX controls this by replacing the base vocabulary with a smaller domain-specific one and admitting macro tokens under a fixed budget. In the grader, the authors report the output layer shrank from about 152,000 rows to about 65,000. Because sequence length and vocabulary size pull on runtime in opposite directions, the adapted model has to be measured on the intended server. Neither the smallest vocabulary nor the shortest response is, by itself, the runtime optimum.

Two extra tokens exposed the wrong optimization target

The authors trained seven grader variants with the same base model, data, training recipe and seed. Only the tokenizer differed. Each variant served the same 500 requests through vLLM on an RTX 5090, and the reported cost was calculated from measured runtime.

The team's bench score combines grading accuracy with accuracy at screening off-topic or rule-breaking essays. The production grader, with its original vocabulary, scored 0.703. The grader returns a grade for each of several competencies, and the first specialized version fused each competency's grade with its surrounding JSON into one macro token. That version scored 0.689. Giving each grade its own token produced a score of 0.810. The revised model averaged 101 output tokens against the baseline's 350, and the runtime-based cost reduction rose from 73.3% for the fused version to 77.0% for the separated one. These are the team's own figures from a research preview of a paper still under review, and the authors say the numbers may change before publication.

The result does not establish that any label embedded in a macro token will fail. It establishes that token reduction was an insufficient selection criterion in this experiment. The most compact representation lost task quality, and keeping a small amount of extra structure did not remove the cost benefit.

The explanation also has a limit. Separating a value creates a distinct prediction step, but the preview does not isolate every reason the separated model scored higher or ran cheaper. The measurements support choosing the separated version in that test. They do not show that the two added tokens caused the runtime improvement on their own.

For another structured-output task, a sensible candidate design would keep explicit choices for the values the application acts on, such as a classification label or a routing destination, while compressing the stable text around them. That is a design hypothesis to evaluate, not a rule that separate tokens guarantee correct decisions. The comparison has to examine those values, not just whether the output parses.

The adaptation has a cost before it serves a request

A specialized tokenizer cannot simply be placed in front of an unchanged model. New tokens need representations that agree with the new mapping, and the resulting model then needs training and task evaluation.

Earlier research makes that work concrete. In Fast Vocabulary Transfer for Language Model Compression, Leonidas Gee and colleagues initialized each new token's embedding, the numeric representation the model stores for it, by averaging the embeddings of the pieces the original tokenizer would have split it into. Their BERT experiments then ran in-domain masked-language-model training before fine-tuning on the downstream task. They found that performance did not fall steadily as the vocabulary shrank, and that on their news task the smallest vocabulary removed the inference speed benefit entirely. They treat vocabulary size as a hyperparameter to tune against the application's accuracy, size and latency requirements. That paper studies a BERT encoder on sentence classification, document classification and named-entity recognition. It supports treating vocabulary size and adaptation as choices to test; it does not validate VocabX's generative results.

For a serving team, the economic question therefore covers the life of the specialized model. A stable language, a fixed response format and sustained request volume give a vocabulary more opportunity to repay its preparation. Frequent changes to the output schema or the supported languages create more occasions to rebuild and retest it. A base-model upgrade also requires checking that the tokenizer, token representations and trained weights still form a compatible release.

The VocabX preview does not provide a full accounting of those training and maintenance costs. Its 77% result should inform an experiment on inference runtime, not be entered as a 77% cut to the whole service budget. The relevant comparison sets the time needed to prepare and maintain the specialized version against the runtime saved over the requests it actually serves.

A team evaluating this approach can keep the experiment narrow: use the real response format, hold out an evaluation set untouched by vocabulary mining, and compare task outcomes and measured serving cost across candidate vocabularies on the production hardware. Include uncommon but supported values and formats, because the frequent spans used to build a vocabulary do not define everything the service must handle.

The goal is a cheaper model that still makes the required decisions. VocabX's most instructive result is that a few tokens carrying those decisions were worth keeping. In the reported grader, the vocabulary that kept the grades separate was the one that both scored higher and measured cheaper, although the preview does not apportion the saving between the compressed structure and the separated values. The practical rule that follows is to select a candidate vocabulary on two measurements together: task quality on the values the application acts on, and runtime on the server that will serve it.