On November 8 an overnight run on my local Qwen3.8 27B GGUF finished, and before looking at the score I checked one number per task: how many prompt tokens the server reported beyond the tokens of the message body. The pre-registration said that number should be about 14 on all 10 tasks. It was 52 on all 10. By the rule I had written two days earlier, the run was invalid and got no verdict.
The variable I was testing was a prompt change made through the Modelfile TEMPLATE. The template change was real: the new model had a different template blob. The prompt the server built did not change. This note is about how that was caught, what the cause appears to be, and what I am doing differently for the retry, which has not run yet.
The experiment sits inside a small benchmark I run on this Mac with Ollama: the same 10 build-fix tasks, the same decoding (temperature 0.6, top_p 0.95, max_tokens 12,288, fixed seed), and a pass threshold fixed in advance. Applicable patches at 5/10 or more support the hypothesis, under 5/10 rejects it, and no adjustment after the fact.
In the baseline run, 8/10 tasks ended with finish_length: the model ran out of budget before producing an edit. The next step, pre-registered on November 6, had two parts. First, save the raw reasoning text for the two shortest failing tasks and classify it. Second, pick exactly one variable from that classification and rerun the same 10 tasks overnight.
The classifier is a function called classify_reasoning with fixed thresholds. In the last 25% of the reasoning (minimum 2,000 characters), if any line of 40+ characters repeats 5 or more times, or the zlib compression ratio is under 0.15, it is loop. Otherwise, if the reasoning contains a <<<<<<< SEARCH marker, it is drafting. Otherwise progress. The rule for the variable: any loop means try a presence penalty of 1.5; anything else means try no_xhigh.
no_xhigh meantThe GGUF ships a chat template that inserts an extra paragraph into the system prompt about reasoning effort, at the xhigh level. Before the run I confirmed that this paragraph really was in the baseline prompts. On all 10 baseline tasks, usage.prompt_tokens minus the message body tokens came out to 52. Tokenizing the first task with candidate renderings and comparing: renderings that included the xhigh paragraph were off by −1 to +3, renderings without it by −35 to −39. The paragraph is 38 tokens. Whether it made the model think longer, I did not know.
no_xhigh was the plan to remove exactly that paragraph and nothing else. I built it as a second Ollama model: same weights blob, same stop tokens, same context size, and a Modelfile TEMPLATE equal to the baseline template minus the paragraph. The run script checked that the new template was byte-identical to "baseline template minus paragraph" before starting.
On the same day I ran a control on a different model family, qwen3:4b. Applying the same template there, the prompt token count stayed the same as the original model's. My note at the time was that Ollama seems to treat templates differently per model family, so the no_xhigh run needed its own check that the paragraph was really gone.
That check went into the pre-registration as a validity condition:
usage.prompt_tokens minus body tokens ≈ 14 (the baseline's 52 minus 38).It is a cheap check. The server already returns prompt_tokens, and the body token count comes from the same tokenizer. It also checks the thing the experiment is about, the prompt the model actually saw, and not the configuration I believed I had sent.
The first attempt started at 00:49:02 on November 7 and stopped itself at 00:51. The start guard passed with 22GB free on the internal disk, but loading the 27B model pushed swap from 8 to 13.4GB and free space fell to 16GB, below the 20 GB safety line, so the script aborted. Zero generations were written. I added disk re-measurement after model load and before phase two, and kept the configuration, tasks and thresholds unchanged.
The second attempt ran from 01:42:16 to 10:14:42 on November 8. The diagnosis part was valid:
| task | class | reasoning chars | tail chars | max same-line repeats | tail zlib ratio | finish | completion tokens | wall s |
|---|---|---|---|---|---|---|---|---|
| A | progress | 46,417 | 11,604 | 3 | 0.347 | length | 12,288 | 2,270 |
| B | progress | 45,405 | 11,351 | 3 | 0.302 | length | 12,288 | 2,305 |
Both were far from the loop thresholds and had no SEARCH markers. The reasoning was not repeating; it kept going and was cut off at the 12,288 budget. So the rule picked no_xhigh.
The experiment part failed validity. Prompt overhead was 52 on 10/10 tasks, the same as the baseline.
Same seed, same decoding, and the outputs were byte-identical to the baseline on 10/10 tasks. The two tasks that passed even had the same completion token counts, 7,965 and 859. That is independent evidence that the server received the same prompt as in the baseline. It also means the numbers the run produced (applicable 2/10, pass@1 2/10, finish_length:edit_no_blocks 8/10) are a replay of the baseline and say nothing about no_xhigh.
If I had looked only at the score, I would have seen "no change from removing the paragraph" and could easily have written that the effort paragraph does not matter. The token check said the variable was never applied.
The model registration was different. The new model used the same weights blob as the baseline, but its template blob was 201 B against the baseline's 410 B. My script compared that registered template against the expected bytes, and it passed, because the registered template was correct. What it did not check was whether the server used that template to build the prompt. The token count on the actual requests is what caught it.
The qwen3:4b control two days earlier, where changing the template left the prompt tokens unchanged, was an early sign of the same thing. I recorded it and used it to justify the validity check, but I did not follow it further at the time.
From reading the static files: the jinja tokenizer.chat_template embedded in the GGUF inserts the paragraph into the system message through reasoning_effort|default('xhigh'). With medium there is no paragraph; with low there is a shorter one.
Removing the paragraph from the Modelfile TEMPLATE did not change the server's prompt. So Ollama 0.32.14 renders this model's /v1 requests (its model_family is qwen35) through some path other than the Modelfile TEMPLATE. My guess is that it uses the embedded jinja template. I did not confirm which component does the rendering.
If that is right, editing the template is the wrong lever. The effort level has to change on the request side: either chat_template_kwargs: {"reasoning_effort": "medium"} or the OpenAI-compatible reasoning_effort field. Whether Ollama 0.32 forwarded either one to the template was also unknown, so the next step was a probe and not another overnight run.
At 10:41 on November 8 I sent the same short message several ways and read the server's prompt_tokens each time:
| request | prompt_tokens |
|---|---|
| baseline, no effort field | 61 |
/v1, top-level reasoning_effort: medium | 23 |
/v1, top-level reasoning_effort: low | 49 |
chat_template_kwargs with reasoning_effort | 61 |
Medium dropped 38 tokens: the 37-token paragraph plus 1 separator. Low dropped 12, recorded as 37 minus 25, which fits the shorter paragraph that low inserts. chat_template_kwargs changed nothing, so on this setup it is ignored. The native /api/chat with think behaved the same as the top-level field.
So the working lever is the top-level reasoning_effort on the /v1 request. Adding reasoning_effort: medium there is a valid implementation of no_xhigh, the same variable the rule picked, not a new one.
The retry runs the same 10 tasks with the same model, blob, template and parameters as the baseline. The only difference is RunConfig.reasoning_effort = "medium", which enters the config hash and the request only when set, so the baseline's hash does not move. Because invalid runs stay in the generation cache and a rerun with the same config hash would read only the cache, the retry had to be a new configuration anyway.
Two checks are built in this time:
The thresholds stay where they were: 5/10 applicable patches to support, under 5/10 to reject. If it is rejected and finish_length is still the majority, the write-up stops at "removing the effort paragraph did not get the reasoning inside the budget", and the next candidates (think: false, low) get their own pre-registration.
For any experiment that changes the prompt, I now confirm the change with the server's prompt_tokens on the real requests. A check on the configuration I registered can pass while the prompt the model sees stays exactly the same, and that is what happened here.
What is still open: as of November 12 the retry has not run, so I have no result on whether removing the paragraph changes anything. I have not confirmed which component inside Ollama renders this model's prompt, only that the Modelfile TEMPLATE was not used on this path. And the qwen3:4b control suggests this is not limited to one model, but I have checked it on only those two.
The pre-registration required usage.prompt_tokens minus body tokens to be about 14 on all 10 tasks (the baseline's 52 minus the 38-token paragraph). The run measured 52 on 10/10, so it was declared invalid and given no verdict.
Yes. With the same seed and decoding, outputs were byte-identical to the baseline on 10/10 tasks, including completion token counts of 7,965 and 859 on the two passing tasks. That is independent evidence the server received the same prompt as the baseline.
The script compared the registered template against the expected bytes, and the registered template was correct: its blob was 201 B against the baseline's 410 B. It did not check whether the server used that template to build the prompt.
The GGUF's embedded jinja chat template inserts the paragraph through reasoning_effort|default('xhigh'). Ollama 0.32.14 renders this model's /v1 requests through a path other than the Modelfile TEMPLATE, presumably the embedded jinja template; which component does the rendering was not confirmed.
A probe with a short message measured prompt_tokens 61 at baseline, 23 with top-level /v1 reasoning_effort medium, 49 with low, and 61 with chat_template_kwargs, which was ignored. The retry uses top-level reasoning_effort medium and had not run as of November 12.