DeepSeek V4.1 Flash on DGX Spark with Open WebUI.

Hardware

DGX Spark-class GB10 machines, one model per machine. The front end and proxy run on a separate small server and reach the Sparks over a private network (Tailscale in the measured deployment).

Serving

vLLM nightly (September–October 2026) with the DeepSeek V4.1 tokenizer, reasoning and tool-call parsers from the vLLM recipe. Speculative decoding with 3 draft tokens; about 65% of drafts accepted.

Observed throughput

About 22 output tokens per second per stream, with two sequences running at once. A long tool-using turn of 10,000–20,000 tokens therefore takes 8–15 minutes on one stream.

Proxy

LiteLLM Proxy v1.99.4, non-root image pinned by digest, config-file mode with no database. The loopguard callback is one Python file mounted read-only.

Front end

Open WebUI with native function calling and its built-in tools (notes, memory, chat search, knowledge bases).

When a turn ends in one large deliverable — for example, “write a full project scaffold into a note” — the model's thinking fills with commitments to act (“Let me write it.”, “Writing now.”, “Now write.”) followed by more planning. In the measured deployment, three of three replays of such a conversation looped: 23–29 commitment phrases and 5,000–11,000 reasoning tokens before the first tool call, and one replay was still thinking when stopped at 15 minutes.

Two features make the loop hard to catch. The text never repeats verbatim, so n-gram repetition detectors stay silent; and each commitment phrase is followed by new, plausible planning. In the recorded traces, 22–44% of the thinking text was code drafted inside the reasoning, against none in healthy thinking blocks from the same chat. The model also returns repeatedly to the length of the output it is about to write.

The model sometimes describes itself as Claude in these conversations. Other users report the same identity claim for the V4 Flash family; an identity line in the system prompt made no measurable difference to the loop here.

Replays

The conversation state just before a looping turn was exported from Open WebUI — messages, tool calls and tool results — and replayed directly against vLLM with each setting under test. A scorer counts commitment phrases, words of thinking after the first one, and code blocks inside the thinking. Short replays stop after about 1,200 tokens; full replays stop shortly after the first tool call.

One-token exit probes

To measure the loop without long generations, a recorded trace is cut at each commitment phrase and sent to /v1/completions with the chat template's own rendering of the prompt, asking for one token and its top log-probabilities. The probability of the end-of-thinking token at that point is the model's propensity to stop planning there. Each probe costs one to three single-token requests, and the result is independent of sampling.

At 21 of the 26 points the model was almost certain to keep planning; the most likely next step was a paragraph break or another “I'll…” sentence. Willingness to stop appeared only after about 2,400 words, once much of the deliverable had been drafted in the thinking. Probing the same trace under different effort levels and system prompts moved these values by no more than the probe's own run-to-run variation (±0.005).

Tokenizer trap. DeepSeek V4.1 encodes “.”, “.\n” and “.\n\n” as separate tokens, and the choice among them carries the decision: after a bare “.” the model ends its thinking with probability above 0.99, after “.\n\n” almost never. Cut traces just before the final punctuation and sum over the punctuation branches. A first version of this probe cut after the period, which silently selected the exit branch and reported exit probabilities near 1.0 at points the live run had passed.

Short replays: first ~1,200 tokens

The trailing reminder is the one setting that changed the mechanism. With both reminder arms pooled against the default, 13 of 16 replays skipped in-thinking code drafts against 5 of 16 (Fisher one-sided p = 0.006), and 6 of 16 started the tool call within about 600 words against 1 of 16 (p = 0.04). The same instruction at the top of the system prompt had no measurable effect. Effort 75 against 50 showed a weaker signal on code drafting (6 of 16 against 11 of 16, p ≈ 0.08) and no change in commitment phrases. Section-by-section tools changed nothing; in the first 600 words only two of eight traces mentioned them, because drafting starts before the model plans delivery.

The reminder raises the count of commitment phrases while the drafting drops, so that count stops being a useful loop measure once the reminder is on. Measure code blocks in the thinking and words before the first tool call instead.

Full replays

With the recommended settings (effort 75, temperature 1.0, top_p 0.95 and the trailing reminder), all six full replays reached the tool call, with a median of about 1,800 words of thinking against about 2,600 at defaults (Mann–Whitney one-sided p ≈ 0.05). The runs split into two groups: three acted after 350–1,500 words with little or no drafting, and three still drafted 7–17 code blocks before acting, in fewer words than at defaults.

Each item notes whether it was measured in this deployment or comes from other sources.

LiteLLM model entry

model_list:
  - model_name: deepseek-v4.1-flash
    litellm_params:
      model: hosted_vllm/deepseek-v4.1-flash
      api_base: http://SPARK_HOST:8000/v1
      api_key: "none"
      temperature: 1.0
      top_p: 0.95
      extra_body:
        chat_template_kwargs:
          reasoning_effort: 75

litellm_settings:
  callbacks: loopguard.proxy_handler_instance
  drop_params: false

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Container

docker run -d --name litellm --restart unless-stopped --network host \
  --read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges \
  --env-file /opt/litellm/litellm.env --env-file /opt/litellm/loopguard.env \
  -e LITELLM_LOCAL_MODEL_COST_MAP=True \
  -v /opt/litellm/config.yaml:/app/config.yaml:ro \
  -v /opt/litellm/loopguard.py:/app/loopguard.py:ro \
  -v /opt/litellm/data:/data \
  ghcr.io/berriai/litellm-non_root@sha256:63dfaf8232b7b524bc2ebca12bd77e997baec0179719c340443c74f14fd8822e \
  --config /app/config.yaml --host 127.0.0.1 --port 4000

The digest is litellm-non_root:v1.99.4 as of October 2026; re-resolve it from the registry before use. litellm.env holds only the master key (mode 600). loopguard.env sets the reminder text and the log path. Host networking with a localhost bind keeps the proxy reachable only from the front-end machine; Open WebUI then uses http://127.0.0.1:4000/v1 with the master key and the Default provider.

What loopguard does

• Caches each streamed response's reasoning, keyed by tool-call ID and a hash of the visible reply, and re-attaches it to the matching assistant message on later requests.
• Leaves out any reasoning block that crosses its loop thresholds (six or more commitment phrases, or 800 or more words after the first one).
• Appends the trailing reminder as a system message when LOOPGUARD_TAIL_REMINDER is set.
• Writes one JSON line of loop metrics per response for the configured models, and passes every chunk through unchanged.

Checks after install

1. Reasoning passthrough. Send the same tool-call conversation twice with max_tokens: 1, once with reasoning_content on the assistant turn and once without, directly to vLLM and through the proxy. The prompt-token difference must match (17 tokens in the measured setup).
2. Replay. Let the model make a real tool call through the proxy, then send the follow-up without reasoning; the proxied request should carry more prompt tokens than the direct one.
3. Defaults. Start a throwaway instance with --detailed_debug on another port, send one request, and confirm chat_template_kwargs, temperature and top_p in the logged upstream request. Remove the instance afterwards; its log contains prompt text.
4. Reminder. A one-word prompt costs more prompt tokens through the proxy than directly (31 against 71 here).

• One prompt family (a large code scaffold written into a note through a tool) and one deployment. Other deliverables and other serving stacks may behave differently.
• Small samples: 3 to 16 runs per setting. The p-values are one-sided and uncorrected for the number of settings tried.
• Answer quality was read in samples and was not scored.
• Exit probes on a recorded trace are off-policy: they measure the propensity to stop given that history, which is the quantity compared across settings.
• The thinking-off writing step, speculative decoding settings and the proxy's behavior on aborted streams were not tested.