Tackling random foreign characters

By Chen-Chen Yeh, Member of Technical Staff at Mirai Labs

Sep 22, 2026

We randomly saw some Chinese and Russian characters in the middle of coherent English outputs, while benchmarking our new dflashtfm release. The benchmark runs 20 MT-Bench prompts through uzu at temperature 1 with no top-p or top-k, and generates tens of thousands of tokens in total.

What we saw

Fixed sampler

Qwen through uzu · MT-Bench prompt · temperature 1, no top-p, no top-k0 tokens · 0 forced

Troubleshooting

Since the benchmark was designed to test our new speculative decoding (specdec) implementation, that was the first suspect: new kernels, new numerics, maybe a precision issue. Our team developed a cool, efficient tree verification algorithm for gated deltanet (GDN) ¹, which stores the recurrent state in bf16 to leverage fast tensor cores for CUDA (Nvidia) and neural accelerators for Metal (Apple) respectively ² ³. In comparison, the sequential GDN implementation in uzu uses fp32 for state because state-space models are sensitive to numerical errors ⁴. We guessed that numerical drift could let foreign characters slip into the top-k candidates and get picked via sampling randomness. So we replaced the tree-verification algorithm with the precise sequential implementation. Foreign characters still popped out randomly in the output.

If precision wasn’t it, maybe specdec wasn’t involved at all. We had never run plain autoregressive generation through the new benchmark, so we tried that next. Surprisingly, generating text without specdec didn’t remove the foreign characters, which means the issue already existed long before our new release.

Another possible explanation is that this is an intrinsic property of Qwen models, similar to the doom loop observed in small Qwen/LFM models ⁵ ⁶. However, we did not observe the same artifacts in outputs from MLX or llama.cpp. These comparisons pointed toward uzu rather than an intrinsic model behavior.

That left the uzu sampler, the only stage involving randomness, which coincidentally matches the random Chinese/Russian characters pattern observed. It’s also easy to test, since greedy decoding skips the sampler entirely. Finally! Outputs have no foreign characters at all when testing greedily. This narrowed our investigation to the uzu sampler.

Four suspects, one test each

Suspect 1
Specdec numerics
bf16 state in the tree verification
Test
Swap the tree verification for the fp32 sequential path.
cleared
Characters still appear.
Suspect 2
Specdec at all
New kernels in the release
Test
Run plain autoregressive generation through the same benchmark.
cleared
Characters still appear. The bug predates the release.
Suspect 3
The model
A Qwen quirk, like the doom loop
Test
Same weights through MLX and llama.cpp.
cleared
Not a single foreign character. The model is fine.
Suspect 4
The sampler
The only stage with randomness
Test
Greedy decoding skips the sampler entirely.
culprit
No foreign characters at all.

The culprit: the sampler

uzu samples tokens with Gumbel max: it adds independent noise g = −ln(−ln u), u ~ Uniform(0, 1), to every logit in parallel and takes the argmax, which is provably equivalent to sampling from softmax ⁷, but needs no softmax or CDF.

G_i = −ln(−ln u_i), u_i ~ Uniform(0, 1)

(1)

token = argmax_i (ℓ_i + G_i)

(2)

Note that u must lie between 0 and 1, but our implementation didn’t. It could return exactly 0 and exactly 1, and produce noise with −∞ and +∞, which means always losing or winning, no matter the logit. This matches the pattern we observed: random nonsensical text in the output, which a precision issue shouldn’t cause, but a wrong random generator will.

Gumbel noise, and where it blows up

u0.500

u = 0.500 → g = 0.367 · added to the logit before the argmaxInside (0, 1) the noise is finite. The generator must never return the endpoints.

Figure 1. The Gumbel noise term is finite inside (0, 1) and diverges at both endpoints.

So where do 0 and 1 come from? Our random generator outputs a 32-bit integer, and we convert it to a float and divide by 2³².

inline float uniform_float(thread PhiloxState* state) {
  return float(philox_next(state)) * (1.0f / 4294967296.0f); /* (0,1) */
}

The largest integer is 2³² − 1, so this looks like it can never reach 1. But float can’t hold a 32-bit integer exactly. It only keeps 24 bits, so near the top of the range it has to round, and the largest integers round up to 2³² itself ⁸. Divide that by 2³² and u is exactly 1.

The last 512 integers before 2³²

word = 2³² −100

32-bit word

4,294,967,196

float(word)

4,294,967,296(rounded)

u = float(word) / 2³²

1.0 exactly→ noise is +∞

Figure 2. Near 2³², float cannot tell the top 128 integers apart from 2³² itself.

The numbers tell the same story as well! We draw one independent u_i for each vocabulary entry per generated token, so even a tiny chance per draw adds up:

P(bad token) = 1 − (1 − 2⁻²⁵)^V ≈ 4.5 × 10⁻³ for V = 151 936

(3)

Rate	Interval
Theoretical endpoint hit	1 every 221 tokens
Observed non-English word	1 every 430 tokens

Table 1. The script detector sees roughly half of the forced random tokens; English-looking random tokens are not flagged.

We expect one random token every 221 tokens, and the observed occurrence of non-English words is one every 430 tokens. Why not the same? The −∞ (u = 0) case is invisible, since it causes a candidate logit never to be selected, but the selected one could still make sense. The +∞ (u = 1) case is easier to catch: our detector flags unexpected scripts, but it misses random tokens that look like English. Thus, the observed counts of non-English words are roughly half of the probability of generating 1.

The fix

The fix is simple: keep only the top 24 bits of the integer, which float can hold exactly, so nothing rounds up to 1, and clamp the low end so u never hits 0 either.

fn unit_interval(word: u32) -> f32 {
    ((word >> 8).max(1)) as f32 * (1.0 / 16777216.0)
}

Keep 24 bits, clamp the bottom

32-bit word from the generator4,294,967,295

solid · top 24 bits, what a float can hold exactlydashed · low 8 bits, dropped

Before · all 32 bitsfloat(word) / 2³²

u = 1.0 exactly

float rounds this word up to 2³², so u = 1 and the noise is +∞

After · top 24 bitsmax(word ≫ 8, 1) / 2²⁴

u = 0.999999940

(2²⁴ − 1) / 2²⁴, the largest value, still below 1

Before, u could be 0 or 1. After, it stays inside [2⁻²⁴, 1 − 2⁻²⁴]: the Gumbel noise is always finite.

Figure 3. Dropping the low 8 bits keeps every value exactly representable; the clamp keeps 0 out. Both endpoints are gone.

To summarize, this is one of the nastiest bugs, dormant for a long time in the codebase, and only uncovered in very long outputs. Troubleshooting problems like this in an inference engine like uzu is nontrivial and requires an understanding of the model, numerical errors, software, math, and more. I think it’s a fun troubleshooting experience, but I hope I won’t encounter one in the near future.

References

Y. Oda, R. Mathieu, R. Knyazhitskiy, and A. Chakhvadze, “Trees from Marginals: Autoregressive Drafting with Factorized Priors,” arXiv preprint arXiv:2607.06763, 2026.
NVIDIA, “CUDA C++ Programming Guide: Alternate Floating Point.”
Apple, “Accelerate Your Machine Learning Workloads with the M5 and A19 GPUs.”
State Spaces, “Mamba: Precision Troubleshooting.”
QwenLM contributors, “Repetition/Looping Issue Observed in Qwen3.5-35B-A3B.”
Liquid AI, “Antidoom: Boundary-Level Preference Tuning for Repetition Loops.”
M. Oberst and D. Sontag, “Counterfactual Off-Policy Evaluation with Gumbel-Max Structural Causal Models,” in Proceedings of ICML, 2019, pp. 4881–4890.
NVIDIA, “CUDA Programming Guide: Floating-Point Computation.”