Skip to content
The AI Research Digest is included with Super Saver. See Super Saver
Tech AI Magazine Research
Digest
Sign in Subscribe

Research Digest » October 2026 research » Breakthrough 2 of 5

Breakthrough 2 of 5 · October 2026

Soft-MTP: a one-line change strips noise out of multi-token prediction training

A new preprint finds that the standard way of training multi-token prediction heads hides a noisy gradient term. Removing it improved perplexity, speculative decoding and reasoning at no measurable extra cost.

ML systems
Soft-MTP’s illustration, pseudocode, and loss performance compared to MTP. Blue line is the one-line code change . We refer readers to Algorithm 1 for the full code implementation of Soft-MTP, including the MTP loss weighting.
Soft-MTP’s illustration, pseudocode, and loss performance compared to MTP. Blue line is the one-line code change . We refer readers to Algorithm 1 for the full code implementation of Soft-MTP, including the MTP loss weighting. Figure from “Multi-Token Prediction Done Right” (arXiv)
RD
AI Research Digest research desk
October 2026 research · 4 min read · fact-checked against the source

Many recent language models use multi-token prediction (MTP): extra heads that guess several tokens ahead and can speed up inference through speculative decoding. A new preprint shows that the standard MTP training gradient splits exactly into two parts. One is a useful self-distillation term. The other is a noisy residual that works against next-token learning. The proposed method, Soft-MTP, keeps only the useful term with a one-line code change. It adds no parameters and no measurable overhead. In models from 167M to 1.2B parameters, it beat standard MTP on perplexity, zero-shot accuracy and speculative decoding.

Super Saver

Keep reading, and get the Explain it Like I'm 12 version

The full article and its Explain it Like I'm 12 version are for Super Saver members. Sign in if you're a member.

Join Super Saver Sign in

Figures from the research

Illustration of position-aligned NTP distributions.
Figure 2. Illustration of position-aligned NTP distributions. Figure from “Multi-Token Prediction Done Right” (arXiv)
((a)) 167M–1.2B: Trunk + MTP gradient SNR.
Figure 3. ((a)) 167M–1.2B: Trunk + MTP gradient SNR. Figure from “Multi-Token Prediction Done Right” (arXiv)
((b)) 1.2B: Trunk gradient SNR.
Figure 4. ((b)) 1.2B: Trunk gradient SNR. Figure from “Multi-Token Prediction Done Right” (arXiv)
More from October 2026

Keep exploring this issue

Figure S1: The Reed–Solomon decoder. (a) Measured forward Toffoli cost per step: one cycle of the 222-cycle synchronized EEA baseline of Khattar et al. [13] , or one round of our 70-round BM decoder. (b) Locator degree (top) and number of stored discrepancies…Paper of the Month

LLM-guided search finds cheaper quantum circuits in seven of eight benchmark tasks

A language model proposes circuit designs, a strict classical checker verifies and scores them, and the results beat prior designs while making a previously impractical classical algorithm usable in quantum form.

7 min+ ELI12
Positives that join less of a scene need more data ( Section 2.3 ). A synthetic world of five classes of six states (literal data, as in Figure 12 ): alignment of a text JEPA with the image side against the samples per state and the reach of its bad views…Breakthrough 1 of 5

A theorem for the Platonic Representation Hypothesis points to positive pairs, not scale

Balestriero and LeCun prove when separately trained JEPAs learn matching representations, then show a 109M-parameter text model aligning with vision better than a 72B language model.

Randall Balestriero, Yann LeCun4 min+ ELI12
Qualitative comparison. Rows show Causal Forcing, Rolling Forcing, and 4-step and 5-step Wave Forcing.Breakthrough 3 of 5

Wave Forcing overlaps denoising stages to stream video faster, with more motion

By letting each video chunk see only earlier, cleaner chunks, a new method pipelines generation across GPUs. It matches a strong baseline on quality and shows much more motion.

Yijie Jin, Yuxuan Liu et al.4 min+ ELI12