workrr workrr

Apple has introduced a breakthrough in large language model (LLM) efficiency with its new “multi-token prediction” framework, enabling models to generate multiple tokens in a single step—rather than strictly one at a time as in traditional autoregressive LLMs. This innovation allows for significant speed improvements:

• Up to 5x faster text generation for coding and math tasks, and about 2–3x gains for general chat and knowledge tasks.

• Speed gains are achieved without sacrificing output quality or accuracy.machinelearning.apple+2

How It Works

Apple’s approach builds on several key techniques:

• Masked input formulation: The model is prompted with special mask tokens representing upcoming words (e.g., “The cat is <MASK1> <MASK2>”), allowing the model to fill in several words simultaneously.

• Gated LoRA formulation: This fine-tuning preserves the base LLM’s original abilities while enabling accurate multi-token prediction, avoiding the degradation seen in naïve fine-tuning.

• Lightweight learnable sampler: Ensures the sequence of predicted tokens stays coherent when filling multiple masks.

• Speculative decoding and verification: The model’s guesses for multiple tokens are promptly checked against what standard, one-token-per-step decoding would have produced. If a guess doesn’t pass, generation safely reverts to the classic approach for maximum reliability.

• Auxiliary training losses: These further refine output quality and consistency when predicting tokens in batches.9to5mac+3

Results

Apple tested this framework on its models and reported:

• Average speedups of 2–3x for general Q&A and chat.

• Up to 5x for structured, predictable domains like code and mathematics.

• No observed decrease in response quality compared to standard single-token generation techniques.ainvest+3

Impact and Applications

This advance is especially relevant for on-device AI, enabling more responsive assistants, real-time code assistants, and math solvers, even on consumer hardware where computation is at a premium.webpronews

In summary, Apple’s research shows pretrained LLMs “know” about future tokens well enough to accurately predict several at once, allowing order-of-magnitude improvements in latency and efficiency—most notably for tasks with predictable structure, like programming and math, but also with solid improvements for general conversation and knowledge retrieval.machinelearning.apple+2