Apple Feature Trails

Feature trailActive

Normalizing Trajectory Models

This post introduces Normalizing Trajectory Models (NTM), a novel approach to generative modeling that treats each reverse step as a conditional normalizing flow. This allows for exact likelihood training over the entire generative trajectory, enabling self-distillation into a lightweight denoiser that produces high-quality samples in just four steps. NTM matches or outperforms existing baselines on text-to-image benchmarks while retaining exact likelihood.

1 post · 2026

Timeline

Feature trailActive

Agentic Data Cataloging

This post introduces RISED (Rubrics for Agentic Multi-Environment Selection and Self-Distillation), a novel framework for training LLM agents across diverse interactive environments. RISED repurposes rubrics, originally used for reward signals, to guide online data selection and policy supervision. An LLM judge tags rollouts with a predefined rubric vocabulary, which then guides the selection of data aligning with the overall behavioral composition of the mixed-environment batch. Positive rubrics provide privileged context for self-distillation, while negative rubrics guide generation away from failure modes. RISED achieves state-of-the-art performance across environments and offers rubric-based analysis of behavioral changes.

4 posts · 2025–2026

Timeline

20252026

Feature trailActive

Confidence Limits in Diffusion Models

This post introduces a theoretical framework for understanding the limits of confidence in discrete diffusion models. It demonstrates that current sampling methods often violate conditional independence assumptions, leading to significant deviations from the training distribution. The work proposes a new understanding of how to measure and potentially improve confidence in diffusion model outputs.

1 post · 2026

Timeline

Feature trailActive

Self-Improvement via Internalized Self-Generated Feedback

This post introduces RLTL;DR, a novel self-improvement paradigm for reinforcement learning agents facing extremely difficult tasks with no external guidance. It enables agents to generate their own 'TL;DR' insights after failed attempts, condition subsequent rollouts on these insights, and internalize a direct task-to-insight mapping through backpropagation. This approach significantly breaks through learning barriers where standard methods fail, achieving notable success rates even when insights are not present at evaluation time. The core contribution is the demonstration that learning from self-generated, internalized feedback is a powerful and efficient training paradigm.

1 post · 2026

Timeline

Feature trailActive

Dynamically Scaled Activation Steering

This post investigates the effectiveness-fluency trade-off in LLM conditioning methods. It finds that efficient steering methods often degrade fluency, and that activation steering is less effective on instruction-tuned models compared to base models. Prompting and fine-tuning are better for concept injection but not removal. Textual metrics correlate well with LLM-as-judge scores and provide insights into conditioning behavior.

2 posts · 2026

Timeline

Feature trailActive

Tree-Structured Expression Serialization in Language Models

This post introduces a round-trip protocol to empirically evaluate the lossy and asymmetric nature of serializing tree-structured expressions into natural language by language models. It identifies generation as the primary source of failure and demonstrates that the communication channel is trainable, with fine-tuning significantly improving performance. The work highlights tree-structured expression serialization as a key bottleneck for models communicating hierarchical structure.

1 post · 2026

Timeline

Feature trailActive

Semi-Supervised Federated ASR Training

This post details advancements in Semi-Supervised Federated Learning (SSFL) for Automatic Speech Recognition (ASR). It introduces a practical recipe involving online pseudo-labeling per client and server-side update stabilization. Key findings include the effectiveness of per-client online teachers over broadcast global teachers, the critical role of server-side training on labeled data for convergence, and the sensitivity of stabilization to data augmentation and batch size. The work provides guidelines for improving SSFL in ASR, significantly narrowing the gap to fully-supervised FL.

1 post · 2026

Timeline

Feature trailActive

Streaming Neural Audio Encoder Compression

This post introduces a method for compressing streaming neural audio encoders by distilling their latent-space representations. This technique aims to reduce the memory footprint and latency of these encoders, which are crucial for on-device speech transcription systems. The distilled encoders maintain performance close to their teachers while achieving significant compression ratios.

1 post · 2026

Timeline

Feature trailActive

Diffusion vs. Autoregressive Language Model Performance

This post introduces probe guidance, a method to guide flow matching models using frozen internal states of an existing diffusion model. This eliminates the need for an additional forward pass at inference time and ensures that weak and strong models share similar dynamics. Applied to continuous diffusion language models, probe guidance sets a new state-of-the-art performance on unconditional generation. When applied to a 1.7B diffusion language model, probe guidance consistently improves on multiple choice question answering benchmarks. The post also studies the autoguidance setting, finding that the weak model must come from a low-entropy region of training. This provides a practical way to improve diffusion language models and sheds light on the mechanism behind autoguidance.

3 posts · 2026

Timeline

Feature trailActive

Reset-Free Reinforcement Learning Benchmarking

This post introduces REVERSAL-BENCH, a benchmark designed to measure the impact of environmental irreversibility on reinforcement learning agents, particularly those aiming for reset-free continuous training. It quantifies a 'reversibility cliff' where agents fail as environmental reversibility decreases, and evaluates the effectiveness of safety shields in predicting and mitigating irreversible failures.

1 post · 2026

Timeline

Feature trailActive

AI Agent Scenario Synthesis

Agentic LLM systems require effective memory management to retain productive context across sessions. This post introduces shared selective persistent memory, which identifies and retains reusable context categories (task specifications, data schemas, tool configurations, output constraints) while discarding session-specific reasoning traces. This memory is shared across users with role-based access control for collaborative reuse. The system is implemented in a collaborative workspace platform for producing git-versioned artifacts. A zero-token data refresh mechanism decouples generated programs from runtime data, enabling artifact reuse without re-invocation. This approach significantly improves task completion rates, reduces token costs, and outperforms naive full-history persistence.

2 posts · 2026

Timeline

Feature trailActive

Distillation of Tool-Calling Capabilities

This post introduces Trajectory-Shaped Discrete Flow Matching (TS-DFM), a method to improve the efficiency and performance of discrete flow matching for text generation. TS-DFM uses a lightweight energy compass to guide the generation process at each step, selecting the most coherent continuations. This training-only shaping significantly reduces the number of forward passes required, leading to substantial speedups and lower perplexity compared to standard discrete diffusion models. The method achieves state-of-the-art performance among discrete generation baselines.

2 posts · 2026

Timeline

Feature trailActive

Value Induction in LLMs

This post investigates the unintended consequences of value induction in conversational LLMs. It fine-tunes models using curated value subsets of preference datasets and measures the impact on the expression of other values, model safety, anthropomorphic language, and QA benchmarks. Findings indicate that inducing values leads to the expression of related and sometimes contrasting values, increases safety, and all values increase anthropomorphic language use, making models more validating and sycophantic.

1 post · 2026

Timeline

Feature trailActive

Joint Protein Sequence and Structure Codesign

SimpleDesign introduces a single-stage end-to-end objective for joint protein sequence and structure codesign, directly modeling in the data space. It combines discrete cross-entropy for sequences and a regression objective for structures using Transformer-based multimodal backbones with modality-specific processing and global self-attention. Trained on over 2M sequence-structure pairs, it achieves competitive performance in co-design and unconditional generation benchmarks.

1 post · 2026

Timeline

Feature trailActive

Unsupervised Library Learning of Typed Motor Programs

REFACTOR-VLA introduces a novel 'wake/sleep' architecture for unsupervised learning of reusable skills in vision-language-action (VLA) models. It uses a Behavioral-Equivalence Kernel (BEK) based on a learned latent world model to cluster action segments into skills. These skills are represented as typed lambda terms and used by a library-conditioned rectified-flow action decoder. Skills are accepted based on Minimum Description Length (MDL) and return-preservation criteria. The system employs a three-phase training schedule: world-model warmup, wake-phase policy optimization, and sleep-phase skill discovery. Empirical results show that increasing world model size does not always improve performance, and an auxiliary supervised contrastive loss during warmup significantly enhances skill clustering quality.

1 post · 2026

Timeline

Feature trailActive

3D Asset Generation with Relightable Gaussians

This post introduces Luce, a novel 3D representation that unifies geometry and physically based rendering (PBR) materials within a voxelized multimodal Gaussian cloud. Luce utilizes dedicated Gaussian primitives for each PBR modality (albedo, metallic-roughness, surface normals) and compresses this representation into a material-aware latent space using a variational autoencoder. A rectified-flow transformer generates this latent from a single image, enabling the creation of relightable, geometrically accurate, and materially faithful 3D assets with preserved fine details. Luce achieves state-of-the-art performance on single-image-to-3D generation benchmarks.

1 post · 2026

Timeline

Feature trailActive

Generative Language Model Pruning

IDEA Prune introduces an integrated enlarge-and-prune pipeline for generative language model pretraining. It combines enlarged model pretraining, pruning, and recovery under a single cosine annealing learning rate schedule, complemented by an iterative structured pruning method for gradual parameter removal. This approach mitigates knowledge loss and facilitates smooth compression and enhanced performance, achieving superior performance of pruned models.

1 post · 2026

Timeline

Feature trailActive

Unified Multimodal Generation with Autoregressive Flows

STARFlow2 builds upon the Pretzel architecture, vertically interleaving a frozen pretrained VLM stream with a TARFlow stream via residual skip connections. This design preserves pretrained multimodal understanding, enables high-fidelity continuous image generation, and achieves structural unification under a single causal mechanism. Combined with a deep-shallow flow design and a unified FAE latent space, STARFlow2 supports cache-friendly interleaved generation where both text and visual outputs directly enter the KV-cache without re-encoding.

1 post · 2026

Timeline

Feature trailActive

Internalized Visual Thinking for Video Reasoning

Internalized Visual Thinking (IVT) is a post-training framework for multimodal large language models that enables proactive video reasoning. It jointly optimizes textual prediction and next-embedding prediction over unlabeled videos, encouraging the model to capture motion, object transitions, interactions, and latent intent. At inference, IVT generates the answer directly without synthesizing or re-encoding future frames, significantly reducing latency compared to Visual CoT.

1 post · 2026

Timeline

Feature trailActive

Code-Switching ASR Training

This post introduces an iterative pseudo-labeling approach for Mandarin-English Code-Switching Automatic Speech Recognition (ASR). The method leverages unlabeled data to improve ASR performance by generating pseudo-labels, followed by a two-stage bilingual model training (pre-training and fine-tuning on supervised CS data), and iterative refinements. This approach significantly reduces Mix Error Rate (MER) on SEAME datasets.

1 post · 2026

Timeline

Feature trailActive

Mixture Pretraining Scaling Laws

This post introduces a repetition-aware mixture scaling law for language model pretraining under data constraints. It accounts for the decreasing value of repeated target tokens and the regularizing role of generic data, providing principled recommendations for effective mixture configurations. The work studies the trade-off between target data repetition and generic data regularization across various model and dataset sizes, finding that repetition is a central driver of target-domain performance and that mixture training tolerates higher repetition than single-source training.

1 post · 2026

Timeline

Feature trailActive

Multilingual Knowledge Transfer via Lexical Intervention

This post introduces LINK, a data-level intervention method to improve cross-lingual knowledge transfer during model pretraining. By performing lexical substitutions in the high-resource (English) training corpus using bilingual vocabularies, LINK enhances knowledge acquisition for downstream tasks in low-resource languages without requiring additional model training or complex translation systems. This method has shown significant improvements on downstream tasks across eight languages and five model sizes, achieving up to a 2x speedup in training to reach equivalent performance.

1 post · 2026

Timeline

Feature trailActive

Boolean Query DAG Evaluation over Inverted Indices

This post establishes the theoretical boundaries of executing complex logic natively over an inverted index. It formalizes a retrieval language (L_R) based on Directed Acyclic Graphs (DAGs) and proves that its evaluation problem is strictly P-Complete. To make evaluation tractable, it introduces ComputePN, a deterministic, sparsity-aware evaluation algorithm. By decoupling logical negation from universe-scale materialization via a novel Positive-Negative dual representation, and utilizing native DAG memoization, ComputePN strictly bounds evaluation time to O(|Q| · |U_active|). This approach successfully evaluates P-Complete queries natively over the index, avoiding both the combinatorial tree-expansion bottleneck and the universal scan penalty, laying the formal foundation for computational retrieval.

1 post · 2026

Timeline

Feature trailActive

Low Influence Point Unlearning

This post introduces a method for unlearning specific data points from trained machine learning models by identifying and leveraging low-influence points. This approach aims to reduce the computational costs associated with unlearning by pre-filtering datasets, leading to significant savings (up to ~50%) on real-world empirical examples.

2 posts · 2025–2026

Timeline

20252026

Feature trailActive

Categorical Flow Maps for Language Modeling

Categorical Flow Maps (CFMs) are scaled to 1.7B parameters and trained on 2.1T tokens, demonstrating competitive text generation quality in few inference steps. A likelihood bound for CFMs in the semi-discrete setting is introduced, enabling scoring on standard LM benchmarks. Challenges and insights for training at scale, including loss weighting and time scheduling, are provided.

1 post · 2026

Timeline

Feature trailActive

Speculative Decoding for LLM Inference

ARBITRAGE is a novel step-level speculative generation framework for Large Language Models (LLMs) that dynamically routes generation based on the relative advantage between draft and target models. Instead of applying a fixed acceptance threshold, ARBITRAGE uses a lightweight router trained to predict when the target model is likely to produce a meaningfully better step. This routing approximates an ideal ARBITRAGE ORACLE that always chooses the higher-quality step, achieving near-optimal efficiency–accuracy trade-offs. Across multiple mathematical reasoning benchmarks, ARBITRAGE consistently surpasses prior step-level SD baselines, reducing inference latency by up to ∼ 2× at matched accuracy.

2 posts · 2026

Timeline

Feature trailActive

Weight Locking via Deep Low-Rank Residual Distillation

This post introduces DLR-Lock, a novel defense mechanism for open-weight language models that prevents unauthorized modification of pretrained weights. DLR-Lock replaces standard MLPs with Deep Low-Rank Residual Networks (DLR-Nets), which exploit the inference-training asymmetry of automatic differentiation. This defense forces increased activation memory during backpropagation and creates architectural mismatches that complicate fine-tuning, effectively withstanding adaptive attackers while preserving original model capabilities. Experiments on LLMs validate its efficacy.

1 post · 2026

Timeline

Feature trailActive

Outlier Token Control in Diffusion Transformers

This post introduces Dual-Stage Registers (DSR) as a method to control outlier tokens in Diffusion Transformers (DiTs) for image generation. DSR is applied to both the encoder and denoiser components, utilizing trained registers when available and recursive test-time registers otherwise. This approach aims to reduce outlier artifacts and improve generation quality across various image generation tasks.

1 post · 2026

Timeline

Feature trailActive

UMAP kNN Graph Analysis

UMAP's internal kNN graph is being leveraged for enhanced data sensemaking. Standard graph algorithms like PageRank, k-core decomposition, and clustering coefficient are applied to this graph to identify representative data points, reveal dense core regions, and detect tight-knit neighborhoods of similar data points. These methods offer practical and competitive alternatives to existing sensemaking techniques.

1 post · 2026

Timeline

Feature trailActive

On-device Audio Synthesis

Siri Expressive Voices synthesizes rich, configurable speech in real time and entirely on device, powered by AFM 3 Core Advanced. This work presents the memory-efficient audio synthesis architecture: a detokenizer that converts semantic audio tokens into high-fidelity audio within the tight compute and memory budget of the Apple Matrix Coprocessor (AMX). The architecture decouples temporal and depth processing using a residual vector quantization (RVQ) representation with a three-component design—a streaming encoder, a temporal decoder, and a depth decoder. A single reusable depth decoder with Diffusion Transformer (DiT)-style stage conditioning generates all RVQ levels autoregressively, replacing dedicated per-level decoders. Causal sliding window attention with fixed-window key-value caching yields constant memory complexity. Deployed on the AMX, the detokenizer sustains roughly 10ms per generation step with a peak runtime memory of only ~21MB, enabling continuous streaming synthesis. This constant, small footprint replaces the linear and quadratic memory scaling of conventional transformer- and GAN-based approaches. Audio quality assessment confirms that the proposed architecture maintains synthesis fidelity while achieving computational efficiency gains. The proposed architecture is deployed in production as part of Siri Expressive Voices, powering a voice overhaul with Pace and Expressivity customizations sliders in Apple Devices and support for custom assistant voices. Operating at a 1-billion-parameter activation size within AFM 3 Core Advanced, it improves Mean Opinion Score (MOS) by +0.28 overall (4.15 vs. 3.87) and by +0.42 on conversational speech (4.24 vs. 3.82) over the prior on-device text-to-speech system.

1 post · 2026

Timeline

Feature trailActive

Sparse Attention for Video Generation

CalibAtt, a training-free method, accelerates text-to-video generation by identifying and skipping negligible token-to-token connections in transformer-based diffusion models. It achieves up to 1.58x end-to-end speedup while maintaining video generation quality and text-video alignment by performing an offline calibration pass to compile block-level sparsity and repetition patterns into optimized attention operations.

1 post · 2026

Timeline

Feature trailCooling

Token-Level Length Modeling

This post introduces the Length Value Model (LenVM), a token-level framework for modeling remaining generation length at each decoding step. LenVM formulates length modeling as a value estimation problem, assigning a constant negative reward to each generated token to predict a bounded, discounted return as a proxy for the remaining generation horizon. This provides annotation-free, dense, unbiased, and scalable supervision. Experiments show LenVM significantly improves exact length matching on LIFEBench and enables continuous control over the performance-efficiency trade-off on GSM8K. It also offers interpretable insights into generation dynamics, revealing how specific tokens influence reasoning towards shorter or longer regimes. LenVM supports length control, prediction, and interpretation, suggesting its potential as a general framework for length modeling and a length-specific value signal for future RL training.

1 post · 2025

Timeline

20252026