AgentFlareScholar · long context language models
Long-context research shifts from window length to using context well and cheaply
Early work showed models use long inputs unevenly. Current work targets better training recipes, retrieval and memory alternatives, and cutting the serving cost of long inputs.
AISA Scholar API · arXiv
You're reading the full report — free for humans and search engines.
The same content ships as a structured JSON dataset (normalized facts, sources and
citation arrays, no HTML parsing) so AI agents can consume it directly — metered per fetch via the open
HTTP 402 payment protocol.
crawlerAutoPrice=true · verify at https://esa.aisa.one/api/v1/access/verify
Having a long window is not the same as using it
Foundational evaluations questioned whether long inputs are used well. Lost in the Middle found that performance can degrade significantly when the position of relevant information changes in multi-document question answering and key-value retrieval. LooGLE built a benchmark of post-2022 documents with over 24,000 tokens each and reported that most LLMs have shockingly bad long context ability on long-dependency questions. Gao and colleagues argue for downstream tasks evaluated after SFT instead of perplexity or simple needle-in-a-haystack tests. Together these papers shape how the field measures progress.
Cheaper ways to extend and train for long context
A second line of work makes context extension affordable. LongLoRA uses shifted sparse attention during fine-tuning, with dense attention still available at inference. CEPE adds a small encoder that processes long inputs chunk by chunk so a frozen decoder can use them via cross-attention, extending LLAMA-2 to 128K tokens. ProLong studies data mixes, finding code repositories and books are strong long-data sources when combined with high-quality short-context data. LongAlign addresses instruction fine-tuning for long inputs. A recent instance is Apertus 1.5, which extends its context window fourfold to 262,144 tokens.
Retrieval, agents and memory as alternatives or partners
Several papers ask whether a long window should be replaced or supplemented. Xu and colleagues found a 4K-window LLM with simple retrieval can match a finetuned 16K-window model, and that retrieval helps regardless of window size. Chain-of-Agents splits text among sequential worker agents followed by a manager. A further paper asks whether long-context models can subsume retrieval, RAG and SQL. Recent work continues this theme: NoteBank adds a sparse episodic memory to linear attention, Titan-Llama retrofits external memory modules into pre-trained LLMs, and Copy-Paste Language Models add a copy head.
Serving long inputs: KV cache and attention design
Recent preprints focus heavily on inference cost. Long-context inference makes the KV cache a major memory bottleneck, and new methods attack it through quantization with budgeted precision allocation, low-rank compression, and redesigned attention heads. A systems paper studies how tensor and pipeline parallelism trade off time-to-first-token against inter-token latency as context length and workload vary. These efforts treat long-context ability as established and concentrate on making it practical to deploy.
What’s new on arXiv
Monthly arXiv submissions matching the field ranged from 35 in Nov 2025 to 93 in Sep 2026, with Sep 2026 the highest of the twelve months listed, up from 77 in Aug 2026.
MultiAlloc allocates a fixed KV cache budget between layer-wise precision and token-wise preservation. BELT does low-rank key compression with a learned rank allocator, reporting up to 70% compression and up to 17.8x overall reduction with quantization. Virtual Head Attention balances cache size, compute and quality.
NoteBank gives Gated DeltaNet a training-free sparse episodic memory that archives text and reads it back via replay. Titan-Llama retrofits attention models with a hierarchical memory updated at test time. Copy-Paste Language Models add a lightweight copy head, tested from 135M to 1.7B parameters.
One preprint builds a framework for choosing tensor, pipeline or hybrid parallelism under latency objectives and varying context length. Apertus 1.5 extends fully open models to image and audio understanding and a 262,144-token context window.
Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off
Javad Mirzaei, Jeebak Mitra
Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consists of two fundamentally different phases: prefill, which determines time-to-first-token (TTFT), and decode, which determines inter-token latency (ITL). The use of tensor parallelism (TP), pipeline parallelism (PP), and hybrid configurations exposes different trade-offs between these latency dimensions, making the optimal parallelism configuration highly dependent on workload characteristics, latency requirements, and context length. Moreover, despite the widespread adoption of distributed inference, there is still limited understanding of how parallelism configurations should be selected under varying workload conditions and service-level objectives (SLO). In this paper, we develop a systematic framework for characterizing LLM inference parallelism via throughput-latency trade-offs, SLO-aware serving, and long-context scalability. We introduce the notion of parallelism operating regions by constructing TTFT-ITL feasible sets and Pareto frontiers across different workload compositions, thereby revealing a systematic transition from TP-dominant to PP-dominant configurations as workloads become increasingly prefill-heavy. We further formulate SLO-aware parallelism selection using goodput, showing that stringent TTFT requirements favor PP-dominated configurations, whereas relaxed latency constraints progressively shift the optimality toward TP-dominated configurations with higher serving capacity. Finally, we demonstrate that increasing context length fundamentally changes the relative advantages of parallelism configurations, with PP-dominant configurations exhibiting more favorable scalability and SLO attainment in the long-context regime.
Copy-Paste Language Models
Nathan Godey, Yoav Artzi
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.
Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules
Sarah Pan, Ali Behrouz
Transformers are efficient to pre-train at scale because attention permits parallel computation across sequence positions, but long-context inference requires increasingly large KV caches. Efficient recurrent alternatives reduce the asymptotic complexity of inference, but their sequential dependence limits token-wise parallelism during training, making them harder to scale. Further, the current paradigm of extending long context tasks creates competing notions of retaining information precisely and keeping long-context inference efficient. To address this gap, we introduce Titan-Llama, a post-training method for retrofitting attention-based language models with a hierarchical memory architecture that updates parameters at test-time. Titan-Llama applies a blockwise attention mask and preserves full-fidelity attention over recent tokens while compressing historical context into recurrent, fixed-size neural memory modules (NMMs). Our formulation allows pre-trained attention layers to be replaced without training a new language model from scratch. On Llama-3.2-1B-Instruct, Titan-Llama raises accuracy over segmented attention windows from 23.87\% to 40.51\% on QuALITY and from 14.50\% to 40.75\% on LongHealth. Inference benchmarks show more than 20\% faster prefill, 37\% faster decode at medium context lengths, and 24\% lower steady-state decode memory usage in the evaluated configurations. Together, these results demonstrate that incorporating memory architecture mechanisms that utilize test-time training effectively modulates long-context behavior of pre-trained models, offering a practical route to adapting pre-trained transformers for efficient long-context inference. Our code is released on Github hrefhttps://github.com/sarahpannn/titans-pytorch/tree/mainGitHub
MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization
Yangchengyu Zhou, Zhiping Wu, Runchang Jiang, Dongdong Ren, Youfang Han, Wei Zhao, Dongqi Cai, Wenbin Li
The Key-Value (KV) cache constitutes a dominant source of memory consumption during large language model (LLM) inference, particularly in long-context and complex reasoning scenarios. Existing KV cache quantization methods have explored different compression granularities, including layer-wise precision adjustment and token-level information preservation. In practical deployment, however, the key challenge is not merely compression, but how to allocate limited quantization resources among heterogeneous precision preservation strategies under a fixed KV cache memory budget. We propose MultiAlloc, a profiling-guided precision allocation framework for KV cache quantization that adaptively distributes limited quantization resources between layer-wise precision enhancement and token-wise preservation. MultiAlloc employs lightweight offline profiling to characterize layer sensitivity and token importance, enabling adaptive precision allocation under a fixed memory constraint. Guided by the profiled memory--accuracy trade-off, it automatically derives an effective precision allocation strategy without online allocation search, and applies the resulting policy during inference. Extensive experiments on reasoning and long-context benchmarks demonstrate that MultiAlloc consistently improves over layer-wise, token-wise, and recent multi-granularity KV cache quantization methods under matched memory budgets across diverse models, including Mistral, Llama, DeepSeek-Qwen, and QwQ.
BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation
Yaoyang Liu, Shuwei Shen, Jincheng Tang, Jinguang Liu, Jinbao Xue, Hailong Yang, Ke Zhang, Yinjun Wu
Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side objective targets attention distributions rather than raw keys or pre-softmax logits, yielding a quadratic surrogate with a closed-form low-rank solution. A learned global rank allocator uses differentiable soft selection during training and hard selection at deployment to meet a user-specified component budget. Experiments demonstrate improved model quality under aggressive compression and increased decoding throughput at comparable accuracy. Evaluated across three LLMs and multiple benchmarks, BELT achieves up to 70% low-rank KV cache compression while outperforming the strongest baseline by up to 23.3 points on LongBench, and up to 17.8x overall KV cache reduction when combined with quantization. With Triton-based GPU kernels and a quantized cache, BELT delivers up to 2.7x speedup for the attention module and 3.3x end-to-end decoding throughput.
Virtual Head Attention
Xibo Ding, Guoxia Wang, Jinle Zeng, Jiabin Yang, Sirui Tao, Chao Yang, Dianhai Yu
The design of attention mechanisms in long-context large language models faces an ``impossible triangle'' among KV cache size, training and prefill FLOPs, and model quality. GQA achieves strong quality but incurs high KV cache overhead; alternative approaches such as MLA and MFA can compress the cache, but MLA suffers from quality loss while MFA substantially increases training and prefill compute. We propose Virtual Head Attention (VHA), which simultaneously improves all three vertices through two lightweight linear operations: Q Premix applies a near-identity transformation to queries within each KV group to recover virtual head diversity from halved physical query heads, and Linear Postmix fuses inter-head features via a low-rank I + AB^top residual structure. Since VHA introduces no nonlinear operations, Premix can be folded into the query projection and Postmix into the output projection at inference time, reducing the model to a standard GQA-2 architecture that directly reuses all existing GQA inference optimizations. With only two KV heads, the group size in large models readily reaches 64, fully exploiting Tensor Core parallelism during decoding. Across four scales with 4K-context 50B-token pretraining, VHA consistently surpasses the standard GQA baseline at every scale (+0.05\% to +0.81\%), while reducing KV cache by 4× vs.\ GQA-8 at dense scales and 2× vs.\ GQA-4 at the MoE scale, and lowering training and prefill FLOPs by 7.9\% (1.7B, S=4096). On an industrial 30B-A3B MoE model trained on 500B tokens at 8K context, VHA outperforms both MLA and GQA-4 by +0.70\,/\,+2.60\,\% on a 12-benchmark base-model evaluation suite; in a separate matched 10B-A1B MoE comparison after long-context mid-training, VHA attains an average RULER score of 84.4 over 8K--128K versus 73.0 for GQA, while using two rather than four KV heads in the full-attention layers.
History Replays Itself: Unleashing Long-Context Capability in Linear Attention
Li Nanbo, Firas Laakom, Yilan Zhang, Piotr Piękos, Yimeng Chen, Liangyu Wang, Yifan Xu, Mateusz Ostaszewski, Dandan Guo, Wenyi Wang, Yannin…
Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve information from an ever-growing history. This coupling has reinforced a prevailing view that the state's memory capacity defines the model's long-context capability. We challenge this presumed ceiling: recurrent computation need not bear the full burden of long-term storage. We introduce NoteBank, a sparse episodic memory that archives selected text before future queries are known, uses the backbone's native features for writing and addressing, and reads retrieved evidence through its native forward computation via replay. This training-free, plug-and-play interface requires no external embedding model, separately trained retriever, or separate neural reader. At one million tokens, it raises Gated DeltaNet's mean accuracy across six RULER retrieval tasks from 1.7% to 70.3%, with no observed significant retrieval degradation from 4K to 1M tokens. This recovery adds leqslant0.025% to analytical FLOPs, with persistent context storage approximately 1/2,600 that of a Transformer's full MHA KV cache at matched width, depth, and precision. Experiments extend from three 1.3B recurrent backbones to Mamba-2-8B and post-trained Qwen3.5-9B, demonstrating gains on LongBench v1/v2 and MemoryAgentBench. Together, these results show that sparse evidence storage and native replay can substantially expand the long-context capability of pretrained linear attention models while retaining the efficiency of recurrent computation.
Apertus 1.5: From Open Text Models to Open Multimodal Reasoners
Project Apertus, :, Yixuan Xu, Yassine Turki, Vinko Sabolčec, Valentina Pyatkin, Tommy Chu, Sven Najem-Meyer, Sean F. Carroll, Robert Smith…
Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale. Apertus 1.5 extends the fully open Apertus family from text to multimodal reasoning. Continued pretraining of the Apertus 1.0 models, on 4T additional tokens for the 8B and 2T for the 70B, adds image and audio understanding and extends the context window fourfold to 262,144 tokens; multimodality enters through the tokenizer, the language-model head remains restricted to text, and the architecture is unchanged. Supervised finetuning, long-context finetuning, reinforcement learning with verifiable rewards, and preference alignment complete the released models. Thinking is a flag of the chat template rather than a separate model, and lifts mathematical reasoning by up to 20 points on the 8B model. The released models improve over their predecessors on nearly every benchmark; among fully open models of its class, the 8B leads on knowledge, instruction following, and cultural benchmarks, and its instruction following surpasses all compared models, including open-weight ones. On image and audio understanding, a first for the family, the models are competitive with fully open multimodal models, strongest on document understanding and long-form and lower-resource speech recognition, while specialized vision and audio models remain ahead on perception and transcription of English and other high-resource languages. As in Apertus 1.0, we train only on openly available data, respect crawler opt-outs retroactively, and exclude restrictively licensed material. We release the model weights, data preparation and training recipes, and evaluation suites under permissive licenses; openness extends to values, with the Apertus charter guiding alignment stages published alongside the models.
Foundational papers
- 2024Lost in the middle: How language models use long contexts6,966 citations
- 2024
- 2024
- 2024Retrieval meets long context large language models379 citations
- 2024
- 2024
- 2024
- 2024
- 2025
- 2025
- 2026-10
Lost in the middle: How language models use long contexts
NF Liu, K Lin, J Hewitt, A Paranjape et al.
Contribution. Shows that performance can degrade significantly when the position of relevant information changes in multi-document QA and key-value retrieval, questioning how well models use long contexts.
Loogle: Can long-context language models understand long contexts?
J Li, M Wang, Z Zheng, M Zhang
Contribution. Introduces LooGLE, a benchmark of post-2022 documents over 24,000 tokens with 6,000 new questions; reports most LLMs have shockingly bad long context ability.
A comprehensive survey on long context language modeling
J Liu, D Zhu, Z Bai, Y He, H Liao, H Que et al.
… long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (… recent advances in long-context modeling for large language models. Our …
Chain of agents: Large language models collaborating on long-context tasks
Y Zhang, R Sun, Y Chen, T Pfister et al.
Contribution. Proposes Chain-of-Agents, where worker agents sequentially handle text segments and a manager agent aggregates, as an alternative to input reduction or window extension.
How to train long-context language models (effectively)
T Gao, A Wettig, H Yen, D Chen
Contribution. Studies continued training and SFT for long context using downstream-task evaluation; yields ProLong-8B, trained on 40B tokens from Llama-3, with data mix and sequence length findings.
Retrieval meets long context large language models
P Xu, W Ping, X Wu, L McAfee, C Zhu et al.
Contribution. Finds a 4K-window LLM with retrieval can match a finetuned 16K-window model, and that retrieval improves performance regardless of extended window size.
Longlora: Efficient fine-tuning of long-context large language models
Y Chen, S Qian, H Tang, X Lai, Z Liu et al.
Contribution. Presents LongLoRA, which uses shifted sparse attention and revisits parameter-efficient fine-tuning to extend context sizes at limited computation cost.
Longalign: A recipe for long context alignment of large language models
Y Bai, X Lv, J Zhang, Y He, J Qi, L Hou et al.
Contribution. Presents LongAlign, a recipe for long-context alignment, motivated by the need for instruction fine-tuning on input sequences of similar length.
Long-context language modeling with parallel context encoding
H Yen, T Gao, D Chen
Contribution. Introduces CEPE, a small encoder that processes chunks so a frozen decoder uses them via cross-attention, extending LLAMA-2 to 128K tokens with 10x throughput.
Can long-context language models subsume retrieval, rag, sql, and more?
J Lee, A Chen, Z Dai, D Dua, DS Sachan et al.
… Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging …
Open problems
- Closing the gap between nominal window size and effective use of information across long dependency ranges.
- Balancing precise information retention against efficient long-context inference, since recurrent states and KV caches pull in opposite directions.
- Choosing among parallelism configurations as workload, latency targets and context length vary.
- Combining retrieval with long windows so retrieved evidence helps rather than degrades extended-context models.
FAQ
Do long-context models use all of their input equally?
Not according to Lost in the Middle, which found performance can degrade significantly when the position of relevant information changes. LooGLE likewise reported that most LLMs have shockingly bad long context ability.
Is retrieval still useful with long windows?
Xu and colleagues found retrieval significantly improves performance regardless of context window size, and that a 4K model with retrieval can match a finetuned 16K model.
What are recent papers working on?
Many target inference cost, including KV cache quantization, low-rank compression and attention redesign, alongside external memory modules and parallelism trade-offs for serving.
How this page is made. Foundational papers and citation counts come from academic search via the AISA Scholar API, with abstracts read from each paper’s page; recent preprints and monthly volumes come from the public arXiv API (metadata CC0). The digest is written by a language model from those abstracts only — every theme links the papers it draws on, and figures not in the inputs are removed.
Sources & citations
- Lost in the middle: How language models use long contexts 6,966 citations
- Loogle: Can long-context language models understand long contexts? 337 citations
- A comprehensive survey on long context language modeling 145 citations
- Chain of agents: Large language models collaborating on long-context tasks 386 citations
- How to train long-context language models (effectively) 160 citations
- Retrieval meets long context large language models 379 citations
- Longlora: Efficient fine-tuning of long-context large language models 744 citations
- Longalign: A recipe for long context alignment of large language models 186 citations
- Long-context language modeling with parallel context encoding 107 citations
- Can long-context language models subsume retrieval, rag, sql, and more? 95 citations
- Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off arXiv · 2026-10-14
- Copy-Paste Language Models arXiv · 2026-10-13
- Titan-Llama: Retrofitting Pre-trained LLMs with External Neural Memory Modules arXiv · 2026-10-13
- MultiAlloc: Budget-Aware Precision Allocation for KV Cache Quantization arXiv · 2026-10-13
- BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation arXiv · 2026-10-13
- Virtual Head Attention arXiv · 2026-10-13
- History Replays Itself: Unleashing Long-Context Capability in Linear Attention arXiv · 2026-10-11
- Apertus 1.5: From Open Text Models to Open Multimodal Reasoners arXiv · 2026-10-11