Skip to content
World clockEU--:--UK--:--USA--:--CN--:--PLDEFRIT中文EN

portal about AI and technologyevents · analysis · interviews · technical background

Search
LIVE
›

Memory plugins are the wrong fix for long AI context, a new paper and two proxies argue

AI agent tools are treating the context window problem as a memory problem. A compaction proxy called Gobstopper claims 29% fewer input tokens than Claude Code's built-in step. A September paper argues the real bottleneck is hardware, not recall.

AI & modelsAnalysisRachel NwosuPublished: 3 October 20264 min readSources 8
Memory plugins are the wrong fix for long AI context, a new paper and two proxies argue

The pitch for Gobstopper is narrow and numeric. It replaces the compaction built into Claude Code and Codex with a local proxy. It compacts more often and keeps the full history in a searchable database on your disk. On Terminal-Bench 2.1, the project says, that saved 29% of input tokens, 84.3M against 118.6M.

Task counts barely moved. Gobstopper solved 61 of 89 tasks; Claude Code without the proxy solved 60. The site admits the two-task gap is within single-trial noise. It also says 21 of the 89 tail-0 trials may have run an earlier build. So the headline is a token bill, not a capability jump.

That distinction matters for anyone paying per token to run coding agents. A 29% cut in input tokens is a 29% cut in that part of the bill. It also matters for anyone who assumed longer context windows would end the problem. They don't. A proxy still has to decide what to throw away. According to Gobstopper's own limitations section, it can't promise a provider will resume a copied session, or that a summary kept every fact.

The hardware side of the same problem got a rigorous airing in a paper by Tejinder Singh of Dell Technologies, arXiv:2609.30854.

The paper is a systematization of knowledge rather than a new algorithm. Its central claim is that decode is memory-bound, not compute-bound. At batch 1, BF16 weights have an arithmetic intensity of 1 FLOP/B against the H100's ridge point of 295 FLOP/B. Past a computable context length, the KV cache overtakes the weights as the thing the GPU must read every step. The threshold falls hyperbolically as batch size rises. Llama-3-70B needs 327,680 bytes per token in BF16, the paper says. A single 128k-token sequence therefore adds 42 GB on top of 140 GB of weights, and a million-token sequence needs 328 GB. That is more than four H100s hold in total.

Five families of fix, three regimes

The paper groups the workarounds into one footprint model: quantization, token eviction, paging, prefix caching and tiering. It then derives crossover lengths and speedup ceilings on roofline maps for the H100, B200 and MI300X. Its point is that results measured under different batching policies and cache layouts were never comparable. The same 2-bit quantization gives 1.25x at batch 1 and 4.82x at batch 32. In the author's phrasing, the algorithm didn't change, the regime did.

Architecture tweaks shrink the constant, not the curve. Moving from multi-head to grouped-query or multi-head latent attention cuts per-token cost by up to 12x. DeepSeek-V2's MLA is lowest at 0.07 MB per token. But every variant still grows linearly with context length.

If that framing is right, the practical question for buyers is not who has the biggest window. It is which layer absorbs the waste. Vendors can charge for tokens, for memory bandwidth or for clever eviction. The paper suggests all five technique families scale factors that are largely orthogonal, so a stack can combine them.

The store is unauditable.

That is roughly the argument Kevin Liao made on 3 October against the memory-plugin category. His target is the vector database that ingests past conversations. It then injects the five most similar snippets into every prompt. Similarity, he writes, tells you nothing about which snippet is correct, current or missing, and the codebase changes daily.

Liao's alternative is documentation: files a person writes and an agent reads, rather than recall over transcripts. It is an opinion piece with no benchmark behind it, and it competes with vendors selling the opposite. But it lands on the same practical note as the paper and the proxy. The scarce resource is curated context, and every layer between the model and the truth costs tokens or bandwidth.

None of the three settles what a long session should keep.

Gobstopper's 29% comes from one model, GLM 5.3 Flash. It used a 45,000-token threshold rather than the 128,000 default, over 89 tasks and one trial per arm. The paper's crossover points are derived, not measured on every workload. The debate over where context lives is now about arithmetic, and the arithmetic is not settled.

Comments 0

Sources

8
  1. 01Gobstopper: context compaction proxy for Claude Code and CodexEN
  2. 02The KV Cache Is the New Memory WallEN
  3. 03Agents Don't Need Memory. They Need Documentation.EN
  4. 04Memory safety for Postgres extensions in C/C++EN
  5. 05Memory-Safe WebP DecodingEN
  6. 06An Update on Orion for Linux and WindowsEN
  7. 07warply: Unofficial one-click Cloudflare WARP applicationEN
  8. 08Windows on Itanium also provided for hot-patchingEN

All figures and quotations in this text come from the sources listed below.

Content prepared by the editorial team with AI assistance.

Rachel Nwosu

Rachel Nwosu

AI, models and technology

Rachel Nwosu covers AI, models and technology for FLASH24, working from public model documentation, benchmark releases and repository histories rather than press summaries, and she skips announcements that arrive without reproducible numbers. She checks training-data claims against dataset cards and reruns reported metrics where code is available. She spends much of her week interviewing researchers and engineers, tracking model launch calendars, and comparing vendor benchmarks with independent evaluations. Outside the desk she runs 3D printers, restores old computers, and tests how models learn from internet junk. She does not publish benchmark figures she cannot trace to a source.

Newsroom →

Comments

0
  1. No comments yet — be the first.

Write a comment

Comments are public. We do not publish abuse, spam or advertising.