How Compaction Works in Pi
Conversation activity · last 37 hours presspostscomments the crowd ↓
Every item, in order 31
Clustered from 31 items across 3 sources. Not yet parsed — the coverage below is the raw record.
Press coverage 1
Social posts 2
The conversation unedited, verbatim
88 comments claimed on the threads · 28 collected here from 2 sites · 24 voices below over time ↑
- Today
-
How Compaction Works in Pi: https:// earendil.com/posts/compaction- in-pi/ Discussion: http:// news.ycombinator.com/item?id=4 9289654
-
There are some very interesting latent compaction approaches like this[1] for when you can control the whole inference stack. i.e in on-device and datacenter inference.[1]:
-
enforcing markdown files is much better. In a perfect world these docs are curated by humans, but if you can't be arsed then at least have some ground rules so these do not get flooded with context. Agents are pretty good at finding what they need for a task, if they're instructed correctly.I stopped compacting and started making new chats every…
-
Does anyone find compaction useful? I have always been a big believer of many small chats instead of large ones. There is certainly a time and a place where maybe I need to load in a lot to initial context BUT I often find that quality goes does as we near context limits and of course costs go massively up. I would rather repeat small notarized…
-
In juggler (https://news.ycombinator.com/item?id=48883305) I spent a lot of effort on some compaction tricks that are elegant, but in reality find myself not really using them..Because juggler has a nested-thread architecture, I had a great ah-ha! moment when I realised that with sub-threads, compaction becomes almost free: you just take the list…
-
Great thread, I was just thinking about compaction. My current line of thought is that compaction/pruning/ctx management in general should be something ongoing and maybe recursive. For example:User:'How is auth implemented?' -> [thinking] [codebase exploration with [thinking] in between, 10 file reads, 3 of which were "wrong"] [thinking] ->…
-
the way compaction works is pretty simple and highly relies on another model to do it, the only part the user has control of is when to do the compaction, which actually means when doing large amount of work in one shot, it should be planned from the beginning to be separated into works that can be reviewed, afterwards the model can compact
-
I want to like Pi but compaction is why I had to go back to Open Code…My problem is that when in a loop and it’s calling tools, it won’t check how close it is to the compacting limit until the whole loop returns to you for the next prompt. And so if you have a run that could go for hours, it’s a gamble if you’ll OOM or an interrupting compaction…
-
I implemented a few additional strategies in https://github.com/rcarmo/piclaw/tree/main/runtime/src/exten... - including Codex-native server-side compaction. They all have slightly different trade-offs, but I run very long sessions quite successfully
-
My summarization creation functions over batches of 50 messages, and I don’t often lose important context any more. The loss comes from trying to stick a whole conversation in a single compaction request (at least in my case)
-
Potentially: remove thinking blocks, and keep the rest. At least this would ensure that the entire context of the conversation is still there, and anything said isn't lost.Having a second model also iterate the resulting messages and remove low-value tool calls could also be interesting. Especially failed calls which add no value.
- Yesterday
-
Ampcode used a handoff feature for a while that I found genuinely useful [1] and then they removed it. Anecdotally, I felt it worked better than compaction.[1]
-
In my experience, the best approach to compaction is to never get to the point where you need compaction and to generally stay below about 30% context window utilization. Even for long agentic workflows this can be accomplished for quite a while, much longer than most people might think.Here's what I do for each of my sessions:1. For asides…
-
OMP has this, it's called "/shake" where bloat from tool calls and such is turned into an artifact ref instead of remaining in the context.There are other pruning or compaction strategies you can configure too.You can also use "/btw" to have side tangent conversations with the current context but with the benefit that it will not bloat your…
-
I have a few variations of pruning (trimming all long tool outputs, agent-directed trimming, etc) in my aichat collection of session tools:
-
I'm very interested in this too. I feel like when my Claude session compacts I immediately have to re-alert it to critical aspects of the task, but it should be possible even to have a secondary low-skill agent crawl over the whole context window semi-continuously and nominate pieces for removal or summarization. Or at a higher level, have…
-
It is kinda a combination of the two:https://github.com/spott/pi-task-compactionI’m still playing with it, but it essentially has the model define a region, and mark it with begin_task, and end_task. End_task also requires a fairly hefty summery of the entire region.It then pulls that entire region and replaces it with the summary. If the model…
-
I think there are a lot of strategies that will open up when costs come down but right now you take a hit on cache rate and thus costs every time you do anything other than wait until the last minute to compact
-
I think the way prompt caching works really discourages more creative compaction techniques. Like perhaps some kind of heuristic progressive compaction that replaces tool results and thinking traces after use with pointers could potentially keep the model smart for much longer, but that'd mean breaking cache every turn, and possibly even within a…
-
OMP changed the default compaction to images! Kinda nuts to read about. Saves the generation cost of the traditional compaction step and writes the context as tiny text to an image, if I was following correctly.
-
Instead of compaction, has anyone seen a successful implementation of pruning? That is, the agent looks at the conversation history and removes any low-value messages.For example, sometimes context will be taken up by a side tangent, tool call outputs, or low-value codebase exploration.Much of the time, I prefer to preserve the history of my…
-
Was expecting the article to go more in-depth.Say, what happens when chain of summaries grows so long, that it still overflows context window. Is summarization runned over the summaries in the context window?
-
I don't like any of current solutions when it comes to compaction. I'd love to have a way to say what exactly should be summarized, because most of the time I just need to compact some noisy MCP tool calls, test runs and things like that. Just let me pick what should be summarized and keep the rest as is.
-
Compaction is painful if you run just one local LLM, the best way to avoid it is to keep context as small as possible.One trick I find useful is to have one model with two KV caches running and while first cache has produced tokens, second cache immediately summarizes them during input tokens are being generated (tools time), then harness switches…