Annie Sexton explains how compression and LLMs solve the same problem
ngrok developer educator publishes guide arguing that data compressors and large language models are functionally equivalent at their core.
Conversation activity · last 6 hours peak 2/15m
Summary, timeline and people extracted by Claude from 18 items across 4 sources · 44m ago. Quotes are verbatim.
Annie Sexton, a developer educator at ngrok, published a technical guide titled "Compression Is Prediction" arguing that compressors and large language models fundamentally solve the same problem: modeling probability distributions over data to minimize representation size. The post walks through compression basics—minification, run-length encoding, transforms, models, and entropy coders—to show how LLMs operate as sophisticated compression algorithms.
- Sexton argues that compression algorithms and LLMs both model probability distributions to minimize representation size—the core problem is equivalent.
- HN discussion surface significant prior work: Shannon's information theory, David MacKay's frameworks, Grant Sanderson's video series, and Schmidhuber's compression progress principle.
- Key tension: commenters debate whether the equivalence holds only for representative data distributions or generalizes to out-of-distribution scenarios.
- Some commenters criticize the post for inadequate attribution and suggesting novelty of an established idea from information theory.
How it unfolded
-
A commenter notes that ngrok's blog content—particularly work from Sam Rose and now Sexton—has become more compelling than their products.
-
A commenter expresses concern that the post presents established ideas without adequate citation or acknowledgment of prior work, suggesting it creates impression of novel discovery.
“I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it”
chermi · Hacker News ↗ -
Analysis Information theory grounding cited
A commenter frames the compression-prediction link as rooted in Shannon's information theory and standard computer science curricula.
-
A commenter notes parallels between the compression framework and geometric packing discoveries in neural networks, particularly regarding sparse feature representation.
-
Some commenters argue the equivalence holds only under specific conditions (representative data distribution) and breaks down when generalization is required, while others defend the framework as a useful intuition for understanding LLM capabilities.
“Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems.”
ssivark · Hacker News ↗ -
HN commenters reference Grant Sanderson's video series on compression and intelligence, David MacKay's information theory work, and Jürgen Schmidhuber's compression progress principle, suggesting the core idea is established in academic literature.
“Grant Sanderson has an excellent video on the same topic. It's part of a series that is ongoing.”
sheeeeesh · Hacker News ↗ -
Sexton, a developer educator at ngrok with prior experience at Heroku, Render, and Fly.io, publishes a technical article explaining the connection between data compression and language models.
-
The post reaches Hacker News frontpage with 198 points and 91 comments by the timestamp, indicating significant developer interest in the compression-prediction thesis.
What people are saying verbatim
“compressors and LLMs are, at their core, trying to solve the exact same problem”
Annie Sexton, Developer Educator at ngrok · ngrok blog ↗
“Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems.”
ssivark, HN commenter · Hacker News ↗
“I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it”
chermi, HN commenter · Hacker News ↗
“This perspective is a useful source of intuition against the "LLMs can't have new ideas, they're just next-token-predictors" style arguments.”
throwaway_7274, HN commenter · Hacker News ↗
“Grant Sanderson has an excellent video on the same topic. It's part of a series that is ongoing.”
sheeeeesh, HN commenter · Hacker News ↗
“Information theory and machine learning still belong together. Brains are the ultimate compression and communication systems.”
farfatched, HN commenter (quoting Cambridge course) · Hacker News ↗
The conversation positions from the crowd, verbatim
The HN discussion centers on validating (and contextualizing) Sexton's core thesis. Most commenters affirm the compression-prediction connection but highlight that this is well-established in information theory and machine learning; the energy goes toward excavating prior work and debating the scope of the equivalence, with a minority criticism about attribution and novelty framing.
The dispute Whether compression-prediction equivalence holds universally or only under the condition that training and test data distributions align.
The compression-prediction equivalence is sound and illuminating for understanding LLMs, though rooted in established theory.
-
“This perspective is a useful source of intuition against the "LLMs can't have new ideas, they're just next-token-predictors" style arguments.”
throwaway_7274 · Hacker News ↗ -
“This is basically a thesis supported by Shannon's information theory. Any rigorous CS program should cover this in depth.”
melenaboija · Hacker News ↗
The equivalence has important nuances and breaks down when generalization to out-of-distribution data is required.
-
“Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization.”
ssivark · Hacker News ↗
The post inadequately cites prior work and frames established ideas as novel without acknowledging their sources.
-
“I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it”
chermi · Hacker News ↗
- How does the compression framework account for LLM behavior on adversarial or out-of-distribution inputs?
- Does ngrok intend this as educational exposition of established theory, or as novel research contribution?
Voices from the web unedited
-
Nope; there is a bit more nuance and the distinction is important.Compression is functionally equivalent to prediction when the data distribution is exactly representative of all future problems. The story changes drastically if you want generalization -- because the test distribution could be arbitrarily different, even if it had the same…
-
Compression is prediction Link: https:// ngrok.com/blog/compression-is- prediction Comments: https:// news.ycombinator.com/item?id=4 9263497
-
This is the thesis behind the "Information Theory, Inference, and Learning Algorithms" course that was taught at Cambridge University.> Why unify information theory and machine learning? Because they are two sides of the same coin. In the 1960s, a single field, cybernetics, was populated by information theorists, computer scientists, and…
-
Compression is prediction: https:// ngrok.com/blog/compression-is- prediction Discussion: http:// news.ycombinator.com/item?id=4 9263497
-
I stumbled across a connection between LLMs and compression when researching N-dim polytope emergence in neural networks. Toy Models of Superposition (Anthropic, 2022) suggests that gradient descent can independently discover efficient geometric packing arrangements for sparse features. LVQ compression uses regular lattice structures, including…
-
This is a lot less surprising when you learn how non-LZ compressors work, that is, by modeling a probability distribution and using those probabilities to encode information in the minimum number of bits required to transmit the data. A less obvious conclusion is that LZ compressors do this to implicitly, the length of each symbol they could emit…
-
I had a long ranting comment I deleted. I just don't like this trend of people presenting work in a way that makes you think some combo of 1) they discovered from scratch themselves 2) it's new 3) they didn't try to cite or acknowledge where they learned it/point to good sources 4) they don't really care about trying to teach something deeply…
-
Ah Sir David MacKay. I so respect him. Great explainer and speaker. He had built this text entry tool called Dasher [0] - that I'd heard him introduce at Princeton around 2006. It was basically an early language model that predicted which characters are more likely than others, given what you've already types and it would adjust the sizes of the…
-
This perspective is a useful source of intuition against the “LLMs can’t have new ideas, they’re just next-token-predictors” style arguments. What if you shift your perspective to thinking of training as optimization over a vast parametrized family of compression algorithms? Well, it suddenly looks a lot more plausible that “new” “ideas” can…
-
Schmidhuber did it first:Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes