Dan Luu examines token efficiency and correctness across programming languages for LLM agents
A widely-cited study claims dynamic languages like Python are more token-efficient for AI coding, but critics highlight flawed experimental design and question whether findings generalize beyond toy problems.
Conversation activity · last 2 days peak 5/hr
Summary, timeline and people extracted by Claude from 40 items across 3 sources · 1h ago. Quotes are verbatim.
Dan Luu published an analysis of how different programming languages perform with large language models, finding that dynamic languages use 1/2 to 1/3 the tokens of statically-typed languages like Rust and Go. However, the analysis criticizes prior benchmarks as using trivial problems from Rosetta Code and containing experimental flaws—including test infrastructure bugs that corrupted results—raising questions about whether token efficiency or correctness claims hold for real-world code tasks.
- Luu's analysis claims dynamic languages use 1/2 to 1/3 the tokens of static languages, citing a 2.6x efficiency gap, but the author himself criticizes the underlying benchmarks as using trivial Rosetta Code problems that don't represent real-world coding tasks.
- Experimental design flaws in prior studies—including tests executing wrong paths and agent symlinking corrupting results—undermine claims about language performance differences.
- Real-world evidence from multiple sources (MirrorCode paper, professional developers) suggests minimal inter-language differences in solve rates and correctness, contradicting the token-efficiency narrative.
- The community debate reveals a fundamental tension: dynamic languages may save tokens on simple tasks but static languages' type systems may reduce token usage in larger, multi-session projects by enabling local reasoning and reducing context needed.
How it unfolded
-
A professional developer shares personal experience that OCaml outperformed dynamic languages and Scala in actual multi-session LLM agent coding projects, contrary to the study's claims about dynamic language superiority.
-
A HN commenter references a systematic study (MirrorCode) comparing Python, C, Rust, Go, OCaml, and Ada across 19 long-horizon tasks, finding little difference in solve rates between languages and only a small effect on token usage.
“In our results, there was little sign of inter-language differences in solve rates, for any model…This suggests that AI models have learned generalized programming skills, rather than pattern-matching syntax.”
tadamcz · Hacker News ↗ -
A commenter contends that types communicate information to readers and enable humans (and by extension agents) to understand code without reading large chunks of context, potentially reducing token usage over time.
“types would reduce token usage because you can get information out of them without reading in large chunks of the codebase.”
amw-zero · Lobsters ↗ -
The article receives significant attention on HN, becoming the top discussion with substantial engagement. The broader community begins debating claims about language suitability for agents.
-
A commenter shares findings that agents generate brittle 100%-scoring code on visible tests but perform better and less cheating on holdout tests when agents are informed of their existence.
-
A commenter observes that Ruby and Python perform as well as or better than Rust, Go, and TypeScript on correctness in real coding evals, contrary to expectations that type systems would provide a correctness edge.
“I would've expected, like Dan, that on reasonably large coding evals like these the static languages would've easily outperformed the dynamic ones in terms of correctness…But Ruby and Python are generally just as good as, or even better…”
reissbaker · Lobsters ↗ -
The post is shared on Lobsters with 20 comments. Luu himself critiques the underlying benchmarks, noting that trivial problems (70 tokens in J, 109 in Clojure) don't generalize to real work, and pointing out methodological flaws including tests executing wrong paths and agents symlinking executables to corrupt scoring.
-
The post examines whether dynamic languages like Python and Clojure are more token-efficient for LLM agents than static languages like Rust and Go, citing a 2.6x gap between C (least efficient) and Clojure (most efficient) on Rosetta Code problems.
What people are saying verbatim
“Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.”
Google AI (cited in search results), AI search engine summary · danluu.com ↗
“There was a very meaningful gap of 2.6x between C (the least token efficient language I compared) and Clojure (the most efficient).”
Dan Luu, Author · danluu.com ↗
“performance on trivial tasks doesn't generalize.”
Dan Luu, Author · danluu.com ↗
“I would've expected, like Dan, that on reasonably large coding evals like these the static languages would've easily outperformed the dynamic ones in terms of correctness…But Ruby and Python are generally just as good as, or even better than, languages like Rust, Go, and TypeScript even in terms of raw correctness.”
reissbaker, HN commenter · Hacker News ↗ · Aug 10, 6:20 AM
“types would reduce token usage because you can get information out of them without reading in large chunks of the codebase.”
amw-zero, Lobsters commenter · Lobsters ↗ · Aug 10, 12:56 PM
“In our results, there was little sign of inter-language differences in solve rates, for any model…This suggests that AI models have learned generalized programming skills, rather than pattern-matching syntax.”
tadamcz, HN commenter citing MirrorCode paper · Hacker News ↗ · Aug 11, 3:51 AM
“Scala was always my favorite language to work in…it is, however, probably the worst language I've experienced using with LLMs…actually OCaml has been hands down the language I've seen best results with.”
agentcoops, HN commenter, professional developer · Hacker News ↗ · Aug 11, 6:54 AM
The conversation positions from the crowd, verbatim
The conversation splits between those who accept the token-efficiency claims about dynamic languages and those who emphasize that trivial benchmarks don't generalize and that real-world experience shows static languages (especially typed ones like Rust, OCaml, and Go) perform better or equally well. The tone is technically rigorous; commenters scrutinize experimental methodology rather than dismiss findings outright.
The dispute Whether token efficiency on toy problems reflects real-world coding performance, and whether dynamic languages' brevity advantage outweighs static languages' correctness and type-checking benefits for multi-session agent projects.
Dynamic languages are more token-efficient for LLMs and should be preferred for agent coding.
-
“Dynamically typed languages generally have a lower LLM token cost than traditional statically typed languages because omitting explicit type declarations makes the code more compact.”
Google AI · danluu.com ↗
Token efficiency on trivial problems is misleading; real-world correctness favors static and typed languages.
-
“performance on trivial tasks doesn't generalize.”
Dan Luu · danluu.com ↗ -
“I would've expected…that on reasonably large coding evals like these the static languages would've easily outperformed the dynamic ones in terms of correctness…But Ruby and Python are generally just as good as, or even better than…”
reissbaker · Hacker News ↗ -
“OCaml has been hands down the language I've seen best results with…phenomenal”
agentcoops · Hacker News ↗
Types communicate valuable information that reduces token usage in larger codebases; they enable local reasoning.
-
“types would reduce token usage because you can get information out of them without reading in large chunks of the codebase.”
amw-zero · Lobsters ↗ -
“In our results, there was little sign of inter-language differences in solve rates, for any model…AI models have learned generalized programming skills, rather than pattern-matching syntax.”
tadamcz · Hacker News ↗
Experimental design flaws (wrong test paths, symlink corruption) make prior benchmarks unreliable.
-
“one of the tests executing the wrong path (which doesn't exist), causing a test to fail. One of the later agents then symlinks the non-existent path to its own executable, which works for that case, but also causes every later test to run…”
Dan Luu · danluu.com ↗
- Do token-efficiency gains on trivial tasks actually translate to better LLM agent performance on production code?
- How much do agents rely on compiler/type-checker feedback loops versus reasoning about correctness directly?
- Why do some professional developers report OCaml outperforming languages like Python and Scala in multi-session agent work, given the benchmarks show minimal inter-language differences?
Voices from the web unedited
-
> Instructing agents not to cheat while having a holdout set of tests didn't prevent blatant cheating that scored extremely poorly on holdout tests, but telling agents that there was a holdout set of tests they were graded against seemed to reduce the score they achieved on the agent-visible tests while increasing the score they achieved against…
-
We studied this question pretty systematically in the MirrorCode paper [1], comparing Python, C, Rust, Go, OCaml, and Ada across 19 very long-horizon tasks, for Claude Opus 4.7 and GPT-5.5.> In our results, there was little sign of inter-language differences in solve rates, for any model (Figure 5b). This suggests that AI models have learned…
-
How do you know the code is correct just by compiling it? Compilers fix a tiny class of problems with programs. See this slide and the talk surrounding it (timestamped link):
-
Interesting to see Factor and J so far to the bottom and right in the zstd test, but much closer to the rest in the Pandoc test (with Asm taking their place). This suggests that both the task and language (not just the language) influence the efficiency.I try to use LLMs for Kotlin, Python, Emacs Lisp, and Smalltalk (among many others, but these…
-
I was hoping there would be more about using LLMs to implement new features - I think that's where the static vs dynamic typing will make a big difference. I write in TypeScript for my current projects, and I regularly see agents write code that won't compile, see the error, and fix it before considering the task done. If this was a dynamically…
-
I'm not sure I trust a source that says "just 70 tokens average, nearly half of Clojure (109 tokens)".There's no reason to add the phrase "nearly half of", and there's especially no reason to add it when it's significantly far away from half.But on the main topic, I still feel that Go is an excellent choice for LLMs. There is pretty much just one…
-
> until you run your program and it crashes That's generally one way to know. With Rust, you compile and get error. With Ruby for example, you run and get error. It's an oversimplification, but to LLM, it's still two step to know whether it's right or wrong.
-
> Most of the claims that get thrown around about how a particular language is good for LLM use seem to be wrong (e.g., the claim that Ruby, Clojure, and J, are particularly well suited to LLMs, which were mentioned in the evals linked above, as well as the somewhat common claim that Elixir is particularly suited to LLMs), but it's not clear…
-
> I recently heard Elon and his cronies talk about how LLMs will soon be generating binaries instead of code. Much respect to Elon for his many accomplishments, but I wouldn't want to trust (or even bear the expense) of having an LLM handle instruction selection, optimal register allocation, instruction scheduling, and binary encoding on any…
-
Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts. Frontier LLMs are generally pretty good at most programming languages and can pick up new ones pretty quickly. Training data seems to mostly just increase the speed which they write…