llama.cpp
Conversation activity · last 7 hours peak 1/15m
Clustered from 16 items across 1 source. Not yet parsed — the coverage below is the raw record.
Social posts 1
Voices from the web unedited
-
llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit.Two examples:- https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other…
-
Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...).Git clone llama.cpp and build it, it's not…
-
This MacOS app used to be called LlamaBarn. Really excellent to see the fast progress being made.Official repo, also has documentation how to configure server parameters:https://github.com/ggml-org/Llama-macOSSmall tip, install llama.cpp with brew before llama.app, which will pick up the existing llama.cpp. That way it's easier to stay up to date…
-
Old news by now, but you might not be aware that llama-server can do multi-model for a while now,Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that…
-
I tried curl -LsSf https://llama.app/install.sh | sh and then llama serve -hf unsloth/Qwen3-4B-GGUF:Q4_0 Then I get: W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden Terminated And the web interface says Server unavailable Maybe it gets killed by the OS because it…
-
Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm…
-
Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.
-
Vanilla llama.cpp leaves a lot of performance on the table. I'm reaching 120 t/s with a custom inference engine for a model that llama.cpp can barely run at 70 t/s. Theoretical maximum on this hardware is around 147 t/s according to measured memory bandwidth.
-
Caveat: this kind-of-works. There are plenty of subtle bugs around this. For example, it's hard to have a default model, and clients need to select the model explicitly in every API request, which makes it difficult to work with several models and multiple clients.
-
I think I can probably run Gemma 3 12B on my macbook M3 pro with 18GB. The question is, should I do it? This small model is probably not capable of doing a lot or advanced coding or reasoning. What else could it be used for, since it can run locally and privately?