FreeToken Just Shipped. Is It Better Than Ollama and llama.cpp?

Testing the radical new MoE serving engine against the reigning champions on a 6GB laptop FreeToken shipped recently. As a new serving engine for Mixture of Experts (MoE) models, its headline claim of running a 753B parameter model on a single GPU garnered a lot of attention fast. I have a laptop with an RTX 3050 and 6GB of VRAM. I installed FreeToken on it alongside Ollama and llama.cpp, ran all three against the same model and the same three questions, and measured tokens per second and time to first token. This piece is a breakdown of that comparison. But more than that, it is an attempt to explain what each of these three tools is actually doing under the hood, because the performance numbers only make sense once you understand the underlying mechanism. Three Engines, Two Designs An MoE model has a very large number of parameters, but only a small number of them i.e. the “experts” gets used for any given token. Most of the model sits idle at any moment. The three engines disagree about what to do with those idle weights. llama.cpp and Ollama share the same answer: Split the model once, at load time, and leave the split alone. Some layers go on the GPU, and the rest go to system RAM and run on the CPU. With llama.cpp, you set that split yourself with a flag. Ollama runs llama.cpp underneath, but it looks at your hardware first and picks the split for you. How llama.cpp and Ollama split a model across CPU and GPU. (Image by Author) I saw this directly. Running ollama ps while a model was loaded showed a 71% CPU and 29% GPU split, chosen automatically. With llama.cpp, I had to choose, and I chose badly on my first attempt. More on that shortly. FreeToken’s answer is different: Do not commit to one split at all. According to the team’s August 2026 paper, “ FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution ”, FreeToken keeps the complete set of experts in CPU RAM as the source of truth. It then turns the GPU’s spare memory into a shared LRU cache that holds whichever experts were recently used. When the model needs an expert that is not in that VRAM cache, FreeToken uses a “bandwidth adaptive” policy i.e. it decides on the spot whether to pull the weights over PCIe or compute them directly on the CPU, based on the actual bandwidth it measures on your machine. FreeToken’s dynamic expert cache and routing mechanism. (Image by Author) This is a genuinely more complicated thing to build than a fixed split. That complexity is the entire story of what follows. Setup Hardware: GPU: RTX 3050 Laptop GPU, 6GB VRAM CPU: Intel i5–13450HX (6 performance cores, 4 efficiency cores) RAM: 24GB OS: Windows 11 (all three engines running natively) The Model: gpt-oss-20b in MXFP4 It takes up about 13GB on disk and was identical across all three engines. I checked rather than assumed: Ollama reports MXFP4 for its build, and the GGUF I used for llama.cpp is Unsloth’s MXFP4 quant, published under an F16 filename with a note in their README explaining why. I ran everything natively on Windows rather than WSL. FreeToken’s command line version stalled partway through loading under WSL, which shares memory with Windows and left too little room. The desktop app worked fine natively, so I moved everything to Windows rather than testing one engine on Linux and two on Windows, which would have measured the platform instead of the engines. The Prompts: I tested the engines against three distinct prompts to evaluate different types of generation: a direct factual answer, logical reasoning, and longer prose. Short explainer: “Explain the difference between MoE and dense models in three sentences.” Trick question: “Seventeen sheep, all but nine die; how many are left?” Long explainer: “Write two hundred words on how MoE routing works.” I ran each prompt four times, prefacing the sequence with a throwaway “hi” so that cold-start times wouldn’t skew the results. The numbers reported below are the medians. Because all three engines expose an OpenAI-compatible API on localhost, I drove the entire benchmark using a single Python script. This guaranteed that the timing code was perfectly identical across all three contenders. Results Generation speed by engine across three different prompt types. (Image by Author) Results (Image by Author) Time to first token (TTFT) compared across engines. (Image by Author) Ollama started answering in about 0.38 seconds . llama.cpp took 0.5 seconds . FreeToken took 1.17 seconds, roughly three times the wait. Reading the Results Against the Mechanism None of this is surprising once you look at how each engine operates. Ollama and llama.cpp, with the split set correctly, do almost nothing extra during generation. The layers are where they are, the boundary does not move, and the only cost is the fixed computational cost of some layers living on a slower device. FreeToken, on the other hand, is doing continuous work that the other two skip entirely: checking whether an expert is cached, deciding what to evict, measuring bandwidth, and calculating whether a miss goes over PCIe or gets computed on the CPU. On a 20B model, that overhead has almost nothing to buy back. A 20B model’s experts do not change fast enough, or often enough, to make dynamic caching pay for itself. You are paying FreeToken’s machinery cost with none of its benefit. The 1.17 second wait before the first token fits the exact same story. That startup work, checking the cache state and deciding on a routing strategy happens before generation can begin. It is the same overhead showing up as latency instead of throughput. Where FreeToken’s Approach Could Actually Pay Off FreeToken lost every measurement here, but I do not think a 20B model on a 6GB laptop is a fair test of what it was built to do. The paper behind FreeToken is explicit that its target is agent workloads whose execution patterns keep changing, scaling up to workstation GPUs, not a single 20B model answering three static questions. Its own model list makes this point without needing the paper: when I checked what else was available in the desktop app, a 35B model and a 27B model were both marked “Insufficient RAM” on my 24GB machine. That matters for the argument I want to make, because it also limits it: FreeToken’s own numbers show it needs more system RAM than I have just to load a 35B model, well before you reach the scale where its caching and bandwidth-adaptive machinery is supposed to start earning its keep. I cannot tell you FreeToken wins on bigger models, because I could not get a bigger model to load on this machine under any engine, FreeToken included. The 753B headline that got everyone’s attention sits well outside anything I was able to test. What I can say is narrower and more honest: the overhead I measured losing here is not wasted effort in general, it is effort with no problem to solve at this specific size. Whether it becomes worth its cost on a workstation running a model that genuinely does not fit any other way is an open question and it is the one question this test could not answer. What I Would Use Tomorrow On a hardware like mine, running models in this size range, my pick is Ollama . It was the fastest, picked sensible settings without being asked, and took about four minutes to set up. If you want control over the settings yourself, llama.cpp gets within the Ollama range once configured properly and may even outperform it. That performance gap is simply the price of doing it by hand. FreeToken is something I would revisit with a machine that has enough RAM to load something the other two cannot. That is the test its design is actually built for, and a 6GB laptop with 24GB of RAM simply cannot run it. Practical Notes & Loose Ends If you plan on benchmarking this yourself, keep a few oddities in mind: FreeToken on Windows: Use the desktop app, not the command line path. My first load failed with an error saying it needed about 1.01GB for its cache and had 756MB available. Raising memory_ratio from 0.9 to 0.95 in settings fixed it. The error told me exactly what to change. llama.cpp on Windows: Setting this up took longer than everything else combined. The binaries are not code-signed, and Windows Memory Integrity blocked the DLLs with a Bad Image error that produced no output at all in a normal terminal. I only saw the real error by running as administrator. Turning Memory Integrity off in Windows Security fixed it. The benchmarking script, the raw numbers, and both diagrams are available on GitHub . I would love to see this run on a machine with real RAM headroom, since that is the test I couldn’t do. References Yang, S., Fan, X., Pan, M., et al. (2026). “FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution.” arXiv preprint. arXiv:2608.16157 . FlashML / FreeToken GitHub Repository: The open-source edge-native MoE serving engine. Available at: https://github.com/FlashML-org/FreeToken llama.cpp GitHub Repository: Port of Facebook’s LLaMA model in C/C++. Available at: https://github.com/ggml-org/llama.cpp Ollama: The official runtime and model manager for local AI. Available at: https://ollama.com FreeToken Just Shipped. Is It Better Than Ollama and llama.cpp? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.