Run Gemma 4 26B on 2 GB RAM via Metal API

Running a 26B parameter model usually requires a beefy workstation or a cloud budget you aren't prepared to spend. Most of us just accept that high-parameter inference is out of reach for a base 8GB M-series Mac. But a custom Swift runtime is changing that, making it possible to actually run these models on hardware that should, by all rights, be choking.

The setup is surprisingly lean. The Swift Package Manager handles the tokenizer builds, and the final release is just a Mac app paired with a decode-service executable. It doesn't even force you to materialize a full source checkpoint on your drive, which is a relief given how tight storage is on these machines.

I'm still figuring out if this is a sustainable way to handle memory, or if we're just pushing the hardware to a breaking point. The model lives in a scratch directory, and the app handles the installation sizes upfront so you aren't guessing. It works, but the real question is whether the performance trade-offs are worth the convenience of local execution.

The 2 GB Memory Constraint

Running a 26B parameter model in 2 GB of RAM is a technical anomaly. Normally, a model of this size requires significantly more memory just to hold the weights, let alone handle the KV cache during inference. This implementation manages it by using a custom Swift and Metal runtime that bypasses the standard memory overhead found in heavier frameworks like PyTorch. It's a clever bit of engineering that makes high-parameter models viable on entry-level Apple Silicon, specifically the 8 GB MacBooks.

The magic here is in how the runtime interacts with the GPU. Instead of loading the entire model into active memory, it manages the budget aggressively. This part is genuinely confusing because the math for 26B parameters usually doesn't add up to 2 GB, even with heavy quantization. It's likely using a highly optimized paging or streaming mechanism that keeps only the necessary tensors active in the Metal buffers.

To get this running, you have to repack the model into the .gturbo format. This is a prerequisite for the runtime to handle the memory constraints correctly.

Swift run -c release TurboFieldfareRepack \
  --output scratch/gemma4.gturbo \
  --overwrite

If you're on an 8 GB machine, this is a huge win. You're no longer locked out of larger models because of a hardware ceiling. I'm not sure if this is the most efficient way to run models if you actually have 64 GB of RAM, but for the "MacBook Air" crowd, it's a practical way to get 26B-scale reasoning without the system swapping to disk and crawling to a halt.

Practical Implementation

The most impressive part of this setup is that Gemma 4 26B-A4B runs in about 2 GB of RAM. It's a custom Swift and Metal runtime designed for Apple Silicon, which means it actually works on 8 GB Macs without swapping to death. I'm still wrapping my head around how a 26-billion-parameter model fits into that kind of budget, but the math checks out in practice.

To get the model running, you have to prepare it using TurboFieldfareRepack. If you're doing a fresh install, use the --discard-partial flag. This ensures you aren't building on top of a corrupted or incomplete previous attempt.

Swift run -c release TurboFieldfareRepack --discard-partial --output scratch/gemma4.gturbo

If the repack process crashes or your laptop dies mid-way, you don't have to start from zero. The --resume flag picks up where the process stopped. If you've already got a file at the output path and just want to start over, use --overwrite.

Swift run -c release TurboFieldfareRepack --output scratch/gemma4.gturbo --overwrite --resume

Architecture and Performance

The 26B-A4B variant of Gemma 4 runs in about 2 GB of RAM. That's an aggressive memory budget for a model of this size, but it makes the model usable on Apple Silicon Macs with only 8 GB of unified memory. It's a custom Swift and Metal runtime that bypasses the usual overhead of heavier frameworks. I'm not entirely sure how they squeezed a 26B parameter model into that footprint without catastrophic quality loss, but the benchmarks suggest it works.

The system splits the workload between a foreground Mac app and a separate decode-service executable. This isn't just for organization; the decode-service handles the heavy lifting of the Metal kernels to keep the UI from freezing. If you're setting up the model for the first time, you have to repack the weights into a specific format the runtime understands.

The performance is impressive, but there's a trade-off. This runtime is optimized for low-memory environments. If you have a Mac Studio with 128 GB of RAM, this engine is probably the wrong choice because it's designed to survive on a budget, not to maximize the throughput of high-end hardware.

Installation and Model Setup

The decision to bundle a separate decode-service executable alongside the Mac app is the real story here. It suggests the developers aren't treating the model as a simple library call, but as a persistent piece of infrastructure. By decoupling the decoding logic from the UI, they're avoiding the typical "app freeze" when the model hits a heavy computation spike. I suspect this is also a hedge for future-proofing; if they want to move the engine to a local server or a different process entirely, the plumbing is already there.

I've seen a few people on X praising the expert caching, but I think the excitement is premature. Caching is great in a vacuum, but in practice, it usually comes down to how the model handles context switching. If the "expert switching" the community is asking about involves heavy memory swapping, any gains from the cache will be eaten by latency. I'm not convinced yet that this will feel seamless on base-model MacBooks with limited unified memory.

The real question is whether this architecture allows for actual hardware flexibility or if it's just a polished wrapper for Apple Silicon. If the decode-service is too tightly coupled to Metal, the "various hardware" hopes the community has are dead on arrival. I want to know if this can run on a machine without a M-series chip, or if we're just looking at a very efficient way to use a Neural Engine.

Conclusion

Squeezing a 26-billion-parameter model into 2 GB of RAM is a hell of a feat, and the custom Swift + Metal runtime actually pulls it off. It's a relief to see a tool that doesn't immediately crash an 8 GB MacBook Air the moment you try to load a decent-sized model.

I'm still curious if the trade-offs for this memory budget hit the actual utility of the model's reasoning. It's one thing to get the decode-service running; it's another to see if the output remains coherent when you're pushing the limits of the hardware this hard.

If you have an Apple Silicon Mac gathering dust, go ahead and point it at scratch/gemma4.gturbo and see for yourself. Does the speed hold up once you actually start hitting it with long-context queries?