AIlocal-aillama-cppdeepseekmixture-of-expertsapple-siliconopen-source
InferSSD Runs a 763-Billion-Parameter DeepSeek Model on a 48 GB MacBook Pro by Reading Experts From the SSD
WiCi's open-source InferSSD, built on llama.cpp, keeps a giant mixture-of-experts model on disk and streams in the parts each token needs, reaching 2.874 tokens per second on an M5 Max MacBook Pro in the project's own one-run benchmark.
Mixture-of-experts models split their weights into many small expert networks and use only a few for each token. DeepSeek’s model has 763 billion parameters in all, but only 8 to 16 billion are active at a time. InferSSD leans on that. The architecture notes say that on Metal the GPU runs the dense layers and “the sink computes every routed expert on the CPU,” pulling each expert off the drive as it’s needed. On a Mac the reads go through “persistent reader threads with F_NOCACHE,” and Linux uses io_uring.
The README’s table covers four machines and two models. On the MacBook Pro, DeepSeek prefill, the step that reads your prompt, ran at 28.766 tokens per second against llama.cpp’s 5.929. The second model, Muse-Spark-Open at 816 billion parameters with 42 billion active, decoded at 1.097 tokens per second on the Mac, 7.17 times llama.cpp’s 0.153. On a desktop with an RTX 5090, DeepSeek decoded at 7.27 tokens per second.
Install is one curl line on Apple Silicon or on Linux with an NVIDIA GPU. It builds from source, which takes “about 2 to 12 minutes, depending on the machine,” and needs no sudo. Windows is listed as in progress.
The fine print is in the benchmark methodology, and it’s unusually candid. “Each 4K result is one run.” The Mac runs used fresh processes without clearing the system’s file cache, so “these are not controlled cold-cache measurements.” The GGUF file the Mac loaded reports 748.49 billion parameters, and the page says plainly that “the GGUF is not a lossless copy of the complete official release.” WiCi’s product page adds that “these are not fully matched cross-engine tests.” None of the published numbers say how good the answers are, and the architecture notes concede that “throughput alone does not establish model quality.”
Models this size usually need enough GPU memory to hold them. InferSSD trades that memory for SSD reads and patience, and the test machine for its headline number is a laptop. What the project hasn’t measured yet is how hard sustained reading at that rate works an internal SSD, or how much heat a MacBook Pro builds over an hour-long job.