Faysal AzizPyTorch on your Mac: less memory trouble, more useful local AI
PyTorch 2.15’s release candidate makes Apple-silicon Macs more useful for local AI work. Its MPS backend can store FP8 tensors, runs parts of language-model decoding through faster Metal paths, and fixes costly behaviour in some image and video workloads. These are useful improvements to test—not a promise that every model or Mac suddenly becomes fast enough.
By George the bot
Edited and approved by Faysal Aziz
Published
What changes when memory is tight?
FP8 stores each value in one byte, half the space of BF16 or FP16. PyTorch 2.15 adds float8_e4m3fn tensors on MPS and scaled matrix multiplication on the GPU. That can reduce the storage occupied by suitable weights or intermediate values, but FP8 is lower precision. A model must use appropriate scaling, and you should check output quality against a higher-precision baseline. Merely installing the release candidate does not convert an existing model to FP8 or halve its entire memory footprint.
One fix addresses a more surprising source of memory pressure: constant padding in a tiled video VAE decoder. The Hugging Face PyTorch team measured roughly 270 GiB of peak driver memory on the old MPSGraph path versus about 8 GiB with a Metal implementation for one Wan2.2 VAE decode. That is a measurement of this decoding workload, not of a whole video-generation pipeline, and it should not be generalized to every Mac or video model. The same padding change also fixes an image encode–decode corruption reported for Qwen-Image-2.1 on MPS.
Faster tokens are workload-specific
For language models, PyTorch now connects common single-row linear operations to dedicated matrix-vector kernels. Its new grouped matrix multiplication avoids repeatedly copying mixture-of-experts routing offsets back to the CPU. Softmax over a large vocabulary also gets a dedicated Metal kernel. Together these remove overhead from parts of decoding that run once per generated token.
The team reports strong end-to-end gains for particular mixture-of-experts models on an M3 Ultra with torch.compile, and an approximately eightfold improvement for one large-vocabulary softmax microbenchmark. Neither number is a universal speedup. Model architecture, context length, compilation, precision, memory capacity and Mac chip all matter. If a dense model is already limited by memory bandwidth, expect a smaller gain.
A sensible way to try it
Start in a separate Python environment and follow the PyTorch release-candidate installation instructions for your Mac; do not replace a working production environment. PyTorch 2.15 requires Python 3.11 or newer, and MPS requires macOS 15 or newer. Confirm that `torch.backends.mps.is_available()` is true, then record your current version and run the same model, prompt, context length and precision on both versions.
Measure peak memory and tokens per second after warm-up, synchronizing MPS before timing GPU work. For an image or video workflow, compare the actual output as well as memory use. If you experiment with FP8, compare errors and representative results against BF16, not just byte counts. The new public MPS streams and event timing controls should make closer GPU measurements easier.
The practical gain is more room to experiment locally—and a better way to identify whether a specific bottleneck has been fixed. Keep the release-candidate label and workload-specific evidence attached to any results you share.