Blog Network

Huawei Computing Systems Lab · 2026-11-10 · notable

MACaQ — Huawei open-sources a 4-bit quantizer for GGUF, vLLM and Diffusers

MACaQ is Huawei CSL's Apache-2.0 quantizer that calibrates 4-bit weights and activations one layer at a time. Its Qwen3.6-35B-A3B GGUF is closer to FP16 than Unsloth's file of the same size.

MACaQ social cover from Huawei Computing Systems Lab's blog

Huawei's MACaQ shrinks text, image, video and audio models to 4 bits with one layer-by-layer recipe.

What is it?

MACaQ (Model Agnostic Calibrated Quantization) is an open-source toolkit from Huawei Computing Systems Lab that compresses models to 4-bit weights and activations (MXFP4 W4A4) or to 2–4-bit GGUF files. One method covers LLMs, MoE models, FLUX.2 [klein] image, Wan2.2 video and Qwen3-TTS speech models, with backends for llama.cpp, vLLM, Diffusers and PyTorch.

How does it work?

Each linear layer is handled on its own: a per-channel scale and a block-Hadamard rotation are applied to the activations at inference time, and the inverse transforms are folded into the weights, so nothing crosses a layer boundary. A Hessian-aware step called Flip-Refinement then adjusts individual FP4 codes to beat round-to-nearest. The paper reports a 122B model quantized in 64 minutes with 32.5 GB of memory.

Why does it matter?

Most strong quantization methods need model-specific engineering for every new architecture. A layer-local recipe works on new models as they ship. For local users, the README shows a refined Qwen3.6-35B-A3B Q4_K_XL GGUF at KL 0.013367 versus 0.015731 for Unsloth's UD-Q4_K_XL at the same 22.36 GB, and W4A4 Qwen3.8-27B in vLLM decoding 2.58x faster than BF16.

Who is it for?

people running open models locally and inference engineers

Try it

https://github.com/huawei-csl/MACaQ

Sources · 5 outlets

Tags

  • quantization
  • gguf
  • llama-cpp
  • vllm
  • diffusers
  • mxfp4
  • w4a4
  • local-inference
  • huawei
  • open-source

← All releases