Optimizing large-model inference for speed and safety
🔧 Cloudflare describes techniques to serve demanding long-context models like Moonshot's Kimi and Z.ai's GLM efficiently by reducing memory use and protecting shared caches. They run experiments with SGLang and separate prefill and decode phases. Key optimizations are FP8 KV-cache quantization, INT4 weight compression for decode, and KV cache integrity checks to prevent cross-request corruption, all while preserving model accuracy.
