A Review of Joint Optimization Methods for Neural Network Compression

Abstract

As AI models scale, storage, compute, and energy block edge deployment. Single prunin as deep neural networks grow in size and complexity, deploying them on resource-constrained edge devices remains a critical challenge. This paper systematically investigates joint model compression techniques—specifically the integration of pruning, quantization, and knowledge distillation—to achieve efficient inference while preserving accuracy. In this work, I analyze three hybrid frameworks: pruning with distillation, quantization with distillation, and pruning with quantization. By comparing sequential and parallel optimization strategies, this work demonstrates that joint methods consistently outperform single-compression approaches, delivering higher compression ratios and better accuracy retention across benchmark models. Our results further reveal that co-optimizing multiple compression dimensions enables more effective model scaling-down, yet challenges such as stage-wise optimization gaps and hardware-aware design remain. This study not only synthesizes recent advances but also outlines practical pathways toward lightweight, hardware-friendly neural networks for edge AI deployment, highlighting the importance of integrated compression in real-world applications. g, quantization, or distillation struggles with accuracy vs. hardware trade-offs, pushing “pruning-quantization-distillation” combos. This paper surveys core compressors, summarizes three joint schemes—distillation + pruning, distillation + quantization, pruning + quantization—outlines their ideas, and charts future directions.