In machine learning, pruning is the practice of removing parameters from an existing neural network. The goal of this process is to reduce the size (parameter count) of the neural network (and therefore the computational resources required to run it) while maintaining accuracy. This can be compared to the biological process of synaptic pruning which takes place in mammalian brains during development.

Pruning Pipeline A basic algorithm for pruning is as follows:Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440. Evaluate the importance of each parameter. Rank them according to their importance metric. Generate a pruning mask to optimally remove parameters. Check a termination condition to see whether to continue pruning.

History

Early Pruning Techniques Introduced in the 1980s, Optimal Brain Damage prunes neural networks using the diagonal of the Hessian matrix of the loss function as a scoring mechanism for the "saliency" or importance of a weight to the overall accuracy of the model. Improving on this, Optimal Brain Surgeon uses the inverse of the full Hessian matrix and does lightweight weight-updates to unpruned parameters to minimize error and compensate for pruned weights. However, computing the Hessian and its inverse requires significant computational cost making these methods infeasible for larger networks resulting in pruning being secondary to other compression techniques.

One-Shot Magnitude Pruning In the 2010s, magnitude pruning was introduced as an alternative, one-shot technique which prunes models directly using the magnitude of weights as their ranking for a fast approach to introduce sparsity that does not perform model retraining. Originally used for CNNs and other vision networks, magnitude pruning was sufficient and accurate at pruning networks to high sparsity. This method served as a standard for pruning neural networks; however, its simplicity made it fail to maintain accuracy on Large Language Models especially when pruning to higher sparsity percentages.

Pruning Large Language Models To improve on the scoring, recent methods like SparseGPT and Wanda enhance metrics using local gradient reconstruction and activation-aware scoring, respectively. These methods allow for accurate and efficient approaches to pruning models with negligible degradation even on modern LLMs.

Hardware Efficient Pruning Pruning is heavily constrained by hardware. Much of earlier research focused on methods for scoring and ranking parameters to decide their importance; however, little consideration was given to how to actually compress the model once the weights to prune are decided.

Unstructured Pruning In unstructured pruning, low-importance parameters are chosen to be removed before high-importance ones; however, this creates pruning masks where there is no pattern in how weights are removed. This non-standard pattern makes it difficult to compress models meaningfully in hardware as leaving pruned weights as zeroes does not alleviate its need to be stored and accumulated, resulting in little to no actual improvements when implemented in practice.

Structured Pruning On the other hand, structured pruning methods utilize regular patterns like removing entire structures from networks like neurons or layers to improve on practical efficiency. However, the coarse granularity and unforgiving structure of structured pruning results in significant accuracy loss compared to its unstructured counterpart.

Semi-Structured Pruning Further research developments have been done to decrease the constraints of structured pruning by modifying underlying hardware, allowing for some of the hardware benefits of structured and freedom of unstructured pruning. Nvidia developed a semi-structured pruning format known as 2:4 pruning where for each group of 4 values, two are pruned. This significantly increases flexibility of pruning masks and in turn accuracy of pruned models. This is instantiated in hardware using specialized designs that support these semi-structured sparse matrix operations supported by modern AMD and Nvidia GPUs, allowing for both memory and compute improvements.

See also Quantization Knowledge distillation Model Compression Neural Darwinism

References