Model Distillation

Model distillation is the process of transferring knowledge from a large model to a smaller one. While large models (such as very deep neural networks or ensembles of many models) have more knowledge capacity than small models, this capacity might not be fully utilized.

Difference with model compression?

Model compression generally preserves the architecture and the nominal parameter count of the model, while decreasing the bits-per-parameter.