Deep neural networks (DNNs) have become indispensable as tools for the processing of image classification, text recognition or language transcription tasks. In practice, however, the models used must always be adapted to the specific challenge in order to deliver the best possible results. Google introduced a new approach under the name of MorphNet last year, which allows the structure of an existing DNN to be developed and optimized automatically, step by step. Now, Google releases MorphNet’s TensorFlow implementation as open source for the community.
Model Optimization through Targeted Regularization
MorphNet supports model optimization among others in terms of scalability and portability. In the course of an iterative process of shrinking and expansion phases, not only can the training costs be reduced, but the models can also be used again for other tasks and re-trained. One of the core functions of MorphNet, however, is the targeted regularization, in which relevant parameters such as FLOPS by inference or the model size are optimized for the intended task. The following graphic illustrates the procedure by way of example using the ResNet 101 model, which was trained with the JFT data set.
Topology Morphing
In topology morphing, MorphNet may remove whole branches from a network. This occurs, for example, when optimizing the neurons per layer, the algorithm should reduce the number of neurons to zero. In the case of ResNet models, MorphNet then removes residual connections; in Inception models, complete parallel strands can be affected.
Depending on the number of optimization cycles that have been taken, MorphNet models can either be improved in terms of FLOP-related costs or trimmed to higher accuracy. Google itself uses MorphNet primarily for image classification in order to be able to downsize DNNs while maintaining the same quality or operate at lower FLOP costs. More details about MorphNet can be found in the blog post , the technical background is described in an article ( PDF download ). If you want to try MorphNet, download the TensorFlow implementation from the GitHub project site.