As artificial intelligence (AI)- or deep-learning-based technologies become more popular,\nthe main research interest in the field is not only on their accuracy, but also their efficiency, e.g., the\nability to give immediate results on the usersâ?? inputs. To achieve this, there have been many attempts\nto embed deep learning technology on intelligent sensors. However, there are still many obstacles in\nembedding a deep network in sensors with limited resources. Most importantly, there is an apparent\ntrade-off between the complexity of a network and its processing time, and finding a structure\nwith a better trade-off curve is vital for successful applications in intelligent sensors. In this paper,\nwe propose two strategies for designing a compact deep network that maintains the required level of\nperformance even after minimizing the computations. The first strategy is to automatically determine\nthe number of parameters of a network by utilizing group sparsity and knowledge distillation (KD)\nin the training process. By doing so, KD can compensate for the possible losses in accuracy caused\nby enforcing sparsity. Nevertheless, a problem in applying the first strategy is the unclarity in\ndetermining the balance between the accuracy improvement due to KD and the parameter reduction\nby sparse regularization. To handle this balancing problem, we propose a second strategy: a feedback\ncontrol mechanism based on the proportional control theory. The feedback control logic determines\nthe amount of emphasis to be put on network sparsity during training and is controlled based on\nthe comparative accuracy losses of the teacher and student models in the training. A surprising fact\nhere is that this control scheme not only determines an appropriate trade-off point, but also improves\nthe trade-off curve itself. The results of experiments on CIFAR-10, CIFAR-100, and ImageNet32 *32\ndatasets show that the proposed method is effective in building a compact network while preventing\nperformance degradation due to sparsity regularization much better than other baselines.
Loading....