MobiSR

In recent years, convolutional networks have demonstrated unprecedented performance in the image restoration task of super-resolution (SR). SR entails the upscaling of a single low-resolution image in order to meet application-specific image quality demands and plays a key role in mobile devices. To comply with privacy regulations and reduce the overhead of cloud computing, executing SR models locally on-device constitutes a key alternative approach. Nevertheless, the excessive compute and memory requirements of SR workloads pose a challenge in mapping SR networks on resource-constrained mobile platforms. This work presents MobiSR, a novel framework for performing efficient super-resolution on-device. Given a target mobile platform, the proposed framework considers popular model compression techniques and traverses the design space to reach the highest performing trade-off between image quality and processing speed. At run time, a novel scheduler dispatches incoming image patches to the appropriate model-engine pair based on the patch's estimated upscaling difficulty in order to meet the required image quality with minimum processing latency. Quantitative evaluation shows that the proposed framework yields on-device SR designs that achieve an average speedup of 2.13x over highly-optimized parallel difficulty-unaware mappings and 4.79x over highly-optimized single compute engine implementations.

• Paper: Official Preprint
• Date: October 2019
• Venue: 25th Annual International Conference on Mobile Computing and Networking (MobiCom), 2019.
• Domains: Super-resolution, Deep Neural Networks, Mobile Computing, Heterogeneous Computing, Scheduling

ShrinkML

End-to-end automatic speech recognition (ASR) models are increasingly large and complex to achieve the best possible accuracy. In this paper, we build an AutoML system that uses reinforcement learning (RL) to optimize the per-layer compression ratios when applied to a state-of-the-art attention based end-to-end ASR model composed of several LSTM layers. We use singular value decomposition (SVD) low-rank matrix factorization as the compression method. For our RL-based AutoML system, we focus on practical considerations such as the choice of the reward/punishment functions, the formation of an effective search space, and the creation of a representative but small data set for quick evaluation between search steps. Finally, we present accuracy results on LibriSpeech of the model compressed by our AutoML system, and we compare it to manually-compressed models. Our results show that in the absence of retraining our RL-based search is an effective and practical method to compress a production-grade ASR system. When retraining is possible, we show that our AutoML system can select better highly-compressed seed models compared to manually hand-crafted rank selection, thus allowing for more compression than previously possible.

• Paper: Official Preprint
• Date: September 2019
• Venue: Interspeech, 2019.
• Domains: ASR, Compression, Embedded AI, ML

Deployment of DNNs on IAS

With their unprecedented performance in major AI tasks, deep neural networks (DNNs) have emerged as a primary building block in modern autonomous systems. Intelligent systems such as drones, mobile robots and driverless cars largely base their perception, planning and application-specific tasks on DNN models. Nevertheless, due to the nature of these applications, such systems require on-board local processing in order to retain their autonomy and meet latency and throughput constraints. In this respect, the large computational and memory demands of DNN workloads pose a significant barrier on their deployment on the resource-and power-constrained compute platforms that are available on-board. This paper presents an overview of recent methods and hardware architectures that address the system-level challenges of modern DNN-enabled autonomous systems at both the algorithmic and hardware design level. Spanning from latency-driven approximate computing techniques to high-throughput mixed-precision cascaded classifiers, the presented set of works paves the way for the on-board deployment of sophisticated DNN models on robots and autonomous systems.

• Paper: Official Preprint
• Date: July 2019
• Venue: IEEE Computer Society Annual Symposium on VLSI (ISVLSI), 2019
• Domains: Autonomous Systems, Intelligent Systems, Embedded Processing, FPGAs, Robotics, On Board Processing, Deep Learning, Convolutional Neural Networks, Long Short Term Memory Networks

Approximate LSTMs for Time-Constrained Inference

The need to recognise long-term dependencies in sequential data such as video streams has made Long Short-Term Memory (LSTM) networks a prominent Artificial Intelligence model for many emerging applications. However, the high computational and memory demands of LSTMs introduce challenges in their deployment on latency-critical systems such as self-driving cars which are equipped with limited computational resources on-board. In this paper, we introduce a progressive inference computing scheme that combines model pruning and computation restructuring leading to the best possible approximation of the result given the available latency budget of the target application. The proposed methodology enables mission-critical systems to make informed decisions even in early stages of the computation, based on approximate LSTM inference, meeting their specifications on safety and robustness. Our experiments on a state-of-the-art driving model for autonomous vehicle navigation demonstrate that the proposed approach can yield outputs with similar quality of result compared to a faithful LSTM baseline, up to 415x faster (198x on average, 76x geo. mean).

• Paper: Preprint
• Date: July 2019
• Venue: IEEE Consumer Electronics Magazine, 2019
• Domains: Autonomous Systems, Intelligent Systems, Embedded Processing, FPGAs, Robotics, On Board Processing, Deep Learning, Convolutional Neural Networks, Long Short Term Memory Networks

EmBench

In recent years, advances in deep learning have resulted in unprecedented leaps in diverse tasks spanning from speech and object recognition to context awareness and health monitoring. As a result, an increasing number of AI-enabled applications are being developed targeting ubiquitous and mobile devices. While deep neural networks (DNNs) are getting bigger and more complex, they also impose a heavy computational and energy burden on the host devices, which has led to the integration of various specialized processors in commodity devices. Given the broad range of competing DNN architectures and the heterogeneity of the target hardware, there is an emerging need to understand the compatibility between DNN-platform pairs and the expected performance benefits on each platform. This work attempts to demystify this landscape by systematically evaluating a collection of state-of-the-art DNNs on a wide variety of commodity devices. In this respect, we identify potential bottlenecks in each architecture and provide important guidelines that can assist the community in the co-design of more efficient DNNs and accelerators.

• Paper: Official Preprint
• Date: June 2019
• Venue: 3rd International Workshop on Embedded and Mobile Deep Learning (EMDL), MobiSys, 2019
• Domains: Embedded AI, Benchmarking

All publications

• MobiSR: Efficient On-Device Super-resolution through Heterogeneous Mobile Processors - Royson Lee, Stylianos I. Venieris, Lukasz Dudziak, et al. • 25th Annual International Conference on Mobile Computing and Networking (MobiCom), 2019.
• ShrinkML: End-to-End ASR Model Compression Using Reinforcement Learning - Łukasz Dudziak, Mohamed S. Abdelfattah, Ravichander Vipperla, et al. • Interspeech, 2019.
• Towards Efficient On-Board Deployment of Deep Neural Networks on Intelligent Autonomous Systems - Alexandros Kouris, Stylianos I. Venieris, Christos-Savvas Bouganis • IEEE Computer Society Annual Symposium on VLSI (ISVLSI), 2019
• Approximate LSTMs for Time-Constrained Inference: Enabling Fast Reaction in Self-Driving Cars - Alexandros Kouris, Stylianos I. Venieris, Michalis Rizakis, et al. • IEEE Consumer Electronics Magazine, 2019
• EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices - Mario Almeida, Stefanos Laskaridis, Ilias Leontiadis, et al. • 3rd International Workshop on Embedded and Mobile Deep Learning (EMDL), MobiSys, 2019