HAPI: Hardware-Aware Progressive Inference
Convolutional neural networks (CNNs) have recently become the state-of-the-art in a diversity of AI tasks. Despite their popularity, CNN inference still comes at a high computational cost. A growing body of work aims to alleviate this by exploiting the difference in the classification difficulty among samples and early-exiting at different stages of the network. Nevertheless, existing studies on early exiting have primarily focused on the training scheme, without considering the use-case requirements or the deployment platform. This work presents HAPI, a novel methodology for generating high-performance early-exit networks by co-optimising the placement of intermediate exits together with the early-exit strategy at inference time. Furthermore, we propose an efficient design space exploration algorithm which enables the faster traversal of a large number of alternative architectures and generates the highest-performing design, tailored to the use-case requirements and target hardware. Quantitative evaluation shows that our system consistently outperforms alternative search mechanisms and state-of-the-art early-exit schemes across various latency budgets. Moreover, it pushes further the performance of highly optimised hand-crafted early-exit CNNs, delivering up to 5.11× speedup over lightweight models on imposed latency-driven SLAs for embedded devices.
Authors: S. Laskaridis*, S. I. Venieris*, H. Kim, N. D. Lane
Published at: International Conference on Computer-Aided Design (ICCAD’20)
Overview
We developed HAPI to systematically design early-exit networks that are optimized for specific hardware targets and latency constraints. Rather than treating exit placement and inference policy independently, HAPI co-optimizes both using efficient design-space exploration.
Across multiple embedded platforms, HAPI consistently outperforms prior early-exit strategies and lightweight models, achieving significant speedups while meeting strict latency SLAs.
Links
Reference
@inproceedings{laskaridis2020hapi,
title={HAPI: Hardware-aware progressive inference},
author={Laskaridis, Stefanos and Venieris, Stylianos I and Kim, Hyeji and Lane, Nicholas D},
booktitle={Proceedings of the 39th International Conference on Computer-Aided Design},
pages={1--9},
year={2020}
}
