FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout

06 Dec 2021 in Projects

Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes a primary problem for fairness, training performance and accuracy. Although significant efforts have been made into tackling statistical data heterogeneity, the diversity in the processing capabilities and network bandwidth of clients, termed as system heterogeneity, has remained largely unexplored. Current solutions either disregard a large portion of available devices or set a uniform limit on the model’s capacity, restricted by the least capable participants. In this work, we introduce Ordered Dropout, a mechanism that achieves an ordered, nested representation of knowledge in deep neural networks (DNNs) and enables the extraction of lower footprint submodels without the need of retraining. We further show that for linear maps our Ordered Dropout is equivalent to SVD. We employ this technique, along with a self-distillation methodology, in the realm of FL in a framework called FjORD. FjORD alleviates the problem of client system heterogeneity by tailoring the model width to the client’s capabilities. Extensive evaluation on both CNNs and RNNs across diverse modalities shows that FjORD consistently leads to significant performance gains over state-of-the-art baselines, while maintaining its nested structure.

Authors: S. Horvath*, S. Laskaridis*, M. Almeida*, I. Leontiadis, S. I. Venieris, N. D. Lane

Published at: Conference on Neural Information Processing Systems (NeurIPS’21), Spotlight (top 3%)

Overview

We proposed Ordered Dropout, a structured dropout technique that produces a nested family of sub-models from a single network. This allows us to extract smaller models without retraining, while preserving a clear connection to low-rank structure.

Building on this idea, we introduced FjORD, a federated learning system that assigns different-sized sub-models to clients based on their capabilities and uses self-distillation to maintain accuracy and fairness. FjORD consistently outperforms prior federated approaches and was recognized as a NeurIPS Spotlight.

Links

• paper
• poster
• talk @ FLOW Seminar
• talk @ flwr Summit
• code

Reference

@inproceedings{horvath2021fjord,
  title     = {FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout},
  author    = {Horv{\'a}th, Samuel and Laskaridis, Stefanos and Almeida, Mario and Leontiadis, Ilias and Venieris, Stylianos I. and Lane, Nicholas Donald},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2021},
  url       = {https://openreview.net/forum?id=4fLr7H5D_eT}
}