Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering

Federated learning is generally used in cross-device settings if a task has readily available labels (e.g., next word prediction). Relaxing this constraint requires design of unsupervised learning methods that can support desirable properties for federated training: robustness to statistical/systems heterogeneity, scalability with number of participants, and communication efficiency. Prior work on this topic has focused on directly extending centralized self-supervised learning techniques, which are not designed to have the properties listed above. To address this situation, we propose emph{Orchestra}, a novel unsupervised federated learning technique that exploits the federation's hierarchy to orchestrate a distributed clustering task and enforce a globally consistent partitioning of clients' data into discriminable clusters. We show the algorithms underlying Orchestra guarantee good generalization performance under a linear probe, allowing it to outperform alternative techniques in a broad range of conditions, including variation in heterogeneity, number of clients, participation ratio, and local epochs.