Step-by-step guide to building and comparing FedAvg and FedProx on non-IID CIFAR-10 with NVIDIA FLARE
In this tutorial, we build an advanced federated learning experiment using NVIDIA FLARE. We compare two popular federated optimization algorithms: Federated Averaging (FedAvg) and Federated Proximal Gradient Descent (FedProx), on a non-IID CIFAR-10 dataset. The data is split using the Dirichlet distribution to simulate realistic label imbalance across federated sites.
Setup
We install necessary libraries and import essential packages for our experiment, including TensorFlow, PyTorch, and Matplotlib. We define key parameters such as the number of clients, training rounds, batch size, learning rate, and non-IID Dirichlet alpha value. Additionally, we create directories for data and results storage, download CIFAR-10 once to ensure all simulated clients have access to a consistent dataset.
Client-Side Training Script
We create a separate Python script for the client-side training process, which is imported by NVFlare. The script includes a small CNN model for CIFAR-10 classification and defines a deterministic function to partition data into non-IID shards. We also define an evaluation function that measures the global model’s accuracy on the shared CIFAR-10 test set.
Server-Side Experiment Definition
We use NVIDIA FLARE’s Job API to configure and launch our federated learning experiment. We create a FedAvg job, attach the client training script to each simulated site, and specify various parameters such as the number of clients, training rounds, batch size, learning rate, and non-IID Dirichlet alpha value. The experiment is run with both FedAvg and FedProx algorithms, allowing us to compare their performance on this non-IID dataset.
Key takeaways
- The use of NVIDIA FLARE simplifies the setup and management of federated learning experiments.
- Federating a small CNN model across clients with non-IID data demonstrates how different optimization methods can affect performance.
- This experiment provides insights into how label imbalance impacts federated learning outcomes.
Stay ahead of AI. Get the most important stories delivered to your inbox — no spam, no noise.
