Implementation of dynamic aggregation for heterogeneous quantization in federated learning.

The experiements refer to the papers as following. The paper link and GitHub link are given.

• Communication-Efficient Learning of Deep Networks from Decentralized Data : GitHub
• SWALP: Stochastic Weight Averaging in Low-Precision Training : GitHub

Requirments.txt gives the detail requirements.

• Python3
• Pytorch
• Torchvision

• The experiments of FedHQ are run on MNIST and Cifar.
• You can choose download the data through the code.

• --epochs: Number of communication rounds (T in the paper). Default is 150.
• --num_users:Number of clients (n in the paper). Default is 100.
• --frac: Fraction of users to be used for federated updates (C in the paper). Default is 0.1.
• --local_ep: Number of local training epochs in each user (K in the paper). Default is 1.
• --local_bs: Batch size of local updates in each user (B in the paper). Default is 600.
• --lr: Learning rate (η in the paper). Default is 0.1.
• --optimizer: The optimizer used. Default is sgd.
• --momentum: Momentum of optimizer (M in the paper). Default is 0.5.
• --weight_decay: Weight decay of optimizer (λ in the paper). Default: 0.0005.
• --average_scheme: Decide the average scheme. Default is FedHQ.
• --dataset: Name of dataset. Default is mnist.
• --gpu: To use CPU or GPU. Default set 1 to use GPU.
• --iid: Distribution of data amongst clients. Default set 1 for IID.
• --bit_4_ratio: The ratio for 4-bit quantization clients.
• --bit_8_ratio: The ratio for 8-bit quantization clients.

In our experiment, the sum of 'bit_4_ratio' and 'bit_8_ratio' is 1.

The detail results of our experiment refer to the Section 6 of the paper. All the commands are given when running directory is FedHQ folder.

• To run the FedHQ experiment with MNIST under IID condition using GPU:

python src/FedHQ_main.py --dataset=mnist --frac=1 --local_bs=600 --average_scheme=FedHQ --bit_4_ratio=0 --bit_8_ratio=1

• To run the FedHQ experiment with MNIST under non-IID condition using GPU:

python src/FedHQ_main.py --dataset=mnist --iid=0 --frac=1 --local_ep=1 --local_bs=600 --average_scheme=FedHQ --bit_4_ratio=0 --bit_8_ratio=1

Parameters setting as follows(only list the parameters differing from default):

• frac: 1
  Learning-rate decay is 0.9 per ten rounds. The ratios of 4-bit quantization clients are [0,0.2,0.4,0.6,0.8,1].

Table 1: Number of communication round to reach different target accuracy on MNIST dataset, IID partition.

Table 2: Number of communication round to reach different target accuracy on MNIST dataset, non-IID partition.

• To run the FedHQ experiment with MNIST under IID condition using GPU:

python src/FedHQ_main.py --dataset=cifar --epochs=300 --frac=0.1 --local_ep=5 --local_bs=128 --average_scheme=FedHQ --bit_4_ratio=0 --bit_8_ratio=1

• To run the FedHQ experiment with MNIST under non-IID condition using GPU:

python src/FedHQ_main.py --dataset=cifar --epochs=150 --iid=0 --frac=0.1 --local_ep=5 --local_bs=64 --momentum=0.2 --average_scheme=FedHQ --bit_4_ratio=0 --bit_8_ratio=1

Parameters setting as follows(only list the parameters differing from default):

• --epochs: 300 for IID. 150 for non-IID.
• frac: 0.1
• local_ep: 5
• local_bs: 128
  Learning-rate decay is 0.9 per ten rounds. The ratios of 4-bit quantization clients are [0,0.2,0.4,0.6,0.8,1].

Table 3: Number of communication round to reach different target accuracy on CIFAR dataset, IID partition.

Table 4: Number of communication round to reach different target accuracy on CIFAR dataset, non-IID partition.