Question about FedAvg strategy. about flamby HOT 4 CLOSED

Hello @Arctic-Xiangjian, you are completely right we do not average BN parameters thus effectively creating $n$ different models in the sense that they all share the same parameters but have different BN statistics.
However for evaluation we always use the first model so in a way we shoot ourselves in the foot by doing that so our baseline is "fair".
In fact there is no general consensus in the literature on what to do with BN layers in FL settings, you have FedBN (https://arxiv.org/abs/2102.07623) or the SiloedBN work you mentioned (disclaimer I am first author of this paper) and some other approaches but usually in the literature most people use non-batch-wise normalization techniques.
However with siloedBN we have a more private instantiation of Federated Averaging because BN statistics are key to state-of-the-art data reconstruction attacks (https://arxiv.org/abs/2104.07586).
This repository is not meant to force everyone to use one technique or the other but to give the user the liberty to choose what she or he think is more relevant.
Maybe it would be nice to be able to pass how to average BN statistics as an option to the strategy ?

from flamby.

Hello @Arctic-Xiangjian, you are completely right we do not average BN parameters thus effectively creating n different models in the sense that they all share the same parameters but have different BN statistics. However for evaluation we always use the first model so in a way we shoot ourselves in the foot by doing that so our baseline is "fair". In fact there is no general consensus in the literature on what to do with BN layers in FL settings, you have FedBN (https://arxiv.org/abs/2102.07623) or the SiloedBN work you mentioned (disclaimer I am first author of this paper) and some other approaches but usually in the literature most people use non-batch-wise normalization techniques. However with siloedBN we have a more private instantiation of Federated Averaging because BN statistics are key to state-of-the-art data reconstruction attacks (https://arxiv.org/abs/2104.07586). This repository is not meant to force everyone to use one technique or the other but to give the user the liberty to choose what she or he think is more relevant. Maybe it would be nice to be able to pass how to average BN statistics as an option to the strategy ?

Thank you for detailed reply~ The reconstruction paper is very helpful, I am very interested in this direction.

And I was wondering if we do not average the BN layers, we would have different models. May be it will be better when we test the result on each dataset, we can use their own model insteated of Model 0?(If we have a big gap thought different clinents this might be more fair.)

from flamby.

Your idea to use each model on its own distribution is exactly what we are doing in SiloedBN and very close to works on personalization in FL.
It would indeed be probably better in terms of performances but is not what we implemented in the article. Feel free to test it by running something along the lines of (similarly as the personalization example you can find in the repository):

models = strat(**kwargs).run()
perfs = []
for i in range(n_clients):
    perf_dict = evaluate_model_on_tests(models[i], [test_dls[i]], metric)
    perfs.append(perf_dict["client_0"])

from flamby.

@Arctic-Xiangjian closing now that the question is answered. Feel free to reopen an issue if needed.

from flamby.