Benchmarking Track's WP2: How do we ensure reproducibility and robust comparisons ? about flamby HOT 4 CLOSED

Thanks @jeandut for the proposal.

Some comments:

1. It is quite common in FL papers to compare algorithms under different budgets of communication rounds, with some approaches aiming to reach reasonably good models under a small number of rounds (there is even work on "one-shot FL"). So perhaps we could set a small number of regimes between 1 and the maximum number of rounds you propose.
2. There is quite some work on compressing model updates (through quantization, low-rank approximations, etc). In the above proposal, these methods would not provide any gain as they would be compared to methods that can communicate much more per round. Here again, perhaps a few regimes could be considered (e.g., 10 times the gradient size, 1 time the gradient size, 10% the gradient size).

Of course these suggestions would make things a bit more complex, so as always there is a trade-off between generality and complexity of the benchmark.

from flamby.

An example of a function monitoring the number of bits exchanged would be:

import numpy as np
import torch
LIMIT = 1000
def check_exchange_compliance(tensors_list, units="bytes"):
    """
    Check that for each round the quantities exchanged are below the dataset specific limit.
    Parameters
    ----------
    tensors_list: List[Union[torch.Tensor, np.ndarray]]
        The list of quantities sent by the client.
    units: str
        The units in which to return the result. Default to bytes.
    """
    assert units in ["bytes", "bits", "megabytes", "gigabytes"]
    assert isinstance(tensors_list, list), "You should provide a list of tensors."
    assert all([(isinstance(t, np.ndarray) or isinstance(t, torch.Tensor)) for t in tensors_list])
    bytes_count = 0
    for t in tensors_list:
        if isinstance(t, np.ndarray):
            bytes_count += t.nbytes
        else:
            bytes_count += t.shape.numel() * torch.finfo(t.dtype).bits // 8
        if bytes_count > LIMIT:
            raise ValueError(f"You cannot send more than {LIMIT} bytes, this "
            f"round. You tried sending more than {bytes_count} bytes already")
    if units == "bytes":
        res = bytes_count
    elif units == "bits":
        res = bytes_count * 8
    elif units == "megabytes":
        res = 1e-6 * bytes_count
    elif units == "gigabytes":
        res = 1e-9 * bytes_count

    return bytes_count


l = [torch.ones((2, 2)), torch.zeros((2,2), dtype=torch.float64), torch.Tensor(1).long()]
print(check_exchange_compliance(l))

from flamby.

Thank you @bellet for your insights !

I will describe it this afternoon but we propose benchmarking guidelines and not rules to give the liberty to users to do as they please: if they are not satisfied with the constraints, we will provide users with ways to disable these constraints.

For 1. Good point it's doable easily in the current implementation, for 2. you indeed have a point maybe by adding a total number of bits exchanged counter in the class and by disabling constraints per round maybe we can facilitate the arrival of users from the quantization community.

from flamby.

Closing this issue for now as the decisions were made.

from flamby.