Comments (3)
Hello @ybouilla !
The default LR
works only for the pooled case aka when using traditional pytorch without any FL strategy, luckily we have an appendix detailing all hyper parameters that were used for FL experiments. See page 14-15 of our arxiv submission. For instance for Scaffold and Fed-Heart Disease we used 0.001, which matches default LR (Table 16 page 14).
As those parameters were cross-validated only by looking at the final metric maybe the trainings losses will still fluctuate.
Fluctuations of the loss may come from:
- Limited batch-size
- Too large learning rate
Try using a larger batch-size and a smaller learning rate and see if the oscillations dampen.
Also the loss do decrease in your curves, maybe simply using more rounds would allow to better optimize the loss.
from flamby.
See https://github.com/owkin/FLamby/blob/main/flamby/config_heart_disease.json for the full-list of hyper-parameters used for the benchmark. We were trying to optimize the metrics given a budget in rounds:
For num_updates=100
and nrounds = get_nb_max_rounds(100, batch_size=BATCH_SIZE)
from flamby.
@ybouilla we are planning on working on benchmarks in FLamby in general using the https://benchopt.github.io/ initiative. This will allow to compare optimizers and hyperparameters more easily. Until then I am closing this issue.
from flamby.
Related Issues (20)
- Downloading IXI dataset: link broken HOT 5
- CI issue: numba does not support Python3.11 HOT 4
- fed-ixi dataset download issue HOT 4
- Question about FedAvg strategy. HOT 4
- Pip install with or without -e should install the full FLamby suite
- Dummy Dataset is nor reproducible nor flexible enough HOT 2
- Strategies should accept optimizer arguments
- Doc enhancement: explain more clearly datasets and associated hyperparameters HOT 1
- Strategies Monitoring Improvements: average loss and metrics
- RFC: Should we allow to do epochs instead of batch-updates in FLamby's strategies ?
- C-index computation should not be batched HOT 1
- Fed_KiTS19 code generates negative loss values HOT 6
- Mismatching evaluation code for FedKiTS19 HOT 4
- Adding docs on metrics and evaluation function for each dataset
- Setuptools is set to an old version HOT 1
- Mismatch in python version between `environment.yml` and the CI
- CI is not operational HOT 1
- KITS results mismatch with paper HOT 13
- Caching preprocessed features in Kits19 HOT 1
- RuntimeError: Discrete mean differs significantly from continuous mean. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flamby.