Comments (11)
Hi. In Per-FedAvg paper, there are 2 ways to implement a second update. We implemented the first one when the hessian is approximated by gradient descent update with a new mini-batch. So the second step simply is just new gradient descent update with new batch size and parameter \beta.
Hi Canh, Thanks for your reply. But I could not agree with you. Please see below:
from pfedme.
Hi. In Per-FedAvg paper, there are 2 ways to implement a second update. We implemented the first one when the hessian is approximated by gradient descent update with a new mini-batch. So the second step simply is just new gradient descent update with new batch size and parameter \beta.Hi Canh, Thanks for your reply. But I could not agree with you. Please see below:
![]()
I agree with your point. In FO Per-FedAvg, the second update (i.e., update the meta-model) relies on the loss and the gradient
. The following matlab code is provided by the author of Per-FedAvg.
%Local updates
for i=A
lw12 = w12;
lw23 = w23;
lw34 = w34;
lb12 = b12;
lb23 = b23;
lb34 = b34;
for t=1:tau
B_1 = randperm(user_l(i), D_i);
[lgw12,lgw23,lgw34, lgb12,lgb23, lgb34] = grad_batch (lw12,lw23,lw34,lb12,lb23,lb34, Dat(:,B_1,i), Lab(:,B_1,i), D_i);
B_2 = randperm(user_l(i), D_o);
[lgw12,lgw23,lgw34, lgb12,lgb23, lgb34] = grad_batch (lw12-al*lgw12,lw23-al*lgw23,lw34-al*lgw34,lb12-al*lgb12,lb23-al*lgb23,lb34-al*lgb34, Dat(:,B_2,i), Lab(:,B_2,i), D_o);
lw12 = lw12 - be*lgw12;
lw23 = lw23 - be*lgw23;
lw34 = lw34 - be*lgw34;
lb12 = lb12 - be*lgb12;
lb23 = lb23 - be*lgb23;
lb34 = lb34 - be*lgb34;
end
slw12 = slw12 + lw12/(r*n);
slw23 = slw23 + lw23/(r*n);
slw34 = slw34 + lw34/(r*n);
slb12 = slb12 + lb12/(r*n);
slb23 = slb23 + lb23/(r*n);
slb34 = slb34 + lb34/(r*n);
end
from pfedme.
Hi @sshpark,
Thanks for your commit.
I just merged it to the main branch. Have you tested the performance of PerfedAvg?
Hi @CharlieDinh,
I am sorry for not providing a full performance test for this commit. I have applied it to my current project. In my project, the updated version of Per-FedAvg does not differ much from its previous version in some aspects such as accuracy. I would give a full performance test after finishing my current project.
from pfedme.
Hi. In Per-FedAvg paper, there are 2 ways to implement a second update. We implemented the first one when the hessian is approximated by gradient descent update with a new mini-batch. So the second step simply is just new gradient descent update with new batch size and parameter \beta.
from pfedme.
When the code was implemented, the previous version of this paper only propose the first way to approximate Hessian, so we didn't implement the second approach.
from pfedme.
The fundamental idea of both pFedMe and Per-FedAvg is to generate a well-generalized global model which allowed personalizing this global model to make the personalized model using 1-k steps (small steps). Also in Per-Fedavg, it is trained one batch of test data before evaluation. So, the personalized model is always better than the global only.
from pfedme.
Please see above for details.
from pfedme.
Hi @sshpark and @chuanting;
Thank all.
Yes, I think my implementation is not correct in terms of the second update in PerfedAvg and @chuanting found out this issue.
I am planning to update it after I finished my current project. But if anyone can contribute to fix this issue, it would be wonderfull.
Regards,
Canh Dinh
from pfedme.
Hi @sshpark and @chuanting;
Thank all.
Yes, I think my implementation is not correct in terms of the second update in PerfedAvg and @chuanting found out this issue.
I am planning to update it after I finished my current project. But if anyone can contribute to fix this issue, it would be wonderfull.Regards,
Canh Dinh
Hi @CharlieDinh;
Actually, I have tried to fix this issue and pulled a request. You could check it out if you have a minute.
Regards,
sshpark
from pfedme.
Hi @sshpark,
Thanks for your commit.
I just merged it to the main branch. Have you tested the performance of PerfedAvg?
from pfedme.
Thank @sshpark.
from pfedme.
Related Issues (19)
- Experiment image
- Unable to generate non-iid MNIST Data HOT 4
- why train loss will be nan? HOT 1
- pfedme Optimizer Probelem HOT 6
- In the Per-FedAvg experiments, there is always an unignorable gap between the accuracy of the actual experimental results and the expected accuracy provided under the same conditions. HOT 5
- Something maybe wrong in data/mnist/generate_niid_20users.py HOT 1
- Some mistakes in generating niid mnist data HOT 11
- Client's train method HOT 2
- A question in PerAvg algorithm HOT 1
- Question about pFedMeOptimizer. HOT 2
- About the Hessian Approximation
- Hello author, why did Loss become Nan after more than a dozen rounds of training?
- some questions about the results HOT 1
- UserpFedMe class HOT 4
- A question about train_one_step() method. HOT 3
- Some questions about your peper and your code HOT 2
- Why does FedAvg only train on 1 batch in each local epoch? HOT 2
- Cifar-10 running error HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pfedme.