Comments (4)
Hi, @liuxiaozhu01. Thank you for your interest in our work! Your observation about the implementation detail is correct. The reason I apply a square operation to the attn_metric but not to the mlp_metric is because I aim to aggregate the group channel metrics of attn_head using the L2 norm, which is represented by the code snippet "W_metric = W_metric.reshape(-1, 128).sum(dim=1)". Ideally, this operation would be followed by a square root to complete the L2 norm calculation. However, I omitted the square root step since it does not affect the ordering of the elements, which is our primary concern in this context. This detail stems from our experimental observations and theoretical considerations, suggesting that treating these components differently could yield optimal pruning results.
I hope this explanation clarifies the reasoning behind our implementation choices. Should you have any further questions or need additional details, feel free to ask!
from flap.
Thanks for your reply! I think i get your idea.
From my understanding, the WIFV
correspond the Eq. (5) in your paper, and after the group channel metrics of attn_head is aggregated, the metrics for every attn_head(attn_metric
in the code) is squared actually while mlp's(mlp_metric
in the code) is not. Due to the standarlization, attn_metric
and mlp_metric
are in the same scale so global sort, which means concat and sort, is reasonable.
Is that a correct understanding?
from flap.
Thanks for your reply! I think i get your idea.
From my understanding, the
WIFV
correspond the Eq. (5) in your paper, and after the group channel metrics of attn_head is aggregated, the metrics for every attn_head(attn_metric
in the code) is squared actually while mlp's(mlp_metric
in the code) is not. Due to the standarlization,attn_metric
andmlp_metric
are in the same scale so global sort, which means concat and sort, is reasonable.Is that a correct understanding?
Yes, it is correct. To elaborate, the most meticulous implementation would indeed entail an initial standardization of the attn_metric
. Following this, we would proceed to square the attn_metric
values. Next, we'd aggregate the group channel metrics derived from attn_head, and take the square root of the resulting value. Finally, we'd carry out a combined sorting of the refined attn_metric
and mlp_metric
.
In practice, the performance impact resulting from the differences between the two is minimal. This is largely due to the fact that standardization ensures both metrics are normalized to the same scale, which validates the use of a global sorting procedure. I appreciate your astute observation.
from flap.
It's been very rewarding. Thank you for your generous help!
from flap.
Related Issues (12)
- ModuleNotFoundError: No module named 'models' HOT 1
- Compute sparsity problem
- Could you please explain the purpose behind the following code? HOT 1
- can you support baffo32/decapoda-research-llama-7B-hf
- Question about the number of parameters HOT 1
- Question about the wikitext2 data loader HOT 1
- how much data was used in the pruning process? HOT 2
- support for other models HOT 2
- pruning 之后使用 无法读取模型 HOT 5
- request for the inference speed test script
- AttributeError: Can't pickle local object 'add_hook_to_module.<locals>.new_forward'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flap.