Comments (1)
Nana2929
commented on July 19, 2024
Configs
# reference: https://tsmatz.wordpress.com/2022/11/25/huggingface-japanese-summarization/
# Note : Do not use FP16 precision in mT5 fine-tuning.
seed: 42
data:
data_root: 'data/diaasq/speaker_dataset'
train_split_name: 'train'
test_split_name: 'valid'
lang_src : 'en'
proc_data:
data_root: 'data/diaasq/speaker_dataset/proc'
train_ic_name: 't5_in_context' # use t5/create_kshot_dataset_split.py
t5_train_split_name: 't5_train'
test_ic_name: 't5_in_context' # use the same in-context examples as in training
t5_test_split_name: 't5_valid'
dataset:
k: 1 # number of in-context examples
prompt_path: 'prompt/experiment/diaasq-speaker-spec-en-t5'
in_context_strategy: None
model:
model_name: 'allenai/tk-instruct-base-def-pos'
max_length: 512
output_dir: 'output/diaasq/t5-en'
tokenizer:
truncation_side: 'left'
trainer:
gpu_id: "1" # if training on server 9102, only 1 gpu at a time can be used otherwise it crashes
push_to_hub: false
per_device_train_batch_size: 4
per_device_eval_batch_size: 2
epochs: 20
weight_decay: 0.01
learning_rate: 0.00004
# https://github.com/huggingface/transformers/blob/v4.29.1/src/transformers/trainer_utils.py#L362
lr_scheduler_type: 'cosine_with_restarts'
warmup_steps: 100
optim : 'adamw_hf'
generation_max_length: 128
save_strategy: 'epoch'
load_best_model_at_end: true
metric_for_best_model: 'eval_quad_micro_f1'
logging_steps: 500
save_total_limit: 2
logger:
logging_dir: 'logs/diaasq/t5-en' # logs/diaasq/t5-en/exp_20230524-143733
report_to: 'tensorboard'
run_name: 'exp'
Results
有稍微調過參數（用 ReducedOnPlateau 的 Scheduler）
trainset length: 2236
testset length: 290
[tensorboard] logging to: logs/diaasq/t5-en/exp_20230524-143733
{'loss': 2.7081, 'learning_rate': 3.855595667870036e-05, 'epoch': 0.89}
{'eval_loss': 0.3480522930622101, 'eval_target_f1': 0.46002805044075473, 'eval_aspect_f1': 0.40658049348693526, 'eval_opinion_f1': 0.346368715033774, 'eval_pair_ta_f1': 0.2229102166684056, 'eval_pair_to_f1': 0.1981424148108132, 'eval_pair_ao_f1': 0.19875776392520864, 'eval_quad_micro_f1': 0.09233791743557149, 'eval_runtime': 20.0959, 'eval_samples_per_second': 14.431, 'eval_steps_per_second': 7.215, 'epoch': 1.0}
{'loss': 0.2968, 'learning_rate': 3.675090252707581e-05, 'epoch': 1.79}
{'eval_loss': 0.3033939301967621, 'eval_target_f1': 0.5027624308891013, 'eval_aspect_f1': 0.4617142856641855, 'eval_opinion_f1': 0.3844492440103924, 'eval_pair_ta_f1': 0.27794561928535355, 'eval_pair_to_f1': 0.23176823171821667, 'eval_pair_ao_f1': 0.24799999994995361, 'eval_quad_micro_f1': 0.13523809518818286, 'eval_runtime': 9.1547, 'eval_samples_per_second': 31.678, 'eval_steps_per_second': 15.839, 'epoch': 2.0}
{'loss': 0.3704, 'learning_rate': 3.494584837545127e-05, 'epoch': 2.68}
{'eval_loss': 0.28762921690940857, 'eval_target_f1': 0.5572413792602071, 'eval_aspect_f1': 0.4582843712777574, 'eval_opinion_f1': 0.4099099098598998, 'eval_pair_ta_f1': 0.2848232847734609, 'eval_pair_to_f1': 0.26375908613915194, 'eval_pair_ao_f1': 0.25078369900965075, 'eval_quad_micro_f1': 0.15476190471229057, 'eval_runtime': 9.0573, 'eval_samples_per_second': 32.018, 'eval_steps_per_second': 16.009, 'epoch': 3.0}
{'loss': 0.2206, 'learning_rate': 3.314079422382672e-05, 'epoch': 3.58}
{'eval_loss': 0.2941332459449768, 'eval_target_f1': 0.5564971750910873, 'eval_aspect_f1': 0.47990543730216034, 'eval_opinion_f1': 0.43230944249838305, 'eval_pair_ta_f1': 0.3039832284617675, 'eval_pair_to_f1': 0.27643979052613465, 'eval_pair_ao_f1': 0.2776025236094272, 'eval_quad_micro_f1': 0.16350947153567927, 'eval_runtime': 8.9849, 'eval_samples_per_second': 32.276, 'eval_steps_per_second': 16.138, 'epoch': 4.0}
{'loss': 0.2011, 'learning_rate': 3.133574007220217e-05, 'epoch': 4.47}
{'eval_loss': 0.278605192899704, 'eval_target_f1': 0.551626591180398, 'eval_aspect_f1': 0.4941995359127635, 'eval_opinion_f1': 0.45879732734414913, 'eval_pair_ta_f1': 0.31115660179244714, 'eval_pair_to_f1': 0.2827868851959749, 'eval_pair_ao_f1': 0.2986611739973849, 'eval_quad_micro_f1': 0.17170731702340492, 'eval_runtime': 8.9753, 'eval_samples_per_second': 32.311, 'eval_steps_per_second': 16.156, 'epoch': 5.0}
{'loss': 0.1837, 'learning_rate': 2.953068592057762e-05, 'epoch': 5.37}
{'eval_loss': 0.2796546220779419, 'eval_target_f1': 0.5495750707713776, 'eval_aspect_f1': 0.5234741783536466, 'eval_opinion_f1': 0.4527027026526829, 'eval_pair_ta_f1': 0.33367875642682787, 'eval_pair_to_f1': 0.2880829015045442, 'eval_pair_ao_f1': 0.3041666666167283, 'eval_quad_micro_f1': 0.18163869689011578, 'eval_runtime': 8.9184, 'eval_samples_per_second': 32.517, 'eval_steps_per_second': 16.259, 'epoch': 6.0}
{'loss': 0.1756, 'learning_rate': 2.772563176895307e-05, 'epoch': 6.26}
{'eval_loss': 0.2843130826950073, 'eval_target_f1': 0.5605633802315326, 'eval_aspect_f1': 0.5174825174324013, 'eval_opinion_f1': 0.4864864864364591, 'eval_pair_ta_f1': 0.3302373580512523, 'eval_pair_to_f1': 0.3061013443141352, 'eval_pair_ao_f1': 0.3204994796586897, 'eval_quad_micro_f1': 0.1793103447779015, 'eval_runtime': 8.825, 'eval_samples_per_second': 32.861, 'eval_steps_per_second': 16.431, 'epoch': 7.0}
{'loss': 0.1555, 'learning_rate': 2.5920577617328523e-05, 'epoch': 7.16}
{'eval_loss': 0.28299134969711304, 'eval_target_f1': 0.5643835615937114, 'eval_aspect_f1': 0.5254813136531838, 'eval_opinion_f1': 0.48636859318871906, 'eval_pair_ta_f1': 0.33099297888679236, 'eval_pair_to_f1': 0.3055276381409419, 'eval_pair_ao_f1': 0.3245967741434959, 'eval_quad_micro_f1': 0.19293218715161944, 'eval_runtime': 8.8512, 'eval_samples_per_second': 32.764, 'eval_steps_per_second': 16.382, 'epoch': 8.0}
{'loss': 0.1513, 'learning_rate': 2.4115523465703976e-05, 'epoch': 8.05}
{'loss': 0.1359, 'learning_rate': 2.2310469314079425e-05, 'epoch': 8.94}
{'eval_loss': 0.2927750051021576, 'eval_target_f1': 0.5804676753281284, 'eval_aspect_f1': 0.5249709639452341, 'eval_opinion_f1': 0.4849833147441465, 'eval_pair_ta_f1': 0.32823649332415594, 'eval_pair_to_f1': 0.3108384457578278, 'eval_pair_ao_f1': 0.3268242548318063, 'eval_quad_micro_f1': 0.1966893865130186, 'eval_runtime': 8.8726, 'eval_samples_per_second': 32.685, 'eval_steps_per_second': 16.342, 'epoch': 9.0}
{'loss': 0.1281, 'learning_rate': 2.0505415162454874e-05, 'epoch': 9.84}
{'eval_loss': 0.29930731654167175, 'eval_target_f1': 0.5903448275360602, 'eval_aspect_f1': 0.5389908256379522, 'eval_opinion_f1': 0.49234135662385886, 'eval_pair_ta_f1': 0.3534136545684545, 'eval_pair_to_f1': 0.3101711983387138, 'eval_pair_ao_f1': 0.34949494944489545, 'eval_quad_micro_f1': 0.20478468894531462, 'eval_runtime': 8.8807, 'eval_samples_per_second': 32.655, 'eval_steps_per_second': 16.328, 'epoch': 10.0}
{'loss': 0.1216, 'learning_rate': 1.8700361010830327e-05, 'epoch': 10.73}
{'eval_loss': 0.30461862683296204, 'eval_target_f1': 0.5733882029677025, 'eval_aspect_f1': 0.5294797687360048, 'eval_opinion_f1': 0.46954595786797404, 'eval_pair_ta_f1': 0.35829959509170983, 'eval_pair_to_f1': 0.30831642997030373, 'eval_pair_ao_f1': 0.3374233128334136, 'eval_quad_micro_f1': 0.20656370651385747, 'eval_runtime': 8.8613, 'eval_samples_per_second': 32.727, 'eval_steps_per_second': 16.363, 'epoch': 11.0}
{'loss': 0.1205, 'learning_rate': 1.6895306859205777e-05, 'epoch': 11.63}
{'eval_loss': 0.2945067286491394, 'eval_target_f1': 0.5710344827084793, 'eval_aspect_f1': 0.5173210161161623, 'eval_opinion_f1': 0.48720800884870674, 'eval_pair_ta_f1': 0.33096446695510423, 'eval_pair_to_f1': 0.31460674152308465, 'eval_pair_ao_f1': 0.3347022586768916, 'eval_quad_micro_f1': 0.19825072881317327, 'eval_runtime': 8.8625, 'eval_samples_per_second': 32.722, 'eval_steps_per_second': 16.361, 'epoch': 12.0}
{'loss': 0.1129, 'learning_rate': 1.5090252707581228e-05, 'epoch': 12.52}
{'eval_loss': 0.3026190400123596, 'eval_target_f1': 0.6030013642063567, 'eval_aspect_f1': 0.5315005726875676, 'eval_opinion_f1': 0.4901098900597943, 'eval_pair_ta_f1': 0.34406438626789665, 'eval_pair_to_f1': 0.3195146612240195, 'eval_pair_ao_f1': 0.3414634145841067, 'eval_quad_micro_f1': 0.2040423483620676, 'eval_runtime': 8.8855, 'eval_samples_per_second': 32.637, 'eval_steps_per_second': 16.319, 'epoch': 13.0}
{'loss': 0.1044, 'learning_rate': 1.3285198555956679e-05, 'epoch': 13.42}
{'eval_loss': 0.304652601480484, 'eval_target_f1': 0.5928961748132633, 'eval_aspect_f1': 0.5622857142355911, 'eval_opinion_f1': 0.4841182912971082, 'eval_pair_ta_f1': 0.3581488933101474, 'eval_pair_to_f1': 0.3174924165324123, 'eval_pair_ao_f1': 0.34517766492457647, 'eval_quad_micro_f1': 0.2173076922578169, 'eval_runtime': 8.8082, 'eval_samples_per_second': 32.924, 'eval_steps_per_second': 16.462, 'epoch': 14.0}
{'loss': 0.1056, 'learning_rate': 1.1480144404332132e-05, 'epoch': 14.31}
{'eval_loss': 0.3082675039768219, 'eval_target_f1': 0.6109589040594521, 'eval_aspect_f1': 0.544202066540003, 'eval_opinion_f1': 0.4972375690106856, 'eval_pair_ta_f1': 0.35829959509170983, 'eval_pair_to_f1': 0.3313008129581538, 'eval_pair_ao_f1': 0.3496932514837178, 'eval_quad_micro_f1': 0.2166344293505486, 'eval_runtime': 8.8181, 'eval_samples_per_second': 32.887, 'eval_steps_per_second': 16.443, 'epoch': 15.0}
{'loss': 0.1003, 'learning_rate': 9.675090252707581e-06, 'epoch': 15.21}
{'eval_loss': 0.3161661624908447, 'eval_target_f1': 0.5997248967861699, 'eval_aspect_f1': 0.5506912441895048, 'eval_opinion_f1': 0.5071982280783754, 'eval_pair_ta_f1': 0.35532994918860183, 'eval_pair_to_f1': 0.3367346938275924, 'eval_pair_ao_f1': 0.3514902363323152, 'eval_quad_micro_f1': 0.21595330734319937, 'eval_runtime': 8.8343, 'eval_samples_per_second': 32.827, 'eval_steps_per_second': 16.413, 'epoch': 16.0}
{'loss': 0.0962, 'learning_rate': 7.870036101083033e-06, 'epoch': 16.1}
{'loss': 0.093, 'learning_rate': 6.064981949458484e-06, 'epoch': 16.99}
{'eval_loss': 0.3212610185146332, 'eval_target_f1': 0.6101231189649148, 'eval_aspect_f1': 0.5425287355820603, 'eval_opinion_f1': 0.4934210525814794, 'eval_pair_ta_f1': 0.35448136953710047, 'eval_pair_to_f1': 0.3238866396261218, 'eval_pair_ao_f1': 0.34658511717728624, 'eval_quad_micro_f1': 0.20616570322566852, 'eval_runtime': 8.8367, 'eval_samples_per_second': 32.818, 'eval_steps_per_second': 16.409, 'epoch': 17.0}
{'loss': 0.0904, 'learning_rate': 4.259927797833936e-06, 'epoch': 17.89}
{'eval_loss': 0.32145148515701294, 'eval_target_f1': 0.6092896174362096, 'eval_aspect_f1': 0.5448275861567724, 'eval_opinion_f1': 0.4879120878619925, 'eval_pair_ta_f1': 0.3572149343596847, 'eval_pair_to_f1': 0.3221884497980371, 'eval_pair_ao_f1': 0.3448979591336439, 'eval_quad_micro_f1': 0.20849420844435906, 'eval_runtime': 8.8475, 'eval_samples_per_second': 32.777, 'eval_steps_per_second': 16.389, 'epoch': 18.0}
{'loss': 0.0909, 'learning_rate': 2.4548736462093864e-06, 'epoch': 18.78}
{'eval_loss': 0.32269975543022156, 'eval_target_f1': 0.6101231189649148, 'eval_aspect_f1': 0.5419058552885683, 'eval_opinion_f1': 0.49504950490039973, 'eval_pair_ta_f1': 0.35555555550555734, 'eval_pair_to_f1': 0.3286004056295288, 'eval_pair_ao_f1': 0.34933605715119825, 'eval_quad_micro_f1': 0.21062801927382796, 'eval_runtime': 8.8492, 'eval_samples_per_second': 32.771, 'eval_steps_per_second': 16.386, 'epoch': 19.0}
{'loss': 0.0882, 'learning_rate': 6.498194945848375e-07, 'epoch': 19.68}
{'eval_loss': 0.3236048221588135, 'eval_target_f1': 0.6073871408527405, 'eval_aspect_f1': 0.5366972476563013, 'eval_opinion_f1': 0.4977973567780998, 'eval_pair_ta_f1': 0.34747474742475093, 'eval_pair_to_f1': 0.3286004056295288, 'eval_pair_ao_f1': 0.3472931562318932, 'eval_quad_micro_f1': 0.20676328497431182, 'eval_runtime': 8.854, 'eval_samples_per_second': 32.754, 'eval_steps_per_second': 16.377, 'epoch': 20.0}
{'train_runtime': 2227.8463, 'train_samples_per_second': 20.073, 'train_steps_per_second': 5.018, 'train_loss': 0.26315246779931467, 'epoch': 20.0}
以 eval_quad_micro_f1 來挑選 best model 的話，第14個 epoch 最佳。
{'eval_loss': 0.304652601480484, 'eval_target_f1': 0.5928961748132633, 'eval_aspect_f1': 0.5622857142355911, 'eval_opinion_f1': 0.4841182912971082, 'eval_pair_ta_f1': 0.3581488933101474, 'eval_pair_to_f1': 0.3174924165324123, 'eval_pair_ao_f1': 0.34517766492457647, 'eval_quad_micro_f1': 0.2173076922578169, 'eval_runtime': 8.8082, 'eval_samples_per_second': 32.924, 'eval_steps_per_second': 16.462, 'epoch': 14.0}
Tensorboard
from dialogue-absa.
Related Issues (15)
- 10. chatgpt raw replies generation
- 11. chatgpt replies post-processing
- 12. tableau gpt replies visualization
- 13. LLaMA family spike
- 14. compile challenging absa datasets and add readme for source credit HOT 2
- [survey] prompt
- [survey] DiaASQ HOT 1
- [data prep] DiaASQ HOT 1
- [test] DiaASQ dataset
- [experiment] llm + diaASQ [en] HOT 1
- [exp] DiaASQ T5 compute_metrics in Trainer
- [paper] Personalized Showcases: Generating Multi-Modal Explanations for Recommendations (An Yan, et al. SIGIR-23)
- ⏰ [exp] DiaASQ HOT 1
- Full-Dialogue DiaASQ zh
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dialogue-absa.