Hi , Thanks for sharing your code. I have a quick question, in the paper it is mention

back-propagate the discriminator loss through the generator about electra_pytorch HOT 1 CLOSED

richarddwang commented on May 22, 2024

back-propagate the discriminator loss through the generator

from electra_pytorch.

Comments (1)

richarddwang commented on May 22, 2024

Hi.
There is a sampling between generator and discriminator and sampling is not able to be backward propagated.

electra_pytorch/pretrain.py

Lines 294 to 302 in 9b2533e

 with torch.no_grad(): 

 # sampling 

 pred_toks = self.sample(mlm_gen_logits) # ( #mlm_positions, ) 

 # produce inputs for discriminator 

 generated = masked_inputs.clone() # (B,L) 

 generated[is_mlm_applied] = pred_toks # (B,L) 

 # produce labels for discriminator 

 is_replaced = is_mlm_applied.clone() # (B,L) 

 is_replaced[is_mlm_applied] = (pred_toks != labels[is_mlm_applied]) # (B,L)

from electra_pytorch.

Related Issues (20)

How do I extract and save the discriminator from the checkpoint? HOT 3
GPU utilization falls back to 0% when training with multiple GPUs HOT 5
Sequence length too long for `ELECTRADataProcessor`. HOT 3
Small typo in the README.md HOT 1
How do I continue language model training? HOT 3
How can I pretrain ELECTRA starting from weights from google ? HOT 9
Is multi_task.py in a working state and if so how should one use it? HOT 1
Is it possible to perform the fine-tuning within the HuggingFace library? HOT 3
How to load the cached data from ELECTRAProcessor? HOT 1
如何多卡并行训练模型（How to train multi-card models in parallel？） HOT 1
Training time and ++ version HOT 1
Pyarrow dataloading issue HOT 2
Use different tokenizer (and specify special tokens) HOT 1
SST-2 accuracy is 50% after finetuning HOT 10
Relative importance of different "tricks" in README HOT 1
NameError: name 'sort_by_run' is not defined HOT 2
Custom Dataset
License info is required
Python version
Restarting from previous checkpoint

back-propagate the discriminator loss through the generator about electra_pytorch HOT 1 CLOSED

Comments (1)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	with torch.no_grad():
	# sampling
	pred_toks = self.sample(mlm_gen_logits) # ( #mlm_positions, )
	# produce inputs for discriminator
	generated = masked_inputs.clone() # (B,L)
	generated[is_mlm_applied] = pred_toks # (B,L)
	# produce labels for discriminator
	is_replaced = is_mlm_applied.clone() # (B,L)
	is_replaced[is_mlm_applied] = (pred_toks != labels[is_mlm_applied]) # (B,L)