how good or bad are the predicted probabilities?? low probability

torch.nn.CrossEntropyLoss VS <code class="notranslat

conventional CrossEntropy <a target="_blank" rel="noopener norefer

cross-entropy (log loss) & KL Divergence about knowledge-distillation-pytorch HOT 2 OPEN

jl749 commented on July 19, 2024

cross-entropy (log loss) & KL Divergence

from knowledge-distillation-pytorch.

Comments (2)

jl749 commented on July 19, 2024

`torch.nn.CrossEntropyLoss` VS `torch.nn.NLLLoss`

same operation but different approaches

CrossEntropy contains both LogSoftmax and NLLLoss (more descriptive)

NLLLoss on the other hand takes logit output of F.log_softmax() (more imperative)

from knowledge-distillation-pytorch.

jl749 commented on July 19, 2024

conventional CrossEntropy

knowledge-distillation-pytorch/#playground/torch_cee_imperatively.py

Lines 11 to 14 in 4a7c13d

 def cross_entropy(predictions, targets, epsilon=1e-12): 

 N = predictions.shape[0] 

 cee = -np.sum(targets * np.log(predictions + epsilon)) / N 

 return cee

torch.nn.NLLLose

takes input, target; target must be 1d tensor directing to the max indexes
e.g. input = [[0.25, 0.25, 0.5], [0.1, 0.2, 0.7]], target = [2, 2]

knowledge-distillation-pytorch/#playground/torch_cee_imperatively.py

Lines 17 to 26 in 4a7c13d

 def nlllose(log_logit, targets): 

 """ 

  negative log likelihood loss 

  """ 

 targets = np.argmax(targets, axis=1) 

 out = np.empty(targets.shape, dtype=np.float32) 

 for batch_idx in range(len(targets)): # for every batch 

 out[batch_idx] = log_logit[batch_idx][targets[batch_idx]] 

 return -np.mean(out)

torch.nn.CrossEntropyLoss

torch.nn.CrossEntropyLoss consists of nn.LogSoftmax and nn.NLLLoss

knowledge-distillation-pytorch/#playground/torch_cee_imperatively.py

Lines 29 to 35 in 4a7c13d

 def cross_entropy_better(predictions, targets): 

 """ 

  easier to backprop 

  """ 

 log_predictions = log_softmax(predictions) 

 return nlllose(log_predictions, targets)

) (see the implementation, equations will make more sense

wait, why log softmax to calculate cross entropy err?

https://datascience.stackexchange.com/questions/40714/what-is-the-advantage-of-using-log-softmax-instead-of-softmax
https://stats.stackexchange.com/questions/436766/cross-entropy-with-log-softmax-activation

TODO

read more about the references above

from knowledge-distillation-pytorch.

cross-entropy (log loss) & KL Divergence about knowledge-distillation-pytorch HOT 2 OPEN

Comments (2)

`torch.nn.CrossEntropyLoss` VS `torch.nn.NLLLoss`

conventional CrossEntropy

torch.nn.NLLLose

torch.nn.CrossEntropyLoss

wait, why log softmax to calculate cross entropy err?

TODO

Related Issues (4)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def cross_entropy(predictions, targets, epsilon=1e-12):
	N = predictions.shape[0]
	cee = -np.sum(targets * np.log(predictions + epsilon)) / N
	return cee

	def nlllose(log_logit, targets):
	"""
	negative log likelihood loss
	"""
	targets = np.argmax(targets, axis=1)
	out = np.empty(targets.shape, dtype=np.float32)
	for batch_idx in range(len(targets)): # for every batch
	out[batch_idx] = log_logit[batch_idx][targets[batch_idx]]

	return -np.mean(out)

	def cross_entropy_better(predictions, targets):
	"""
	easier to backprop
	"""
	log_predictions = log_softmax(predictions)

	return nlllose(log_predictions, targets)

Comments (2)

torch.nn.CrossEntropyLoss VS torch.nn.NLLLoss

conventional CrossEntropy

torch.nn.NLLLose

torch.nn.CrossEntropyLoss

wait, why log softmax to calculate cross entropy err?

TODO

Related Issues (4)

Recommend Projects

Recommend Topics

Recommend Org

`torch.nn.CrossEntropyLoss` VS `torch.nn.NLLLoss`