Currently the most complex optimiser is at the the top of the page and the simplest is at the bottom. Given that more complex optimizers are evolutions of the simple ones, it would make more sense to have the simplest first.

Thanks

Given the loss functions calculate the loss of a batch (which is common),

def MSE(yHat, y):
    return np.sum((yHat - y)**2) / 2.0

should be:

def MSE(yHat, y):
    return np.sum((yHat - y)**2) / y.size 

and for float values:

def MSE(yHat, y):
    return np.sum((yHat.astype(float) - y)**2) / y.size

#Take the error when label=1
class1_cost = -labels*np.log(predictions)

#Take the error when label=0
class2_cost = (1-labels)*np.log(1-predictions)

#Take the sum of both costs
cost = class1_cost + class2_cost

In this code, it seem like class1 return positive cost and class2 return negative cost, wouldn't they cancel when added?

[Feel free to close this issue without responding, but I thought it might be worth mentioning]

In Safari, High Sierra, Adblock 2.70.0 (Betafish) blocked the link to the Advertising.csv dataset.

In this section:
https://github.com/bfortuner/ml-cheatsheet/blob/master/docs/linear_regression.rst#simple-regression

The link to a dataset:
http://www-bcf.usc.edu/~gareth/ISL/Advertising.csv

Logistic Regression classify function: local variable 'decision_boundary' referenced before assignment. It seems like we are referring to variable decision_boundary before it is defined.

Wikipedia defines the cost function MSE as follows:

Yet, the ml cheatsheet uses the following formulas in the backpropogation section.

This is particularly confusing since the Linear Regression and Gradient Descent section defines it correctly:

Do you have a PDF of this ML cheatsheet project ?

PDF will make easy to read and faster.

In page ml-glossary/docs/math_notation.rst the example for $$\epsilon$$ is given as learning rate. But the "e" in learning rate denotes exponent (10^-4).

I see that the probability page has a CC-BY-NC-SA license on it. Is that the license for the whole repository?

https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#cross-entropy
The second paragraph mentions that log loss penalizes "both types of error" without prior description of these types of errors.

git clone https://github.com/bfortuner/ml-glossary
Error:
Cloning into 'ml-glossary'...
remote: Repository not found.
fatal: repository 'https://github.com/bfortuner/ml-glossary/' not found

Hey, I am currently working on section "Activation Functions". Since I want to keep the consistency of the figures of the function, could you note the tools that you use to generate them? 😄 @bfortuner

In the Loss-Functions > Cross Entropy "code" example the arguments seem reversed.

Location: http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html#cross-entropy

It currently reads:

import numpy as np

def CrossEntropy(yHat, y):
    if yHat == 1:
        return -np.log(y)
    else:
        return -np.log(1 - y)
    
print 'true: ', CrossEntropy(.1, 1)
print 'false:', CrossEntropy(.8, 1)

# true:  inf
# false: inf

It should read?

def CrossEntropy2(yHat, y):
    if y == 1:
        return -np.log(yHat)
    else:
        return -np.log(1 - yHat)
    
print 'true: ', CrossEntropy2(.1, 1)
print 'false:', CrossEntropy2(.8, 1)

# true:  2.3025850929940455
# false: 0.2231435513142097

It would be really useful if the descriptions for linear regression, logistic regression and algorithms had a section that described data requirements/expectations. For example, from what I understand, least squares estimates for regression models are highly sensitive to (not robust against) outliers.

I just want to clarify my understanding before making any clarifying changes. In the Linear Regression article under 'Bias Term', it reads:

Below we add a constant 1 to our features matrix. By setting this value to 1, it turns our bias term into a constant.

bias = np.ones(shape=(len(features),1))
features = np.append(bias, features, axis=1)

So the purpose of adding the 1 along with the other features in each example is so that the 1 will be multiplied by the 'bias weight' when the dot product of the features and weights is performed in the predict() function. Is that accurate?

confusing definition for cross-entropy loss
referring to Stanford lecture notes
http://cs231n.github.io/linear-classify/

you are calling log loss same as cross entropy loss

https://github.com/bfortuner/ml-glossary/blob/master/docs/forwardpropagation.rst#id15

Should the Zh shape not be (3,2)?

Hey

I was going through the Calculas Cheatsheet here https://ml-cheatsheet.readthedocs.io/en/latest/calculus.html

Don't you think we should mention Integration Formulas, and Differentiation Rules as well?
Since its a cheat sheet, I think it would be useful for people to understand.

The following comment really confused me:

# Use matrix cross product (*) to simultaneously
# calculate the derivative for each weight
d_w1 = -x1*(targets - predictions)
...

https://ml-cheatsheet.readthedocs.io/en/latest/linear_regression.html#id4

In Numpy the cross product method would be np.cross() and not *, both versions give different results. Which is the correct version?

the dataset address have changed to http://faculty.marshall.usc.edu/gareth-james/ISL/Advertising.csv
It does not exist on old address anymore.

I would like to start translating your ml-glossary repo into Vietnamse and possibly contributing to both existing English and future Vietnamese versions.

Can I start my own repo and start translating like the mlbvn/d2l-vn translation of d2l-ai/d2l-en?

Also, I wondered if this is something already pursued by someone else?

For the dot product between matrices, The number of columns of the 1st matrix must equal the number of rows of the 2nd matrix. Hence, the example given here is wrong:

https://ml-cheatsheet.readthedocs.io/en/latest/linear_algebra.html#dot-product

The following link for downloading dataset is not valid:
http://scilab.io/wp-content/uploads/2016/07/data_classification.csv

Where can I get the data? Thanks!

I liked the idea of the ml-cheatsheets to give a quick but concise overview of the concept. I would like to add the explanations for the softmax, Dying RELU problem. Is there any template I should follow?

This page, section iii, has an excellent interactive lab of how neural network weight initializations work. I previously understood the "generally accepted wisdom", but this shows why.

http://www.deeplearning.ai/ai-notes/initialization/

It might be possible to steal it for this repository.

In the Epub download, inline equations aren't rendered/formatted. They are just raw Sphinx syntax e.g.:

\[\begin{split}\begin{align} f'(W_1) = -x_1(y - (W_1 x_1 + W_2 x_2 + W_3 x_3)) \\ f'(W_2) = -x_2(y - (W_1 x_1 + W_2 x_2 + W_3 x_3)) \\ f'(W_3) = -x_3(y - (W_1 x_1 + W_2 x_2 + W_3 x_3)) \end{align}\end{split}\]

This makes the Epub, which would otherwise be super useful, unreadable :(

In docs/backpropagation.rst the first formula should be f'(x) = A'(B)*B'(C)*C'(x) and the rest following that should reflect the change.

In https://github.com/bfortuner/ml-glossary/blob/master/docs/linear_regression.rst at Multivariable regression - Normalization section mentioned:

Our output is a normalized matrix of the same shape with all values between -1 and 1.

while the provided normalize code normalize the input values between -0.5 and 0.5.

Hi,

I find the following description in the Simple Network to be a bit confusing:

Prediction = A(\;A(\;X W_h\;)W_o\;)
Where A is an activation function like :ref:`activation_relu`, X is the input and W_h and W_o are weights.

I think I get your point, namely that the expression on the right hand side of Prediction is the approximate pseudo mathematical expression of the simple network, and where A is used to representation an arbitrary mathematical function that takes a matrix as an argument and that returns a matrix (there's a lot of capital letters in that expression, not all of which are matrices). Unfortunately, for me at least, the use of the capital A is confusing, and takes a few moments to figure out that A is itself not a matrix. It might be clearer to replace A with f or some other lower case letter. Just a suggestion.

@bfortuner, thanks for getting us organized to put this together.
I cloned and installed the dependencies and found one to be missing: recommonmark
I am running the usual anaconda3 stack used in fast.ai/part2.
Perhaps you could add this dependency to the install guide.
pip install recommonmark

Currently we have only master branch. We should have dev branch for the changes to be committed so that master is clean and production version.

Currently the glossary of activation functions is limited to only a few functions. There are many newer functions such as Swish, Mish, Phish, Softplus, GELU, etc. that are missing from the glossary.