aapatel09 / handson-unsupervised-learning Goto Github PK

def view_digit(example):
label = y_train.loc[example] #compared to old label = y_train.loc[0]

X_train has 30 columns but the input layer to the autoencoder expects 29.
np.shape(X_train)
(193823, 30)

model = Sequential()
model.add(Dense(units=40, activation='linear',
activity_regularizer=regularizers.l1(10e-5),
input_dim=29,name='hidden_layer'))

Were you supposed to drop the Amount column in addition to Class and Time?
dataX.columns
Index(['Unnamed: 0', 'V1', 'V2', 'V3', 'V4', 'V5', 'V6', 'V7', 'V8', 'V9',
'V10', 'V11', 'V12', 'V13', 'V14', 'V15', 'V16', 'V17', 'V18', 'V19',
'V20', 'V21', 'V22', 'V23', 'V24', 'V25', 'V26', 'V27', 'V28',
'Amount'],
dtype='object')

NameError Traceback (most recent call last)
in
----> 1 im = impute.SimpleImputer(strategy='mean')
2 data.loc[:,fillWithMean] = im.fit_transform(data[fillWithMean])
3
4 data.loc[:,fillWithZero] = data.loc[:,fillWithZero].fillna(value=0,axis=1)

NameError: name 'impute' is not defined

I have to configure an anomaly detection system. I found very useful what you wrote but I have a doubt.
I train PCA on the the train set. Then I get anomalyScores with the following function

def anomalyScores(originalDF, reducedDF):
    loss = np.sum((np.array(originalDF)-np.array(reducedDF))**2, axis=1)
    loss = pd.Series(data=loss,index=originalDF.index)
    loss = (loss-np.min(loss)) / (np.max(loss)-np.min(loss))
    return loss

At this point I try to use this trained algorithm for the test set.

The point is that I'am not sure that anomalyScores so defined is suitble for this task.
Maybe using a MinMaxScaler would be better (or using this function but annotating the minimum and maximum obtained during the first evaluation) ?
This because consider the following example at inference time:

#during training I got the following losses
rec_error = np.array([0, 0.1, 1])
loss = (loss-np.min(loss)) / (np.max(loss)-np.min(loss)) #returns [0, 0.1, 1]

#now during testing suppose I got these reconstruction errors
rec_error = np.array([0, 0.1, 0.2])
loss = (loss-np.min(loss)) / (np.max(loss)-np.min(loss)) #returns [0, 0.5, 1]

So a reconstruction error of 0.1 wuold be considered much larger in the second example even if the trained model is the same.
While with a MinMaxScaler I would get:

from sklearn.preprocessing import MinMaxScaler
from sklearn.utils.validation import check_is_fitted

rec_error_train = np.array([ [0.0], [0.1], [1]])
rec_error_test = np.array([ [0.0], [0.1], [0.2]])
scaler = MinMaxScaler((0,1)).fit(rec_error_train)

loss_train = scaler.transform(rec_error_train) #returns [0, 0.1, 1]
loss_test = scaler.transform(rec_error_test) #returns [0, 0.1, 0.2]

What do you think?

I do not know if this is an issue of the code, or just the consequence of my limited understanding of what is going on in a piece of code. In the notebook 02_end_to_end_machine_learning_project.ipynb, why the for loop is like this in the section about logistic regression:

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)), y_train.ravel()):

Instead of just

for train_index, cv_index in k_fold.split(X_train, y_train):

What is behind that np.zeros and the call to the method ravel()?

If there is no reason for doing it the way it is done, I think the simpler version would be better, given that this is code to teach people how to use scikitlearn library.

On line 36, you are passing two arguments to anomalyScores; X_test and predictions. Is that right? Shouldn't the arguments be y_test and predictions?

I am trying to do anomaly detection with Normal PCA and 27 principal components as presented in the book but if I run the model on the entire dataset (without splitting into train and set) I get a completely different result:

data = pd.read_csv('credit_card.zip')
X = data.copy().drop(["Class"],axis=1)
y = data['Class'].copy()

featuresToScale = X.columns
scaler = StandardScaler(copy=True)
X.loc[:,featuresToScale] = scaler.fit_transform(X[featuresToScale])

pca = PCA(n_components=n_components, random_state=42)

X_train_PCA = pca.fit_transform(X)
X_train_PCA = pd.DataFrame(data=X_train_PCA, index=X.index)

X_train_PCA_inverse = pca.inverse_transform(X_train_PCA)
X_train_PCA_inverse = pd.DataFrame(data=X_train_PCA_inverse, index=X.index)

anomalyScoresPCA = anomalyScores(X, X_train_PCA_inverse)
plotResults(y, anomalyScoresPCA, False) 

The score training PCA only on the train set (after splitting into train and test) is 0.69

Do you have any ideas on why this appens ?

In your book, chapter2, if you scaling datasets first, it will leak the information of testing dataset.

In[50] of 08_autoencoders.ipynb cannot be reproduced.
I'm using Python 3.7 tensorflow 1.14.

Missing

https://github.com/aapatel09/handson-unsupervised-learning/tree/master/datasets/credit_card_data/credit_card.csv

file.

Otherwise very nice book. 🙂

the easiest way for me to install xgboost was through conda:
conda install -c conda-forge xgboost

In Chapter 4 (Anomaly Detection), in the PCA part, it is mentioned the the code for finding the optimal number of principal components to be used (reduced from 30 to 27 in the book) is in this repository, but I could not find it.

Where is it?

In model 9-10 of chapter 8 I don't understand why the target label of the fitting phase is the noisy input. Doing this the autoencoders learns to reconstruct the noisy input, isn't it ?
Should instead be written in this way ?

history = model.fit(x=X_train_AE_noisy, y=X_train_AE,  #using the original input as target
                            epochs=10,
                            batch_size=32,
                            shuffle=True,
                            validation_data=(X_val_AE_noisy, X_val),
                            verbose=1)

Original code is:

history = model.fit(x=X_train_AE_noisy, y=X_train_AE_noisy,
                        epochs=num_epochs,
                        batch_size=batch_size,
                        shuffle=True,
                        validation_data=(X_train_AE, X_train_AE),
                        verbose=1)

I have also replaced
validation_data=(X_train_AE, X_train_AE)
with
validation_data=(X_val_AE_noisy, X_val)
Where X_val is obtained by splitting the train set into 2 subsets. What do you think ?

Moreover why predictions are made on the noised test set ? I tried to predict the original test set.
Executing the code with these updates I got:

Mean average precision over 10 runs: 0.6327761266275229
Coefficient of variation over 10 runs: 0.018682160539079985
[0.6243557484686052,
0.6346261618115446,
0.6111520781820431,
0.6294291611519081,
0.6387461341507094,
0.6258058349652793,
0.6550653293459515,
0.6473704778074738,
0.6260815948098096,
0.6351287455819044]

In [32]:
preds = pd.concat([y_train,predictionsBasedOnKFolds.loc[:,1]], axis=1)

the above is wrong... it would always grab prediction for class 1

so if we had predict_proba give us something like
index prob0 prob1
1 0.9 0.1

and label is
index label
1 0

so above would produce preds as
index trueLabel prection
1 0 0.1

should be 0.9, not 0.1

Just trying to execute the code as is and I'm getting the following error:
trainingScores = []
cvScores = []
predictionsBasedOnKFolds = pd.DataFrame(data=[], index=y_train.index,
columns=['prediction'])

for train_index, cv_index in k_fold.split(np.zeros(len(X_train)),
y_train.ravel()):
X_train_fold, X_cv_fold = X_train.iloc[train_index,:],
X_train.iloc[cv_index,:]
y_train_fold, y_cv_fold = y_train.iloc[train_index],
y_train.iloc[cv_index]

lgb_train = lgb.Dataset(X_train_fold, y_train_fold)
lgb_eval = lgb.Dataset(X_cv_fold, y_cv_fold, reference=lgb_train)
gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
               valid_sets=lgb_eval, early_stopping_rounds=200)

loglossTraining = log_loss(y_train_fold, gbm.predict(X_train_fold, \
                            num_iteration=gbm.best_iteration))
trainingScores.append(loglossTraining)

predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'] = \
    gbm.predict(X_cv_fold, num_iteration=gbm.best_iteration) 
loglossCV = log_loss(y_cv_fold, \
    predictionsBasedOnKFolds.loc[X_cv_fold.index,'prediction'])
cvScores.append(loglossCV)

print('Training Log Loss: ', loglossTraining)
print('CV Log Loss: ', loglossCV)

loglossLightGBMGradientBoosting = log_loss(y_train,
predictionsBasedOnKFolds.loc[:,'prediction'])
print('LightGBM Gradient Boosting Log Loss: ',
loglossLightGBMGradientBoosting)

LightGBMError Traceback (most recent call last)
in
14 lgb_eval = lgb.Dataset(X_cv_fold, y_cv_fold, reference=lgb_train)
15 gbm = lgb.train(params_lightGB, lgb_train, num_boost_round=2000,
---> 16 valid_sets=lgb_eval, early_stopping_rounds=200)
17
18 loglossTraining = log_loss(y_train_fold, gbm.predict(X_train_fold, \

~/.local/lib/python3.6/site-packages/lightgbm/engine.py in train(params, train_set, num_boost_round, valid_sets, valid_names, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, evals_result, verbose_eval, learning_rates, keep_training_booster, callbacks)
226 # construct booster
227 try:
--> 228 booster = Booster(params=params, train_set=train_set)
229 if is_valid_contain_train:
230 booster.set_train_data_name(train_data_name)

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in init(self, params, train_set, model_file, model_str, silent)
1712 self.handle = ctypes.c_void_p()
1713 _safe_call(_LIB.LGBM_BoosterCreate(
-> 1714 train_set.construct().handle,
1715 c_str(params_str),
1716 ctypes.byref(self.handle)))

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in construct(self)
1083 init_score=self.init_score, predictor=self._predictor,
1084 silent=self.silent, feature_name=self.feature_name,
-> 1085 categorical_feature=self.categorical_feature, params=self.params)
1086 if self.free_raw_data:
1087 self.data = None

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in _lazy_init(self, data, label, reference, weight, group, init_score, predictor, silent, feature_name, categorical_feature, params)
913 raise TypeError('Wrong predictor type {}'.format(type(predictor).name))
914 # set feature names
--> 915 return self.set_feature_name(feature_name)
916
917 def __init_from_np2d(self, mat, params_str, ref_dataset):

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in set_feature_name(self, feature_name)
1366 self.handle,
1367 c_array(ctypes.c_char_p, c_feature_name),
-> 1368 ctypes.c_int(len(feature_name))))
1369 return self
1370

~/.local/lib/python3.6/site-packages/lightgbm/basic.py in _safe_call(ret)
43 """
44 if ret != 0:
---> 45 raise LightGBMError(decode_string(_LIB.LGBM_GetLastError()))
46
47

LightGBMError: Do not support special JSON characters in feature name.