nfmcclure / tensorflow_cookbook Goto Github PK

Code for Tensorflow Machine Learning Cookbook

Home Page: https://www.packtpub.com/big-data-and-business-intelligence/tensorflow-machine-learning-cookbook-second-edition

License: MIT License

Python 12.30% Jupyter Notebook 87.70%

tensorflow tensorflow-cookbook linear-regression neural-network tensorflow-algorithms rnn cnn svm nlp packtpub

tensorflow_cookbook's Introduction

TensorFlow Machine Learning Cookbook

A Packt Publishing Book

By Nick McClure

=================

Build:

=================

Ch 1: Getting Started with TensorFlow
Ch 2: The TensorFlow Way
Ch 3: Linear Regression
Ch 4: Support Vector Machines
Ch 5: Nearest Neighbor Methods
Ch 6: Neural Networks
Ch 7: Natural Language Processing
Ch 8: Convolutional Neural Networks
Ch 9: Recurrent Neural Networks
Ch 10: Taking TensorFlow to Production
Ch 11: More with TensorFlow

Ch 1: Getting Started with TensorFlow

This chapter intends to introduce the main objects and concepts in TensorFlow. We also introduce how to access the data for the rest of the book and provide additional resources for learning about TensorFlow.

General Outline of TF Algorithms

Here we introduce TensorFlow and the general outline of how most TensorFlow algorithms work.

Creating and Using Tensors

How to create and initialize tensors in TensorFlow. We also depict how these operations appear in Tensorboard.

Using Variables and Placeholders

How to create and use variables and placeholders in TensorFlow. We also depict how these operations appear in Tensorboard.

Working with Matrices

Understanding how TensorFlow can work with matrices is crucial to understanding how the algorithms work.

Declaring Operations

How to use various mathematical operations in TensorFlow.

Implementing Activation Functions

Activation functions are unique functions that TensorFlow has built in for your use in algorithms.

Working with Data Sources

Here we show how to access all the various required data sources in the book. There are also links describing the data sources and where they come from.

Additional Resources

Mostly official resources and papers. The papers are TensorFlow papers or Deep Learning resources.

Ch 2: The TensorFlow Way

After we have established the basic objects and methods in TensorFlow, we now want to establish the components that make up TensorFlow algorithms. We start by introducing computational graphs, and then move to loss functions and back propagation. We end with creating a simple classifier and then show an example of evaluating regression and classification algorithms.

One Operation as a Computational Graph

We show how to create an operation on a computational graph and how to visualize it using Tensorboard.

Layering Nested Operations

We show how to create multiple operations on a computational graph and how to visualize them using Tensorboard.

Working with Multiple Layers

Here we extend the usage of the computational graph to create multiple layers and show how they appear in Tensorboard.

Implementing Loss Functions

In order to train a model, we must be able to evaluate how well it is doing. This is given by loss functions. We plot various loss functions and talk about the benefits and limitations of some.

Implementing Back Propagation

Here we show how to use loss functions to iterate through data and back propagate errors for regression and classification.

Working with Stochastic and Batch Training

TensorFlow makes it easy to use both batch and stochastic training. We show how to implement both and talk about the benefits and limitations of each.

Combining Everything Together

We now combine everything together that we have learned and create a simple classifier.

Evaluating Models

Any model is only as good as it's evaluation. Here we show two examples of (1) evaluating a regression algorithm and (2) a classification algorithm.

Ch 3: Linear Regression

Here we show how to implement various linear regression techniques in TensorFlow. The first two sections show how to do standard matrix linear regression solving in TensorFlow. The remaining six sections depict how to implement various types of regression using computational graphs in TensorFlow.

Using the Matrix Inverse Method

How to solve a 2D regression with a matrix inverse in TensorFlow.

Implementing a Decomposition Method

Solving a 2D linear regression with Cholesky decomposition.

Learning the TensorFlow Way of Linear Regression

Linear regression iterating through a computational graph with L2 Loss.

Understanding Loss Functions in Linear Regression

L2 vs L1 loss in linear regression. We talk about the benefits and limitations of both.

Implementing Deming Regression (Total Regression)

Deming (total) regression implemented in TensorFlow by changing the loss function.

Implementing Lasso and Ridge Regression

Lasso and Ridge regression are ways of regularizing the coefficients. We implement both of these in TensorFlow via changing the loss functions.

Implementing Elastic Net Regression

Elastic net is a regularization technique that combines the L2 and L1 loss for coefficients. We show how to implement this in TensorFlow.

Implementing Logistic Regression

We implement logistic regression by the use of an activation function in our computational graph.

Ch 4: Support Vector Machines

This chapter shows how to implement various SVM methods with TensorFlow. We first create a linear SVM and also show how it can be used for regression. We then introduce kernels (RBF Gaussian kernel) and show how to use it to split up non-linear data. We finish with a multi-dimensional implementation of non-linear SVMs to work with multiple classes.

Introduction

We introduce the concept of SVMs and how we will go about implementing them in the TensorFlow framework.

Working with Linear SVMs

We create a linear SVM to separate I. setosa based on sepal length and pedal width in the Iris data set.

Reduction to Linear Regression

The heart of SVMs is separating classes with a line. We change tweek the algorithm slightly to perform SVM regression.

Working with Kernels in TensorFlow

In order to extend SVMs into non-linear data, we explain and show how to implement different kernels in TensorFlow.

Implementing Non-Linear SVMs

We use the Gaussian kernel (RBF) to separate non-linear classes.

Implementing Multi-class SVMs

SVMs are inherently binary predictors. We show how to extend them in a one-vs-all strategy in TensorFlow.

Ch 5: Nearest Neighbor Methods

Nearest Neighbor methods are a very popular ML algorithm. We show how to implement k-Nearest Neighbors, weighted k-Nearest Neighbors, and k-Nearest Neighbors with mixed distance functions. In this chapter we also show how to use the Levenshtein distance (edit distance) in TensorFlow, and use it to calculate the distance between strings. We end this chapter with showing how to use k-Nearest Neighbors for categorical prediction with the MNIST handwritten digit recognition.

Introduction

We introduce the concepts and methods needed for performing k-Nearest Neighbors in TensorFlow.

Working with Nearest Neighbors

We create a nearest neighbor algorithm that tries to predict housing worth (regression).

Working with Text Based Distances

In order to use a distance function on text, we show how to use edit distances in TensorFlow.

Computing Mixing Distance Functions

Here we implement scaling of the distance function by the standard deviation of the input feature for k-Nearest Neighbors.

Using Address Matching

We use a mixed distance function to match addresses. We use numerical distance for zip codes, and string edit distance for street names. The street names are allowed to have typos.

Using Nearest Neighbors for Image Recognition

The MNIST digit image collection is a great data set for illustration of how to perform k-Nearest Neighbors for an image classification task.

Ch 6: Neural Networks

Neural Networks are very important in machine learning and growing in popularity due to the major breakthroughs in prior unsolved problems. We must start with introducing 'shallow' neural networks, which are very powerful and can help us improve our prior ML algorithm results. We start by introducing the very basic NN unit, the operational gate. We gradually add more and more to the neural network and end with training a model to play tic-tac-toe.

Introduction

We introduce the concept of neural networks and how TensorFlow is built to easily handle these algorithms.

Implementing Operational Gates

We implement an operational gate with one operation. Then we show how to extend this to multiple nested operations.

Working with Gates and Activation Functions

Now we have to introduce activation functions on the gates. We show how different activation functions operate.

Implementing a One Layer Neural Network

We have all the pieces to start implementing our first neural network. We do so here with regression on the Iris data set.

Implementing Different Layers

This section introduces the convolution layer and the max-pool layer. We show how to chain these together in a 1D and 2D example with fully connected layers as well.

Using Multi-layer Neural Networks

Here we show how to functionalize different layers and variables for a cleaner multi-layer neural network.

Improving Predictions of Linear Models

We show how we can improve the convergence of our prior logistic regression with a set of hidden layers.

Learning to Play Tic-Tac-Toe

Given a set of tic-tac-toe boards and corresponding optimal moves, we train a neural network classification model to play. At the end of the script, you can attempt to play against the trained model.

Ch 7: Natural Language Processing

Natural Language Processing (NLP) is a way of processing textual information into numerical summaries, features, or models. In this chapter we will motivate and explain how to best deal with text in TensorFlow. We show how to implement the classic 'Bag-of-Words' and show that there may be better ways to embed text based on the problem at hand. There are neural network embeddings called Word2Vec (CBOW and Skip-Gram) and Doc2Vec. We show how to implement all of these in TensorFlow.

Introduction

We introduce methods for turning text into numerical vectors. We introduce the TensorFlow 'embedding' feature as well.

Working with Bag-of-Words

Here we use TensorFlow to do a one-hot-encoding of words called bag-of-words. We use this method and logistic regression to predict if a text message is spam or ham.

Implementing TF-IDF

We implement Text Frequency - Inverse Document Frequency (TFIDF) with a combination of Sci-kit Learn and TensorFlow. We perform logistic regression on TFIDF vectors to improve on our spam/ham text-message predictions.

Working with Skip-Gram

Our first implementation of Word2Vec called, "skip-gram" on a movie review database.

Working with CBOW

Next, we implement a form of Word2Vec called, "CBOW" (Continuous Bag of Words) on a movie review database. We also introduce method to saving and loading word embeddings.

Implementing Word2Vec Example

In this example, we use the prior saved CBOW word embeddings to improve on our TF-IDF logistic regression of movie review sentiment.

Performing Sentiment Analysis with Doc2Vec

Here, we introduce a Doc2Vec method (concatenation of doc and word embeddings) to improve out logistic model of movie review sentiment.

Ch 8: Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are ways of getting neural networks to deal with image data. CNN derive their name from the use of a convolutional layer that applies a fixed size filter across a larger image, recognizing a pattern in any part of the image. There are many other tools that they use (max-pooling, dropout, etc...) that we show how to implement with TensorFlow. We also show how to retrain an existing architecture and take CNNs further with Stylenet and Deep Dream.

Introduction

We introduce convolutional neural networks (CNN), and how we can use them in TensorFlow.

Implementing a Simple CNN.

Here, we show how to create a CNN architecture that performs well on the MNIST digit recognition task.

Implementing an Advanced CNN.

In this example, we show how to replicate an architecture for the CIFAR-10 image recognition task.

Retraining an Existing Architecture.

We show how to download and setup the CIFAR-10 data for the TensorFlow retraining/fine-tuning tutorial.

Using Stylenet/NeuralStyle.

In this recipe, we show a basic implementation of using Stylenet or Neuralstyle.

Implementing Deep Dream.

This script shows a line-by-line explanation of TensorFlow's deepdream tutorial. Taken from Deepdream on TensorFlow. Note that the code here is converted to Python 3.

Ch 9: Recurrent Neural Networks

Recurrent Neural Networks (RNNs) are very similar to regular neural networks except that they allow 'recurrent' connections, or loops that depend on the prior states of the network. This allows RNNs to efficiently deal with sequential data, whereas other types of networks cannot. We then motivate the usage of LSTM (Long Short Term Memory) networks as a way of addressing regular RNN problems. Then we show how easy it is to implement these RNN types in TensorFlow.

Introduction

We introduce Recurrent Neural Networks and how they are able to feed in a sequence and predict either a fixed target (categorical/numerical) or another sequence (sequence to sequence).

Implementing an RNN Model for Spam Prediction

In this example, we create an RNN model to improve on our spam/ham SMS text predictions.

Implementing an LSTM Model for Text Generation

We show how to implement a LSTM (Long Short Term Memory) RNN for Shakespeare language generation. (Word level vocabulary)

Stacking Multiple LSTM Layers

We stack multiple LSTM layers to improve on our Shakespeare language generation. (Character level vocabulary)

Creating a Sequence to Sequence Translation Model (Seq2Seq)

Here, we use TensorFlow's sequence-to-sequence models to train an English-German translation model.

Training a Siamese Similarity Measure

Here, we implement a Siamese RNN to predict the similarity of addresses and use it for record matching. Using RNNs for record matching is very versatile, as we do not have a fixed set of target categories and can use the trained model to predict similarities across new addresses.

Ch 10: Taking TensorFlow to Production

Of course there is more to TensorFlow than just creating and fitting machine learning models. Once we have a model that we want to use, we have to move it towards production usage. This chapter will provide tips and examples of implementing unit tests, using multiple processors, using multiple machines (TensorFlow distributed), and finish with a full production example.

Implementing Unit Tests

We show how to implement different types of unit tests on tensors (placeholders and variables).

Using Multiple Executors (Devices)

How to use a machine with multiple devices. E.g., a machine with a CPU, and one or more GPUs.

Parallelizing TensorFlow

How to setup and use TensorFlow distributed on multiple machines.

Tips for TensorFlow in Production

Various tips for developing with TensorFlow

An Example of Productionalizing TensorFlow

We show how to do take the RNN model for predicting ham/spam (from Chapter 9, recipe #2) and put it in two production level files: training and evaluation.

Ch 11: More with TensorFlow

To illustrate how versatile TensorFlow is, we will show additional examples in this chapter. We start with showing how to use the logging/visualizing tool Tensorboard. Then we illustrate how to do k-means clustering, use a genetic algorithm, and solve a system of ODEs.

Visualizing Computational Graphs (with Tensorboard)

An example of using histograms, scalar summaries, and creating images in Tensorboard.

Working with a Genetic Algorithm

We create a genetic algorithm to optimize an individual (array of 50 numbers) toward the ground truth function.

Clustering Using K-means

How to use TensorFlow to do k-means clustering. We use the Iris data set, set k=3, and use k-means to make predictions.

Solving a System of ODEs

Here, we show how to use TensorFlow to solve a system of ODEs. The system of concern is the Lotka-Volterra predator-prey system.

Using a Random Forest

We illustrate how to use TensorFlow's gradient boosted regression and classification trees.

Using TensorFlow with Keras

Here we show how to use the Keras sequential model building for a fully connected neural network and a CNN model with callbacks.

tensorflow_cookbook's People

Contributors

Stargazers

Watchers

Forkers

chao-jiang vyraun dkfromsd qicst23 rushidaoyhs bgshin seindlut matheweis zihaow21 josejamilena phpmind dhananjay11 rumary fangwq oscar-lu ozgurgundogan mphat10 pku-wuwei manishperiwal2009 afrendeiro dagangwood163 sajal14 e-lin yunitata ringtail inventionzhang deeplearningresource sachinprabhu007 ouyangshourui wlmwwx fitrialif lovaya abhishekkodi havk64 jgabriellima snowcicala inimah dolaameng akansal1 guohaoqiang wanjinchang wuqixiaobai hundred06 scholltan yiweiyihang irwenqiang falconzyx aracthon benjamesbabala thomasdic2000 kevin369ml reinhardhsu marlboro233544951 cxf2015 ouya-bytes xiximeng caiweining markz2z tandychao lyk125 qibin-google rainygoblin xsongx cosmozhang wuzhongdehua jiehu567 wustmeiming jinlianglovehi chuangchuangtan donnyyou leezqcst libcorner yiqinggit george-wu509 vincentsohjh kv91 laventura alantsev ngineer247 qwshy maxwellzfl robustfengbin wl-tsui rokhoang gokingstones xxlest ipanivko menxia weiwei5444 xzm2004260 hassyma frannetty c4k-tensorflow zhenyu duanliying woodstone121 vanova hitwsl undfined zwang96-dl

tensorflow_cookbook's Issues

shouldn't use softmax

https://github.com/nfmcclure/tensorflow_cookbook/blob/master/09_Recurrent_Neural_Networks/02_Implementing_RNN_for_Spam_Prediction/02_implementing_rnn.py#L120

shouldn't use softmax activation in output layer, when use sparse_softmax_cross_entropy_with_logits

refer:https://www.tensorflow.org/api_docs/python/tf/nn/sparse_softmax_cross_entropy_with_logits

Error running 09 Recurrent_Neural_Networks/05_seq2seq

When running:

print('Creating Translation Model')
input_vocab_size = vocab_size
output_vocab_size = vocab_size

translate_model = translation_model(sess=sess,
                                    input_vocab_size=vocab_size,
                                    output_vocab_size=vocab_size,
                                    buckets=buckets,
                                    rnn_size=rnn_size,
                                    num_layers=num_layers,
                                    max_gradient=max_gradient,
                                    learning_rate=learning_rate,
                                    lr_decay_rate=lr_decay_rate,
                                    forward_only=False)

# Tell TensorFlow to reuse the variables for the test model
with tf.variable_scope(tf.get_variable_scope(), reuse=True):
    #Reuse the variables for the test model
    test_model = translation_model(sess=sess,
                                    input_vocab_size=vocab_size,
                                    output_vocab_size=vocab_size,
                                    buckets=buckets,
                                    rnn_size=rnn_size,
                                    num_layers=num_layers,
                                    max_gradient=max_gradient,
                                    learning_rate=learning_rate,
                                    lr_decay_rate=lr_decay_rate,
                                    forward_only=True)
    test_model.batch_size = 1

Got output:

Creating Translation Model
TypeError Traceback (most recent call last)
in ()
12 learning_rate=learning_rate,
13 lr_decay_rate=lr_decay_rate,
---> 14 forward_only=FCreating Translation Model

TypeError Traceback (most recent call last)
in ()
12 learning_rate=learning_rate,
13 lr_decay_rate=lr_decay_rate,
---> 14 forward_only=False)
15
16 # Tell TensorFlow to reuse the variables for the test model

in translation_model(sess, input_vocab_size, output_vocab_size, buckets, rnn_size, num_layers, max_gradient, learning_rate, lr_decay_rate, forward_only)
14 lr_decay_rate,
15 forward_only=forward_only,
---> 16 dtype=tf.float32)
17 return(model)

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in init(self, source_vocab_size, target_vocab_size, buckets, size, num_layers, max_gradient_norm, batch_size, learning_rate, learning_rate_decay_factor, use_lstm, num_samples, forward_only, dtype)
177 self.target_weights, buckets,
178 lambda x, y: seq2seq_f(x, y, False),
--> 179 softmax_loss_function=softmax_loss_function)
180
181 # Gradients and SGD update operation for training the model.

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in model_with_buckets(encoder_inputs, decoder_inputs, targets, weights, buckets, seq2seq, softmax_loss_function, per_example_loss, name)
1204 variable_scope.get_variable_scope(), reuse=True if j > 0 else None):
1205 bucket_outputs, _ = seq2seq(encoder_inputs[:bucket[0]],
-> 1206 decoder_inputs[:bucket[1]])
1207 outputs.append(bucket_outputs)
1208 if per_example_loss:

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in (x, y)
176 self.encoder_inputs, self.decoder_inputs, targets,
177 self.target_weights, buckets,
--> 178 lambda x, y: seq2seq_f(x, y, False),
179 softmax_loss_function=softmax_loss_function)
180

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in seq2seq_f(encoder_inputs, decoder_inputs, do_decode)
140 output_projection=output_projection,
141 feed_previous=do_decode,
--> 142 dtype=dtype)
143
144 # Feeds for inputs.

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in embedding_attention_seq2seq(encoder_inputs, decoder_inputs, cell, num_encoder_symbols, num_decoder_symbols, embedding_size, num_heads, output_projection, feed_previous, dtype, scope, initial_state_attention)
846 dtype = scope.dtype
847 # Encoder.
--> 848 encoder_cell = copy.deepcopy(cell)
849 encoder_cell = core_rnn_cell.EmbeddingWrapper(
850 encoder_cell,

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
159 copier = getattr(x, "deepcopy", None)
160 if copier:
--> 161 y = copier(memo)
162 else:
163 reductor = dispatch_table.get(cls)

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/python/layers/base.py in deepcopy(self, memo)
474 setattr(result, k, copy.copy(v))
475 else:
--> 476 setattr(result, k, copy.deepcopy(v, memo))
477 return result
478

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_list(x, memo, deepcopy)
213 append = y.append
214 for a in x:
--> 215 append(deepcopy(a, memo))
216 return y
217 d[list] = _deepcopy_list

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
167 reductor = getattr(x, "reduce_ex", None)
168 if reductor:
--> 169 rv = reductor(4)
170 else:
171 reductor = getattr(x, "reduce", None)

TypeError: can't pickle _thread.lock objectsalse)
15
16 # Tell TensorFlow to reuse the variables for the test model

in translation_model(sess, input_vocab_size, output_vocab_size, buckets, rnn_size, num_layers, max_gradient, learning_rate, lr_decay_rate, forward_only)
14 lr_decay_rate,
15 forward_only=forward_only,
---> 16 dtype=tf.float32)
17 return(model)

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in init(self, source_vocab_size, target_vocab_size, buckets, size, num_layers, max_gradient_norm, batch_size, learning_rate, learning_rate_decay_factor, use_lstm, num_samples, forward_only, dtype)
177 self.target_weights, buckets,
178 lambda x, y: seq2seq_f(x, y, False),
--> 179 softmax_loss_function=softmax_loss_function)
180
181 # Gradients and SGD update operation for training the model.

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in model_with_buckets(encoder_inputs, decoder_inputs, targets, weights, buckets, seq2seq, softmax_loss_function, per_example_loss, name)
1204 variable_scope.get_variable_scope(), reuse=True if j > 0 else None):
1205 bucket_outputs, _ = seq2seq(encoder_inputs[:bucket[0]],
-> 1206 decoder_inputs[:bucket[1]])
1207 outputs.append(bucket_outputs)
1208 if per_example_loss:

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in (x, y)
176 self.encoder_inputs, self.decoder_inputs, targets,
177 self.target_weights, buckets,
--> 178 lambda x, y: seq2seq_f(x, y, False),
179 softmax_loss_function=softmax_loss_function)
180

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in seq2seq_f(encoder_inputs, decoder_inputs, do_decode)
140 output_projection=output_projection,
141 feed_previous=do_decode,
--> 142 dtype=dtype)
143
144 # Feeds for inputs.

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in embedding_attention_seq2seq(encoder_inputs, decoder_inputs, cell, num_encoder_symbols, num_decoder_symbols, embedding_size, num_heads, output_projection, feed_previous, dtype, scope, initial_state_attention)
846 dtype = scope.dtype
847 # Encoder.
--> 848 encoder_cell = copy.deepcopy(cell)
849 encoder_cell = core_rnn_cell.EmbeddingWrapper(
850 encoder_cell,

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
159 copier = getattr(x, "deepcopy", None)
160 if copier:
--> 161 y = copier(memo)
162 else:
163 reductor = dispatch_table.get(cls)

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/python/layers/base.py in deepcopy(self, memo)
474 setattr(result, k, copy.copy(v))
475 else:
--> 476 setattr(result, k, copy.deepcopy(v, memo))
477 return result
478

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_list(x, memo, deepcopy)
213 append = y.append
214 for a in x:
--> 215 append(deepcopy(a, memo))
216 return y
217 d[list] = _deepcopy_list

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
167 reductor = getattr(x, "reduce_ex", None)
168 if reductor:
--> 169 rv = reductor(4)
170 else:
171 reductor = getattr(x, "reduce", None)

TypeError: can't pickle _thread.lock objects

Creating Translation Model

TypeError Traceback (most recent call last)
in ()
12 learning_rate=learning_rate,
13 lr_decay_rate=lr_decay_rate,
---> 14 forward_only=False)
15
16 # Tell TensorFlow to reuse the variables for the test model

in translation_model(sess, input_vocab_size, output_vocab_size, buckets, rnn_size, num_layers, max_gradient, learning_rate, lr_decay_rate, forward_only)
14 lr_decay_rate,
15 forward_only=forward_only,
---> 16 dtype=tf.float32)
17 return(model)

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in init(self, source_vocab_size, target_vocab_size, buckets, size, num_layers, max_gradient_norm, batch_size, learning_rate, learning_rate_decay_factor, use_lstm, num_samples, forward_only, dtype)
177 self.target_weights, buckets,
178 lambda x, y: seq2seq_f(x, y, False),
--> 179 softmax_loss_function=softmax_loss_function)
180
181 # Gradients and SGD update operation for training the model.

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in model_with_buckets(encoder_inputs, decoder_inputs, targets, weights, buckets, seq2seq, softmax_loss_function, per_example_loss, name)
1204 variable_scope.get_variable_scope(), reuse=True if j > 0 else None):
1205 bucket_outputs, _ = seq2seq(encoder_inputs[:bucket[0]],
-> 1206 decoder_inputs[:bucket[1]])
1207 outputs.append(bucket_outputs)
1208 if per_example_loss:

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in (x, y)
176 self.encoder_inputs, self.decoder_inputs, targets,
177 self.target_weights, buckets,
--> 178 lambda x, y: seq2seq_f(x, y, False),
179 softmax_loss_function=softmax_loss_function)
180

~/cookbook_py3/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in seq2seq_f(encoder_inputs, decoder_inputs, do_decode)
140 output_projection=output_projection,
141 feed_previous=do_decode,
--> 142 dtype=dtype)
143
144 # Feeds for inputs.

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in embedding_attention_seq2seq(encoder_inputs, decoder_inputs, cell, num_encoder_symbols, num_decoder_symbols, embedding_size, num_heads, output_projection, feed_previous, dtype, scope, initial_state_attention)
846 dtype = scope.dtype
847 # Encoder.
--> 848 encoder_cell = copy.deepcopy(cell)
849 encoder_cell = core_rnn_cell.EmbeddingWrapper(
850 encoder_cell,

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
159 copier = getattr(x, "deepcopy", None)
160 if copier:
--> 161 y = copier(memo)
162 else:
163 reductor = dispatch_table.get(cls)

~/anaconda2/envs/cookbook_py3/lib/python3.6/site-packages/tensorflow/python/layers/base.py in deepcopy(self, memo)
474 setattr(result, k, copy.copy(v))
475 else:
--> 476 setattr(result, k, copy.deepcopy(v, memo))
477 return result
478

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_list(x, memo, deepcopy)
213 append = y.append
214 for a in x:
--> 215 append(deepcopy(a, memo))
216 return y
217 d[list] = _deepcopy_list

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
178 y = x
179 else:
--> 180 y = _reconstruct(x, memo, *rv)
181
182 # If is its own copy, don't memoize.

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy)
278 if state is not None:
279 if deep:
--> 280 state = deepcopy(state, memo)
281 if hasattr(y, 'setstate'):
282 y.setstate(state)

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
148 copier = _deepcopy_dispatch.get(cls)
149 if copier:
--> 150 y = copier(x, memo)
151 else:
152 try:

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in _deepcopy_dict(x, memo, deepcopy)
238 memo[id(x)] = y
239 for key, value in x.items():
--> 240 y[deepcopy(key, memo)] = deepcopy(value, memo)
241 return y
242 d[dict] = _deepcopy_dict

~/anaconda2/envs/cookbook_py3/lib/python3.6/copy.py in deepcopy(x, memo, _nil)
167 reductor = getattr(x, "reduce_ex", None)
168 if reductor:
--> 169 rv = reductor(4)
170 else:
171 reductor = getattr(x, "reduce", None)

TypeError: can't pickle _thread.lock objects

I use ubuntu 16.04, tensorflow1.2.1, python3.6,

Siamese RNN - target values and model configuration

Hi Nick,
get_bach function in 06_siamese_similarity_driver.ipynb generates data with targets -1 or 1, however the loss function in siamese_similarity_model.py seems to be configured to work with targets 0 or 1:

#If y-target is -1 to 1, then do the following
#pos_mult = tf.add(tf.multiply(0.5, y_target), 0.5)
#Else if y-target is 0 to 1, then do the following
pos_mult = tf.cast(y_target, tf.float32)

This part of code also remains commented:

#Change targets from (0,1) --> (-1, 1)
#via (2 * x - 1)
#y_target_int = tf.sub(tf.mul(y_target_int, 2), 1)

I wonder if I should change the loss function if I want to run this model on my data with target -1/1. Best regards!

running error in 09_Recurrent_Neural_Networks/05 seq2seq

when defining translate_model, running this cell:

print('Creating Translation Model')
input_vocab_size = vocab_size
output_vocab_size = vocab_size

translate_model = translation_model(sess=sess,
                                    input_vocab_size=vocab_size,
                                    output_vocab_size=vocab_size,
                                    buckets=buckets,
                                    rnn_size=rnn_size,
                                    num_layers=num_layers,
                                    max_gradient=max_gradient,
                                    learning_rate=learning_rate,
                                    lr_decay_rate=lr_decay_rate,
                                    forward_only=False)

# Tell TensorFlow to reuse the variables for the test model
with tf.variable_scope(tf.get_variable_scope(), reuse=True):
    #Reuse the variables for the test model
    test_model = translation_model(sess=sess,
                                    input_vocab_size=vocab_size,
                                    output_vocab_size=vocab_size,
                                    buckets=buckets,
                                    rnn_size=rnn_size,
                                    num_layers=num_layers,
                                    max_gradient=max_gradient,
                                    learning_rate=learning_rate,
                                    lr_decay_rate=lr_decay_rate,
                                    forward_only=True)
    test_model.batch_size = 1

I got error as:

AttributeErrorTraceback (most recent call last)
in ()
12 learning_rate=learning_rate,
13 lr_decay_rate=lr_decay_rate,
---> 14 forward_only=False)
15
16 # Tell TensorFlow to reuse the variables for the test model

in translation_model(sess, input_vocab_size, output_vocab_size, buckets, rnn_size, num_layers, max_gradient, learning_rate, lr_decay_rate, forward_only)
14 lr_decay_rate,
15 forward_only=forward_only,
---> 16 dtype=tf.float32)
17 return(model)

/home/wenyanli/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in init(self, source_vocab_size, target_vocab_size, buckets, size, num_layers, max_gradient_norm, batch_size, learning_rate, learning_rate_decay_factor, use_lstm, num_samples, forward_only, dtype)
177 self.target_weights, buckets,
178 lambda x, y: seq2seq_f(x, y, False),
--> 179 softmax_loss_function=softmax_loss_function)
180
181 # Gradients and SGD update operation for training the model.

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.pyc in model_with_buckets(encoder_inputs, decoder_inputs, targets, weights, buckets, seq2seq, softmax_loss_function, per_example_loss, name)
1204 variable_scope.get_variable_scope(), reuse=True if j > 0 else None):
1205 bucket_outputs, _ = seq2seq(encoder_inputs[:bucket[0]],
-> 1206 decoder_inputs[:bucket[1]])
1207 outputs.append(bucket_outputs)
1208 if per_example_loss:

/home/wenyanli/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in (x, y)
176 self.encoder_inputs, self.decoder_inputs, targets,
177 self.target_weights, buckets,
--> 178 lambda x, y: seq2seq_f(x, y, False),
179 softmax_loss_function=softmax_loss_function)
180

/home/wenyanli/tensorflow_cookbook/09_Recurrent_Neural_Networks/05_Creating_A_Sequence_To_Sequence_Model/temp/tutorials/rnn/translate/seq2seq_model.py in seq2seq_f(encoder_inputs, decoder_inputs, do_decode)
140 output_projection=output_projection,
141 feed_previous=do_decode,
--> 142 dtype=dtype)
143
144 # Feeds for inputs.

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.pyc in embedding_attention_seq2seq(encoder_inputs, decoder_inputs, cell, num_encoder_symbols, num_decoder_symbols, embedding_size, num_heads, output_projection, feed_previous, dtype, scope, initial_state_attention)
846 dtype = scope.dtype
847 # Encoder.
--> 848 encoder_cell = copy.deepcopy(cell)
849 encoder_cell = core_rnn_cell.EmbeddingWrapper(
850 encoder_cell,

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
172 copier = getattr(x, "deepcopy", None)
173 if copier:
--> 174 y = copier(memo)
175 else:
176 reductor = dispatch_table.get(cls)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/site-packages/tensorflow/python/layers/base.pyc in deepcopy(self, memo)
474 setattr(result, k, copy.copy(v))
475 else:
--> 476 setattr(result, k, copy.deepcopy(v, memo))
477 return result
478

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_list(x, memo)
228 memo[id(x)] = y
229 for a in x:
--> 230 y.append(deepcopy(a, memo))
231 return y
232 d[list] = _deepcopy_list

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
332 if state is not None:
333 if deep:
--> 334 state = deepcopy(state, memo)
335 if hasattr(y, 'setstate'):
336 y.setstate(state)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
332 if state is not None:
333 if deep:
--> 334 state = deepcopy(state, memo)
335 if hasattr(y, 'setstate'):
336 y.setstate(state)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
332 if state is not None:
333 if deep:
--> 334 state = deepcopy(state, memo)
335 if hasattr(y, 'setstate'):
336 y.setstate(state)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
332 if state is not None:
333 if deep:
--> 334 state = deepcopy(state, memo)
335 if hasattr(y, 'setstate'):
336 y.setstate(state)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_list(x, memo)
228 memo[id(x)] = y
229 for a in x:
--> 230 y.append(deepcopy(a, memo))
231 return y
232 d[list] = _deepcopy_list

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
332 if state is not None:
333 if deep:
--> 334 state = deepcopy(state, memo)
335 if hasattr(y, 'setstate'):
336 y.setstate(state)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
332 if state is not None:
333 if deep:
--> 334 state = deepcopy(state, memo)
335 if hasattr(y, 'setstate'):
336 y.setstate(state)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
332 if state is not None:
333 if deep:
--> 334 state = deepcopy(state, memo)
335 if hasattr(y, 'setstate'):
336 y.setstate(state)

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_list(x, memo)
228 memo[id(x)] = y
229 for a in x:
--> 230 y.append(deepcopy(a, memo))
231 return y
232 d[list] = _deepcopy_list

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_tuple(x, memo)
235 y = []
236 for a in x:
--> 237 y.append(deepcopy(a, memo))
238 d = id(x)
239 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
161 copier = _deepcopy_dispatch.get(cls)
162 if copier:
--> 163 y = copier(x, memo)
164 else:
165 try:

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _deepcopy_dict(x, memo)
255 memo[id(x)] = y
256 for key, value in x.iteritems():
--> 257 y[deepcopy(key, memo)] = deepcopy(value, memo)
258 return y
259 d[dict] = _deepcopy_dict

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in deepcopy(x, memo, _nil)
188 raise Error(
189 "un(deep)copyable object of type %s" % cls)
--> 190 y = _reconstruct(x, rv, 1, memo)
191
192 memo[d] = y

/home/wenyanli/anaconda2/envs/cookbook/lib/python2.7/copy.pyc in _reconstruct(x, info, deep, memo)
341 slotstate = None
342 if state is not None:
--> 343 y.dict.update(state)
344 if slotstate is not None:
345 for key, value in slotstate.iteritems():

AttributeError: 'NoneType' object has no attribute 'update'

previous codes all work fine. How can I get this code work?

Solution: 02_Ten/04_loss_..., the last part with tf.nn.sparse_softmax_cross_entropy_with_logits

I'm not quite understand how this function work, but in my computer. Do the following changes can make code works.Otherwise, errors are given.
Tensorflow 1.0.0

sparse_xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=unscaled_logits,labels= sparse_target_dist)
print(sess.run(sparse_xentropy))

The solution when 02_TensorFlow_way , 02_layering_nested_... can't work

I'm not sure how many people have the same problem with me, when to finish the 02_layering_nested_operations. In my version , the default type of numpy and the constant of tensorflow is not the same. So when you meet the problem, you can try to use np.float32 or dtype=tf.float32, to make sure all variables have the same types to help sess.run() .

Love this cookbook!
and a suggestion. The last part in 02_Tensorflow_... can be replaced as:

merged = tf.summary.merge_all()
if not os.path.exists('tensorboard_logs/'):
os.makedirs('tensorboard_logs/')

my_writer = tf.summary.FileWriter('tensorboard_logs/', sess.graph)

How do we predict data without targets in SVM Kernels?

Hello, thanks for the excellent work.

I am confused in 04_svm_kernels.ipynb of Chapter 4 Support Vector Machine, where we always require y_target in prediction because of dual calculation. However, in real world practice, if we only have X_test to predict Y_test, how do we get prediction without the labels?

Do you have any insight about this?

Thanks

403 Forbidden when trying to access Low Birthweight Data

Clicking on the link "Low Birthweight Data" in the 07_Working_with_Data_Sources directory gets:

You don't have permission to access /statdata/statdata/data/lowbwt.txt on this server.

I would like to request your permission as author so that I can translate some parts of your book

Dear Nick,
I started reading your book entitled “TensorFlow Machine Learning Cookbook” about a month ago, and I should say it's a well-written book.

I really liked some of the ideas that you wrote and I would like to share them with students in my country. I would like to request your permission as author so that I can translate some parts of your book into the Persian language with some modifications. The modifications are required to provide context to ensure the translation is understood correctly. I am not doing this for financial gain; I would just like students in my country become more familiar with the topic and your book provides an excellent resource on this subject. Needless to say, I will keep all links and give proper attribution to you as the author. The translated sections will be published on my blog, and I can provide you with the link once complete.

I really appreciate you considering my request. If you have any questions regarding the intent of my translation or the exact sections I wish to translate, please do not hesitate to ask. I look forward to receiving your response.

06_Neural_Networks/06_using_a_multiple_layer_network.ipynb (list index out of range)

During compiling following piece of code:

birth_weight_file = 'birth_weight.csv'

download data and create data file if file does not exist in current directory

if not os.path.exists(birth_weight_file):
birthdata_url = 'https://github.com/nfmcclure/tensorflow_cookbook/raw/master/01_Introduction/07_Working_with_Data_Sources/birthweight_data/birthweight.dat'
birth_file = requests.get(birthdata_url)
birth_data = birth_file.text.split('\r\n')
birth_header = birth_data[0].split('\t')
birth_data = [[float(x) for x in y.split('\t') if len(x)>=1] for y in birth_data[1:] if len(y)>=1]
with open(birth_weight_file, "w") as f:
writer = csv.writer(f)
writer.writerows(birth_data)
f.close()

read birth weight data into memory

birth_data = []
with open(birth_weight_file, newline='') as csvfile:
csv_reader = csv.reader(csvfile)
birth_header = next(csv_reader)
for row in csv_reader:
birth_data.append(row)

birth_data = [[float(x) for x in row] for row in birth_data]

y_vals = np.array([x[10] for x in birth_data])

I faced a problem:

IndexError Traceback (most recent call last)
in ()
28
29 # Extract y-target (birth weight)
---> 30 y_vals = np.array([x[10] for x in birth_data])

in (.0)
28
29 # Extract y-target (birth weight)
---> 30 y_vals = np.array([x[10] for x in birth_data])

IndexError: list index out of range

It is obviously because

print(birth_data[10]) is given as empty row. However, I do not know fix it, could please someone help me?

I am new in python before I was working with Matlab and a little bit with C++, therefore a piece of code which is responsible for '# download data and creates data file if file does not exist in current directory'
is quite unclear. I will be extremely thankful if someone explains briefly what is happing there.

1.0.0rc2 tensorflow has no attribute 'merge_all_summaries'

This error will show up when I run the example 02_tensors.ipynb

Inconsistent results in LASSO Regression - Chapter 3, Section 6

@nfmcclure ... while working on the LASSO regression I noticed one issue. As with the other notebooks, I added code for reproducibility. Depending on the random seed, the LASSO regression returns nan results.

Here are some examples.

LASSO code fragment:

# For Lasso, uncomment the following four lines (and comment out the Ridge Regression loss below)
lasso_param = tf.constant(0.9)
heavyside_step = tf.truediv(1., tf.add(1., tf.exp(tf.multiply(-100., tf.subtract(A, lasso_param)))))
regularization_param = tf.multiply(heavyside_step, 99.)
loss = tf.add(tf.reduce_mean(tf.square(y_target - model_output)), regularization_param)

With this seed setting:

seed = 13
np.random.seed(seed)
tf.set_random_seed(seed)

The training code

# Initialize variables
init = tf.global_variables_initializer()
sess.run(init)

# Training loop
loss_vec = []
for i in range(1500):
    rand_index = np.random.choice(len(x_vals), size=batch_size)
    rand_x = np.transpose([x_vals[rand_index]])
    rand_y = np.transpose([y_vals[rand_index]])
    sess.run(train_step, feed_dict={x_data: rand_x, y_target: rand_y})
    temp_loss = sess.run(loss, feed_dict={x_data: rand_x, y_target: rand_y})
    loss_vec.append(temp_loss[0])
    if (i+1)%300==0:
        print('Step #' + str(i+1) + ' A = ' + str(sess.run(A)) + ' b = ' + str(sess.run(b)))
        print('Loss = ' + str(temp_loss))
        #print('regularization_param='+ str(regularization_param))
        #print('heavyside step=' + str(sess.run(heavyside_step)))
        #print('\n')

Results in this output:

Step #300 A = [[ nan]] b = [[ nan]]
Loss = [[ nan]]
Step #600 A = [[ nan]] b = [[ nan]]
Loss = [[ nan]]
Step #900 A = [[ nan]] b = [[ nan]]
Loss = [[ nan]]
Step #1200 A = [[ nan]] b = [[ nan]]
Loss = [[ nan]]
Step #1500 A = [[ nan]] b = [[ nan]]
Loss = [[ nan]]

With this random seed setting:

seed = 31
np.random.seed(seed)
tf.set_random_seed(seed)

Results in this output:

Step #300 A = [[ 0.82903057]] b = [[ 1.62271953]]
Loss = [[ 10.61604691]]
Step #600 A = [[ 0.82230437]] b = [[ 3.08032537]]
Loss = [[ 3.35721326]]
Step #900 A = [[ 0.81690294]] b = [[ 3.88035631]]
Loss = [[ 1.26691902]]
Step #1200 A = [[ 0.81232369]] b = [[ 4.32818317]]
Loss = [[ 0.69326025]]
Step #1500 A = [[ 0.80592763]] b = [[ 4.57499456]]
Loss = [[ 0.33951065]]

Additional note, there are no issues with the Ridge Regression.

Question in 02_Tensor/03_multiple_... , feed_dict={x_data=x_val}

First, I change the type of x_val with np.float32 into y_val. When I run:
mov_avg_layer= tf.nn.conv2d(y_val, my_filter, my_strides,
padding='SAME', name='Moving_Avg_Window')
print(sess.run(mov_avg_layer))
I get the result. But when I try the feed_dict, my kernel runs out.

When I put the codes in my terminal python envirment, my operation system runs wrong.
I use the Gpu size of tensorflow, did this double? Or other things make my feed_dict goes out?
mov_avg_layer= tf.nn.conv2d(x_data, my_filter, my_strides,
padding='SAME', name='Moving_Avg_Window')
print(sess.run(mov_avg_layer,feed_dict={x_data:y_val}))
The above command make my kernel die without any information.

06_Neural_../04_single_..., batch size maters

The examle with default 50 batch size can't converge, so I replace it with 10 or 15 receiving better condition.
I think the batch size matters, but don't understand.
And I have another question, the y_target has the shape [None, 1], and the final_ouput has the shape [1].
Why can they subtract with each other?

ValueError: Only call `sparse_softmax_cross_entropy_with_logits` with named arguments (labels=..., logits=..., ...)

when I run the chapter8：08_Convolutional_Neural_Networks，02_Intro_to_CNN_MNIST ,the example give me a error ,I don't know how to sovle.

the error as follow:

ValueError Traceback (most recent call last)
in ()
109
110 # Declare Loss Function (softmax cross entropy)
--> 111 loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(model_output, y_target))
112
113 # Create a prediction function

E:\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py in sparse_softmax_cross_entropy_with_logits(_sentinel, labels, logits, name)
1682 """
1683 _ensure_xent_args("sparse_softmax_cross_entropy_with_logits", _sentinel,
-> 1684 labels, logits)
1685
1686 # TODO(pcmurray) Raise an error when the label is not an index in

E:\Anaconda3\lib\site-packages\tensorflow\python\ops\nn_ops.py in _ensure_xent_args(name, sentinel, labels, logits)
1531 if sentinel is not None:
1532 raise ValueError("Only call %s with "
-> 1533 "named arguments (labels=..., logits=..., ...)" % name)
1534 if labels is None or logits is None:
1535 raise ValueError("Both labels and logits must be provided.")

ValueError: Only call sparse_softmax_cross_entropy_with_logits with named arguments (labels=..., logits=..., ...)

Question: 06_Neural_../05_implement_.. , about the function tf.nn.conv2d()

tf.nn.conv2d(input,filter,strides,padding) is the function of convolutional network in tensorflow.
Here, the author says the input of 4-D data should have the size [batch#, width, height, channels],
but the official web gives the description:
input tensor of shape [batch, in_height, in_width, in_channels]
Will this make difference , if we replace the place of width and height in our experience?

best way is only use transform,not fit_transform.

https://github.com/nfmcclure/tensorflow_cookbook/blob/master/07_Natural_Language_Processing/02_Working_with_Bag_of_Words/02_bag_of_words.py#L138

the use of np.roll in Shakespeare Example

Hello, I am trying to understand this operiaton in Shakespeare example

# Create targets from shuffled batches
targets = [np.roll(x, -1, axis=1) for x in batches]

From my test below

import numpy as np
arr = np.array([
    [1, 2, 3, 4, 5],
    [6, 7, 8, 9, 10],
    [11, 12, 13, 14, 15],
])
np.roll(arr, -1, axis=1)

which results in

array([[ 2,  3,  4,  5,  1],
       [ 7,  8,  9, 10,  6],
       [12, 13, 14, 15, 11]])

I don't really understand what does this mean, because I think logically we should move the first world of next sentence to the end of current sentence (and in that case we are predicting the next word).

I even think put axis=0 may be better, because we are trying to predict the next sentence in the batch in that case.

Thank you for the time.

Error at chapter 7 Working with CBOW Embeddings

Hi, there are some errors in the CBOW example but this one is something I can't solve.

Traceback (most recent call last):
File "05_Working_With_CBOW.py", line 125, in
window_size, method='cbow')
File "/home/test/cbow/text_helpers.py", line 97, in generate_batch_data
batch, labels = [list(x) for x in zip(*batch_and_labels)]
ValueError: not enough values to unpack (expected 2, got 0)

Thanks,

cross-validation with multiclass SVM

Hi,
I have adapted the code of multiclass SVM to my dataset, it is working well, however I don't know how to use k-fold cross validation in the training loop. Any help or guidance would be very appreciated. Thanks

some questions about the detail of "05_nonlinear_svm.py"

the example can work well, but I wonder the mathematical theory of the code, could you show me how the code solve the svm problem.
To be specific, I tried the SVM in sklearn, but it is too slow. But in sklearn there is a parameter "C" which can control the softmargin, I want to know if the code support the softmargin and how it works.
thanks!

The switch problem in jupyter notebook .

When I finish one example, then switch to another. My jupyter notebook shows the problem
The kernel appears to die
Restart can't help, and I have to close all of them. After restarting jupyter notebook and run into the same example, it can work. But do this is not comfortable. Why my condition happens?

AttributeError: 'module' object has no attribute 'rnn_cell' in 09_Recurrent_Neural_Networks/03_Implementing_LSTM/03_implementing_lstm.py

Hi, sorry, i'm absoultely new in Tensorflow and may be telling nonesence.

This example
https://github.com/nfmcclure/tensorflow_cookbook/blob/master/09_Recurrent_Neural_Networks/03_Implementing_LSTM/03_implementing_lstm.py

raise with this error:

AttributeError                            Traceback (most recent call last)
<ipython-input-1-2bd8df52813f> in <module>()
    211     # Define LSTM Model
    212     lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,
--> 213                             training_seq_len, vocab_size)
    214     scope.reuse_variables()
    215     test_lstm_model = LSTM_Model(rnn_size, batch_size, learning_rate,

<ipython-input-1-2bd8df52813f> in __init__(self, rnn_size, batch_size, learning_rate, training_seq_len, vocab_size, infer_sample)
    134             self.training_seq_len = training_seq_len
    135 
--> 136         self.lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(rnn_size)
    137         self.initial_state = self.lstm_cell.zero_state(self.batch_size, tf.float32)
    138 

AttributeError: 'module' object has no attribute 'rnn_cell'

>>> import pkg_resources
>>> pkg_resources.get_distribution("tensorflow").version
'1.0.0'

import platform
print(platform.python_version())
2.7.6

i'm using this Docker image

tensorflow/tensorflow   latest              ea40dcc45724        2 weeks ago         1.029 GB

thanks

https://github.com/nfmcclure/tensorflow_cookbook/blob/master/09_Recurrent_Neural_Networks/03_Implementing_LSTM/03_implementing_lstm.py

Version Semantic and other Bugs in tensorflow_cookbook/09_Recurrent_Neural_Networks/04_Stacking_Multiple_LSTM_Layers/04_stacking_multiple_lstm.py

In line 136:
self.lstm_cell = tf.contrib.rnn.BasicLSTMCell(rnn_size)
change to
self.lstm_cell = tf.contrib.rnn.BasicLSTMCell(rnn_size, reuse=tf.get_variable_scope().reuse)

In line 137:
self.lstm_cell = tf.contrib.rnn.MultiRNNCell([self.lstm_cell] * self.num_layers)
change to
self.lstm_cell = tf.contrib.rnn.MultiRNNCell([self.lstm_cell for self.lstm_cell in range(num_layers)])

In line 138:
self.initial_state = self.lstm_cell.zero_state(self.batch_size, tf.float32)

Please help, as I am getting the error:
/core_rnn_cell_impl.py", line 929, in
return tuple(cell.zero_state(batch_size, dtype) for cell in self._cells)

AttributeError: 'int' object has no attribute 'zero_state'

Or else:

In line 137:
change to
self.lstm_cell = tf.contrib.rnn.MultiRNNCell([self.lstm_cell for _ in range(num_layers)])

Then an error occurs:

self._value = int(value)

TypeError: int() argument must be a string, a bytes-like object or a number, not 'tuple'

Double softmax in cross entropy with logits

Just wanted to point out that you have some examples where you use
logits_out = tf.nn.softmax(tf.matmul(last, weight) + bias)

and then define the loss function as:
losses = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits_out, labels=y_output)

You are calling softmax twice. From TensorFlow documentation:

WARNING: This op expects unscaled logits, since it performs a softmax on logits internally for efficiency. Do not call this op with the output of softmax, as it will produce incorrect results.

Have a nice day :)

dynamic_rnn_decoder

Hi, I am wondering Is it possible to use the new dynamic_rnn_decoder to replace the legacy rnn decoder in rnn chapter for text generation, and update the code?

In 07_Natural_Language_Processing / 02_Working_with_Bag_of_Words / 02_bag_of_words.py

From Line 29 to Line 34 these codes block show as follows:
"
if os.path.isfile(save_file_name):
text_data = []
with open(save_file_name, 'r') as temp_output_file:
reader = csv.reader(temp_output_file)
for row in reader:
text_data.append(row)
"
There lack a little small code when the 'temp_spam_data.csv' have been exists in local.
There needs to split the "row" data with ',' to give to text_data. Otherwise, there will be some error message like:
"---> 53 texts = [x[1] for x in text_data]
54 target = [x[0] for x in text_data]
IndexError: list index out of range
"

06_multiclass_svm.py ValueError: Dimensions

Python 3.5
Tensorflow GPU 1.0.1
Commit: 69396a0

File: 06_multiclass_svm.py
Line: 74
second_term = tf.reduce_sum(tf.multiply(my_kernel, tf.multiply(b_vec_cross, y_target_cross)),[1,2])

Error:
ValueError: Dimensions must be equal, but are 2 and 3 for 'Mul_3' (op: 'Mul') with input shapes: [2,?,1], [3,50,50].

06_Neural_Networks/05_Implementing_Different_Layers (

NonLinear SMV

Hi ,
you are using formulas given below to calculate your accuracy.

prediction_output = tf.matmul(tf.mul(tf.transpose(y_target),b), pred_kernel)
prediction = tf.sign(prediction_output-tf.reduce_mean(prediction_output))
accuracy = tf.reduce_mean(tf.cast(tf.equal(tf.squeeze(prediction), tf.squeeze(y_target)), tf.float32))

you are using directly y_target (ground truth) to make a prediction and then comparing the prediction with y_target.
Firstly I dont understand what is going on there. I mean "predicting by using target value" sounds meaningless a little bit. Probably I am missing something, but I couldnt find what it is.

At the end of the day, if I want to predict output when test input data are applied without knowing output how can I predict my outputs ?

By the way, there is a line in the code

model_output = tf.matmul(b, my_kernel)

But you dont use model_output anywhere in your code.

Thanks.

question about 06_using_a_multiple_layer_network.py

Hello! I have a question about the mlp network which has around 200 parameters. But I check the sample size is less than 200. Depending on the seeds of random number, it's able to get converged results sometimes. my question is that how is it possible and is it the optimal model to run regression analysis to get the birth weight? Thanks!

Replace tf.initialize_all_variables() with tf.global_variables_initializer()?

They seem to be making changes with TensorFlow

It looks like the following initialization should be used..

init = tf.global_variables_initializer()

Below is the older code, which gives the following warning...

init = tf.initialize_all_variables()

WARNING:tensorflow:From <ipython-input-15-961de099ec6b>:61 in <module>.: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be 
removed after 2017-03-02.

Solution in 02_Ten_../04_loss_.., sigmoid entropy loss don't work well

tf.nn.sigmoid_cross_entropy_with_logits(_sentinel=None, labels=None, logits=None, name=None) don't suggest using _sentinel , because it refer to positional parameters.
But in my computer and python environment, if just put in
tf.nn.sigmoid_cross_entropy_with_logits(x_vals,targets)
It gives warnings, so I think maybe this function receive x_vals as _sentinel. When add the labels , I get the resule:
xentropy_sigmoid_y_vals = tf.nn.sigmoid_cross_entropy_with_logits(labels=x_vals,logits= targets)
This can work on python3.5 and tensorflow 1.0.0

Missing CIFAR-10 Image Data

/01_Introduction/07_Working_with_Data_Sources/ is missing a link to the CIFAR-10 dataset

[08\05_stylenet.py] cannot show the correct picture.

When I run "tensorflow_cookbook-master\08_Convolutional_Neural_Networks\05_Stylenet_NeuralStyle\05_stylenet.py" , it comes out the below message:
D:\Anaconda3\lib\site-packages\scipy\misc\pilutil.py:98: RuntimeWarning: invalid value encountered in greater
bytedata[bytedata > high] = high
D:\Anaconda3\lib\site-packages\scipy\misc\pilutil.py:99: RuntimeWarning: invalid value encountered in less
bytedata[bytedata < 0] = 0
Generation 1000 out of 10000
Generation 1500 out of 10000
Generation 2000 out of 10000
Generation 2500 out of 10000
...
And finally, I got all the black picture. Are the two warning caused the black picture?

08_logistic_regression

I just tried 08_logistic_regression code. However, I got the error in follows.
Give me some advice?

IndexError Traceback (most recent call last)
in ()
26
27 # Pull out target variable
---> 28 y_vals = np.array([x[1] for x in birth_data])
29 # Pull out predictor variables (not id, not target, and not birthweight)
30 x_vals = np.array([x[2:9] for x in birth_data])

in (.0)
26
27 # Pull out target variable
---> 28 y_vals = np.array([x[1] for x in birth_data])
29 # Pull out predictor variables (not id, not target, and not birthweight)
30 x_vals = np.array([x[2:9] for x in birth_data])

IndexError: list index out of range

why 'embedding_size' should be equal to 'rnn_size' in chapter9 03 implementing lstm

rnn_size = 128 # RNN Model size, has to equal embedding size

I don't know why there has to be equal between these two hyper-params? could you be kind to explain it for us, thanks.

tf.nn.seq2seq. cannot be found with tf version 1.0

In 09_Recurrent_Neural_Networks\03_Implementing_LSTM\03_implementing_lstm.py ,
there could not find seq2seq. I think maybe update to version 1.0 of tf caused this problem.
"
--> 166 decoder = tf.nn.seq2seq.rnn_decoder
167 outputs, last_state = decoder(rnn_inputs_trimmed,
168 self.initial_state,

AttributeError: module 'tensorflow.python.ops.nn' has no attribute 'seq2seq'
"

ValueError: Attempt to reuse RNNCell <tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.GRUCell

After running the Seq2Seq example I got:



---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-8e0fa45c722e> in <module>()
    209                                     learning_rate=learning_rate,
    210                                     lr_decay_rate=lr_decay_rate,
--> 211                                     forward_only=False)
    212 
    213 # Tell TensorFlow to reuse the variables for the test model

<ipython-input-1-8e0fa45c722e> in translation_model(sess, input_vocab_size, output_vocab_size, buckets, rnn_size, num_layers, max_gradient, learning_rate, lr_decay_rate, forward_only)
    193           lr_decay_rate,
    194           forward_only=forward_only,
--> 195           dtype=tf.float32)
    196     return(model)
    197 

/Users/user/Jupyter/quora_challenge/temp/tutorials/rnn/translate/seq2seq_model.py in __init__(self, source_vocab_size, target_vocab_size, buckets, size, num_layers, max_gradient_norm, batch_size, learning_rate, learning_rate_decay_factor, use_lstm, num_samples, forward_only, dtype)
    177           self.target_weights, buckets,
    178           lambda x, y: seq2seq_f(x, y, False),
--> 179           softmax_loss_function=softmax_loss_function)
    180 
    181     # Gradients and SGD update operation for training the model.

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in model_with_buckets(encoder_inputs, decoder_inputs, targets, weights, buckets, seq2seq, softmax_loss_function, per_example_loss, name)
   1199           variable_scope.get_variable_scope(), reuse=True if j > 0 else None):
   1200         bucket_outputs, _ = seq2seq(encoder_inputs[:bucket[0]],
-> 1201                                     decoder_inputs[:bucket[1]])
   1202         outputs.append(bucket_outputs)
   1203         if per_example_loss:

/Users/user/Jupyter/quora_challenge/temp/tutorials/rnn/translate/seq2seq_model.py in <lambda>(x, y)
    176           self.encoder_inputs, self.decoder_inputs, targets,
    177           self.target_weights, buckets,
--> 178           lambda x, y: seq2seq_f(x, y, False),
    179           softmax_loss_function=softmax_loss_function)
    180 

/Users/user/Jupyter/quora_challenge/temp/tutorials/rnn/translate/seq2seq_model.py in seq2seq_f(encoder_inputs, decoder_inputs, do_decode)
    140           output_projection=output_projection,
    141           feed_previous=do_decode,
--> 142           dtype=dtype)
    143 
    144     # Feeds for inputs.

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py in embedding_attention_seq2seq(encoder_inputs, decoder_inputs, cell, num_encoder_symbols, num_decoder_symbols, embedding_size, num_heads, output_projection, feed_previous, dtype, scope, initial_state_attention)
    853         embedding_size=embedding_size)
    854     encoder_outputs, encoder_state = core_rnn.static_rnn(
--> 855         encoder_cell, encoder_inputs, dtype=dtype)
    856 
    857     # First calculate a concatenation of encoder outputs to put attention on.

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn.py in static_rnn(cell, inputs, initial_state, dtype, sequence_length, scope)
    195             state_size=cell.state_size)
    196       else:
--> 197         (output, state) = call_cell()
    198 
    199       outputs.append(output)

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn.py in <lambda>()
    182       if time > 0: varscope.reuse_variables()
    183       # pylint: disable=cell-var-from-loop
--> 184       call_cell = lambda: cell(input_, state)
    185       # pylint: enable=cell-var-from-loop
    186       if sequence_length is not None:

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py in __call__(self, inputs, state, scope)
    879         embedded = embedding_ops.embedding_lookup(
    880             embedding, array_ops.reshape(inputs, [-1]))
--> 881     return self._cell(embedded, state)
    882 
    883 

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py in __call__(self, inputs, state, scope)
    951                 state, [0, cur_state_pos], [-1, cell.state_size])
    952             cur_state_pos += cell.state_size
--> 953           cur_inp, new_state = cell(cur_inp, cur_state)
    954           new_states.append(new_state)
    955     new_states = (tuple(new_states) if self._state_is_tuple else

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py in __call__(self, inputs, state, scope)
    144   def __call__(self, inputs, state, scope=None):
    145     """Gated recurrent unit (GRU) with nunits cells."""
--> 146     with _checked_scope(self, scope or "gru_cell", reuse=self._reuse):
    147       with vs.variable_scope("gates"):  # Reset gate and update gate.
    148         # We start with bias of 1.0 to not reset and not update.

/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py in __enter__(self)
     80     def __enter__(self):
     81         try:
---> 82             return next(self.gen)
     83         except StopIteration:
     84             raise RuntimeError("generator didn't yield") from None

/usr/local/lib/python3.6/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.py in _checked_scope(cell, scope, reuse, **kwargs)
     75             "this error will remain until then.)"
     76             % (cell, cell_scope.name, scope_name, type(cell).__name__,
---> 77                type(cell).__name__))
     78     else:
     79       weights_found = False

ValueError: Attempt to reuse RNNCell <tensorflow.contrib.rnn.python.ops.core_rnn_cell_impl.GRUCell object at 0x129464ac8> with a different variable scope than its first use.  First use of cell was with scope 'embedding_attention_seq2seq/embedding_attention_decoder/attention_decoder/multi_rnn_cell/cell_0/gru_cell', this attempt is with scope 'embedding_attention_seq2seq/rnn/multi_rnn_cell/cell_0/gru_cell'.  Please create a new instance of the cell if you would like it to use a different set of weights.  If before you were using: MultiRNNCell([GRUCell(...)] * num_layers), change to: MultiRNNCell([GRUCell(...) for _ in range(num_layers)]).  If before you were using the same cell instance as both the forward and reverse cell of a bidirectional RNN, simply create two instances (one for forward, one for reverse).  In May 2017, we will start transitioning this cell's behavior to use existing stored weights, if any, when it is called with scope=None (which can lead to silent model degradation, so this error will remain until then.)

Any idea of how to fix it?.

Question about “04_Support_Vector_Machines\04_Working_with_Kernels”

Hi, in 04_svm_kernels.py shows below, I have some questions to need your help, please.

In Line 54, the variable "model_output" seems not to be used in the whole py file, is there missed some codes?
Not very understand on how to calculate the loss function when use the my_kernel. From the code I guess the calculation is that: my_kernel *(b^T *b) * (y * y^T) ,and then reduce_sum of it.
And the loss equals "reduce_sum(b)- the above result"
So I don't understand why use this formula to get the loss function calculation result. Could you give me some help to understanding the process, please?
"
# Compute SVM Model
54 model_output = tf.matmul(b, my_kernel)
55 first_term = tf.reduce_sum(b)
56 b_vec_cross = tf.matmul(tf.transpose(b), b)
57 y_target_cross = tf.matmul(y_target, tf.transpose(y_target))
58 second_term = tf.reduce_sum(tf.multiply(my_kernel, tf.multiply(b_vec_cross, y_target_cross)))
59 loss = tf.negative(tf.subtract(first_term, second_term))
"

can't understand the mean of code in SVM kernel?

Can you explain more detail in code SVM kernel, in loss function and predict, i try to read more about SVM to understand but can't, or you can send me some blog with math( in your code) it will better for me. Thank you so much.

ImportError: No module named 'tensorflow.models.rnn' cannot import data_utils

In 09_Recurrent_Neural_Networks\05_Creating_A_Sequence_To_Sequence_Model
05_seq2seq_translation.py
There was some import error as follows:
ImportError Traceback (most recent call last)
in ()
21 from zipfile import ZipFile
22 from collections import Counter
---> 23 from tensorflow.models.rnn.translate import data_utils
24 from tensorflow.models.rnn.translate import seq2seq_model
25 from tensorflow.python.framework import ops

ImportError: No module named 'tensorflow.models.rnn'

And could you please give me some suggestion about how to find the right module in ver 1.0 after the update changed

ipython notebook tensorboard outputs

Looks like the ipynb files are using some deprecated functions:

For section: Create and Format Tensorboard outputs for viewing

Error:

AttributeError Traceback (most recent call last)
in ()
----> 1 merged = tf.merge_all_summaries()
2
3 if not os.path.exists('tensorboard_logs/'):
4 os.makedirs('tensorboard_logs/')
5

AttributeError: module 'tensorflow' has no attribute 'merge_all_summaries'

the following change fixes it:

# merged = tf.merge_all_summaries()
merged = tf.summary.merge_all(key='summaries')

Error:

AttributeError Traceback (most recent call last)
in ()
6
7
----> 8 my_writer = tf.train.SummaryWriter('tensorboard_logs/', sess.graph)
9 # my_writer = tf.summary.FileWriter('tensorboard_logs/', sess.graph)

AttributeError: module 'tensorflow.python.training.training' has no attribute 'SummaryWriter'

the following change fixes it:

# my_writer = tf.train.SummaryWriter('tensorboard_logs/', sess.graph)
my_writer = tf.summary.FileWriter('tensorboard_logs/', sess.graph)

chapter_7:skip_gram and cbow programmes are wrong

when reading codes for this algorithm, I found one of the most important step: calculate similarity( cosine) (in both skip gram and cbow) is wrong, as it is just generated randomly, and never updated when training, you can find the results printed are not different at any time. i.e., the results is wrong. please kindly correct this crucial error, thanks. @nfmcclure @jimthompson5802 @slcott @havk64 @cxxgtxy

Arabic display issue in following your tutorial about machine translation

Hi Nick,

First, thanks for the great book. It's well-written and I like the way you present things.

I have followed "Creating Sequence-to-Sequence Models" and I managed to make it work. It worked for English German, but when I used the same with English Arabic and basically got the same data from "http://www.manythings.org/anki/ara-eng.zip", the Arabic text for the examples is not displayed correctly.

Here is what I am getting:

English: the quick brown fox jumped over the lazy dog arabic: ['ظ‡ظ„', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†', 'ظ…ظ†']

What should I do to get this displayed correctly?

My other question:

once the model is trained and ready to use, how can I pass a sentence for translation? I would like to use some other sentences.

Thanks in advance for your support.

Kind regards

GradientDescentOptimizer example is very sensitive to initial seed for A and learning rate

I've been getting inconsistent results with the Deming regression example given in your text. I'm running Tensorflow locally on an iMac using MacOS Sierra 10.12.4. I then explored using different combinations of initial values for A (rather than the random normal example in the text), and then different learning rates. I found that starting values for A that were negative often led to poor fits (lines with negative slopes and large intercepts, suggesting gradients that were moving in the wrong direction -- obviously diverging from what the raw data would otherwise suggest on inspection), even though the loss function (or Deming distance) improved throughout the optimization given sufficient iterations.

I understand the impact that learning rate has on convergence as well, but wondered whether you could suggest a good way to determine the initial value for the variables and the learn rate. Also, what the best way to determine if the optimization is achieving reasonable results? I've looked at examining the actual gradient calculations during iteration, but thought the must be a better way...

Model Examples Need Saving and Evaluation Added

Make sure that all significant models save the model space (graph, variables, and definitions) properly and have a script that allows for (1) loading of most recent model, and (2) evaluation of new input.

Version Update Bug in siamese_similarity_model.py

anaconda/lib/python2.7/site-packages/tensorflow/contrib/rnn/python/ops/core_rnn_cell_impl.pyc in _checked_scope(cell, scope, reuse, **kwargs)
     91             "To share the weights of an RNNCell, simply "
     92             "reuse it in your second calculation, or create a new one with "
---> 93             "the argument reuse=True." % (scope_name, type(cell).__name__))
     94 
     95     # Everything is OK.  Update the cell's scope and yield it.

ValueError: Attempt to have a second RNNCell use the weights of a variable scope that already has weights: 'bidirectional_rnn/fw/basic_lstm_cell'; and the cell was not constructed as BasicLSTMCell(..., reuse=True).  To share the weights of an RNNCell, simply reuse it in your second calculation, or create a new one with the argument reuse=True.

In the multi-class classification with SVM example code, "reshape_mul" function's last line should be return(tf.batch_matmul(v2, v1)) rather than tf.matmul(v2,v1)

nfmcclure / tensorflow_cookbook Goto Github PK

tensorflow_cookbook's Introduction

By Nick McClure

Table of Contents

tensorflow_cookbook's People

Contributors

Stargazers

Watchers

Forkers

tensorflow_cookbook's Issues

TypeError Traceback (most recent call last) in () 12 learning_rate=learning_rate, 13 lr_decay_rate=lr_decay_rate, ---> 14 forward_only=FCreating Translation Model

Creating Translation Model

During compiling following piece of code:

download data and create data file if file does not exist in current directory

read birth weight data into memory

IndexError: list index out of range

the error as follow:

Recommend Projects

Recommend Topics

Recommend Org

TypeError Traceback (most recent call last)
in ()
12 learning_rate=learning_rate,
13 lr_decay_rate=lr_decay_rate,
---> 14 forward_only=FCreating Translation Model