michaldanieldobrzanski / deeplearningpython Goto Github PK

I've found that you are using sigmoid_prime function in network.py and network2.py to calculate backpropagation. But it is not necessary.

You had calculated activation list in feedforward pass. sigmoid_prime is the derivative of sigmoid function, which is equal to activation * (1 - activation) .

You have no need to repeated calculate this sigmoid derivative function, which is really what old-backpropa paper suggest, just one feedforward pass, and a backward pass.

In the file mnist_svm.py there are missing parentheses in call to 'print'.

This code

for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation)+b
            zs.append(z)
            activation = sigmoid(z)
            activations.append(activation)

throws the following error, and I don't really know why. (I am using Python 3.8.3 and numpy 1.18.4)

ValueError: operands could not be broadcast together with shapes (10,784) (10,30)

Hi Michal,
Thank you for putting up Nielsen's code in Python 3.5. I am trying to build a backpropagation algorithm for a custom physics guided loss function where a neural network works in tandem with a physics-based simulator for a regression problem. Since heavy customizations are required, libraries like Keras are not suitable for the job and I have to build the neural network from scratch. I read the first two chapters of Nielsen's book and found the explanation and code quite suitable to use as a starting template. My clarifications are:
1.) Can I start with network.py or should I use the improved versions (network2.py & network3.py as a starting base?
2.) Which activation function should I use? Can I continue using Sigmoid as in the code or should I implement ReLu or another activation function for regression predictions?
3.) What parts of the code I should change for this regression problem so that the network architecture accepts an np.array of 7 input features and 1 output column, because right now it does not accept multidimensional array in the training data?

Running Mac OS X Sierra High, Python 3.6, got the Error:
ImportError: cannot import name 'downsample', when running python test.py

Solution:
Had to change the following lines to get it work:

#from theano.tensor.signal import downsample
from theano.tensor.signal import pool

#pooled_out = downsample.max_pool_2d(
pooled_out = pool.pool_2d(

and also clear the theano cache: theano-cache purge

Dear Concern,

I am trying to see the data structure inside the .pkl file but i couldn't get until. can you tell me that what type of data is inside this file. as i want to run your example on .csv data. Thanks in advance!

Seems like variable cost_ij = train_mb(minibatch_index) defined in Network3 is never used.

Can this be converted back to the original format? Ideally exactly as in the original files (train labels, train data, test labels, test data), all in ubyte form?

Getting this error on line:
z = np.dot(w, activation)+b

In Network.backprop(), in this snippet:

for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation)+b
            zs.append(z)
            activation = sigmoid(z)
            activations.append(activation)

Anyone got any ideas what's going wrong?

The network1's demo works ok by python352, but when i run the following script:
import mnist_loader
training_data, validation_data, test_data = mnist_loader.load_data_wrapper()
import network2
net = network2.Network([784, 30, 10], cost=network2.CrossEntropyCost)
net.large_weight_initializer()
net.SGD(training_data[:1000], 400, 10, 0.5, evaluation_data=test_data,monitor_evaluation_accuracy=True, monitor_training_cost=True)

something wrong:
net.SGD(training_data[:1000], 400, 10, 0.5, evaluation_data=test_data,monitor_evaluation_accuracy=True, monitor_training_cost=True)
TypeError: 'zip' object is not subscriptable

i have try to change zip() to list(zip()) ,but it's doesn't work. do you have any suggestion?

我用定义[4,3,2]的网络跟踪发现
def backprop(self, x, y):
z = np.dot(w, activation)+b
这里打印出来的z是一个矩阵，不是一个vector，不知道什么原因。
能否指点一下，谢谢！

python 3.6.5 Anaconda still gives this error for

f = gzip.open('mnist.pkl.gz', 'rb')
training_data, validation_data, test_data = pickle.load(f, encoding="latin1")
f.close()

You should update the following lines of code in network3.py:
Line 39 to from theano.tensor.nnet import conv2d
Line 227 to conv_out = conv2d
Line 229 to input_shape=self.image_shape

Note: I recommend to use anaconda to run the code.

https://github.com/MichalDanielDobrzanski/DeepLearningPython35/blob/ea229ac6234b7f3373f351f0b18616ca47edb8a1/network2.py#L253

here it should be
delta = self.cost_derivative(activations[-1], y) * sigmoid_prime(zs[-1])
the code from the book does that

Go to (wherever you have the package installed)/theano/gof/cc.py

Remove this section here (around line 1376):

# We must always add the numpy ABI version here as 
          # DynamicModule always add the include <numpy/arrayobject.h> 
        if np.lib.NumpyVersion(np.__version__)<'1.16.0a': 
            ndarray_c_version = np.core.multiarray._get_ndarray_c_version() 
        else: 
            ndarray_c_version = 
np.core._multiarray_umath._get_ndarray_c_version() 
          sig.append('NPY_ABI_VERSION=0x%X' % 
                   ndarray_c_version) 
                   np.core.multiarray._get_ndarray_c_version()) 

up to but not including this part:

  if c_compiler: 
          sig.append('c_compiler_str=' + c_compiler.version_str()) 

The error happened in the network3.py line 107: training_y[i * self.mini_batch_size: (i + 1) * self.mini_batch_size].the error detail info as you can see:
Traceback (most recent call last):
File "/home/magic/workSpace/neural-networks-and-deep-learning-master/src/test.py", line 179, in
net.SGD(training_data, 60, mini_batch_size, 0.1, validation_data, test_data)
File "/home/magic/workSpace/neural-networks-and-deep-learning-master/src/network3.py", line 107, in SGD
training_y[i * self.mini_batch_size: (i + 1) * self.mini_batch_size]
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/compile/function.py", line 326, in function
output_keys=output_keys)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/compile/pfunc.py", line 486, in pfunc
output_keys=output_keys)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1795, in orig_function
defaults)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1661, in create
input_storage=input_storage_lists, storage_map=storage_map)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/link.py", line 699, in make_thunk
storage_map=storage_map)[:3]
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/vm.py", line 1047, in make_all
impl=impl))
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/op.py", line 935, in make_thunk
no_recycling)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/op.py", line 839, in make_c_thunk
output_storage=node_output_storage)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1190, in make_thunk
keep_lock=keep_lock)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1131, in compile
keep_lock=keep_lock)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1586, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1159, in module_from_key
module = lnk.compile_cmodule(location)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1489, in compile_cmodule
preargs=preargs)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 2325, in compile_str
return dlimport(lib_filename)
File "/home/magic/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 302, in dlimport
rval = import(module_name, {}, {}, [module_name])
ImportError: /home/magic/.theano/compiledir_Linux-4.10--generic-x86_64-with-debian-stretch-sid-x86_64-3.6.1-64/tmprvy0e7qh/m30510001e5ca6bbdd01766b2da67f9bf.so: undefined symbol: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEED1Ev

Process finished with exit code 1

when i run this test code:
import network3
from network3 import Network, ConvPoolLayer, FullyConnectedLayer, SoftmaxLayer
training_data, validation_data, test_data = network3.load_data_shared()
mini_batch_size = 10
net = Network([
ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28),
filter_shape=(20, 1, 5, 5),
poolsize=(2, 2)),
FullyConnectedLayer(n_in=201212, n_out=100),
SoftmaxLayer(n_in=100, n_out=10)], mini_batch_size)
net.SGD(training_data, 60, mini_batch_size, 0.1, validation_data, test_data)

I am facing overflow issues for np.exp while running SGD

When running Network3.py I go the following error:
FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result.
yk[[region_slices[i][r[i]] for i in xrange(nd)]])
I think it is because Theano has updated since the code was written but I don't know what to change to fix the error

This is not even an issue but I'm not sure where else to put it. Basically I'm having a problem with saving the network to a file. What code do I run to start it up?

Which python version is currently supported? Does the related pkg requirements need some change too?

Hello. Thank you for code.
I want to replace sigmoid with with relu, but cant not improve the accuracy_score。
modify code：

def sigmoid(z):
#     return 1.0 / (1.0 + np.exp(-z))
    return z * (z > 0)
def sigmoid_prime(z):
#     return sigmoid(z) * (1 - sigmoid(z))
    return 1.0 * (z > 0)

but ：

Epoch 0: 401 / 4200, Accuracy score: 0.09547619047619048
Epoch 1: 401 / 4200, Accuracy score: 0.09547619047619048
Epoch 2: 401 / 4200, Accuracy score: 0.09547619047619048

Thanks for help me.

import network

net = network.Network([784, 30, 10])
Gives
TypeError: 'builtin function or module' object has no attribute "getitem"
Please help.!

self.weights = [(1-eta*(lmbda/n))*w-(eta/len(mini_batch))*nw for w, nw in zip(self.weights, nabla_w)]
why n is the training dataset size not the batch_size?

def total_cost(self, data, lmbda, convert=False):
        cost = 0.0
        for x, y in data:
            a = self.feedforward(x)
            if convert: y = vectorized_result(y)
            cost += self.cost.fn(a, y)/len(data)
            cost += 0.5*(lmbda/len(data))*sum(np.linalg.norm(w)**2 for w in self.weights) # '**' - to the power of.
        return cost

the cost of regularization should not in the loop?

Hello,

I am running the following commands on Jupyter Notebook (Anaconda) and getting following error. Not able to proceed with programming.

import numpy as np

AttributeError Traceback (most recent call last)
in
----> 1 import numpy as np

~\Anaconda3\lib\site-packages\numpy_init_.py in
138 from . import _distributor_init
139
--> 140 from . import core
141 from .core import *
142 from . import compat

~\Anaconda3\lib\site-packages\numpy\core_init_.py in
96 # do this after everything else, to minimize the chance of this misleadingly
97 # appearing in an import-time traceback
---> 98 from . import _add_newdocs
99 # add these for module-freeze analysis (like PyInstaller)
100 from . import _dtype_ctypes

~\Anaconda3\lib\site-packages\numpy\core_add_newdocs.py in
4402 allocating the array data. Returns the previously set value.
4403 See global_state for more information.
-> 4404 """)
4405
4406 add_newdoc('numpy.core._multiarray_tests', 'format_float_OSprintf_g',

~\Anaconda3\lib\site-packages\numpy\core\function_base.py in add_newdoc(place, obj, doc, warn_on_python)
504 If possible it should be avoided.
505 """
--> 506 new = getattr(import(place, globals(), {}, [obj]), obj)
507 if isinstance(doc, str):
508 _add_docstring(new, doc.strip(), warn_on_python)

AttributeError: module 'numpy.core.multiarray' has no attribute '_set_madvise_hugepage'

lscalar 'i' now can't be used directly to splice the list. Can you please fix it?
Error: "TypeError: slice indices must be integers or None or have an index method"

When I download the data set and script, and just make the change to 'import pickle as cPickle' because without it I was getting: 'NameError: name 'cPickle' is not defined', I now get the error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x90 in position 614: ordinal not in range(128)

I don't think cPickle should be imported in expand_mnist.py. Shouldn't it be pickle instead?

I am getting the error 'zip object is not subscriptable' when I run the following command in python 3:

net.SGD(training_data[:1000], 10, 10, 0.5, evaluation_data=test_data, lmbda = 0.1, # this is a regularization parameter monitor_evaluation_cost=True, monitor_evaluation_accuracy=True, monitor_training_cost=True, monitor_training_accuracy=True)

I have tried changing the mnist_loader by wrapping the zip commands with list, but it's not working.

Problem occurred during compilation with the command line below:
"C:\Users\qhy89\anaconda3\Library\mingw-w64\bin\g++.exe" -shared -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -march=knl -mmmx -mno-3dnow -msse -msse2 -msse3 -mssse3 -mno-sse4a -mcx16 -msahf -mmovbe -maes -msha -mpclmul -mpopcnt -mabm -mno-lwp -mfma -mno-fma4 -mno-xop -mbmi -mbmi2 -mno-tbm -mavx -mavx2 -msse4.2 -msse4.1 -mlzcnt -mno-rtm -mno-hle -mrdrnd -mf16c -mfsgsbase -mrdseed -mprfchw -madx -mfxsr -mxsave -mxsaveopt -mavx512f -mno-avx512er -mavx512cd -mno-avx512pf -mno-prefetchwt1 -mclflushopt -mxsavec -mxsaves -mavx512dq -mavx512bw -mavx512vl -mno-avx512ifma -mno-avx512vbmi -mno-clwb -mno-pcommit -mno-mwaitx --param l1-cache-size=48 --param l1-cache-line-size=64 --param l2-cache-size=6144 -mtune=generic -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -m64 -DMS_WIN64 -I"C:\Users\qhy89\anaconda3\lib\site-packages\numpy\core\include" -I"C:\Users\qhy89\anaconda3\include" -I"C:\Users\qhy89\anaconda3\lib\site-packages\theano\gof\c_code" -L"C:\Users\qhy89\anaconda3\libs" -L"C:\Users\qhy89\anaconda3" -o "C:\Users\qhy89\AppData\Local\Theano\compiledir_Windows-10-10.0.18362-SP0-Intel64_Family_6_Model_126_Stepping_5_GenuineIntel-3.8.5-64\tmpv4y5u640\m5096d6781863c6f6b868780ba93cb6b25dfed943b18b3a27a7e36b0f506f0a3c.pyd" "C:\Users\qhy89\AppData\Local\Theano\compiledir_Windows-10-10.0.18362-SP0-Intel64_Family_6_Model_126_Stepping_5_GenuineIntel-3.8.5-64\tmpv4y5u640\mod.cpp" -lpython38C:\Users\qhy89\AppData\Local\Temp\cce1q7pg.s: Assembler messages:

C:\Users\qhy89\AppData\Local\Temp\cce1q7pg.s:90: Error: invalid register for .seh_savexmm