miladmozafari / spyketorch Goto Github PK
View Code? Open in Web Editor NEWHigh-speed simulator of convolutional spiking neural networks with at most one spike per neuron.
License: GNU General Public License v3.0
spyketorch's Issues
score can't stabilize
Hi miladmozafari, I am trying to implement your SpykeTorch code.I have some issues.
use face and motorbicyle images, during the training, the score will increase first and then decrease with the increase of the number of iterations, can't stabilize, do you know what the problem is?
No learning with STDP
Hello. I am new to SpykeTorch. I am working on an anomaly detection project and I would like to learn features from spectrograms using a Convolutional SNN. I have been struggling so far for the past week with STDP. My model achieves decent performance (~ 70% AUC) but does not learn or very few (gain of 2-5% AUC). I have tried a lot of things : parameter tuning (number of winners, firing threshold, inhibition radius) but also adaptive learning rate, adaptive firing threshold, etc. I don't understand where the problem comes from because I know I can obtain > 90% AUC with a regular CNN.
My pipeline is : signal -> spectrogram (MFSC) -> CSNN -> ML outlier detection classifier
So I have several questions :
- Why do you usually choose a very small number of winners ? If there is at most one winner per feature map, isn't it a good idea to choose one winner per feature map ?
- When to "modify/filter" potentials before choosing winners ? Using things like
sf.pointwise_inhibition(pot)orsf.threshold(pot)leads to worse performance in my case. - Is taking the mean of spikes / potentials from all timesteps a good idea for readout ?
Do you have any intuition about something I am doing wrong ?
Here is my model, I can also send you my whole code if you have the motivation to check it out.
class CSNN(nn.Module):
def __init__(self,
input_shape,
):
super(CSNN, self).__init__()
self.ctx = {}
out_channels = 50
kernel_height = 7
in_nb_spike_bins, in_channels, in_frames, in_freqs = input_shape
output_height = in_frames - kernel_height
self.conv = snn.Convolution(in_channels=in_channels, out_channels=out_channels, kernel_size=(kernel_height,in_freqs), weight_mean=0.8, weight_std=0.05)
self.stdp = snn.STDP(self.conv, learning_rate = (0.004, -0.003))
self.pool = snn.Pooling(kernel_size = (4,1), stride = (4,1), padding = 0)
self.firing_thr = 26
self.nb_winners = 1
self.inhib_rad = 0
self.max_ap = Parameter(torch.Tensor([0.15]))
self.mean_pot = 0
self.counter = 0
def get_thr(self):
return self.mean_pot / self.counter
def forward(self, input):
input = input.float()
if self.training:
pot = self.conv(input)
spk,pot = sf.fire(pot, self.firing_thr, return_thresholded_potentials=True)
#pot = sf.pointwise_inhibition(pot)
#spk = pot.sign() #remove spk where pot is now null
winners = sf.get_k_winners(pot, kwta=self.nb_winners, inhibition_radius=self.inhib_rad, spikes=spk)
self.save_stdp_data(input, pot, spk, winners)
else:
pot = self.conv(input)
pot = self.pool(pot)
self.mean_pot += pot.mean()
self.counter += 1
spk = sf.fire(pot, self.firing_thr)
return spk, pot
def save_stdp_data(self, input_spikes, potentials, output_spikes, winners):
self.ctx['input_spikes'] = input_spikes
self.ctx['potentials'] = potentials
self.ctx['output_spikes'] = output_spikes
self.ctx['winners'] = winners
def update_stdp(self):
self.stdp(self.ctx['input_spikes'], self.ctx['potentials'], self.ctx['output_spikes'], self.ctx['winners'])
def update_learning_rate(self):
ap = torch.tensor(self.stdp.learning_rate[0][0].item(), device=self.stdp.learning_rate[0][0].device) * 2
ap = torch.min(ap, self.max_ap)
an = ap * -0.75
self.stdp.update_all_learning_rate(ap.item(), an.item())Notes :
- My experiments are for a number of epochs < 10
Can traditional STDP weight updating rule be applied in this model?
Hello,
thanks a lot for the library! I am current trying to realize R-STDP with emerging devices, and is wondering if traditional weight updating rule in STDP can be applied in this model. That is, the delta_w calculated by STDP/R-STDP being proportional to exp(-delta_t/tau), instead of proportional to w(1-w). Is it possible for the algorithm to work like that?
(not a native English speaker here, i apologize if anything conveyed being ambiguous)
The system cannot find the path specified: 'facemotortrain'
Hi miladmozafari, I am trying to implement your SpykeTorch code.I have some issues.
First, in your ipynb file kept face and motorbicyle images but i got this error:
ValueError: Unknown resampling filter (64). Use Image.NEAREST (0), Image.LANCZOS (1), Image.BILINEAR (2), Image.BICUBIC (3), Image.BOX (4) or Image.HAMMING (5)
Here i am using GaborFilter, Should i use DoG filters instead? Or i need to chanege some parametes?
Second: i am trying you code "MozafariShallow" when i tried to run your code i got this error:
Waiting for your response.
Thanks
RunTimeError using multiple GPUs.
Hello,
Thank You for this library. I have been using the mozafari.py network to train my spiking network model. Since the dataset is imagenet I wanted to use a multiGPU setup. I used the data parallel module from pytorch to train with 8 GPUs like following:
mozafari = torch.nn.DataParallel(mozafari, device_ids=[1, 7]) File "/data-mount/spiking-CVT/SpykeTorch/snn.py", line 219, in forward
lr[f] = torch.where(pairings[i], *(self.learning_rate[f]))
RuntimeError: Expected condition, x and y to be on the same device, but condition is on cuda:7 and x and y are on cuda:0 and cuda:0 respectivelydataset
hi~, I want to ask a question.
I am very interested in your network, but I want to use my dataset, which is a csv file. However, I don’t know what your MNIST input format looks like, numpy, list, or tensor? If I want to prepare it myself, what kind of format do I need?
Because my data set is not a picture data set, but can be directly input into numpy, list or tensor.
The accuracy of cifar10 is not high
Hello, I changed the training data into cifar10 and then cifar10 into gray-scale image, but the accuracy is not high, only 36%. The general CNN network has an accuracy rate of more than 66%. Why?
about datasets
hi~ I have used this toolbox, but i do not know how to prepare the proper datasets file format in my folder, so could you provide me with you reimplement datasets link, or tell me how can i prepare the datasets?
Thank you!
global pooling in KheradpishehDeep.py
Hi miladmozafari. Thanks for providing such a useful toolbox for building convolutional spiking neural networks. I am trying to implement Kheradpisheh's work using your script (KheradpishehDeep.py). However, I found that in Kheradpisheh's paper the threshold of neurons in the last convolutional layer were set to be infinite and the global pooling was performed for classification, which is inconsistent with your code. Would you please explain this?
Simulation Time
Hi Milad,
How much time did your simulations take for the Deep network to run on MNIST on CPUs vs GPUs?
How to visualise features
Hi, Mila!
I was confused by the feature visualization:
1.in MozafariDeep network, the conv3.weight's kernel size is small(about 5*5), but the visualized features are huge images.
2.I tried some feature visualization methods in CNN, but all methods require gradient (BP).
I don't know if there is a magic method for STDP network's feature visualization.
3. The same confusions for KheradpishehDeep network.
Thanks very very much~
Accuracy problem
Hi, I ran MozafariDeep.py and got the result below. The accuracy is not as good as paper 97.2%.
Current Train: [0.98281667 0.01718333 0. ]
Best Train: [9.83033333e-01 1.69666667e-02 0.00000000e+00 6.68000000e+02]
Current Test: [0.9637 0.0363 0. ]
Best Test: [9.649e-01 3.510e-02 0.000e+00 6.770e+02]
time elapsed: 95007.85 seconds
Parameter Selection: out_channel
Hello,
Thank you for the intuitive tutorial in the included notebook. I am trying to change to modify the provided unsupervised network to learn the MNIST handwritten digit dataset.
I am curious as to how you decided to select your out_channel in the conv layer and also your firing threshold. Why did you choose 20 for both and is it coincidental they are the same? Sorry if dumb question. And do you have any recommendations for parameter selections for MNIST?
Thanks again for this implementation!
No such file or directory: 'facemotortrain'
Hello，
when I run MozafariShallow.py, error occurs
python MozafariShallow.py
Traceback (most recent call last):
File "MozafariShallow.py", line 127, in <module>
trainsetfolder = utils.CacheDataset(ImageFolder("facemotortrain", s1c1))
File "/home/liqp/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 229, in __init__
is_valid_file=is_valid_file)
File "/home/liqp/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 108, in __init__
classes, class_to_idx = self._find_classes(self.root)
File "/home/liqp/.local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 137, in _find_classes
classes = [d.name for d in os.scandir(dir) if d.is_dir()]
FileNotFoundError: [Errno 2] No such file or directory: 'facemotortrain'
Adding and Training a R-STDP Layer in MozafariDeep Model
Hello,
First of all, I appreciate the work done on the SpykeTorch repository. It has been incredibly helpful for my research.
I am working on upgrading the MozafariDeep model to study the behaviour of deeper network, I am trying to add a layer of R-STDP, but I'm encountering problems with poor accuracy. I have tried two different training strategies for the R-STDP layers:
- Simultaneous Update: Updating the weights of both layers based on the last output.
- Separate Update: Updating one layer while fixing the other, and then vice versa.
Both approaches resulted in poor accuracy. Additionally, I suspect there might be parameter issues such as thresholds, the number of winners, and inhibition radius. I understand that the paper mentions there is no significant improvement when adding an R-STDP layer, but I assume there shouldn't be any catastrophic behavior either. Any guidance on these problems would be greatly appreciated.
Snippets
Here some snippets of the code:
def __init__(
self,
num_classes = 10,
learning_rate_multiplier = 1,
):
super(deepSNN, self).__init__()
self.num_classes = num_classes
#### LAYER 1 ####
self.conv1 = snn.Convolution(
in_channels=6,
out_channels=30,
kernel_size=4,
weight_mean=0.8,
weight_std=0.05
)
self.conv1_t = 15
self.k1 = 5
self.r1 = 3
#### LAYER 2 ####
self.conv2 = snn.Convolution(
in_channels=30,
out_channels=250,
kernel_size=3,
weight_mean=0.8,
weight_std=0.05
)
self.conv2_t = 10
self.k2 = 8
self.r2 = 2
#### LAYER 3 ####
self.conv3 = snn.Convolution(
in_channels=250,
out_channels=200,
kernel_size=3,
weight_mean=0.8,
weight_std=0.05
)
self.conv3_t = 400
self.k3 = 15
self.r3 = 0
#### LAYER 4 ####
self.conv4 = snn.Convolution(
in_channels=200,
out_channels=100,
kernel_size=4,
weight_mean=0.8,
weight_std=0.05
)
# STDP
self.stdp1 = snn.STDP(
conv_layer = self.conv1,
learning_rate = (
learning_rate_multiplier * 0.004,
learning_rate_multiplier * -0.003
),
)
self.stdp2 = snn.STDP(
conv_layer = self.conv2,
learning_rate = (
learning_rate_multiplier * 0.004,
learning_rate_multiplier * -0.003
),
)
self.stdp3 = snn.STDP(
conv_layer = self.conv3,
learning_rate = (
learning_rate_multiplier * 0.004,
learning_rate_multiplier * -0.003
),
use_stabilizer = False,
lower_bound = 0.2,
upper_bound = 0.8,
)
self.stdp4 = snn.STDP(
conv_layer = self.conv4,
learning_rate = (
learning_rate_multiplier * 0.004,
learning_rate_multiplier * -0.003
),
use_stabilizer = False,
lower_bound = 0.2,
upper_bound = 0.8,
)
# ANTI STDP
self.anti_stdp3 = snn.STDP(
conv_layer = self.conv3,
learning_rate = (
learning_rate_multiplier * -0.004,
learning_rate_multiplier * 0.0005
),
use_stabilizer = False,
lower_bound = 0.2,
upper_bound = 0.8,
)
self.anti_stdp4 = snn.STDP(
conv_layer = self.conv4,
learning_rate = (
learning_rate_multiplier * -0.004,
learning_rate_multiplier * 0.0005
),
use_stabilizer = False,
lower_bound = 0.2,
upper_bound = 0.8,
)
# adaptive learning rate
self.max_ap = Parameter(torch.tensor([0.15]))
# Decision map
self.decision_map = self.generate_decision_map()
# context parameters
self.ctx = {
'input_spikes': None,
'potentials': None,
'output_spikes': None,
'winners': None,
}
self.spk_cnt1 = 0
self.spk_cnt2 = 0
self.ctx3 = {
'input_spikes': None,
'potentials': None,
'output_spikes': None,
'winners': None,
}
# self.spk_cnt3 = 0
self.ctx4 = {
'input_spikes': None,
'potentials': None,
'output_spikes': None,
'winners': None,
}
def forward(
self,
input,
layer_idx,
):
"""
Forward pass of the network
Parameters
----------
input : torch.Tensor
Input tensor
layer_idx : int
Layer index
Returns
-------
int
Output class
"""
# padding to avoid edge effects
input = sf.pad(
input = input.float(),
pad = (2, 2, 2, 2),
value = 0
)
if self.training:
# Layer 1
# potential and spikes
pot = self.conv1(input)
spk, pot = sf.fire(
potentials = pot,
threshold = self.conv1_t,
return_thresholded_potentials = True,
)
if layer_idx == 1:
self.spk_cnt1 += 1
if self.spk_cnt1 >= 500:
self.spk_cnt1 = 0
ap = torch.tensor(
self.stdp1.learning_rate[0][0].item(),
device = self.stdp1.learning_rate[0][0].device
) * 2
ap = torch.min(ap, self.max_ap)
an = ap * -.75
self.stdp1.update_all_learning_rate(
ap.item(),
an.item()
)
# inhibition
pot = sf.pointwise_inhibition(
thresholded_potentials = pot
)
spk = pot.sign()
winners = sf.get_k_winners(
potentials = pot,
kwta = self.k1,
inhibition_radius = self.r1,
spikes = spk
)
self.ctx.update({
"input_spikes": input,
"potentials": pot,
"output_spikes": spk,
"winners": winners
})
return spk, pot
# Layer 2
# potential and spikes
spk_in = sf.pad(
sf.pooling(
input = spk,
kernel_size = 2,
stride = 2,
),
pad = (2, 2, 2, 2),
)
pot = self.conv2(spk_in)
spk, pot = sf.fire(
potentials = pot,
threshold = self.conv2_t,
return_thresholded_potentials = True,
)
if layer_idx == 2:
self.spk_cnt2 += 1
if self.spk_cnt2 >= 500:
self.spk_cnt2 = 0
ap = torch.tensor(
self.stdp2.learning_rate[0][0].item(),
device = self.stdp2.learning_rate[0][0].device
) * 2
ap = torch.min(ap, self.max_ap)
an = ap * -.75
self.stdp2.update_all_learning_rate(
ap.item(),
an.item()
)
# inhibition
pot = sf.pointwise_inhibition(
thresholded_potentials = pot
)
spk = pot.sign()
winners = sf.get_k_winners(
potentials = pot,
kwta = self.k2,
inhibition_radius = self.r2,
spikes = spk
)
self.ctx.update({
"input_spikes": spk_in,
"potentials": pot,
"output_spikes": spk,
"winners": winners
})
return spk, pot
# Layer 3
# potential and spikes
spk_in = sf.pad(
sf.pooling(
input = spk,
kernel_size = 2,
stride = 2,
),
pad = (2, 2, 2, 2),
)
pot = self.conv3(spk_in)
spk, pot = sf.fire(
potentials = pot,
threshold = self.conv3_t,
return_thresholded_potentials = True,
)
# self.spk_cnt3 += 1
# if self.spk_cnt3 >= 500:
# self.spk_cnt3 = 0
# ap = torch.tensor(
# self.stdp3.learning_rate[0][0].item(),
# device = self.stdp3.learning_rate[0][0].device
# ) * 2
# ap = torch.min(ap, self.max_ap)
# an = ap * -.75
# self.stdp3.update_all_learning_rate(
# ap.item(),
# an.item()
# )
if layer_idx == 3:
winners = sf.get_k_winners(
potentials = pot,
kwta = self.k3,
inhibition_radius = self.r3,
spikes = spk
)
self.ctx3.update({
"input_spikes": spk_in,
"potentials": pot,
"output_spikes": spk,
"winners": winners
})
# Layer 4
# potential and spikes
spk_in = sf.pad(
sf.pooling(
input = spk,
kernel_size = 3,
stride = 3,
),
pad = (2, 2, 2, 2),
)
pot = self.conv4(spk_in)
spk = sf.fire(
potentials = pot,
)
winners = sf.get_k_winners(
potentials = pot,
kwta = 1,
inhibition_radius = 0,
spikes = spk
)
if layer_idx == 4:
self.ctx4.update({
"input_spikes": spk_in,
"potentials": pot,
"output_spikes": spk,
"winners": winners
})
output = -1
if len(winners) != 0:
output = self.decision_map[winners[0][0]]
return output
def train_rl(
network,
data,
target,
max_layer,
):
"""
Train the network using reinforcement learning
Parameters
----------
network : deepSNN
Network to be trained
data : torch.Tensor
Input data
target : int
Target class
max_layer : int
Maximum layer to be trained
Returns
-------
None
"""
network.train()
perf = np.array([0, 0, 0]) # [correct, wrong, silence]
for i in range(len(data)):
data_in = data[i]
target_in = target[i]
if use_cuda:
data_in = data_in.cuda()
target_in = target_in.cuda()
for layer_idx in range(3, max_layer + 1):
d = network(
data_in,
layer_idx,
)
if layer_idx == max_layer:
if d != -1:
if d == target_in:
perf[0] += 1
for j in range(3, max_layer + 1):
network.reward(j)
else:
perf[1] += 1
for j in range(3, max_layer + 1):
network.punish(j)
else:
perf[2] += 1
return perf / len(data)
def train_rl_separate(
network,
data,
target,
max_layer,
):
"""
Train the network using reinforcement learning,
but train each layer separately, fixing the weights
of the previous and next layers
Parameters
----------
network : deepSNN
Network to be trained
data : torch.Tensor
Input data
target : int
Target class
max_layer : int
Maximum layer to be trained
Returns
-------
None
"""
network.train()
perf = np.array([0, 0, 0]) # [correct, wrong, silence]
for i in range(len(data)):
for layer_idx in range(3, max_layer + 1):
data_in = data[i]
target_in = target[i]
if use_cuda:
data_in = data_in.cuda()
target_in = target_in.cuda()
d = network(
data_in,
layer_idx,
)
if d != -1:
if d == target_in:
if layer_idx == max_layer:
perf[0] += 1
network.reward(layer_idx)
else:
if layer_idx == max_layer:
perf[1] += 1
network.punish(layer_idx)
else:
if layer_idx == max_layer:
perf[2] += 1
return perf / len(data)
I apologize for any obvious mistakes or unclear parts in my code or explanation. I am ready to provide any additional information or clarification if needed.
Thank you again for the fantastic work on this repository and for your assistance with my issue.
Best regards,
Aidin
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.