saran-nns / sorn Goto Github PK

LICENSE deprecated. Ref issue #23. The issue will be resolved at v0.5.0 Logs can be found here

Plasticity methods are CPU bounded. Multiprocessing is required instead of threading.

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Add while condition to avoid self-connection indices

Prepare build and pypi package

Hi, I have a couple of suggestions / requests for your documentation. Let me know if you disagree with any of these:

• your documentation references a configuration file which lives inside the installed python package, and is suggested to be edited by the user. This doesn't seem like a good solution for configuring the core part of your library, as the site package directory may not always be editable by the user (e.g. if the library was installed by an administrator, the library is shared across users, etc.)

  I would suggest changing the way this is handled. One way would be to define these in code, in a frozen dictionary inside the modules you have elsewhere. You could then allow users to copy this dictionary, edit it, and pass it as an argument to the SORN class.

• It would be great if you could expand the OpenAI gym example to be "self-contained". Currently it references a pickle file which isn't generated in the documentation. If this is intended to be the pkl file in the github repository, this should be explained, because that pkl file doesn't exist if someone installs the library via pip. It also references computing gradients - is it possible to supply a simple example of this?

• There seems to be some discrepancy between the README and the hosted documentation, for example the OpenAI Gym example. It would be nice if they could match (you could even include your README in your documentation itself).

• It would be nice to include some example pictures of the plots in the documentation, so that users know what to expect

• Running test_sorn.py produces some corrupted graphs, e.g.:
  

Hi there, another question related to the review (openjournals/joss-reviews#3545).

this code is the basis for your master thesis research conducted at University of Osnabrück, correct? Then I was wondering whether there could be any other researches involved in the work underlying this package, e.g., thesis supervisors?

Except for STDP and SN, all other rules can be executed in parallel. Assuming, asynchronous update of connection weights should provide at least 3x speed.

Update docs vers

Hi there,

I was just trying to run the examples as provided in the README, but ran into an error:

from sorn import Simulator
import numpy as np

# Sample input
num_features = 10
time_steps = 200
inputs = np.random.rand(num_features,time_steps)

# To simulate the network;
matrices_dict, Exc_activity, Inh_activity, Rec_activity, num_active_connections = Simulator.simulate_sorn(inputs = inputs, phase='plasticity', matrices=None, noise = True, time_steps=time_steps, _ne = 200, _nu=num_features)

# To resume the simulation, load the matrices_dict from previous simulation;
matrices_dict, Exc_activity, Inh_activity, Rec_activity, num_active_connections = Simulator.simulate_sorn(inputs = inputs, phase='plasticity', matrices=matrices_dict, noise= True, time_steps=time_steps,_ne = 200, _nu=num_features)

Produces the following error:

Network Initialized
Number of connections in Wee 3881 , Wei 1584, Wie 8000
Shapes Wee (200, 200) Wei (40, 200) Wie (200, 40)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-3e49ac279123> in <module>
      8
      9 # To simulate the network;
---> 10 matrices_dict, Exc_activity, Inh_activity, Rec_activity, num_active_connections = Simulator.simulate_sorn(inputs = inputs, phase='plasticity', matrices=None, noise = True, time_steps=time_steps, _ne = 200, _nu=num_features)
     11
     12 # To resume the simulation, load the matrices_dict from previous simulation;

~/github/joss-reviews/.venv/lib/python3.9/site-packages/sorn/sorn.py in simulate_sorn(self, inputs, phase, matrices, time_steps, noise, freeze, **kwargs)
    956
    957             # STDP
--> 958             if 'stdp' not in self.freeze:
    959                 Wee_t = plasticity.stdp(
    960                     Wee[i], x_buffer, cutoff_weights=(0.0, 1.0))

TypeError: argument of type 'NoneType' is not iterable

I assume this is a relatively easy fix for the freeze attribute?

Thanks

this is part of the review: openjournals/joss-reviews#3545

Currently, SORN package includes all 5 plasticity mechanisms. However, it is obvious that researchers might be interested to turn off specific plasticity during simulation.

Hi there, as part of the review at openjournals/joss-reviews#3545 (see also openjournals/joss-reviews#3554) I was following the installation instructions like this:

• create conda env with python 3.8
• install via pip install sorn
• in python shell run import sorn

and got the following error:

Python 3.8.10 (default, May 19 2021, 11:01:55)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sorn
Traceback (most recent call last):
  File "/Users/Jan/anaconda3/envs/sorn-review/lib/python3.8/site-packages/sorn/sorn.py", line 10, in <module>
    from sorn.utils import Initializer
  File "/Users/Jan/anaconda3/envs/sorn-review/lib/python3.8/site-packages/sorn/utils.py", line 9, in <module>
    import networkx as nx
ModuleNotFoundError: No module named 'networkx'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Jan/anaconda3/envs/sorn-review/lib/python3.8/site-packages/sorn/__init__.py", line 1, in <module>
    from .sorn import Simulator, Trainer
  File "/Users/Jan/anaconda3/envs/sorn-review/lib/python3.8/site-packages/sorn/sorn.py", line 12, in <module>
    from utils import Initializer
ModuleNotFoundError: No module named 'utils'

Any suggestions for a fix?

Best,
Jan

Hi @Saran-nns and sorry, it took me a while to get back to review.

Installation and example usage is working now, great!

I have some suggestions regarding the documentation: At the moment the README seems a bit cluttered to me. There is short high level description of SORN, then a set of figures and then the table of contents for the whole documentation. My suggestion would be to reduce this a bit:

1. Give the high-level description of SORN at the beginning and a very simple description of the components of the SORN (illustrated with the schematic figure) and how to set up a SORN using your package. Remove the other figures as it is difficult to understand them in this context / they are too small.
2. Give the installation instruction and just then a link to the full documentation at readthedocs. Or paste the full documentation in the README like Jan Freyberg suggested, also fine, but I would actually prefer the slim version with the link.

Other remarks:

Writing

In the paper itself I would not give the installation instructions, they are online and very clear don't need to be repeated. Rather I would invest in explaining SORN in detail with equations, like you are doing now, but accompanied by some figures like the schematic in the README. It would also be great to add more explanatory text to the equations, or, if this is out of the scope of this paper, to refer the reader to the references where they can read the explanations. Overall, the paper should give the user all they need to understand and then use the package (given only basic knowledge of SORN).

References

For mentioning liquid state machines / echo state networks in summary I think it would be good to cite the initial work on this by Herbert Jaeger and Wolfgang Maass, e.g.

• "Adaptive nonlinear system identification with echo state networks." Advances in neural information processing systems 15 (2002): 609-616. or
• Jaeger, Herbert, Wolfgang Maass, and Jose Principe. "Special issue on echo state networks and liquid state machines." (2007).

Tests

I think it would be great to add some documentation to the tests, e.g., inline comments about what the tests are doing and why / why not they should fail.

Let me know what you think of these suggestions.

In the example using open AI, there is a policy function that is called, could someone give an example of how to pair a policy with SORN?

# Imports

from sorn import Simulator, Trainer
import gym

# Load the simulated network matrices
# Note these matrices are obtained after the network achieved convergence under random inputs and noise

with open('simulation_matrices.pkl','rb') as f:  
    sim_matrices,excit_states,inhib_states,recur_states,num_reservoir_conn = pickle.load(f)

# Training parameters

NUM_EPISODES = 2e6
NUM_PLASTICITY_EPISODES = 20000

env = gym.make('CartPole-v0')

for EPISODE in range(NUM_EPISODES):
    
    # Environment observation
    state = env.reset()[None,:]
    
    # Play the episode
    while True:
      if EPISODE < NUM_PLASTICITY_EPISODE:
...
     # This is the policy... line I am referring to 
      probs = policy(np.asarray(excit_states),output_layer_weights))
      action = np.random.choice(action_space,probs)
      state,reward,done,_ = env.step(action) 
      
      if done:
        break
        
  # YOUR CODE HERE
  # COMPUTE GRADIENTS BASED ON YOUR OBJECTIVE FUNCTION
  # OPTIMIZE `output_layer_weights` BASED ON YOUR OPTIMIZATION METHOD

Traceback (most recent call last):
File "C:\Users\saran\Anaconda3\envs\tfgpu\lib\configparser.py", line 1138, in _unify_values
sectiondict = self._sections[section]
KeyError: 'Network_Config'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\saran\Anaconda3\envs\tfgpu\lib\site-packages\sorn_init_.py", line 1, in
from .sorn import Simulator, Trainer
File "C:\Users\saran\Anaconda3\envs\tfgpu\lib\site-packages\sorn\sorn.py", line 19, in
class Sorn(object):
File "C:\Users\saran\Anaconda3\envs\tfgpu\lib\site-packages\sorn\sorn.py", line 28, in Sorn
nu = int(parser.get('Network_Config', 'Nu')) # Number of input units
File "C:\Users\saran\Anaconda3\envs\tfgpu\lib\configparser.py", line 781, in get
d = self._unify_values(section, vars)
File "C:\Users\saran\Anaconda3\envs\tfgpu\lib\configparser.py", line 1141, in _unify_values
raise NoSectionError(section)
configparser.NoSectionError: No section: 'Network_Config'

Remove lambda_w < num_inhibitory units

Hi there,

as part of the preliminary checks I was going through your License for this repo, and I can see two conflicting license files:
LICENSE and LICENSE.md, one of which as Apache 2.0, the other has MIT.

Could you please choose one, and ensure you only have one license file in the repository?

Thanks!

Execution of the example code from the package PyPi page in Jupyter produces the following error:

TypeError Traceback (most recent call last)
in ()
11 #1.3 To resume the simulation, load the matrices_dict from previous simulation;
12 matrices_dict, Exc_activity, Inh_activity, Rec_activity, num_active_connections = RunSorn(phase='Plasticity', matrices=matrices_dict,
---> 13 time_steps=100).run_sorn(inputs)

/home/tom/.local/lib/python3.6/site-packages/sorn/sorn.py in run_sorn(self, inp)
564
565 """ Fraction of active connections between E-E network"""
--> 566 frac_pos_active_conn.append((Wee[i] > 0.0).sum())
567
568 """ Recurrent drive"""

TypeError: '>' not supported between instances of 'list' and 'float'`

Meanwhile, the source version of sorn/sorn/sorn.RunSorn contains this code at line 608:

""" Fraction of active connections between E-E network""" frac_pos_active_conn.append((np.asarray(Wee[i]) > 0.0).sum())

Was different code compiled into the dist?

Hi there, I'm playing around with the cartpole example as a "hello-world" with sorn. I've made some modifications to optimize output weights based on CMA-ES: https://github.com/CMA-ES/pycma

In my code example however, it seems like the vector of excited states is always stuck at zero, so the agent always takes the same action regardless of the output weights.

I am wondering if I didn't initialize the network correctly, or if the time_steps needs to be set to >1 so that the input signal can propagate through the network.

Here is my code:

from sorn import Simulator, Trainer
import gym
import pickle
import numpy as np
import cma
# Load the simulated network matrices
# Note these matrices are obtained after the network achieved convergence under random inputs and noise

#with open('simulation_matrices.pkl','rb') as f:  
#    sim_matrices,excit_states,inhib_states,recur_states,num_reservoir_conn = pickle.load(f)

# Training parameters

NUM_EPISODES = int(2e6)
NUM_PLASTICITY_EPISODES = 20000

env = gym.make('CartPole-v0')
action_space = env.action_space
import pdb

output_layer_weights = np.random.uniform(-1,1,(200,1))
es = cma.CMAEvolutionStrategy(output_layer_weights,1)
#pdb.set_trace()
# Policy
def policy(state,w):
    "Implementation of softmax policy"
    z = state.dot(w)
    exp = np.exp(z)
    output = exp/np.sum(exp)
    return output

max_reward = 0
state = env.reset()[None,:]
# Note: I don't have a pre-loaded pkl file, so I am using this line to initialize
sim_matrices_init, excit_states_init, inhib_states_init, recur_states_init, num_reservoir_conn_init = Simulator.simulate_sorn(inputs = state, phase ='plasticity', time_steps = 1, noise=False, _nu=1) 

for EPISODE in range(NUM_EPISODES):
    
    # Environment observation
    state = env.reset()[None,:]
    # Play the episode
    sim_matrices = sim_matrices_init
    
    solutions = es.ask()
    reward_vector = []
    for sol_idx in range(0,19):
      output_layer_weights = solutions[sol_idx]
      while True:
        #print("Running")
        if EPISODE < NUM_PLASTICITY_EPISODES:
          # Plasticity phase
          sim_matrices, excit_states, inhib_states, recur_states, num_reservoir_conn = Simulator.simulate_sorn(inputs = state, phase ='plasticity', matrices = sim_matrices, time_steps = 1, noise=False)
        else:
          # Training phase with frozen reservoir connectivity
          sim_matrices,excit_states,inhib_states,recur_states,num_reservoir_conn = Trainer.train_sorn(inputs = state, phase = 'training', matrices = sim_matrices, noise= False)
        
        # Feed excit_states as input states to your RL algorithm, below goes for simple policy gradient algorithm
        # Sample policy w.r.t excitatory states and take action in the environment
        print(excit_states)
        probs = policy(np.asarray(excit_states),output_layer_weights)
        prob = probs[0]
        diff_probs = [prob,1-prob]
        print(diff_probs)
        action = np.random.choice(range(0,action_space.n),p=diff_probs)
        state,reward,done,_ = env.step(action) 
        state = state[None,:] 
        if done:
          reward_vector.append(reward)
          max_reward = np.max([max_reward, reward])
          print("max reward", max_reward)
          break

    # end of Iteration
    es.tell(solutions,reward_vector)

Here is some sample output:

$ python sorn_test.py 
/home/gluser/anaconda3/envs/freqtrade-conda/lib/python3.8/site-packages/cma/utilities/utils.py:343: UserWarning: input x0 should be a list or 1-D array, trying to flatten (200, 1)-array ()
  warnings.warn(msg + ' (' +
(9_w,19)-aCMA-ES (mu_w=5.6,w_1=29%) in dimension 200 (seed=143495, Mon Jul 12 13:44:22 2021)
Network Initialized
Number of connections in Wee 3922 , Wei 1590, Wie 8000
Shapes Wee (200, 200) Wei (40, 200) Wie (200, 40)
[array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])]
[1.0, 0.0]
[array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])]
[1.0, 0.0]
...

As per open issue