pybrain / pybrain Goto Github PK

Thomas:

i think the simple environment should go, together with it's examples. it's a simple function minimization of a square function with noise, which could be done with the FunctionEnvironment as well.

1. Composing tasks (usually benchmark functions) with the same number of inputs (each action gets executed for each task) and return the (weighted) sum of their fitnesses.
2. Composing arbitrary tasks by concatenating their inputs (each segment of an action goes to a different task) and return the (weighted) sum of their fitnesses.

Once (2) is possible, we can learn crazy stuff like playing go with one hand while racing a car and while balancing two poles on the nose!

For example:

• group algorithms by categories
• add NSGA2, PGPE, etc.
• restructure/complete the environment list (as in the googledoc)

Many attributes and methods are not supposed to be accessed from the outside. We could either protect them properly (with wrappers), but as a first step it might already be helpful to make things a bit more consistent w.r.t. to naming conventions: with no underscore in front = public, one underscore = accessible, but should be used from outside as little as possible, two underscores = inaccessible.

Currently some (e.g. in rl/environments/flexcube) are not.

Nested networks are conceptually elegant, but sometimes not the most efficient. Currently they cannot be converted to fast networks neither. Either arac should be extended to handle them or we should write a flatten-network utility.

... to this one and make sure the information is still up-to-date in the process.
The contents are mainly: setup instructions (with/without arac) and developer guidelines.

Details can be found on the mailinglist: http://groups.google.com/group/pybrain/msg/d99e33d89e2844aa

The code that has been used in published papers could indicate those in the comments, this should be particularly useful for understanding some of the algorithms. Maybe for some of the benchmarks too.

We could coordinate the changes that should go into v0.3 here, gathering the info from: JMLR review, mailing list, everyone's private todo list, etc.

It should already be possible to build these (using multiplicative units). Let's get proper Network subclass for them, and a couple of examples in, to make them easily usable.

Hi.
I'm not sure if this is considered an issue and therefore in the right place here.
The rewards received during random exploration with egreey agent do not seem to have any effect on the action values. Also, exploration is not listed in the history of the agent.

Adding
self.lastaction = action
in getAction() of egreedy.py
seems to solve this.
I think the results of the maze example in Q-learning.py make more sense this way.

http://github.com/schmalerc/pybrain/blob/90204cab0ec47c3abb8ec3c5a3df236b9d1863a5/pybrain/rl/agents/egreedy.py

Currently, building is happening via the setup.py script with various hacky methods. The optimal way would be:

1. Check for scipy, drop out of not present
2. Check for BLAS and scons, ignore arac if not present
3. build
4. run unittests
5. install arac if dependencies are there on pythonpath
6. install pybrain on pythonpath

Make setup.py script automatically install dependencies.

Unused import:
http://github.com/pybrain/pybrain/blob/ffc4ee7c9d6eaff07cd693ad8a0f923c02077920/pybrain/rl/learners/discrete/qlambda.py#L4

Unused variable action -- is this a bug?
http://github.com/pybrain/pybrain/blob/ffc4ee7c9d6eaff07cd693ad8a0f923c02077920/pybrain/rl/learners/discrete/qlambda.py#L32

Details in this discussion:
http://groups.google.com/group/pybrain/browse_thread/thread/954c36c242921cf1

"Sequence Classification" seems impossible because :
SequenceClassificationDataSet class expects one target class for
each frame frame in a sequence. No other dataset class exists that
would take only one target class for the entire input sequence.

Correct.

No trainer exists that would compute the error gradients only for
the last timestep and backpropagate it while ignoring gradients at
previous timesteps (since no relevant target should be available for
those previous timesteps)
Is this correct, or is there another way to do it ?

That is correct. However, writing such a trainer would be easy and beneficial.

"Temporal Classification" seems impossible because no CTC
(Connectionist Temporal Classification) is available for the time
beeing. Is that right or did I miss something ?

That's right. I'd like to have it in there. ;)

And have a benign form of failure if things go wrong.

This would replace/be based on
rl/environments/cartpole/fast_version/cartpolecompile.py

Make a baseclass for LSTMLayer and MDLSTMLayer that holds shared functionality, as almost all code is duplicated right now.

because it replaces the ._params array with the new one. We could fix that by using the owner argument whenever the network is made into a fast one.

/docs/tutorials/rl.py

It would be excellent if you append a possibility of usage of mixed
variables for optimization problems.
I mean the capability to describe types and range of definition for
each variable.

For example, I think that PyGene's (http://www.freenet.org.nz/python/pygene/ ) implementation of this feature is excellent: you can describe every variable as a class

# for Integer variable
class Xvar(IntGene):
 # genes get randomly generated within this range
 randMin = 1
 randMax = 190
 # probability of mutation
 mutProb = 0.12

# for float variable
class Yvar(FloatGene):
 # genes get randomly generated within this range
 randMin = 5.5
 randMax = 43.2
 # probability of mutation
 mutProb = 0.12
 # degree of mutation
 mutAmt = 0.15

-- JustNewbie sqwatex@g...

Redesign, prune and unify the evolutionary algorithms framework (currently in rl/learners/search and rl/learners/blackboxoptimizers/evolution), at the same time adding in some standards ones like self-adaptive ES and differential evolution (potentially also NEAT/HyperNEAT?). Other algorithms that might be interesting: CoSyNE, ESP.

This issue is closely related to #3

• how to get it from the cheeseshop
• homogenize between wiki and docs
• new dependency organization for fast networks (arac)

In Sphinx, based on tutorial script(s).

We have lots of init.py files with imports like "from connections.init import *". I guess this is not very common and bears surprises for the user how things work. The way modules can be imported should be unified. Also, the imports are sometimes extremely deep (and flat is better than nested) and thus hard to remember.

Here is my proposal for the structure package.

(1) All structure classes live either in the namespace pybrain.structure.modules, pybrain.structure.networks or pybrain.structure.connections. pybrain end users should only import from there.

(2) The structure package imports explicitly from its submodules/packages like modules.neuronlayer. Absolutely no star imports here, since it makes finding out where source of a specific class lies cumbersome and pollutes the namespace.

(3) structure/init.py contains the line "all = ['modules', 'connections', 'networks']" so these don't have to be imported explicity.

Other packages should be organized in a similar manner imho, although the concrete requirements will probably differ.

-Justin

And improve docstrings...

Currently, Network.indim/outdim, as well as a number of methods are invalid before .sortModules() is called, which might not be obvious to everyone.

If a network is used in an unsorted context, an exception should be thrown.

Although it technically fits there, it is a bit confusing.
I think the split should be along the difference of ontogenetic/phylogenetic with on one side optimization, evolution, pso, etc. (coevolution methods should fit here, but how about multi-objective optimization?) and on the other side policy gradients, and other RL algos.

Thomas:

I also think our supervised trainers are not flexible enough. AFAIK, the mean squared error function is hard-coded, the other day we needed a cross-entropy error and it wasn't apparent how to do that.

And regularization would be nice, in a general form (not just weight decay as option for the trainer). What about conjugate gradient etc?

Martin:

I totally agree with Thomas here (being mostly interested in supervised learning myself). The old-fashioned supervised stuff seems to be quite inflexible so far; I'm not even sure if second order methods like CG or more complicated error functions would fit into the current framework.

Currently some of the Task subclasses are in the /rl/tasks/ folder, others are located with their corresponding environments (e.g. /rl/environments/flexcube). Both are reasonable, but we should choose one way and stick to it consistently.

The UCI Machine Learning Repository contains a lot of semi-standard datasets for supervised and unsupervised learning:

http://archive.ics.uci.edu/ml/datasets.html

It would be nice to have simple access to them from PyBrain (taking into account the differences in tasks and data).

Currently Learners have as their main method .learn(), which is modulated by other parameters.

For Optimizers, this one usually calls a class-specific ._learnStep() method (loosely corresponding to one batch, or generation -- or one sample in online optimizers).

(Supervised) Trainers have a .train() method that does one epoch, and .trainEpochs() that does more.

The question is: should we homogenize all those, and what would be the most consistent/useful/comprehensible scheme for doing so?

One common entry point that gives an outline of the library, the environment/task/agent framework and the module/network/connection structure.

arac could be added as a submodule to PyBrain. By this, user would always be able to track arac in realtime with no additional repository checkout.

On some platforms this message show up when running unittests:

 Exception in thread Thread-1: 
 Traceback (most recent call last):
 File "/sw/lib/python2.5/threading.py", line 460, in __bootstrap
  self.run()
File "/sw/lib/python2.5/threading.py", line 440, in run
  self.__target(*self.__args, **self.__kwargs)
File "/Users/thomas/Work/projects/pybrain/utilities.py", line 280, 
in <lambda>
  target = lambda: callback(func(*args, **kwargs))
File "<doctest unittests.test_utilities[4]>", line 3, in threadname 
NameError: global name 'threading' is not defined

I have been making use of your MultilinePlotter class and came across a problem when using the show() method. The 'title' argument conflicts with the use of the pylab title() function.

I also had a little trouble getting interactive plotting to work. The draw_if_interactive() function seemed not to have any effect and I replaced it with:

if isinteractive():
    draw()

Components that are any of the following:

• stable
• used a lot
• non-trivial to understand

And then include those in the Sphinx API

There are a number of issues with these 2 examples. Even when commenting out missing imports (as suggested here), I can't get them to run.
Also, they could use some explanations.

Should be replaced with it's new and improved version (Daan). Maybe it will need to be adjusted for the v0.3 changes?

I added bidirectional networks (unfolded version), which seem to work fine, the writning to xML and reading from XML does not seem to work properly.

Tree structure, in Sphinx.

Nice to have would be a continuous value-based RL (best would probably be Neuro-fitted Q Iteration) as well as a life-long policy-gradient algorithm (e.g. Natural Actor Critic). Maybe some dynamic programming in addition for the discrete case too...

Currently, it is not clear how to partition code that is not easily categorized as either supervised, unsupervised or reinforcement learning. Such things include

• framework functions (e.g. threaded in utilities.py)
• algorithmic methods (e.g. gradient descent)

an things in between.

Frank!

The code in Julian's Super Mario competition (Java) already has a Python wrapper and some PyBrain-related code -- maybe it could become an environment. It would depend only on one .jar file.

The same holds for the "simplerace" car racing competition code (also Java).
Here as well the connection uses TCP, and here as well that connection is not yet all too robust.

I could make the start, but could use some external TCP expertise to do it well.

Make sure all examples run smoothly, and if they rely on optional dependencies those should be explicitly shown.

Maybe the directory structure of examples can be clarified as well.

Also, add some more examples, for example on optimization. Ultimately, we'd like at least one example for each feature, and the examples structure should be mirrored (approximately) on the features page.

Thomas:

we should have one example for each algorithm (or set of algorithms if they are really interchangable) but at least one for discrete states/discrete actions, continuous states/discrete actions, both continuous

It's currently implemented twice, differently, in VES and in CMA-ES.
It should probably be refactored into a separate file (in its general form, for arbitrary distributions), maybe in the auxiliary directory.