april-org / april-ann Goto Github PK

The official apple version of gcc is llvm-gcc-4.2, which has an important BUG with OpenMP. Currently, the macosx targets are compiled with NO_OMP definition. In the future this BUG could be solved, and the NO_OMP definition will be removed.

Current optimizer branch will be merged with master in a few days. It will lead to a new minor beta release with some minor changes in ANNs API. The most important change is about the customization of optimization algorithms. A new Lua class, in the table/namespace ann.optimizer has been defined. The instances of trainable.supervised_trainer could receive as foruth argument the optimizer that you want to use. By default the optimizer is ann.optimizer.sgd(), which is exactly the same as we have before, the implementation of Stochastic Gradient Descent with momentum, weight decay, and max normalization penalty.

This change leads to an important API modification, because the optimization algorithm is not developed in the ANN components, it is developed in a decoupled class. So, the learning parameters (as learning_rate, momentum, etc) couldn't be defined in the components, they need to be set/get via the optimizer. In order to simplify the API, the trainable.supervised_trainer has methods to perform set/get of the optimizer options. The following code is an example of this last way:

trainer = trainable.supervised_trainer(thenet,
                                       ann.loss.multi_class_cross_entropy(10),
                                       bunch_size)
trainer:build()

trainer:set_option("learning_rate", learning_rate)
trainer:set_option("momentum",      momentum)
trainer:set_option("weight_decay",  weight_decay)
-- bias has weight_decay of ZERO
trainer:set_layerwise_option("b.", "weight_decay", 0)

Note that weight_decay parameter is set layerwise, so all layers has the global option set at trainer:set_option("weight_decay", weight_decay), but the bias has a weight decay of zero, set with the method trainer:set_layerwise_option("b.","weight_decay",0").

The next release will has:

• Deprecated train_holdout_validation and train_wo_validation functions, replaced by a Lua class.
• Removed old weights field from connection objects, instead of that, the old weights will be stored at SGD optimizer.

It is important to review the softmax and cross-entropy. Currently, the derivative of the softmax is asuming cross-entropy error function, and cross-entropy is asuming softmax activation. The use of one without the other is not correct.

In order to convert current April-ANN packages in Lua modules we need to:

1. Remove any lateral effect, as the declaration of GLOBALs. The util package could be an exception to this rule, because it replaces some Lua functions as type or table.insert, and extends other Lua modules, as math, string, etc.
2. Packages must be developed self-contained, in the sense of the Lua tables where the packages bind their functionalities. One package means one Lua table with the name aprilann.PACKAGE_NAME.
3. Bindings must declare ONLY one Lua table in the GLOBALs.
4. Formiga will prepare luaopen_aprilann_PACKAGE_NAME functions to return the table declared with the name aprilann.PACKAGE_NAME.

Any suggestions?

I don't like the implementation of contextualizer in utils/context.h. It is implemented sliding the elements or a fixed size vector, so every insertion, one time the vector ismfull, needs to move all the elelemnts.

It will be better to use a circular vector, so insertion cost will be constant. The oprator[] must be re-written to,return the correct position in the circular vector.

The math functions (cblas, cublas, and others) has a lot of shared code. It could be simplified using templates.

Hi,

This error is happening when I try to execute digits.lua in EXAMPLES folder.

april-ann: [string "luaopen_aprilann_util"]:7: Incorrect type 'boolean' for field 'bunch_size'
stack traceback:
[C]: in function 'error'
[string "luaopen_aprilann_util"]:7: in function 'get_table_fields'
[string "luaopen_aprilann_trainable"]:1: in function 'validate_dataset'
digits.lua:99: in main chunk
[C]: in ?

This function could be useful for many scripts. It should be available on utils. The function gets a randomObject, replacement size and a variable number of datasets and returns a tables with the same replacement for each dataset.

The pool of pointers is never freed, so the memory is stated as allocated
when it is executed with valgrind.

In order to avoid valgrind messages, DEBUG target sets the constant NO_POOL, which indicates to never use the pool.

After pre-training, if you save the neural network using ascii mode, valgrind throws use of uninitialized memory error.

A lot of Matrix methods returns a completely new Matrix. The time of this calls could be improved adding an optional OUTPUT Matrix parameter, which if present will be used as container of the result produced by the method. If not present, the returned Matrix will be totally new. This could be used at ANNs code to improve performance.

Using Doxygen format

EDITED: a new architecture based on Dataflow idea, it replaces graph based ann description by ANNComponents which expose three basic methods (forward, backprop, update) and a Token based input/output communication.

A very interesting improvement will be to develop a GRAPH based description, build over the concept of layers and weights. The graph will describe relations between this two basic components, and depending on this relations an algorithm will create the needed C++ objects.

It is a very future work, but it is important to take it into account. Another time, if anyone want to deal with this idea, we are open to explain all the tricks.

Because of the new design, the optimizer objects has the responsability of update the weigths following an optimization algorithm. So, the momentum term is applied by them, and it is not needed to be stored at ANN component or connection objects.
So, this issue is a proposal for the next version, to remove the oldw field in connection objects, and to introduce it at SGD optimizar.

We tryed to create a single ann.mlp object and save it with ann.mlp.all_all.save method and it fails:

[string "luaopen_aprilann_ann"]:1: Incorrect ANN mode!!!

I've checked out the assert in save method: typeof(model) ~= "table", where model is "ann.components.stack", I've solved this but then there is another error:

stack traceback:
[C]: in function '?'
[string "luaopen_aprilann_util"]:7: in function 'copy_weights'
[string "luaopen_aprilann_ann"]:1: in function 'save'
stdin:1: in main chunk
[C]: in ?

After this update, the wiki master will be merged with the branch. The repo master will also be merged with the token_based_anns branch.

Criterions is a not a valid english word. The plural of criterion is criteria, as for automaton is automata. Take into account this issue, it affects to trainable namespace.

I've implemented on the fly feature for stacked denoising autoencoder. This feature is useful when you have a very big input dataset and you don't have enough memory for generate a big dataset for each hidden layer. Automatically, when you use replacement this feature is enabled. It not make sense use it when you don't use replacement because you need to generate all the dataset for each hidden layer. But, it is not needed to use on the fly when you use pretraining.

My question is, I don't know if keep the automatically on the fly when using replacement, or add a new option.

When the bunch_size at validation is different from bunch_size at test, the validation error increases a lot due to a bug.

april_help() doesn't work with direct classes or namespaces. i,e:

april_help(matrix)
[string "util"]:331: matrix class not found
stack traceback:
[C]: in function 'error'
[string "util"]:331: in function 'april_help'
stdin:1: in main chunk

[C]: ?

april_help("matrix")
matrix

-- static functions or tables

• function fromFilename

• function fromHEX

• function fromPNM

  ....

  april_help(matrix(2,2))
  matrix

  -- static functions or tables

  • function fromFilename

• function fromHEX

• function fromPNM
  ....

I think that, in the future, the class(...) function needs to has the following
interface:

> methods = class("myclass", function(self) print("CONSTRUCTOR") end, superclass)
> function methods:blah() print("IT IS ME") end

So, the constructor is passed as a anonymous function to the class function,
and the methods will be declared in an internal table returned by the class function.

In this way, it is possible to reproduce exactly the behavior of CPP classes.

HMM trainer introduces Hidden Markov Models training in April-ANN :-)

This code:

m = matrix(2,2):uniform(-10,10,random(1234)

Initializes the matrix with an inexplicable very large value at the position (2,1).

• Generalize Math Wrappers to work with Matrices instead of memory blocks
• Instead of TokenMemoryBlock in the future we will use TokenMatrix

I just cloned the repository and tried to compile the CUDA builds, but this is not working because the clip function is undefined.

I just wanted to know if you are checking whether the CUDA versions are compiling or not after making code changes at the wrapper files.

Would be interesting to extract the math methods and write them as a templatized C++ functions which receives matrices of different types?

I think it can reduce the complexity of C++ code, because the matrix code will contain only STRUCTURAL methods (select, slice, sliding_window, etc) and the OPERATIONAL methods will be ad-hoc depending on the type of the data, or templatized if it is allowed.

However, in Lua the binding would be similar, for every matrix type you need to write the full binding, and in Lua the C++ math functions will be seen as methods.

Modify formiga.lua in order to avoid -O0 when debug is set to true.

nvcc warning : Setting optimization level to 0 as optimized debugging is not supported. '--device-debug (-G)' with the optimization level is deprecated and will be removed in the next release. Please use '--device-debug (-G)' without the optimization level

I have seen that there is rlcompleter in april, but for me is not working on the april-ann interpreter on linux. I tried also to require the module "require rlcompleter" on an interpreter and it fails on import the module. Before executing, I executed configure.sh and make on the last version of master branch.

I'm thinking in dataset Lua table reorganization, for example, putting dataset.salt_noise in dataset.filter.salt_noise. It will be better to create some hierarchy in dataset, which could denote better the purpose of the class.

It must to be done at image

Ths histogram per se is interesting, independently of the image cleaning methods.

Basic use of the application will be a good idea. Also the addition of more concrete wiki pages for each package/functionality of the library. In every case, the documentation will be for user reference and tutorial purposes. Any suggestion and improvement will be wellcome

To compute output_error at ANN error_functions

Remove compiler warnings during GZIO package compilation. It is my compiler output:

lua -l formiga build_mkl_release.lua
default target: build
[ 17/ 19 (89.5%) done]: gzio packages/misc/gzio/        c_src/lgziolib.c: En la función ‘io_gc’:
c_src/lgziolib.c:183:22: aviso: la comparación de diferentes tipos de puntero carece de una conversión [activado por defecto]
c_src/lgziolib.c:183:36: aviso: la comparación de diferentes tipos de puntero carece de una conversión [activado por defecto]
c_src/lgziolib.c:183:51: aviso: la comparación de diferentes tipos de puntero carece de una conversión [activado por defecto]
c_src/lgziolib.c: En la función ‘io_popen’:
c_src/lgziolib.c:212:7: aviso: asignación desde un tipo de puntero incompatible [activado por defecto]
c_src/lgziolib.c: En la función ‘io_tmpfile’:
c_src/lgziolib.c:219:7: aviso: asignación desde un tipo de puntero incompatible [activado por defecto]
c_src/lgziolib.c: En la función ‘read_number’:
c_src/lgziolib.c:309:3: aviso: se pasa el argumento 1 de ‘fscanf’ desde un tipo de puntero incompatible [activado por defecto]
In file included from c_src/lgziolib.c:37:0:
/usr/include/stdio.h:449:12: nota: se esperaba ‘struct FILE * __restrict__’ pero el argumento es de tipo ‘gzFile’
c_src/lgziolib.c: En la función ‘g_write’:
c_src/lgziolib.c:453:11: aviso: se pasa el argumento 1 de ‘fprintf’ desde un tipo de puntero incompatible [activado por defecto]
In file included from /usr/include/stdio.h:934:0,
                 from c_src/lgziolib.c:37:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:96:1: nota: se esperaba ‘struct FILE * __restrict__’ pero el argumento es de tipo ‘gzFile’
c_src/lgziolib.c: En la función ‘f_setvbuf’:
c_src/lgziolib.c:497:3: aviso: se pasa el argumento 1 de ‘setvbuf’ desde un tipo de puntero incompatible [activado por defecto]
In file included from c_src/lgziolib.c:37:0:
/usr/include/stdio.h:337:12: nota: se esperaba ‘struct FILE * __restrict__’ pero el argumento es de tipo ‘gzFile’
c_src/lgziolib.c: En la función ‘luaopen_gzio’:
c_src/lgziolib.c:585:3: aviso: se pasa el argumento 2 de ‘createstdfile’ desde un tipo de puntero incompatible [activado por defecto]
c_src/lgziolib.c:553:13: nota: se esperaba ‘gzFile’ pero el argumento es de tipo ‘struct _IO_FILE *’
c_src/lgziolib.c:586:3: aviso: se pasa el argumento 2 de ‘createstdfile’ desde un tipo de puntero incompatible [activado por defecto]
c_src/lgziolib.c:553:13: nota: se esperaba ‘gzFile’ pero el argumento es de tipo ‘struct _IO_FILE *’
c_src/lgziolib.c:587:3: aviso: se pasa el argumento 2 de ‘createstdfile’ desde un tipo de puntero incompatible [activado por defecto]
c_src/lgziolib.c:553:13: nota: se esperaba ‘gzFile’ pero el argumento es de tipo ‘struct _IO_FILE *’
[ 19/ 19 (100.0%) done]: OK!                     

Integrated into trainDataset functions, and which recieves as input two TokenMemoryBlock (ANN output and desired output) and produces as output a number (the loss of the ANN). Another method to compute gradient is needed.

class LossFunction {
public:
  float computeLoss(Token *output, Token *target_output);
  Token *computeGradient();
};

Cross validation normally is k-fold cross validation, and the implemented function is for train with two datasets.

Please, when the master branch is changed, add to the top of CHANGELIST.md
file your updates. In this way, when a release is deployed, it is easier to produce
the list of changes from previous version.

Please fix the website and README.

http://www.lua.org/about.html#name

I've implemented two new functionalities for mlps. Now are two local scripts but it will be very interesting put it in the mlp class code.

• Given an mlp and an input dataset, return a new output dataset with datasetin:numPatterns() x mlp:get_output_size(). There's no need to generate the output dataset outside the code.
• Given a neural network that uses an image data based as an input, sometimes the net is trained where white is 0 or 1. I've implemented a method that takes an mlp and transform the weights of the first layer to work with an inverted notification. It's a kind of converter.

For each function, what will be the correct name for the function on mlp class? In the first case, the function is susceptible to be confused with use_dataset.

My suggestion is compute_dataset(dataset) for the first and reverse_input() for the second.

Eliminate the training method which could work with shuffle, replacement or distribution with DataSet especializations:

• ShuffleDataSet => return the same number of patterns as the underlying DataSet, but in random order. It needs a method RESET to reshuffle the elements.
• ReplacementDataSet => First, in the RESET method, it builds a random sampling replacement of patterns. Every getPattern returns the corresponding pattern in its internal state. The RESET method reinitialize the replacement every time is called.
• DistributionDataSet => as replacement, but also with an apriori distribution over any given number of datasets.

• ANNComponent: dummy component

Basic components => all of them include weights parameter:

• DotProductANNComponent: computes y = w*x
• BiasANNComponent: computes y = x + b
• HyperplaneANNComponent: combination of previous two, computes y = w*x + b

Container components:

• StackANNComponent: a sequential pipeline of ANNComponents
• JoinANNComponent: a parallel pipeline of ANNComponents
• CopyANNComponent: copies input a given number of times

Activation function components:

• ActivationFunctionANNComponent: parent class implementation, includes dropout parameter, so all the childs has the ability to activate dropout
• LogisticANNComponent: computes y = f(x) where f is the sigmoide function
• TanhANNComponent: idem but with f = tanh function
• LogLogisticANNComponent
• LogSoftmaxANNComponent
• SoftmaxANNComponent
• SoftsignANNComponent: idem but with f = softsign function
• SoftplusANNComponent: idem but with f = softplus function
• LinearANNComponent: y = x, but dropout could be applied
• HardtanhANNComponent
• SinANNComponent

Components which could receive a sparse input:

• DotProductANNComponent
• BiasANNComponent

In the method use_dataset, there is an assert that checks if the bunch_size is set in the trainer and if it is not, the argument is mandatory.

If you call a use_dataset like this:

trainer:use_dataset {
input_dataset= dsin,
output_dataset= dsout
}
and bunch_size is in trainer, it fails (type comparision boolean and integer).

if you call,
trainer:use_dataset {
input_dataset= dsin,
output_dataset= dsout,
bunch_size = 32,
}

the error dissapear.

The next release will come with a Lua implementation of automatic differentiation for the fast development of novel machine learning models, regularization methods, loss functions, etc.

It will be implemented in the packages:

• autodiff: all the basis for autodiff of constants, matrices, scalars, ...
• autodiff.ann: an extension of autodiff with new operators related with ANNs.

When the backprop error is computed, and the current bunch size is 1, the call to BLAS is not clear... I'm not sure that it is correct, so it will be nice to check it. Filename dot_product_action.cc, method backpropagateErrors, see the FIXME line.

1. Every feature has to be developed in a new branch. Contributors can have its own fork and work in new branches for new features.
2. Once a feature is finished, a pull requests into devel branch is mandatory, unless commits with bug or memory leak problems solving which can be done directly into devel without a pull request. Please, avoid to do merge of feature branches into devel, a pull request allow other developers and contributors to talk about the feature before it is merged into deve.
3. After the pull request, the branch must to be deleted.
4. Every friday, the stable version in devel branch will be merged with master branch.
5. Every friday, after previous step, your stable changes could be committed to the branch devel.
6. During the week, devel branch must be checked and mark as stable.

Pushes to devel are throwing a travis process to ensure the good compilation and testing the tool. The same is configured to pushes and pull requests to master branch.

Dropout will be converted into an ANN component, instance of stochastic component class. This will complicate the use of dropout, but it makes dropout, gaussian noise, and salt and pepper components more orthogonal with the rest of April-ANN.

Options will be removed from ANN components, instead they has an internal state parametrized at construction time.

The template luabind_template.cc has a NULL pointer check which wasn't needed in lua-5.1.4. It is checked at destructor of binded classes. Due to unknown reasons, the destructor is being called with a NULL pointer when at program exit.

luabind_template.cc, line 185 and so on:

    int lua_delete_$$ClassName$$_$$FILENAME2$$(lua_State *L){
      $$ClassName$$ *obj = lua_rawget$$ClassName$$_$$FILENAME2$$(L,1);
      if (obj != 0) { // LINE 187
        DEBUG_OBJ("lua_delete_$$ClassName$$ (begin)",obj);
        $$class.destructor$$
        DEBUG_OBJ("lua_delete_$$ClassName$$ (end)",obj);
        // Hacemos un DecRef para borrar la referencia a este objeto
        DecRef(obj);

Their are useless at new architecture. AllAllMLP is a Lua function which returns the correspondent hierarchy of ANNComponents. MLP is not needed, its logic will be deleted or written at ANNBase.

ANNBase will be very different, so it only includes methods to train given a bunch of patterns, and other auxiliar stuff. May be it is not necessary...

Deprecated train_holdout_validation and train_wo_validation functions, replaced by a Lua class.