https://www.cs.toronto.edu/~kriz/cifar.html

https://arxiv.org/abs/1409.6070

URGENT: on DenseNets, hyperbolic tangents (activation functions) are incorrectly being used.

As reported by Mika:

    ReadLn;
  end;

should be

    ReadLn;
    Terminate;
  end;

Otherwise it keeps repeating DoRun method over and over.

Add multi gpu/opencl support.

https://sourceforge.net/p/cai/svncode/HEAD/tree/trunk/lazarus/experiments/mining/

Most APIs have the LR decay expressed with 0.01 or 0.001 as examples. It's required to change CAI to conform with this.

As an example from the problem look at this example:
https://github.com/joaopauloschuler/neural-api/tree/master/examples/SimpleImageClassifier

This is the current LR setting:
NeuralFit.LearningRateDecay := 0.995;

The requirement is to change API and examples to:
NeuralFit.LearningRateDecay := 0.005;

Development environment.
Delphi 10.3, targets windows 32 or 64 bits, release and debug.
Last release of Master branch as of June 21rst.

Error : Compiler errors when trying to compile in Release mode.

Error 1:
In NeuralNetwork.pas
Compiler raise error : E2501 : inline not possible for a nested procedure.

This is due to inline directive" in the declaration of TNNetLayer.ComputeErrorDeriv();
It is activated when going in release mode, but not activated in debug.

Error2:
In NeuralThread;pas
Similar problem with function fNTL: TNeuralThreadList; {$IFDEF Release} inline; {$ENDIF}
Cannot be inlined.

https://arxiv.org/pdf/1706.02515.pdf
https://towardsdatascience.com/gentle-introduction-to-selus-b19943068cd9

Have you got cool examples? Feel free to pull and share your example with everyone!!!

It would be nice to provide a callback function to print the output messages in a GUI. For example, a TMemo. I mean the ones that are written with WriteLn.

For example:

  WriteLn
  (
    'Min/Max at 0:',  Min0:4:0,' ',Max0:4:0,
    ' at 1:', Min1:4:0,' ',Max1:4:0,
    ' at 2:', Min2:4:0,' ',Max2:4:0
  );

will be

S := Format('Min/Max at 0: %4.0f %4.0f at 1: %4.0f %4.0f at 2: %4.0f %4.0f', 
           [Min0, Max0, Min1, Max1, Min2, Max2]); 
SomeCallBack(S);

and, by default SomeCallBack point to a simple procedure with a WriteLn to don't break anything

procedure SomeCallBack(S: string);
begin
  WriteLn(S);
end;

There are strange errors when running the simple image classifier example on ARM64/Graviton processors.

TNeuralImageLoadingFit and TNeuralImageFit could have a better training sampling shuffling.

When you try to save neural net settings in string then first float variable is encoded with "," separator, while all others are encoded with ".". This is the is issue with fformatsettings.
Check this sample TNNetFullConnectReLU:4;1;1;0;0;0;0;0>!0,055678159]1;93;1;1;-0.009392704815;-0.06912346184;-0.01935156807;-0.1705754697;

Number 0,055678159 is encoded with colon if you have central european default decimal settings.
So if one computer learn and save result and it is with ',', and the other use that result and it is in US settings (decimal separator i '.') then you will get error while trying to decode this string.

File is TNNetNeuron.LoadFromString. Search for string FBiasWeight := StrToFloat(S[0]) around line 9322. Also check TNNetNeuron.SaveToString(): FloatToStr in same file.

Next issue is
procedure TNeuralDataLoadingFit.FitLoading(pNN: TNNet; TrainingCnt,

Line if Assigned(FThreadNN) then FThreadNN.Free;

is this line exists then

instead of FThreadNN.Free; in line 604
there should be FreeAndNil(FThreadNN);

Issue is visible when you try to run FitLoading for the second time.
First time does not have pointer to thread, second time it thinks then it have pointer but it is actually not available because of FThreadNN.Free

Regards, Dinko

The documentation should include:

• Fitting options (Image classification, volume pairs, callbacks).
• Layer activation functions.
• How to hide or override warnings and errors when optimizing a NN.
• Include an autoencoder example and describe the importance of Hyperbolic Tangent with autoencoders and generative networks.

https://arxiv.org/abs/1605.06489

This is great stuff! Thanks for making this!

My setup: Windows 10 Home 2H20, Ryzen 2500U, latest AMD drivers, Lazarus 2.0.10 with FPC 3.2.0. GPU probe shows following:

Platform info: 0 ---------------------
PROFILE: FULL_PROFILE
VERSION: OpenCL 2.1 AMD-APP (3188.4)
NAME: AMD Accelerated Parallel Processing
VENDOR: Advanced Micro Devices, Inc.
EXTENSIONS: cl_khr_icd cl_khr_d3d10_sharing cl_khr_d3d11_sharing cl_khr_dx9_media_sharing cl_amd_event_callback cl_amd_offline_devices

However, when running, say, opencl-dot-product-test.exe, I get many lines of the following:

Error: TDotProductCL.Compute - Failed setting parameters: -48

Current CSV export isn't compatible with github.

There is a small error in the delphi function

I corrected it like this:

{$IFNDEF FPC}
function SwapEndian(I:integer):integer;
begin
// valid for SmallInt
// result := Swap(I)
Result := ((Swap(Smallint(I)) and $ffff) shl $10) or (Swap(Smallint(I shr $10)) and $ffff)
end;
{$ENDIF}

https://github.com/Pigrecos/neural-api/blob/fa2c0f180a1fd2ce239f908cb7419e8c21380676/neural/neuraldatasets.pas#L217

Add Cyclical Learning Rates property to define the cycle length (epoch count) as an integer number.

Hello,
lot's of errors during build with Delphi10.3 :
[dcc32 Erreur] neuralnetwork.pas(756): E2267 La déclaration précédente de 'Create' n'a pas été marquée de la directive 'overload'
[dcc32 Avertissement] neuralnetwork.pas(778): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetFullConnect'
[dcc32 Erreur] neuralnetwork.pas(778): E2267 La déclaration précédente de 'Create' n'a pas été marquée de la directive 'overload'
[dcc32 Avertissement] neuralnetwork.pas(785): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetFullConnect'
[dcc32 Erreur] neuralnetwork.pas(785): E2267 La déclaration précédente de 'Create' n'a pas été marquée de la directive 'overload'
[dcc32 Avertissement] neuralnetwork.pas(796): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetFullConnectLinear'
[dcc32 Erreur] neuralnetwork.pas(796): E2267 La déclaration précédente de 'Create' n'a pas été marquée de la directive 'overload'
[dcc32 Avertissement] neuralnetwork.pas(805): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetFullConnectReLU'
[dcc32 Erreur] neuralnetwork.pas(805): E2267 La déclaration précédente de 'Create' n'a pas été marquée de la directive 'overload'
[dcc32 Avertissement] neuralnetwork.pas(841): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayerConcatedWeights'
[dcc32 Erreur] neuralnetwork.pas(857): E2169 Définition de champ non autorisée après des méthodes ou propriétés
[dcc32 Erreur] neuralnetwork.pas(858): E2070 Directive inconnue : 'Compute'
[dcc32 Erreur] neuralnetwork.pas(858): E2029 ',' ou ':' attendu(e) mais ';' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(859): E2070 Directive inconnue : 'Backpropagate'
[dcc32 Erreur] neuralnetwork.pas(859): E2029 ';' attendu(e) mais '(' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(859): E2029 ',' ou ':' attendu(e) mais ';' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(860): E2070 Directive inconnue : 'InitDefault'
[dcc32 Erreur] neuralnetwork.pas(860): E2029 ';' attendu(e) mais '(' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(860): E2029 ',' ou ':' attendu(e) mais ';' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(861): E2029 Type attendu(e) mais 'END' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(891): E2169 Définition de champ non autorisée après des méthodes ou propriétés
[dcc32 Erreur] neuralnetwork.pas(892): E2029 Type attendu(e) mais 'DESTRUCTOR' trouvé(e)
[dcc32 Avertissement] neuralnetwork.pas(937): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConvolutionBase'
[dcc32 Erreur] neuralnetwork.pas(937): E2169 Définition de champ non autorisée après des méthodes ou propriétés
[dcc32 Erreur] neuralnetwork.pas(938): E2029 Type attendu(e) mais 'DESTRUCTOR' trouvé(e)
[dcc32 Avertissement] neuralnetwork.pas(956): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConvolutionBase'
[dcc32 Avertissement] neuralnetwork.pas(962): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConvolutionLinear'
[dcc32 Avertissement] neuralnetwork.pas(968): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConvolutionReLU'
[dcc32 Avertissement] neuralnetwork.pas(974): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConvolutionBase'
[dcc32 Avertissement] neuralnetwork.pas(980): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConvolutionReLU'
[dcc32 Avertissement] neuralnetwork.pas(1011): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConvolutionBase'
[dcc32 Avertissement] neuralnetwork.pas(1023): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLocalConnectReLU'
[dcc32 Avertissement] neuralnetwork.pas(1041): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc32 Erreur] neuralnetwork.pas(2873): E2003 Identificateur non déclaré : 'Compute'
[dcc32 Erreur] neuralnetwork.pas(2877): E2003 Identificateur non déclaré : 'FNeurons'
[dcc32 Erreur] neuralnetwork.pas(2880): E2003 Identificateur non déclaré : 'RefreshCalculatePrevLayerError'
[dcc32 Erreur] neuralnetwork.pas(2881): E2003 Identificateur non déclaré : 'FPadding'
[dcc32 Avertissement] neuralnetwork.pas(2882): W1023 Comparaison de types signés et non signés - opérandes élargis
[dcc32 Erreur] neuralnetwork.pas(2882): E2003 Identificateur non déclaré : 'FInputCopy'
[dcc32 Erreur] neuralnetwork.pas(2882): E2066 Opérateur ou point-virgule manquant
[dcc32 Erreur] neuralnetwork.pas(2882): E2003 Identificateur non déclaré : 'FPrevLayer'
[dcc32 Erreur] neuralnetwork.pas(2882): E2066 Opérateur ou point-virgule manquant
[dcc32 Erreur] neuralnetwork.pas(2890): E2029 ';' attendu(e) mais 'ELSE' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(2892): E2003 Identificateur non déclaré : 'FErrorProc'
[dcc32 Erreur] neuralnetwork.pas(2892): E2003 Identificateur non déclaré : 'FNeurons'
[dcc32 Erreur] neuralnetwork.pas(2892): E2029 'END' attendu(e) mais ')' trouvé(e)
[dcc32 Erreur] neuralnetwork.pas(2893): E2029 '.' attendu(e) mais ';' trouvé(e)
[dcc32 Avertissement] neuralnetwork.pas(2894): W1011 Texte après le 'END' final. - ignoré par le compilateur
[dcc32 Erreur] neuralnetwork.pas(106): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.Create'
[dcc32 Erreur] neuralnetwork.pas(107): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.Destroy'
[dcc32 Erreur] neuralnetwork.pas(108): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.Fill'
[dcc32 Erreur] neuralnetwork.pas(109): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.AddInertia'
[dcc32 Erreur] neuralnetwork.pas(110): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.UpdateWeights'
[dcc32 Erreur] neuralnetwork.pas(111): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.SaveToString'
[dcc32 Erreur] neuralnetwork.pas(112): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.LoadFromString'
[dcc32 Erreur] neuralnetwork.pas(113): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.ClearDelta'
[dcc32 Erreur] neuralnetwork.pas(116): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitUniform'
[dcc32 Erreur] neuralnetwork.pas(117): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitGaussian'
[dcc32 Erreur] neuralnetwork.pas(118): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitLeCunUniform'
[dcc32 Erreur] neuralnetwork.pas(119): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitHeUniform'
[dcc32 Erreur] neuralnetwork.pas(120): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitHeGaussian'
[dcc32 Erreur] neuralnetwork.pas(121): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitHeUniformDepthwise'
[dcc32 Erreur] neuralnetwork.pas(122): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitHeGaussianDepthwise'
[dcc32 Erreur] neuralnetwork.pas(123): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuron.InitSELU'
[dcc32 Erreur] neuralnetwork.pas(136): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuronList.GetItem'
[dcc32 Erreur] neuralnetwork.pas(137): E2065 Déclaration forward ou external non satisfaite : 'TNNetNeuronList.SetItem'
[dcc32 Erreur] neuralnetwork.pas(180): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.InitStruct'
[dcc32 Erreur] neuralnetwork.pas(190): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ComputeL2Decay'
[dcc32 Erreur] neuralnetwork.pas(191): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ComputePreviousLayerError'
[dcc32 Erreur] neuralnetwork.pas(192): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.SetPrevLayer'
[dcc32 Erreur] neuralnetwork.pas(193): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ApplyActivationFunctionToOutput'
[dcc32 Erreur] neuralnetwork.pas(194): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.BuildArrNeurons'
[dcc32 Erreur] neuralnetwork.pas(195): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.AfterWeightUpdate'
[dcc32 Erreur] neuralnetwork.pas(197): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.Create'
[dcc32 Erreur] neuralnetwork.pas(198): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.Destroy'
[dcc32 Erreur] neuralnetwork.pas(206): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ComputeOutputErrorForOneNeuron'
[dcc32 Erreur] neuralnetwork.pas(207): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ComputeOutputErrorWith'
[dcc32 Erreur] neuralnetwork.pas(208): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ComputeOutputErrorForIdx'
[dcc32 Erreur] neuralnetwork.pas(209): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ComputeErrorDeriv'
[dcc32 Erreur] neuralnetwork.pas(210): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.Fill'
[dcc32 Erreur] neuralnetwork.pas(211): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ClearDeltas'
[dcc32 Erreur] neuralnetwork.pas(212): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.AddNeurons'
[dcc32 Erreur] neuralnetwork.pas(213): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.AddMissingNeurons'
[dcc32 Erreur] neuralnetwork.pas(214): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.SetNumWeightsForAllNeurons'
[dcc32 Erreur] neuralnetwork.pas(215): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.SetNumWeightsForAllNeurons'
[dcc32 Erreur] neuralnetwork.pas(216): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.SetNumWeightsForAllNeurons'
[dcc32 Erreur] neuralnetwork.pas(217): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetMaxWeight'
[dcc32 Erreur] neuralnetwork.pas(218): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetMinWeight'
[dcc32 Erreur] neuralnetwork.pas(219): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetMaxDelta'
[dcc32 Erreur] neuralnetwork.pas(220): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetMinDelta'
[dcc32 Erreur] neuralnetwork.pas(221): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.ForceMaxAbsoluteDelta'
[dcc32 Erreur] neuralnetwork.pas(222): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetMaxAbsoluteDelta'
[dcc32 Erreur] neuralnetwork.pas(223): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetMinMaxAtDepth'
[dcc32 Erreur] neuralnetwork.pas(224): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetWeightSum'
[dcc32 Erreur] neuralnetwork.pas(225): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetBiasSum'
[dcc32 Erreur] neuralnetwork.pas(226): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.GetInertiaSum'
[dcc32 Erreur] neuralnetwork.pas(227): E2065 Déclaration forward ou external non satisfaite : 'TNNetLayer.CountWeights'
[dcc32 Erreur fatale] neuralnetwork.pas(228): E2226 Compilation abandonnée ; trop d'erreurs

Hello,
I did a software to learn with pair lists (delphi 10.3) . It's work fine.
But if i run the software several times (>=2), to start several learning at the same time, nothing more happens.
Software doesn't do anything anymore, they occupy 0% of the CPU. I need to kill all process.
It's the same problem with or without CUDA OpenCL.
Only one at the same time !
Do you know this problem ?
Thanks

TNNetLayer.ComputeOutputErrorForOneNeuron is incorrectly implemented for 3D activation maps.

Epoch Time Estimation is Wrong Neural Fit

TNNetSplitChannelEvery layer doesn't seem to be saved/loaded properly.

Source of inspiration:
https://towardsdatascience.com/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0

There are plenty of problems that can be solved with neural networks. This is a request to add more source code examples.

Source: https://arxiv.org/pdf/1302.4389.pdf

Requirement: implement maxout activation function or numerically equivalent function.

probably it dont matter so much how activation function but current sigmoid work as relu , and if need to be work as relu why need exp function in it?

function Sigmoid(x: TNeuralFloat): TNeuralFloat;
begin
Result := x / ( 1 + Exp(-x) );
end;

right sigmoid function need to be like that i guess,

function Sigmoid(x: TNeuralFloat): TNeuralFloat;
begin
Result := 1 / ( 1 + Exp(-x) );
end;

PS: probably current activation of sigmoid works so much better then right sigmoid formula, looks like leaky relu

How about an example where you have 2 inputs (x,y) and an output (z) which is the hypotenuse of a right-angle triangle?

So in our dataset, Z^2 = X^2 + Y^2.. use maybe 10 data points.

A simple start-to-finish on "solving" and improving on the solution to this problem could be extrapolated to all other sorts of problems.

Gradient Ascent example fails to show some patterns displaying random dots.

Feature request: add working example for Places365-Standard dataset:
http://places2.csail.mit.edu/

There are 1.8 million train images from 365 scene categories in the Places365-Standard.

Hello,

I have made a basic network, for testing and debugging.

All inputs in the training set are always 0.
All expected output are always the same (3 outputs, 0.1 / 0.25 / 0.50).

So, I would expect the netwrok to learn the bias rapidelly. It does not.
The output is always 0.

If now I set the inputs to a random number, the network will learn.

Network is rather simple, although I used the "multi input" option.

// Create Network
NN := TNNet.Create();

// Create input layers structure
for i := 0 to length(InputLayers) - 1 do
begin
InputLayers[i] := NN.AddLayer(TNNetInput.Create(NbQuotes, NbQuotesData, 1));
end;

for i := 0 to Length(Branch) - 1 do
Branch[i] := NN.AddLayerAfter(TNNetFullConnectReLU.Create(3), InputLayers[i]);

// Merge branches
NN.AddLayer(TNNetConcat.Create(Branch));

// Output layer
NN.AddLayer(TNNetFullConnectReLU.Create(3));
NN.AddLayer(TNNetFullConnectReLU.Create(3));

// Init weights
NN.InitWeights;

// Set learning Rate
NN.SetLearningRate(0.01, 0.95);
NN.ErrorProc := MyErrorProc;

http://yann.lecun.com/exdb/mnist/
https://www.kaggle.com/zalando-research/fashionmnist

Requirement: add option to enable and disable dropout mask for the full batch. When the option is disabled, it means we have one dropout mask per sample (and not per batch).

Hello,

The library is realy moving forward on a lot of topics and gives more and more the ability to process CNN in an efficient way. Thanks for this huge work.

Could you incorporate an example of Yolo Net architecture inside the examples. Seeing the layer types that are now existing, I think a Yolo V1 structure is possible, even V2. For V3, they have a "route" layer type, that is not possible yet I guess. V4 is just released and also integrate a special concat layer. May be possible with your newly added "concatinputoutpout".

Thanks

Hello,

First, thanks for this hard work you did. Promising project.

Development environment is Delphi 10.3, Master branch with no modification as of June 21rst.

I wanted to program a simple NN in a windows form environment. As most of the debug possibilities of the library are sent to console with WriteLn, I copied the concept of "Or Xor And" example, with no NeuralFit.

Then, I started with the simplest possible network. One layer, no bias, fully connected.
Input and output data are identity function --> VOuptput[i]= Vinput[i]
Vinput are in a range 0 to 1, 500 values.

So, 1 neuron linking input to output. Its value should converge to 1 during Training.

Network creation;
NN := TNNet.Create();
NN.AddLayer(TNNetInput.Create(1));
NN.AddLayer(TNNetFullConnect.Create(1, 1));
NN.InitWeights;

Training procedure:
var
Cnt, EpochCnt: integer;
begin
pOutPut := TNNetVolume.Create(1, 1);

NN.SetLearningRate(0.01, 0.9);
for EpochCnt := 1 to 3000 do
begin
for Cnt := Low(vInputs) to High(vInputs) do
begin
NN.Compute(vInputs[Cnt]);
NN.GetOutput(pOutPut);
NN.Backpropagate(vOutput[Cnt]);
end;
end;
pOutPut.Free;

The result of the training is very strange.
Weight starts with a value of 2.68, then increase at each "backprop", instead of decreasing to 1. After 3000 loops of training, it is stuck at a value of 3.22 and no longer change.

I also tested with the NeuralFit process, similar behavior. It runs with no CPU consumption after a short while, and never returns as finished.

I also added more layer, reactivated the bias, ... do difference.. System is not converging. The biggest suprise is that it stops to evoluate, even if in the wrong direction.

I will copy my code in Lazarus (latest release) and look if it does the same.

Add API support for data generators.

Follows an example:
https://gist.github.com/jkjung-avt/48a0e59ea970c9ea6b44bf789e511121

Memory access error in linux when calling CopyResizing.

If I run the XorAndOr example in delphi (or by disabling AVX support in Lazaurus) I get errors while calculating the output of the last layer.

TVolume.MulAdd procedures (Value: T; Original: TVolume);
begin
{$ IFDEF Debug}
if Original.Size <> Self.Size then
raise Exception.Create ('Sizes don''t match at MulAdd:' +
IntToStr (Self.Size) + 'and' + IntToStr (Original.Size) + '.');
{$ ENDIF}
MulAdd (Value, Addr (Original.FData [0]));
end;

Throws the exception which is not the case with AVX.

thank you

Hi, i've just tried your latest example "SuperResolution" with the default training sample file

I've made a little project with Lazarus 2.08/fpc 3.2 under windows 10 64bits for comparing results
(with my own library for image processing BZScene )
The code is exaclty the same as your superresolution example.

The image is resized with a factor of x2
a few image is resized perfectly but some others not.
See attached screenshots

On left the image source not resized, not cropped
On center the image resize with Lanczsos3 resample filter
An on the right the image resized with the NN

(right click and open in new tab for see in full resolution)

• 1st sample : source resolution is 300x300
  

• 2nd : source resolution 215x198
  

• 3rd : source resolution 256x256
  

4th : source resolution 380x300

5th : source resolution 425x425

6th : source resolution 400x306

So do you think is due to the training ? it is not enougth ? or this provide from an other issue ?

Thanks

BTW your library is awesome, i love it 👍

Best regards

I read the code rather intensivelly, as the examples, and was still not able to find how to do the following task.

I have an input, sizex = 20, sizey=2, depth= 1

I want to have a network that will process the inputs depth=1, x from 1 to 10, y=1, another that will process depth=1 y=1, x from 11 to 20, ... same with y=2.
This gives 4 blocks that would pass through several layers.
Then, I want to join the results together so the net continues.

I found that you can keep in memory the created layer with assigning the result of the create to a variable, so you can add layers to it. Fine. This is done with using : NN.AddLayerAfter.

I found you can join the result with Concat, fine. This is done using : TNNetDeepConcat

But how do you force the network to use only some of the input data for a specific portion of the net. I read the "resize" and "split", ... does not seems to be the right answers. The split seems to use only the depth, and nothing to specify the neurons to use.

This example doesn't work correctly:
examples/CaiOptimizedDenseNet/CaiOptimizedDenseNet.lpr

We get access violations when running with "release" setting.

API could support existing word embeddings:
http://vectors.nlpl.eu/repository
https://nlp.stanford.edu/projects/glove/
https://fasttext.cc/docs/en/english-vectors.html
https://fasttext.cc/docs/en/crawl-vectors.html

Hello,
when i compile, i have some errors.
Library Neuralopencl, some errors with type, "t" missing , example : "cl_int" unknown, "tcl_int" is OK.
Library Neuraldatasetsv, function "RGB" is unknown, "Uses Windows" is missing at the top of code.
Library Neuralnetwork, function LocalAddBottleNeck, "UnitCnt+=1" is only for C++ not for Delphi, replaced by inc(UnitCnt)
Thank you very much for your work.
JY

Debug shows Max Output: 0.000 Min Output: 0.000 . It should show actual values.

A significant load of "warnings" with variable unused or set without any need. Should be worth a look. I some case, I think you might have done things... that are at the end not used. Strange.

[dcc64 Avertissement] neuralvolume.pas(389): W1055 PUBLISHED a provoqué l'ajout de RTTI ($M+) au type 'TMObject'
[dcc64 Avertissement] neuralvolume.pas(450): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TMObject'
[dcc64 Avertissement] neuralbit.pas(928): W1011 Texte après le 'END' final. - ignoré par le compilateur
[dcc64 Avertissement] neuralvolume.pas(2348): W1036 La variable 'x' n'a pas été initialisée
[dcc64 Conseil] neuralvolume.pas(4493): H2077 La valeur affectée à 'version' n'est jamais utilisée
[dcc64 Conseil] neuralvolume.pas(4490): H2077 La valeur affectée à 'version' n'est jamais utilisée
[dcc64 Conseil] neuralvolume.pas(4718): H2077 La valeur affectée à 'MissedElements' n'est jamais utilisée
[dcc64 Conseil] neuralvolume.pas(4706): H2164 La variable 'CntAPos' est déclarée mais jamais utilisée dans 'TNNetVolume.DotProducts'
[dcc64 Conseil] neuralvolume.pas(4706): H2164 La variable 'CntBPos' est déclarée mais jamais utilisée dans 'TNNetVolume.DotProducts'
[dcc64 Conseil] neuralvolume.pas(4707): H2164 La variable 'DestPointer' est déclarée mais jamais utilisée dans 'TNNetVolume.DotProducts'
[dcc64 Conseil] neuralvolume.pas(4708): H2164 La variable 'CntBVectorSizePlusCntBPos' est déclarée mais jamais utilisée dans 'TNNetVolume.DotProducts'
[dcc64 Conseil] neuralvolume.pas(4709): H2164 La variable 'vRes' est déclarée mais jamais utilisée dans 'TNNetVolume.DotProducts'
[dcc64 Avertissement] neuralvolume.pas(8264): W1029 constructor 'TNNetVolumePair.CreateCopying' en double avec des paramètres identiques inaccessible dans C++
[dcc64 Avertissement] neuralnetwork.pas(331): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(332): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(333): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(357): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(398): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(426): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(468): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(475): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(495): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(512): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(523): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(647): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(666): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(692): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConcatBase'
[dcc64 Avertissement] neuralnetwork.pas(693): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConcatBase'
[dcc64 Avertissement] neuralnetwork.pas(707): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConcatBase'
[dcc64 Avertissement] neuralnetwork.pas(717): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetConcatBase'
[dcc64 Avertissement] neuralnetwork.pas(730): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(731): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Avertissement] neuralnetwork.pas(755): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayerConcatedWeights'
[dcc64 Avertissement] neuralnetwork.pas(756): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayerConcatedWeights'
[dcc64 Avertissement] neuralnetwork.pas(841): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayerConcatedWeights'
[dcc64 Avertissement] neuralnetwork.pas(1041): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TNNetLayer'
[dcc64 Conseil] neuralnetwork.pas(1838): H2077 La valeur affectée à 'InputCntX' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(1835): H2077 La valeur affectée à 'InputCntY' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(1833): H2077 La valeur affectée à 'CntXYD' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(1818): H2077 La valeur affectée à 'InputCntY' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(1815): H2077 La valeur affectée à 'CntXYD' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(1813): H2077 La valeur affectée à 'InputCntX' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6097): H2077 La valeur affectée à 'Len' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6291): H2077 La valeur affectée à 'InputCntX' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6288): H2077 La valeur affectée à 'InputCntY' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6286): H2077 La valeur affectée à 'CntXYD' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6268): H2077 La valeur affectée à 'InputCntY' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6265): H2077 La valeur affectée à 'CntXYD' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6263): H2077 La valeur affectée à 'InputCntX' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(6979): H2077 La valeur affectée à 'RowSize' n'est jamais utilisée
[dcc64 Conseil] neuralnetwork.pas(7212): H2077 La valeur affectée à 'SmoothLocalOutputErrorDerivPtr' n'est jamais utilisée
[dcc64 Avertissement] neuralnetwork.pas(7387): W1036 La variable 'LocalDestPtr' n'a pas été initialisée
[dcc64 Conseil] neuralnetwork.pas(7194): H2164 La variable 'PtrNeuronDelta' est déclarée mais jamais utilisée dans 'TNNetConvolution.BackpropagateFastCPU'
[dcc64 Conseil] neuralnetwork.pas(7502): H2077 La valeur affectée à 'PtrPreparedInput' n'est jamais utilisée
[dcc64 Avertissement] neuralnetwork.pas(7555): W1036 La variable 'LocalDestPtr' n'a pas été initialisée
[dcc64 Conseil] neuralnetwork.pas(7468): H2164 La variable 'PtrNeuronDelta' est déclarée mais jamais utilisée dans 'TNNetConvolution.BackpropagateFastCPUDev'
[dcc64 Conseil] neuralnetwork.pas(850): H2219 Le symbole privé 'BackpropagateCPU' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(853): H2219 Le symbole privé 'ComputeCPU' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(884): H2219 Le symbole privé 'PrepareInputForConvolution' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(914): H2219 Le symbole privé 'ComputeNeuronCPU' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(915): H2219 Le symbole privé 'AddBiasToRawResult' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(917): H2219 Le symbole privé 'ComputeNeuron' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(919): H2219 Le symbole privé 'ComputeNeuronAtOutputPos3' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(920): H2219 Le symbole privé 'ComputeNeuronAtOutputPos3D3' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(921): H2219 Le symbole privé 'ComputeNeuronAtOutputPosDefault' est déclaré mais jamais utilisé
[dcc64 Conseil] neuralnetwork.pas(922): H2219 Le symbole privé 'ComputeNeuronAtOutputPosDefaultFast' est déclaré mais jamais utilisé
[dcc64 Avertissement] neuralfit.pas(94): W1010 La méthode 'Create' cache une méthode virtuelle de type de base 'TMObject'
[dcc64 Conseil] neuralfit.pas(993): H2077 La valeur affectée à 'MaxDelta' n'est jamais utilisée
[dcc64 Conseil] neuralfit.pas(1352): H2077 La valeur affectée à 'MaxDelta' n'est jamais utilisée

Hello,
I have a problem when i want to use OpenCL with Intel HD Graphics 620 (no problem with CUDA)
OpenCL version 2.1. No error when i test with Geeks 3D. OpenCL is running good.
But when the program is building with opencl, the console windows show:
clCreateContext OK!
clCreateCommandQueue OK!
ClCreateProgramWithSource OK!
Error: Failed to build programm executable : -11
Error: Failed to create compute kernel : cai_dot_product
....
And the programm continue to use OpenCL compute with lot's of errors ...
Do you know what is the problem into neural.cl ? and why the program continue to use opencl with this error ?
Thank you for your help
JY