hiroyuki-kasai / sgdlibrary Goto Github PK

It seems to me that the experiment in demo_paper.m has a mismatch between the result and setting.

Should the variable be info_sag instead? Also, I am wondering if the same issue occurs in the paper? Or the plots in the paper is actually correct.

Dear Prof. Hiroyuki KASAI,

I have some confusion about the damping scheme in obfgs.
You gave a citation of Wang et al. 's "Stochastic Quasi-Newton Methods for Nonconvex Stochastic Optimization," when mentioning the damping, but I found that the code is more likely to be the classical Powell's damping. In the limited memory setting, Wang's damping only requires one two-loop-recursion but in your implementation you call it twice. In addition, the threshold value indicating when to carray out the damping is set to 0.2 in your code, which is the same to Powell's method but different from the Wang's. I also found a probable bug, if it is indeed the Powell's damping scheme. In Line 144, the probablly correct code is "lbfgs_two_loop_recursion(s, s_array, y_array)" (otherwise you may directly reuse the variable calculated before). Is this a typo or something wrong with my opinion?
Thanks!

Hi @hiroyuki-kasai ,

Thanks for publishing this project. This looks great and I would like do some research with this toolbox.

My trouble is in the sgd_cm.m. It looks like containing two momentum schemes, the classic one ('CM') and the Nesterov's ('NAG'), but in the current implementain, they seem to differ only in the setting of the momentum coefficient. See lines 78, 80, and 82.

In my impression NAG should 'look one step ahead' before the gradient calculation, but in the code, the gradient is evaluated just in the current point. This seems to be inconsistent to the original paper. See equations 3 and 4 in Ilya Sutskever, James Martens, George Dahl and Geoffrey Hinton, "On the importance of initialization and momentum in deep learning," ICML, 2013.

Thank you!

Greetings,

First, I would like to thank you for creating such a nice project with so many optimization algorythms!

I'm currently trying to port the online memory limited BFGS algorythm to python ( yes scipy has many algorythm coded in haskell but it's lacking some that I would like to test, namely this one.)

I've got most of it done but I'm confused about the gradient descent part using the weight on the input.

I'm not familiar with matlab and I want to know if calling .cost , .grad or anything else on the input (the variable problem) are integrated method in the problem code itself as a method or is it a matlab built-in?

P.S.:Thanks for all your hard work!

Greetings Hiroyuki,

I'm so excited to have come across such a rich library in SGD, thank you for these great project.

I'm currently learning Python, Matlab and Machine Learning for data training and prediction.

I have collected data as bitmap files (400 files each containing 1000 ~ 1s and 0s), I wanted to do data training and prediction on single file first, and later a couple of files (5 files, 10 files, 20 files... and so on). These bitmap files are in .txt (text files) format.

Kindly advice, how I can use these bitmap files as input?

Lawrence.

Hello,

@hiroyuki-kasai Thank you for creating this project with a wide variety of algorithms.

I went through the code in linear_regression_data_generator.m and was not quite clear how the data is being generated. Also, on running the code I find that all my rows have the same number. Can you explain how the dataset is generated for linear regression?

% set number of dimensions
d = 50;
% set number of samples
n = 7000;
% generate data
std = 0.25
data = linear_regression_data_generator(n, d, std);

Attached below is the data(x_train) and label(y_train) generated

data.xlsx
label.xlsx

Thank you!

Thanks for developing such a library. When trying to use it, I found that when I change the values of "d" and "n" in the file "demo.m", there will be some errors in the convergence results. For instance, if I choose "d = 105", "n = 30000", the returned results are very strange as shown before:

SVRG: Epoch = 000, cost = Inf, optgap = Inf
SVRG: Epoch = 001, cost = NaN, optgap = NaN

Is there any problem with the code for the reusability?

Hello, this library is really good. I am now learning machine learning algorithms. It helps me a lot and I have some problems. When I was learning this library, I saw the data in the file data. I wanted to know what the data meant, where it was collected, and what each data represented. Thank you very much

Thanks for developing such a library. I am trying to learn the SGD through the demon, it helps me a lot. But when I try to run this demo(test_l1_linear_regression.m), the command line throw an error:
The method, attribute or field'prox_flag' of class'l1_linear_regression' is not recognized.