Rating Scale? about nprizeadditions HOT 12 CLOSED

Are you suggesting an input, output or calculation problem?

Thanks
-greg

From: afchin [email protected]
To: gbildson/nprizeadditions [email protected]
Sent: Saturday, November 24, 2012 8:10 PM
Subject: [nprizeadditions] Rating Scale? (#1)

Hi! I'm looking at your rbm.c code and I feel like something's strange. Are the ratings used in the program in the range 0-4? The original netflix data was in the scale 1-5, right?
—
Reply to this email directly or view it on GitHub.

from nprizeadditions.

Without looking too deeply, the ratings in most of these programs are stored as error residuals in the 0 to 4 range. Some output adjustment might have been standard.

from nprizeadditions.

(This is just for the normal rbm) For both input and output. I can tell from the indexing on the arrays that the ratings are expected to be in the 0-4 range. To get the estimated rating back into the 1-5 range, is it just a matter of adding 1 to the output of the program?

I'm trying to run this program right now. My RMSE on the training set started at 1.05325 and after 21 epochs it's only 1.00787. I've looked at Hinton's graphs and the RMSE should decrease rapidly over the first few epochs. I've sort of attributed this to an input issue - I've been using the training set Netflix ratings (scale 1-5) minus one as input ratings.

Or should I be using residuals from an SVD?

Thanks!

from nprizeadditions.

Here's my original post on the subject.   It should get down to a RMSE of 0.918.  I'm rerunning it now to check although the current values look ugly.  

http://www.netflixprize.com//community/viewtopic.php?id=1536

2009-09-17 12:56:15
CommonSense2007
Member

Registered: 2009-07-19
Posts: 102
Restricted Boltzmann Machine code available
I finally got a pure RBM working well enough to open source it.  For those still wanting to take a look at RBM code look here:
  http://code.google.com/p/nprizeaddition … runk/rbm.c

This
code can achieve a probe RMSE of 0.918197 with 100 hidden variables and
slightly modified for 200 hidden variables it can achieve 0.916576.  By
applying global centering on the result for 100 hidden variables, it
can achieve 0.915987.

The code is in C and builds off the tools generously provided by Ehud Ben-Reuven so Kudos to him.

I also posted a version of my Conditional RBM as well:
  http://code.google.com/p/nprizeaddition … /rbmcond.c

I
believe that this contains all the basic elements but I have not been
able to get down to published probe RMSE values.  I'm working on the
tuning now so I will update the parameters there if I achieve better
results. 

If anyone notices a problem in either of these code files, please let me know.   

I
have a whole suite of other models including a Factored RBM, SVD,
SVD++, the integrated model and time dependent SVD++.  If there is
interest, I'll clean those up for publishing as well.

From: afchin [email protected]
To: gbildson/nprizeadditions [email protected]
Cc: gbildson [email protected]
Sent: Sunday, November 25, 2012 3:47 PM
Subject: Re: [nprizeadditions] Rating Scale? (#1)

(This is just for the normal rbm) For both input and output. I can tell from the indexing on the arrays that the ratings are expected to be in the 0-4 range. To get the estimated rating back into the 1-5 range, is it just a matter of adding 1 to the output of the program?
I'm trying to run this program right now. My RMSE on the training set started at 1.05325 and after 21 epochs it's only 1.00787. I've looked at Hinton's graphs and the RMSE should decrease rapidly over the first few epochs. I've sort of attributed this to an input issue - I've been using the training set Netflix ratings (scale 1-5) minus one as input ratings.
Or should I be using residuals from an SVD?
Thanks!
—
Reply to this email directly or view it on GitHub.

from nprizeadditions.

I've actually seen that post and some of your other ones, which is how I got here. :D Thanks so much for posting it, by the way. It really helped me understand Hinton's paper.

So value-wise, do my inputs seem correct? I'm only training on 100 features at the moment, if that affects anything.

Thanks for taking the time to look at this!

Edit: My code finished running and I went from RMSE 1.00846 to RMSE 1.0077 over 30 epochs.

from nprizeadditions.

Your input should be in the 0 to 4 range (from the userent data structure if you use that). Output would need to be + 1 at some point. If you're using my standard setup, you would have needed to generate the user_entry.bin and user_index.bin.

You've probably noticed that it took a great deal of cross referencing for me to understand those hinton papers originally.

from nprizeadditions.

I was getting some really bad results when running this under windows and I thought that some of that initialization looked a little problematic from other experience. So, I changed it to this and it seems to be initializing better and giving better results:

210,219c210,214
< vishid[j][0][i] = 0.02 * randn() - 0.004; // Normal Distribution
< vishid[j][1][i] = 0.02 * randn() - 0.004; // Normal Distribution
< vishid[j][2][i] = 0.02 * randn() - 0.004; // Normal Distribution
< vishid[j][3][i] = 0.02 * randn() - 0.004; // Normal Distribution
< vishid[j][4][i] = 0.02 * randn() - 0.004; // Normal Distribution
< //vishid[j][0][i] = 0.02 * randn() - 0.01; // Normal Distribution
< //vishid[j][1][i] = 0.02 * randn() - 0.01; // Normal Distribution
< //vishid[j][2][i] = 0.02 * randn() - 0.01; // Normal Distribution
< //vishid[j][3][i] = 0.02 * randn() - 0.01; // Normal Distribution

< //vishid[j][4][i] = 0.02 * randn() - 0.01; // Normal Distribution

        vishid[j][0][i] = 0.02 * randn() - 0.01; // Normal Distribution
        vishid[j][1][i] = 0.02 * randn() - 0.01; // Normal Distribution
        vishid[j][2][i] = 0.02 * randn() - 0.01; // Normal Distribution
        vishid[j][3][i] = 0.02 * randn() - 0.01; // Normal Distribution
        vishid[j][4][i] = 0.02 * randn() - 0.01; // Normal Distribution

230,231c225
< //visbiases[j][i] = log( ((double)moviercount[j*SOFTMAX+i]) / ((double) mtot) );

< visbiases[j][i] = 0.02 * randn() - 0.004;

        visbiases[j][i] = log( ((double)moviercount[j*SOFTMAX+i]) / ((double) mtot) );

Results look promising so far:

./rbm -l 1 -se data/r100_012.bin
Loading data/user_index.bin
Train=99072112 Probe=1408395 Qualify=2817131
Loading data/user_entry.bin
Removing global average score 2.603304 99072112
Clipping errors
RMSE Train 1.084575 (-192.2%) Probe 1.129623 (-188.5%) Both 1.085219 (-192.1%)
Clipping errors
RMSE Train 1.084575 (0.0%) Probe 1.129623 (0.0%) Both 1.085219 (0.0%)
Loop 0
0.945398 1.011798 523.898000
0.877517 0.955114 459.798000
0.861968 0.944020 432.324000
0.853362 0.938695 431.780000
0.848100 0.935850 429.346000

from nprizeadditions.

Thanks!

Is there some kind of pre-training going on before the rbm.c is called (based on that output, it looks like something else is happening)? Or are you putting the raw data into the rbm?

I changed the parameters per what you did. I'm seeing greater per-loop improvements, but the starting RMSE was 1.24, and after 5 loops it's at 1.19. All I'm running right now is your doAllFeatures() function.

P.S. I'm also on a Windows machine, and I'm running a C++ version of your code.

from nprizeadditions.

That's weird.  Are you running this on Linux/OSX or Windows?    It should be the raw user/movie data going in.

From: afchin [email protected]
To: gbildson/nprizeadditions [email protected]
Cc: gbildson [email protected]
Sent: Monday, November 26, 2012 3:34 PM
Subject: Re: [nprizeadditions] Rating Scale? (#1)

Thanks!
Is there some kind of pre-training going on before the rbm.c is called (based on that output, it looks like something else is happening)? Or are you putting the raw data into the rbm?
I changed the parameters per what you did. I'm seeing greater per-loop improvements, but the starting RMSE was 1.24, and after 5 loops it's at 1.19.
—
Reply to this email directly or view it on GitHub.

from nprizeadditions.

Sorry, edited my above comment so it might not have shown up for you. I'm running a C++ version on Windows.

from nprizeadditions.

Cygwin?   That should be the same ...

If you don't figure out the diff, make sure your random isn't messed up.  I had that happen to me at least once ...

From: afchin [email protected]
To: gbildson/nprizeadditions [email protected]
Cc: gbildson [email protected]
Sent: Monday, November 26, 2012 3:40 PM
Subject: Re: [nprizeadditions] Rating Scale? (#1)

Sorry, edited my above comment so it might not have shown up for you. I'm running a C++ version on Windows.
—
Reply to this email directly or view it on GitHub.

from nprizeadditions.

Hello Greg,

I just had a look at your C-RBM Code and was wondering if it ever yielded better results than the Basic RBM ? I tried myself an home-made version as close as possible to the what is described in all papers (ie Toronto) but it just -NEVER- yielded any better result whatever the parameters were. Despite the fact that the logic behind it seems more than valid I'm starting to think that there is something about that conditional RBM version for Netflix. A misprint ? What do you think ?

Regards.
Keep

from nprizeadditions.