input.txt max size about char-rnn HOT 10 OPEN

How large was the file?

from char-rnn.

3.2G

from char-rnn.

interesting. 3GB is very large by the way, I'm not even sure if a huge LSTM running on 12GB GPU could fit that.

We have PR in works that will allow people to use multiple input files, so maybe those will be helpful. What is the dataset? (curious)

from char-rnn.

The dataset is an internal one of tag streams from a news website. It probably doesn't need to be that big and could be heavily down sampled but just thought I would try the whole dataset and thus found the error. Hopefully the multiple file work will solve this as you say.

from char-rnn.

Hi, same issue here, only my file is 9 GB big (10 years of eurusd ticks). BTW, the data is freely available here: http://www.ulozto.cz/xpn7H9Td/eurusd-ticks-2003-05-04-2013-08-20-csv-7z

from char-rnn.

I have a 3.4GB input file of user generated fiction and it loads without this error. I'm training on a GPU (GTX 970) 32Gbs RAM.

That said, I am getting a different error while evaluating loss at the split level: .../char-nn/utils/OneHot.lua Index out of bounds at ~/torch/pkg/torch/lib/TH/generic/THTensorMath.c:141

from char-rnn.

@Cmaury one thing you might want to double-check is that all characters in your input file are ascii, in range 32-128 (maybe plus 13, 10 for linefeed), without some strange unicode quotation marks and stuff. Not sure if strange quotation marks will break things, but they might.

from char-rnn.

I had a dataset that included strange quotes and things and it was fine. I just filtered out a bunch of data to reduce my vocab size though, anyway.

from char-rnn.

Thanks @hughperkins! Ensuring the input file was in ASCII resolved the issues. The 3.5gb file is running without any errors.

from char-rnn.

@Cmaury Cool :-)

from char-rnn.