wnzhang / make-ipinyou-data Goto Github PK

arr = l.split('\t')
bid = arr[0]+'-'+arr[1]

the line of bid is useless.

Was already warned in code inspection, because of this code in formalizeua.py:

    if ua.is_mobile:
        device = "m"
    elif ua.is_tablet:
        device = "t"
    elif ua.is_pc:
        device = "c"

Apparently there is a case in the train.log where none of these conditions are met.

Traceback (most recent call last):
File "./python/formalizeua.py", line 41, in
output = output + '\t' + '\t'.join([device, device_family, os_family, browser])
NameError: name 'device' is not defined
make: *** [Makefile:28: train.log] Error 1

Errors occur when running "make all" in the "make-ipinyou-data-master" directory.

`
~/Downloads/make-ipinyou-data-master$ make all
mkdir -p ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training2nd/imp..bz2 ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training2nd/clk..bz2 ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training3rd/imp..bz2 ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training3rd/clk..bz2 ./original-data/ipinyou.contest.dataset/train
bzip2 -d ./original-data/ipinyou.contest.dataset/train/*
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt already exists.
Makefile:11: recipe for target 'init' failed
make: *** [init] Error 2

`

And the result of the command "make all" are as follows:
1458 2261 2997 3386 3476 LICENSE mkyxdata.sh python schema.txt
2259 2821 3358 3427 all Makefile original-data README.md

However, there are only test.log.txt and train.log.txt in "1458","2261",etc.

Hello,
I got "No module named user_agents'"when I used "make all", but I have installed user-agent since I can successfully import the module in command line, what should I do ?

if len(sys.argv) < 5:
print 'Usage: ipinyou.folder 25 train.log.txt test.log.txt'
# python splitadvertisers.py /home/weinan/Project/tqchen-exp/ipinyou/ 25 ../all/train.log.txt ../all/test.log.txt
exit(-1)

" what's format in the file /home/weinan/Project/tqchen-exp/ipinyou/?
and 25 means generate files?

only 4 files after splitadvertisers.py? why it still run?

Please update the download link.

Hello,

When trying to run SNN_RBM on the 1458 campaign, we get the following error.

'''
../../make-ipinyou-data/1458/train.yzx.txt
drop_mlp4da.py|ad:1458|drop:1|b_size:1000 | X:133465 | Hidden 0:300 | Hidden 1:300 | Hidden 2:100 | L_r:0.0006 | activation1:tanh | lambda:0.0001
training RBM
line: 133465
line: 300
Traceback (most recent call last):
File "SNN_RBM.py", line 81, in
ww0,bb0,ww1,bb1,ww2,bb2=gbrbm.get_rbm_weights(train_file,arr,ncases=train_size,batch_size=100000,fm_model_file=fm_model_file)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_gaussian_binary_rbm_sparse.py", line 533, in get_rbm_weights
trainer.train(file,epochs,ncases,cdsteps=1,k=k)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_gaussian_binary_rbm_sparse.py", line 448, in train
thisweightstep -= np.dot(vis.T, hid)
ValueError: operands could not be broadcast together with shapes (4,300) (32,300) (4,300)
'''

Any hard-sought solutions at hand?

Thanks

is this video about your ipinyou-data?

Optimal Real-Time Bidding for Display Advertising
https://www.youtube.com/watch?v=OoI2aCWdWKA&t=2405s

bash ./mkyzxdata.sh
1458
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '1458/train.log.txt'
2261
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '2261/train.log.txt'
2997
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '2997/train.log.txt'
3386
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '3386/train.log.txt'
3476
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '3476/train.log.txt'
2259
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '2259/train.log.txt'
2821
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '2821/train.log.txt'
3358
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '3358/train.log.txt'
3427
Traceback (most recent call last):
File "python/mkyzx.py", line 54, in
fi = open(sys.argv[1], 'r')
IOError: [Errno 2] No such file or directory: '3427/train.log.txt'
make: *** [yzx] 错误 1

Hello Weinan Zhang,
I run "make all" for about seven hours and got the errors above,and under the folder "make-ipinyou-data" I got some folders named "227"、"238"、"241"、“249"、”254"、“277"、”294"、“300" rather than 1458 2261 2997 3386 3476 2259 2821 3358 3427.what should I do?
Thanks.

great code thanks
only can you share description for fields for this data
https://www.dropbox.com/s/txz0ms0axqf7jrl/ipinyou.contest.dataset.7z?dl=0
what meaning is for each feature and target
and hopefully some base line machine learning code how to use it?

Hi,

When running SNN_RBM.py on the datasets from the paper (tried on 1458 and 2261), we get the following error:

'''
../../make-ipinyou-data/2261/train.yzx.txt
drop_mlp4da.py|ad:2261|drop:1|b_size:1000 | X:133465 | Hidden 0:300 | Hidden 1:300 | Hidden 2:100 | L_r:0.0006 | activation1:tanh | lambda:0.0001
Traceback (most recent call last):
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\compile\function_module.py", line 903, in call
self.fn() if output_subset is None else
ValueError: Shape mismatch: x has 37 cols (and 1 rows) but y has 70 rows (and 300 cols)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "SNN_DAE.py", line 85, in
ww0,bb0,ww1,bb1,ww2,bb2=da.get_da_weights(train_file,arr,num_feats=numf,ncases=train_size,batch_size=100000)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 364, in get_da_weights
w,b=sparse_da(num_feats*k,col,file,training_epochs=epochs,sparse_len=row,is_sparse=1,batch_size=1,k=k)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 327, in sparse_da
[cost,z,w,b]=train_da(batcharr,initial_W)
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\compile\function_module.py", line 914, in call
gof.link.raise_with_op(
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "C:\Users\corpa\anaconda3\lib\site-packages\six.py", line 702, in reraise
raise value.with_traceback(tb)
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\compile\function_module.py", line 903, in call
self.fn() if output_subset is None else
ValueError: Shape mismatch: x has 37 cols (and 1 rows) but y has 70 rows (and 300 cols)
Apply node that caused the error: Dot22(Elemwise{Mul}[(0, 0)].0, ww)
Toposort index: 12
Inputs types: [TensorType(float64, matrix), TensorType(float64, matrix)]
Inputs shapes: [(1, 37), (70, 300)]
Inputs strides: [(296, 8), (2400, 8)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwise{Composite{scalar_sigmoid((i0 + i1))}}[(0, 0)](Dot22.0, InplaceDimShuffle{x,0}.0)]]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
File "SNN_DAE.py", line 85, in
ww0,bb0,ww1,bb1,ww2,bb2=da.get_da_weights(train_file,arr,num_feats=numf,ncases=train_size,batch_size=100000)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 364, in get_da_weights
w,b=sparse_da(num_feats*k,col,file,training_epochs=epochs,sparse_len=row,is_sparse=1,batch_size=1,k=k)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 249, in sparse_da
cost, updates,z,w,b = da.get_cost_updates(
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 102, in get_cost_updates
y = self.get_hidden_values(tilde_x)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 92, in get_hidden_values
return T.nnet.sigmoid(T.dot(input, self.W) + self.b)

HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
'''

Ideas of solutions?

Thanks!