wnzhang / make-ipinyou-data Goto Github PK
View Code? Open in Web Editor NEWThis project is to formalise the iPinYou RTB data into a standard format for further researches.
License: Apache License 2.0
This project is to formalise the iPinYou RTB data into a standard format for further researches.
License: Apache License 2.0
arr = l.split('\t')
bid = arr[0]+'-'+arr[1]
the line of bid is useless.
Was already warned in code inspection, because of this code in formalizeua.py:
if ua.is_mobile:
device = "m"
elif ua.is_tablet:
device = "t"
elif ua.is_pc:
device = "c"
Apparently there is a case in the train.log where none of these conditions are met.
Traceback (most recent call last):
File "./python/formalizeua.py", line 41, in
output = output + '\t' + '\t'.join([device, device_family, os_family, browser])
NameError: name 'device' is not defined
make: *** [Makefile:28: train.log] Error 1
Errors occur when running "make all" in the "make-ipinyou-data-master" directory.
`
~/Downloads/make-ipinyou-data-master$ make all
mkdir -p ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training2nd/imp..bz2 ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training2nd/clk..bz2 ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training3rd/imp..bz2 ./original-data/ipinyou.contest.dataset/train
cp ./original-data/ipinyou.contest.dataset/training3rd/clk..bz2 ./original-data/ipinyou.contest.dataset/train
bzip2 -d ./original-data/ipinyou.contest.dataset/train/*
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130606.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130607.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130608.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130609.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130610.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130611.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20130612.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131019.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131020.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131021.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131022.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131023.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131024.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131025.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131026.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt -- using ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/clk.20131027.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130606.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130607.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130608.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130609.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130610.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130611.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20130612.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131019.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131020.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131021.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131022.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131023.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131024.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131025.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131026.txt already exists.
bzip2: Can't guess original name for ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt -- using ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt.out
bzip2: ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt is not a bzip2 file.
bzip2: Output file ./original-data/ipinyou.contest.dataset/train/imp.20131027.txt already exists.
Makefile:11: recipe for target 'init' failed
make: *** [init] Error 2
`
And the result of the command "make all" are as follows:
1458 2261 2997 3386 3476 LICENSE mkyxdata.sh python schema.txt
2259 2821 3358 3427 all Makefile original-data README.md
However, there are only test.log.txt and train.log.txt in "1458","2261",etc.
Hello,
I got "No module named user_agents'"when I used "make all", but I have installed user-agent since I can successfully import the module in command line, what should I do ?
if len(sys.argv) < 5:
print 'Usage: ipinyou.folder 25 train.log.txt test.log.txt'
# python splitadvertisers.py /home/weinan/Project/tqchen-exp/ipinyou/ 25 ../all/train.log.txt ../all/test.log.txt
exit(-1)
" what's format in the file /home/weinan/Project/tqchen-exp/ipinyou/?
and 25 means generate files?
only 4 files after splitadvertisers.py? why it still run?
Please update the download link.
Hello,
When trying to run SNN_RBM on the 1458 campaign, we get the following error.
'''
../../make-ipinyou-data/1458/train.yzx.txt
drop_mlp4da.py|ad:1458|drop:1|b_size:1000 | X:133465 | Hidden 0:300 | Hidden 1:300 | Hidden 2:100 | L_r:0.0006 | activation1:tanh | lambda:0.0001
training RBM
line: 133465
line: 300
Traceback (most recent call last):
File "SNN_RBM.py", line 81, in
ww0,bb0,ww1,bb1,ww2,bb2=gbrbm.get_rbm_weights(train_file,arr,ncases=train_size,batch_size=100000,fm_model_file=fm_model_file)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_gaussian_binary_rbm_sparse.py", line 533, in get_rbm_weights
trainer.train(file,epochs,ncases,cdsteps=1,k=k)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_gaussian_binary_rbm_sparse.py", line 448, in train
thisweightstep -= np.dot(vis.T, hid)
ValueError: operands could not be broadcast together with shapes (4,300) (32,300) (4,300)
'''
Any hard-sought solutions at hand?
Thanks
is this video about your ipinyou-data?
Optimal Real-Time Bidding for Display Advertising
https://www.youtube.com/watch?v=OoI2aCWdWKA&t=2405s
Hello Weinan Zhang,
I run "make all" for about seven hours and got the errors above,and under the folder "make-ipinyou-data" I got some folders named "227"、"238"、"241"、“249"、”254"、“277"、”294"、“300" rather than 1458 2261 2997 3386 3476 2259 2821 3358 3427.what should I do?
Thanks.
great code thanks
only can you share description for fields for this data
https://www.dropbox.com/s/txz0ms0axqf7jrl/ipinyou.contest.dataset.7z?dl=0
what meaning is for each feature and target
and hopefully some base line machine learning code how to use it?
Hi,
When running SNN_RBM.py on the datasets from the paper (tried on 1458 and 2261), we get the following error:
'''
../../make-ipinyou-data/2261/train.yzx.txt
drop_mlp4da.py|ad:2261|drop:1|b_size:1000 | X:133465 | Hidden 0:300 | Hidden 1:300 | Hidden 2:100 | L_r:0.0006 | activation1:tanh | lambda:0.0001
Traceback (most recent call last):
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\compile\function_module.py", line 903, in call
self.fn() if output_subset is None else
ValueError: Shape mismatch: x has 37 cols (and 1 rows) but y has 70 rows (and 300 cols)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "SNN_DAE.py", line 85, in
ww0,bb0,ww1,bb1,ww2,bb2=da.get_da_weights(train_file,arr,num_feats=numf,ncases=train_size,batch_size=100000)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 364, in get_da_weights
w,b=sparse_da(num_feats*k,col,file,training_epochs=epochs,sparse_len=row,is_sparse=1,batch_size=1,k=k)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 327, in sparse_da
[cost,z,w,b]=train_da(batcharr,initial_W)
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\compile\function_module.py", line 914, in call
gof.link.raise_with_op(
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\gof\link.py", line 325, in raise_with_op
reraise(exc_type, exc_value, exc_trace)
File "C:\Users\corpa\anaconda3\lib\site-packages\six.py", line 702, in reraise
raise value.with_traceback(tb)
File "C:\Users\corpa\anaconda3\lib\site-packages\theano\compile\function_module.py", line 903, in call
self.fn() if output_subset is None else
ValueError: Shape mismatch: x has 37 cols (and 1 rows) but y has 70 rows (and 300 cols)
Apply node that caused the error: Dot22(Elemwise{Mul}[(0, 0)].0, ww)
Toposort index: 12
Inputs types: [TensorType(float64, matrix), TensorType(float64, matrix)]
Inputs shapes: [(1, 37), (70, 300)]
Inputs strides: [(296, 8), (2400, 8)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwise{Composite{scalar_sigmoid((i0 + i1))}}[(0, 0)](Dot22.0, InplaceDimShuffle{x,0}.0)]]
Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
File "SNN_DAE.py", line 85, in
ww0,bb0,ww1,bb1,ww2,bb2=da.get_da_weights(train_file,arr,num_feats=numf,ncases=train_size,batch_size=100000)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 364, in get_da_weights
w,b=sparse_da(num_feats*k,col,file,training_epochs=epochs,sparse_len=row,is_sparse=1,batch_size=1,k=k)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 249, in sparse_da
cost, updates,z,w,b = da.get_cost_updates(
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 102, in get_cost_updates
y = self.get_hidden_values(tilde_x)
File "C:\Users\corpa\Master\Sem3\Experiment Design For Data Science\Assignment2\deep-ctr\python\sampling_based_denosing_autoencoder.py", line 92, in get_hidden_values
return T.nnet.sigmoid(T.dot(input, self.W) + self.b)
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
'''
Ideas of solutions?
Thanks!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.