cuai / correctandsmooth Goto Github PK

[ICLR 2021] Combining Label Propagation and Simple Models Out-performs Graph Neural Networks (https://arxiv.org/abs/2010.13993)

Python 96.45% Julia 3.55%

correctandsmooth's People

Contributors

Stargazers

Watchers

correctandsmooth's Issues

I just read title and abstract of the paper, and before I read, I would like to know following things

In which aspect is the comparison is being done? GCNN are highly effective when doing clustering supervised or semi-supervised.
The aspect where GCNN is the first choice is when the input data is point-cloud.
I am not aware of any other case where GCNN is STOA, else if you could simplify the abstract and explain here in a few bullet points?

Hyperparams for LP?

Hi, thanks for this nice work!

I wanted to reproduce your results for label prop but could only find your hyperparameter settings for the two OGB datasets. Could you share the best LP hyperparameters you found for the other datasets as well?

Tara

Question about the implementation of label propagation

Hi, I noticed that a "clamp " operation is applied after each iteration of label propagation. Is there any paper explaining why it is need and what is the new objective function?

FileNotFoundError: [Errno 2] No such file or directory: 'embeddings/diffusionarxiv.pt'

I have installed julia and PyJulia, but there still have some error in run the "gen_models.py"
(pytorch) yq@ubuntu:/mnt/data/yq/CorrectAndSmooth$ python gen_models.py --dataset arxiv --model plain --epochs 1000 WARNING:root:The OGB package is out of date. Your version is 1.3.0, while the latest version is 1.3.1. Namespace(dataset='arxiv', device=0, dropout=0.5, epochs=1000, hidden_channels=256, log_steps=1, lr=0.01, model='plain', num_layers=3, runs=10, use_embeddings=False) embeddings/diffusionarxiv.pt not found or not enough iterations! Regenerating it now Computing adj... Start diffusion processing 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:07<00:00, 1.34it/s] Traceback (most recent call last): File "gen_models.py", line 206, in <module> main() File "gen_models.py", line 144, in main embeddings = torch.cat([preprocess(preprocess_data, 'diffusion', post_fix=args.dataset), File "/mnt/data/yq/CorrectAndSmooth/diffusion_feature.py", line 144, in preprocess torch.save(result, f'embeddings/{preprocess}{post_fix}.pt') File "/home/yq/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 369, in save with _open_file_like(f, 'wb') as opened_file: File "/home/yq/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 234, in _open_file_like return _open_file(name_or_buffer, mode) File "/home/yq/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 215, in __init__ super(_open_file, self).__init__(open(name, mode)) FileNotFoundError: [Errno 2] No such file or directory: 'embeddings/diffusionarxiv.pt'

may somebody share links to get data sets from Table 1: Summary statistics of datasets

may somebody share links to get data sets from
Table 1: Summary statistics of datasets
https://arxiv.org/pdf/2010.13993.pdf

No such file or directory: 'julia'

After my installing the julia package, the following error happened when I run the code 'gen_models'.
FileNotFoundError: [Errno 2] No such file or directory: 'julia'
WHY? thx!

Could you please provide the YML

Hi, could you please provide the set of libraries needed to replicate your experiments?

Something like a YML file would be great :)

Thanks a lot for your time!

Best

About Rice31 and US County datasets

Hi,

Thank you for your good work! May I know where I can find the Rice31 raw data? How do you process the US County datasets from their raw data?

It would be greatly appreciated if you could release your processed data of these two datasets.

Any experimental results on Papers100M?

It's my honor to read your excellent paper which has broadened my view greatly. You conduct associated experiments on two OGB node property prediction datasets, namely, ogbn-arxiv and ogbn-products. I think the two datasets are somewhat small-scale or medium-scale, did you try to conduct experiments on ogbn-papers100M dataset?

what about other datasets.

Hi, I checked the result of the paper and It seems there are more datasets that you used to compare. could you please put the code for that part as well?

Question about PygNodePropPredDataset in run_experiment

When i excute 'run_experiment.py', the code adj, D_isqrt = process_adj(data) in line 31 will build a undirected graph matrix.But, it can't find the edge_index in PygNodePropPredDataset from OGB. I have tried ogb=1.1.0 | 1.2.0 | 1.3.0, which have the same wrong case. Can u do me a favor. Thank you very much!

The strategy of searching hyperparameters for C&S

Hi, thanks for your excellent work. I tried to search hyperparameters for MLP+C&S on arxiv. The performance of base MLP model is:

Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012

With the default hyperparameter setting provided in your example script run_experiments.py, I confirm the similar result:

Valid acc: 0.7401±0.0016 | Test acc: 0.7310±0.0015

However, when I tried to search values of alpha1, alpha2, adj1, adj2 (using autoscale) for better performance by validation accuracy, I found it is easy to obtain obviously higher validation accuracy but lower test accuracy. For example, after 200 trials using Optuna:

[I 2022-02-17 11:13:19,941] Trial 171 finished with value: 0.7397664351152723 and parameters: {'alpha1': 0.9998697337619668, 'adj1': 'AD', 'alpha2': 0.5793203196953342, 'adj2': 'DAD'}. Best is trial 106 with value: 0.741508104298802.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7414±0.0009 | Test acc: 0.7262±0.0014
[I 2022-02-17 11:13:23,649] Trial 172 finished with value: 0.74142085304876 and parameters: {'alpha1': 0.980621104544987, 'adj1': 'AD', 'alpha2': 0.6102579143062772, 'adj2': 'DAD'}. Best is trial 106 with value: 0.741508104298802.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7415±0.0009 | Test acc: 0.7260±0.0013
[I 2022-02-17 11:13:27,362] Trial 173 finished with value: 0.7414510554045438 and parameters: {'alpha1': 0.9845767019485405, 'adj1': 'AD', 'alpha2': 0.5881524805875032, 'adj2': 'DAD'}. Best is trial 106 with value: 0.741508104298802.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7406±0.0011 | Test acc: 0.7249±0.0014
[I 2022-02-17 11:13:31,084] Trial 174 finished with value: 0.7406355917983825 and parameters: {'alpha1': 0.9442333700288734, 'adj1': 'AD', 'alpha2': 0.563141874204418, 'adj2': 'DAD'}. Best is trial 106 with value: 0.741508104298802.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7416±0.0009 | Test acc: 0.7260±0.0013
[I 2022-02-17 11:13:34,819] Trial 175 finished with value: 0.7415550857411323 and parameters: {'alpha1': 0.9879853998605097, 'adj1': 'AD', 'alpha2': 0.5882664075898522, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7392±0.0010 | Test acc: 0.7250±0.0013
[I 2022-02-17 11:13:38,523] Trial 176 finished with value: 0.7392362159804021 and parameters: {'alpha1': 0.9995818269286275, 'adj1': 'AD', 'alpha2': 0.48803181308787474, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7397±0.0011 | Test acc: 0.7261±0.0013
[I 2022-02-17 11:13:42,242] Trial 177 finished with value: 0.7397362327594885 and parameters: {'alpha1': 0.9999159900091027, 'adj1': 'AD', 'alpha2': 0.5940887446932098, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7408±0.0009 | Test acc: 0.7250±0.0014
[I 2022-02-17 11:13:45,950] Trial 178 finished with value: 0.7407798919426826 and parameters: {'alpha1': 0.9543255302847481, 'adj1': 'AD', 'alpha2': 0.5497164972534224, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7414±0.0008 | Test acc: 0.7259±0.0013
[I 2022-02-17 11:13:49,662] Trial 179 finished with value: 0.7413805832410484 and parameters: {'alpha1': 0.983210322945432, 'adj1': 'AD', 'alpha2': 0.5881338038548104, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7405±0.0010 | Test acc: 0.7251±0.0013
[I 2022-02-17 11:13:53,379] Trial 180 finished with value: 0.7404980032887009 and parameters: {'alpha1': 0.9275816290037779, 'adj1': 'AD', 'alpha2': 0.6223246918695156, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7413±0.0009 | Test acc: 0.7258±0.0013
[I 2022-02-17 11:13:57,086] Trial 181 finished with value: 0.7413369576160274 and parameters: {'alpha1': 0.9833652973767343, 'adj1': 'AD', 'alpha2': 0.5736522033930277, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7412±0.0011 | Test acc: 0.7256±0.0014
[I 2022-02-17 11:14:00,802] Trial 182 finished with value: 0.741222859827511 and parameters: {'alpha1': 0.9610791967043985, 'adj1': 'AD', 'alpha2': 0.6077306751373425, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7414±0.0009 | Test acc: 0.7264±0.0013
[I 2022-02-17 11:14:04,526] Trial 183 finished with value: 0.7414040739622135 and parameters: {'alpha1': 0.9801855084637857, 'adj1': 'AD', 'alpha2': 0.6318577630605813, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7409±0.0010 | Test acc: 0.7256±0.0015
[I 2022-02-17 11:14:08,244] Trial 184 finished with value: 0.7408704990100339 and parameters: {'alpha1': 0.9430149061785248, 'adj1': 'AD', 'alpha2': 0.6348513831949518, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7413±0.0008 | Test acc: 0.7263±0.0014
[I 2022-02-17 11:14:11,950] Trial 185 finished with value: 0.7413268901640995 and parameters: {'alpha1': 0.9955916811039495, 'adj1': 'AD', 'alpha2': 0.5967686115392363, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7409±0.0009 | Test acc: 0.7264±0.0014
[I 2022-02-17 11:14:15,659] Trial 186 finished with value: 0.7409040571831269 and parameters: {'alpha1': 0.9986618985811345, 'adj1': 'AD', 'alpha2': 0.6233606296775285, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7413±0.0010 | Test acc: 0.7261±0.0013
[I 2022-02-17 11:14:19,369] Trial 187 finished with value: 0.7412899761736971 and parameters: {'alpha1': 0.9635071926679348, 'adj1': 'AD', 'alpha2': 0.6482700143571917, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7413±0.0009 | Test acc: 0.7258±0.0013
[I 2022-02-17 11:14:23,079] Trial 188 finished with value: 0.7412698412698413 and parameters: {'alpha1': 0.9807221398927547, 'adj1': 'AD', 'alpha2': 0.5715132614425644, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7405±0.0010 | Test acc: 0.7263±0.0014
[I 2022-02-17 11:14:26,796] Trial 189 finished with value: 0.7405382730964126 and parameters: {'alpha1': 0.9992832309685767, 'adj1': 'AD', 'alpha2': 0.6098670231892414, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7361±0.0009 | Test acc: 0.7218±0.0011
[I 2022-02-17 11:14:30,469] Trial 190 finished with value: 0.736108594248129 and parameters: {'alpha1': 0.948309908753022, 'adj1': 'AD', 'alpha2': 0.5250033002666431, 'adj2': 'DA'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7413±0.0010 | Test acc: 0.7267±0.0012
[I 2022-02-17 11:14:34,183] Trial 191 finished with value: 0.7412597738179134 and parameters: {'alpha1': 0.973154541171484, 'adj1': 'AD', 'alpha2': 0.6893967352205037, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7414±0.0009 | Test acc: 0.7265±0.0013
[I 2022-02-17 11:14:37,895] Trial 192 finished with value: 0.7413738716064298 and parameters: {'alpha1': 0.9807824762755216, 'adj1': 'AD', 'alpha2': 0.6396159416388099, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7413±0.0011 | Test acc: 0.7264±0.0012
[I 2022-02-17 11:14:41,608] Trial 193 finished with value: 0.7412597738179134 and parameters: {'alpha1': 0.9612390597009817, 'adj1': 'AD', 'alpha2': 0.6839165981115675, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7408±0.0010 | Test acc: 0.7264±0.0014
[I 2022-02-17 11:14:45,332] Trial 194 finished with value: 0.7407798919426827 and parameters: {'alpha1': 0.999002486052839, 'adj1': 'AD', 'alpha2': 0.61920737978392, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7406±0.0009 | Test acc: 0.7249±0.0013
[I 2022-02-17 11:14:49,049] Trial 195 finished with value: 0.7406020336252894 and parameters: {'alpha1': 0.9337309403205698, 'adj1': 'AD', 'alpha2': 0.5892864918403496, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7413±0.0011 | Test acc: 0.7262±0.0013
[I 2022-02-17 11:14:52,765] Trial 196 finished with value: 0.7413168227121715 and parameters: {'alpha1': 0.9668120293198653, 'adj1': 'AD', 'alpha2': 0.6508764478982865, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7401±0.0010 | Test acc: 0.7257±0.0014
[I 2022-02-17 11:14:56,481] Trial 197 finished with value: 0.7400953052115843 and parameters: {'alpha1': 0.9991146402719119, 'adj1': 'AD', 'alpha2': 0.5498901233953032, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7412±0.0009 | Test acc: 0.7264±0.0013
[I 2022-02-17 11:15:00,205] Trial 198 finished with value: 0.7412295714621296 and parameters: {'alpha1': 0.9766324963371931, 'adj1': 'AD', 'alpha2': 0.6295796864898812, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
Valid acc: 0.7289±0.0008 | Test acc: 0.7150±0.0012
Valid acc: 0.7390±0.0017 | Test acc: 0.7293±0.0015
[I 2022-02-17 11:15:03,918] Trial 199 finished with value: 0.7389811738648947 and parameters: {'alpha1': 0.9529192480940544, 'adj1': 'DA', 'alpha2': 0.6504111365505655, 'adj2': 'DAD'}. Best is trial 175 with value: 0.7415550857411323.
FrozenTrial(number=175, values=[0.7415550857411323], datetime_start=datetime.datetime(2022, 2, 17, 11, 13, 31, 101794), datetime_complete=datetime.datetime(2022, 2, 17, 11, 13, 34, 819114), params={'alpha1': 0.9879853998605097, 'adj1': 'AD', 'alpha2': 0.5882664075898522, 'adj2': 'DAD'}, distributions={'alpha1': UniformDistribution(high=1, low=0), 'adj1': CategoricalDistribution(choices=('DA', 'AD', 'DAD')), 'alpha2': UniformDistribution(high=1, low=0), 'adj2': CategoricalDistribution(choices=('DA', 'AD', 'DAD'))}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=175, state=TrialState.COMPLETE, value=None)

The best hyperparameter setting with the highest validation accruacy has result:

Valid acc: 0.7416±0.0009 | Test acc: 0.7260±0.0013

Would you mind providing your strategy of searching hyperparameters?
Thanks again!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.