awslabs / graphstorm Goto Github PK

We should allow users to specify custom training/validation/test sets during graph construction. #29 (comment)

Some features are categorial values. That is, the original values are strings. We need to convert it to integers.

When DistDGL partition a graph, it will shuffle node ids and edge ids. We need to store the new-id to original-id mapping.

Currently, trainer 0 saves the learnable embeddings to the disk. If the learnable embedding table is very large, trainer 0 will run out of memory. We need to distribute the saving of learnable embeddings.

Our setup.py currently does not have specific requirements of library versions, although the library assumes a couple of libraries have at least some specific versions. In particular:

ogb >= 1.3.6,
torch >= 1.12

The V0.1 release will focus mostly on completing and optimizing the existing functionalities and adding data loading support.

• [Improvement] Saving/restoring model artifact in distributed training process user experience.
  • #28
• [Improvement] Refining all configuration arguments.
  • #78
  • #79
  • #80
  • #81
  • #99
• [New Feature] Provide a single-machine data loading pipeline.
  • #40
  • #51
• [Doc] Tutorial of graph construction, GNN model training and GNN model inference.
  • #61
• [Improvement] Add support for tuning language model with a separate learning rate.
  • #35
• [Improvement] Bug fixes.

Welcome any comments and feedback!

The tutorial will includes:

1. GraphStorm environment setup.
2. Preparing graph data.
3. Training a GNN model using GraphStorm framework.
4. Do GNN inference using GraphStorm framework.
5. Collecting model artifacts and prediction resutls.

We should use a field to store the valid length of a token list instead of using an attention mask.

When GNN embeddings are saved, they are saved based on the node IDs in the partitioned graph, which are different from the original node IDs of users' input data. We need to define a simple way for users to identify the right GNN embeddings from the original node IDs.

Currently, GraphStorm's evaluation code can only run with gloo. We need to support NCCL in the future.

Here are codes can break NCCL:

When the encoder is MLP, GraphStorm fails if fanout is empty.

Remove argument log_report_frequency. It is not useful.

Unable to download OGB datasets required for partitioning and running regression tests.
Affected datasets (arxiv, products, papers100M)
Able to download OGB MAG

There are multiple tests in (https://github.com/awslabs/graphstorm/blob/main/tests/end2end-tests/graphstorm-ec/mgpu_test.sh) that train GNN+BERT in end to end.
BERT+GNN training is expensive. we should reduce the tests.
also, because remove_target_edge_type is turned on by default, the tests don't have fanout.

When doing link prediction training, different edges may have different weights representing their importance. Supporting weighted edge in link prediction loss is required.

We are going to add a new argument --lp-edge-weight-for-loss to specify the edge weight used in loss function.

Related #214

Per our discussion, we need to rename window_for_early_stop to early_stop_rounds.

Running pylint --rcfile tests/lint/pylintrc --errors-only on examples/customized_models/HGT/hgt_nc.py produces:

examples/customized_models/HGT/hgt_nc.py:332:17: E1101: Instance of 'GSgnnNodePredictionTrainer' has no 'get_best_model' member (no-member)

Attempting inference on ogbn-papers100M for link prediction is causing an OOM (out-of-memory) issue, as shown in the attached screenshot. As a result, the system is becoming extremely unresponsive. The OOM happens in the evaluation function (val_mrr, test_mrr = self.evaluator.evaluate(None, test_scores, 0)) function here. System can successfully save the node embedding and relational embeddings and exit the program without any issue when the evaluation function is omitted.

Experiment setup:

Dataset: ogbn-papers100M partitioned into 3
Instance: g4dn.metal
Command to run Inferencing:

python3 -u  ~/dgl/tools/launch.py \
        --workspace /graph-storm/inference_scripts/lp_infer \
        --num_trainers 1 \
        --num_servers 1 \
        --num_samplers 0 \
        --part_config /data/ogbn-papers100M-3p/ogbn-papers100M.json \
        --ip_config  /data/ip_list_p3_metal.txt \
        --ssh_port 2222 \
        "python3 lp_infer_gnn.py --cf  /data/ogbn_papers100M_infer_p3.yaml  --use-node-embeddings false --num-gpus 4 --part-config /data/ogbn-papers100M-3p/ogbn-papers100M.json  --restore-model-path /data/papers100M-lp-p3-model/epoch-0  --feat-name feat --no-validation false"

Reproduced with the following environment:

• DGL 1.0.2 + GSF github/gitlab version
• DGL 1.0.0 + GSF github/gitlab version

Smaller dataset like ogbn-mag works fine on similar setup.

In configuration, argument "verbose" and "debug" have similar functions. So could merge them into one argument, e.g., "verbose" only.

Renaming argument save_model_per_iters to save_model_frequency will add more transparency to users.

Change argument mini_batch_infer to use_mini_batch_infer to increase usability.

I tested with MAG data.

For MAG, the best trainer test MRR is around 0.38, but the infer test MRR is about 0.83.

No clue about why the test MRRs have such big difference.

The interface of predict and forward should be consistent.

These names of configurations/arguments are either intuitive or consistent. Recommend to make changes.

• feat_name: change to “node_feat_name”
  • Use “edge_feat_name” later when support edge features.
• n_layers: change to "num_layers"
• n_epochs change to "num_epochs"
• sparse_lr: change to “sparse_optimizer_lr”
• save_predict_path: change to "save_prediction_path"
• negative_sampler: change to "train_negative_sampler"
• test_negative_sampler: change to "eval_negative_sampler"
• enable_early_stop: change to "use_early_stop"
• lm_infer_batchsize: change to "lm_infer_batch_size"
• evaluation_frequency: change to "eval_frequency"

Setting:

• Two nodes, g4dn.12xlarge.
• Launch node create a folder, e.g., "sharedfolder" and use it as an NFS shared folder.
• In the execution node create an NFS client, and mount the "sharedfolder".
• Copy graphstorm code and data into the sharedfolder.

How to reproduct:

1. Set the "save-model-path", e.g., "/sharedfolder/model"
2. Run an LP task in two partitions.
3. Could set the "evaluation_frequency" and "save_model_per_iters" to a smaller value, e.g. 10, to accelerate this repruducing.

Problem:

1. After evaluation and save mode;
2. The launch node can save sparse embeddings successfully, but
3. The execution node can NOT save the sparse embeddings.

Root cause:

1. The launch node's rank 0 process will make the "/sharedfolder/model" when start to save model artifacts;
2. The "/sharedfolder/model" mode is 755 rwxr-xr-x;
3. The launch node's rank 0 process will make folders for each save-model-per-iters, e.g., "/sharedfolder/model/epoch-0-iter-19" and "/sharedfolder/model/epoch-0-iter-19/item/", whose mode are 755 too.
4. The launc node's rank 0 process will save "model.bin" and "optimizer.bin" to the ""/sharedfolder/model/epoch-0-iter-19"" folder.
5. The launch nodes' process can save sparse embeddings to the ""/sharedfolder/model/epoch-0-iter-19/item"" folder.
6. BUT, the execution node can NOT save sparse embeddings to the ""/sharedfolder/model/epoch-0-iter-19/item"" folder, because the 755 mode do not allow process in the execution nodes to save them.

This is caused by the newly distributed saving of sparse embeddings.

Rename evaluation_frequency to eval_frequency to be consistent with other arguments.

The argument name "n_hidden" is not intuitive. After discussion, suggest to change it to "hidden_size".

Motivation

To run a GraphStorm training task, a user need to take the following steps:

1. Pip install GraphStorm library
2. git clone https://github.com/dmlc/dgl.git
3. git clone https://github.com/awslabs/graphstorm.git
4. Use dgl launch.py script to launch a task.

The steps are to complex as the user needs to download may codebases to get the entrypoint scripts.

Proposal

We will integrate launch.py as well as other entrypoint scripts provided in training_scripts/inference_scripts into GraphStorm library. Then the UX will be:

1. Pip install GraphStorm library
2. Run a training task using built-in training script: python3 -m graphstorm.run.gsf_node_classification xxx or python3 -m graphstorm.run.gsf_link_prediction xxx

For user defined training script: python3 -m graphstorm.run.launch xxx <PATH_TO_SCRIPT> xxx .

Follow the naming convention, we need to replace all n_ to num_ for those use the "n" for the "number".

So, rename "n_bases" and "n_heads" to "num_bases" and "num_heads"

DGL's launch script kills all training processes if a user types Ctrl-C. This doesn't work in GraphStorm. GraphStorm should also handle Ctrl-C signal.

In configuration, the argument "use_dot_product" is not intuitive about its function, and it is easy to get an assertion error of proper setting.

So

• it is better to change to an intuitive argument name, and,
• it is better to automatically to determine its value based on other given arguments or graph information.

Currently, GraphStorm does not save/load optimizer states of learnable sparse embeddings. (https://github.com/awslabs/graphstorm/blob/main/python/graphstorm/model/utils.py#L336-L337). We need to support it.

GraphStorm may need to support evaluation on separated edge types for link prediction

Currently, the labels on edges must be classification task.

Some feature processing generates multiple tensors. For example, tokenziation generates three tensors. We need to define a rule of how to name the output tensors, as discussed here: #29 (comment)

issue description:

In the new launch API, when use "crtl + c" to exit the training/inference processes in the launch instance, the GraphStorm processes in other instances DO NOT exit! The launch script does not kill other instances' processes.

Log track:

Step 40 | Validation mrr: 0.0590
Step 40 | Test mrr: 0.0588
Step 40 | Best Validation mrr: 0.0590
Step 40 | Best Test mrr: 0.0588
Step 40 | Best Iteration mrr: 40.0000
Eval time: 61.0621, Evaluation step: 40.0000
evaluate validation/test: elapsed time: 17.759, mem (curr: 14.181, peak: 14.181, shared: 8.483, global curr: 25.959, global shared: 12.100) GB
successfully save the model to /data/ak/models-test/epoch-0-iter-39
Time on save model 0.0036745071411132812
Epoch 00000 | Batch 040 | Train Loss: 0.3333 | Time: 9.5225
^CProcess Process-1:
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.11.170 'cd /data/ak; (export DGL_ROLE=server DGL_NUM_SAMPLER=0 OMP_NUM_THREADS=1 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo PYTHONPATH=/graphstorm/python DGL_SERVER_ID=3; /usr/bin/python3 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
Traceback (most recent call last):
File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/graphstorm/python/graphstorm/run/launch.py", line 58, in cleanup_proc
remote_pids = get_all_remote_pids_func()
File "/graphstorm/python/graphstorm/run/launch.py", line 279, in get_all_remote_pids
cmds = udf_command.split()
AttributeError: 'list' object has no attribute 'split'
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.2.213 'cd /data/ak; (export DGL_DIST_MODE=distributed DGL_ROLE=client DGL_NUM_SAMPLER=0 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo OMP_NUM_THREADS=6 DGL_GROUP_ID=0 PYTHONPATH=/graphstorm/python ; /usr/bin/python3 -m torch.distributed.launch --nproc_per_node=4 --nnodes=4 --node_rank=1 --master_addr=172.31.11.145 --master_port=1234 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.11.170 'cd /data/ak; (export DGL_DIST_MODE=distributed DGL_ROLE=client DGL_NUM_SAMPLER=0 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo OMP_NUM_THREADS=6 DGL_GROUP_ID=0 PYTHONPATH=/graphstorm/python ; /usr/bin/python3 -m torch.distributed.launch --nproc_per_node=4 --nnodes=4 --node_rank=3 --master_addr=172.31.11.145 --master_port=1234 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.14.145 'cd /data/ak; (export DGL_DIST_MODE=distributed DGL_ROLE=client DGL_NUM_SAMPLER=0 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo OMP_NUM_THREADS=6 DGL_GROUP_ID=0 PYTHONPATH=/graphstorm/python ; /usr/bin/python3 -m torch.distributed.launch --nproc_per_node=4 --nnodes=4 --node_rank=2 --master_addr=172.31.11.145 --master_port=1234 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.11.145 'cd /data/ak; (export DGL_ROLE=server DGL_NUM_SAMPLER=0 OMP_NUM_THREADS=1 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo PYTHONPATH=/graphstorm/python DGL_SERVER_ID=0; /usr/bin/python3 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.11.145 'cd /data/ak; (export DGL_DIST_MODE=distributed DGL_ROLE=client DGL_NUM_SAMPLER=0 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo OMP_NUM_THREADS=6 DGL_GROUP_ID=0 PYTHONPATH=/graphstorm/python ; /usr/bin/python3 -m torch.distributed.launch --nproc_per_node=4 --nnodes=4 --node_rank=0 --master_addr=172.31.11.145 --master_port=1234 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.14.145 'cd /data/ak; (export DGL_ROLE=server DGL_NUM_SAMPLER=0 OMP_NUM_THREADS=1 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo PYTHONPATH=/graphstorm/python DGL_SERVER_ID=2; /usr/bin/python3 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
Called process error Command 'ssh -o StrictHostKeyChecking=no -p 2222 172.31.2.213 'cd /data/ak; (export DGL_ROLE=server DGL_NUM_SAMPLER=0 OMP_NUM_THREADS=1 DGL_NUM_CLIENT=16 DGL_CONF_PATH=/data/ak/ak_lp_4p/ak.json DGL_IP_CONFIG=/data/ip_list.txt DGL_NUM_SERVER=1 DGL_GRAPH_FORMAT=csc,coo PYTHONPATH=/graphstorm/python DGL_SERVER_ID=1; /usr/bin/python3 /graphstorm/python/graphstorm/run/gsgnn_lp/gsgnn_lp.py --cf ak_lp.yaml --evaluation-frequency 20 --save-model-per-iters 20 --save-model-path /data/ak/models-test --save-embed-path /data/ak/embed-test --ip-config /data/ip_list.txt --part-config /data/ak/ak_lp_4p/ak.json --verbose False)'' died with <Signals.SIGINT: 2>.
^C

• Don't require users to provide graph name.
• Verify the node/edge feature names and label names.
• If infer_data.test_idxs is empty, return an error in the inference script.
• We may want to warn users if they want to have mini-batch inference. (mini-batch inference on a large graph can take very long time).

Some node types may not have any node features. In this case, we require users to provide node IDs for all nodes of thes node types.
#29 (comment)

Currently, GraphStorm combines the MRR scores of different relations types into an overall MRR score. It is required to report MRR scores separately.

Rename training_scripts and inference_scripts folders to training_examples and inference_examples respectively.

To be consistent, rename "predict_*type" to "target_*type"

In the configuration file, the "num_gpus" argument is confusing. What is its default behavior?

Because this argument only give a number of GPUs and does not know if some GPUs are shared with other users, its default behavior should be using ALL GPUs.

So please check current codes to see if this is the default behavior. If not, please implement it.

Currently, GraphStorm doesn't pass the edge features.

Current graph construction configuration only allow users to set the split percentages. But users may want to preset the split by their own logics, e.g., timestamp.

Need to allow users to setup their own train/val/test masks in both data files and the configuration JSON file.

The Amazon SageMaker support should includes:

• Support launching data processing task using Amazon SageMaker.
• Support launching distributed training task using Amazon SageMaker.
• Support launching distributed inference task using Amazon SageMaker.
• Tutorial.

In configuration, argument "eval_batch_size" may slow down the speed of evaluation because users normally set this value to 1k-4k as used in the "train_batch_size". And in link prediction task, we have another computation of test score, the smaller batch_size may also slow down the speed.

So,

• set its default value to a larger one, e.g. +10k
• add another argument, e.g. "lp_score_batch_size", to set the batch size of link prediction score computation.

The argument name "call_to_consider_early_stop" is not intuitive. After discussion, suggest to change it to "early_stop_burnin_rounds" and set default value to 0.