Any example show how to do distributed training with data resident on HDFS? about pytorch-biggraph HOT 4 CLOSED

Putting data on NFS like ceph or gluster can fix this issue.

from pytorch-biggraph.

PBG's input and output is file-based: its config, entity counts, edge lists, checkpoints, ... must all be files on the filesystem, i.e, they must have a path which allows to read from and write through them using standards operating system interfaces. And that's really all PBG needs, so in particular it doesn't require that those files are backed by local disk: one can use network storage, as long as it can be accessed by the above means. In fact, for distributed training, the checkpoint directory must be shared across the machines because it's used to transfer data among the trainers. NFS storage works. I'm not familiar with HDFS but I expect it can be used with vanilla PBG if it can be mounted as a filesystem.

from pytorch-biggraph.

Two more things:

• HDF5 (the library we use to read edgelists and store checkpoints) may be able to use some lower-level interfaces to perform faster I/O when the files are on local disk. I'm using the conditional because I've no idea how HDF5 works internally. However it is still able to fall back to "regular" file I/O for remote files.
• During distributed training the checkpoints are only used to transfer files between one epoch and the next. Within one epoch the data is passed around between trainers using parameter servers and partition servers. Except if partition servers are disabled (which, unless you operate under strong memory pressure, they shouldn't), in which case checkpoints are also used within the epoch. Edgelists are however always read from file.

What this boils down to is: in single-machine mode, using local disk may be faster; in distributed mode it doesn't matter that much how the checkpoint directory is stored because anyways I/O to the filesystem is rarer.

from pytorch-biggraph.

@lerks Thanks, that's really helpful.

from pytorch-biggraph.