Comments (1)
Distribute data to cluster is not added in PaddlePaddle now. You can read data directly from a HDFS file path by PyDataProvider2.
PaddlePaddle not handle how to get data file remotely, just pass the file path into a Python function. It is user's job to OPEN the file (or SQL connection string, or HDFS path), and get each
sample one by one from it.
It is welcome to contribute a script to distribute data to cluster. Or we may add it soon if this feature is very necessary.
from paddle.
Related Issues (20)
- 缺少torch.nn.utils.rnn.pad_sequence的API或者实现 HOT 2
- 在瑞芯微3568上已经部署好Fastdeploy。请问example中Yolov5的自己的onnx模型怎么使用呢? HOT 1
- paddlepaddle==2.6.0和2.6.1适配国产化与非国产化代码返回不一致问题 HOT 1
- Torch MultiHeadAttention To Paddle MultiHeadAttention Issue HOT 2
- softmax_with_cross_entropy API在软标签下label是否需要归一化 HOT 3
- 单机多卡问题 HOT 4
- Reported errors after running paddle.utils.run_check() HOT 1
- paddle.sparse.matmul两个参数都是sparse_csr_tensor时报错RuntimeError: (NotFound) The kernel `matmul_csr_csr` is not registered. HOT 2
- 这个教程代码有问题,https://www.paddlepaddle.org.cn/documentation/docs/zh/practices/nlp/transformer_in_English-to-Spanish.html
- Paddle的 Dataloader 遇到纯文本数据时报错 StopIteration HOT 1
- How to solve "license/cla Expected — Waiting for status to be reported" HOT 1
- 使用--use_trt出错 HOT 9
- 单机多卡问题 HOT 1
- templatedoc 机制清理 HOT 2
- 在华为Atlas310pro编译Paddle_npu版本,cmake有警告,make报错 HOT 3
- 【快乐开源】PIR模式下单测问题修复与适配
- 【开源之夏】动转静支持子图高阶微分
- phi::Device::SynchronizeStream传入的stream的raw_stream成员为空指针是否是正常的? HOT 24
- Does PP has a flowchart recognize product? HOT 2
- 如何将一个Tensor在内存中(不要保存到硬盘)序列化成可以通过网络传输的二进制格式? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from paddle.