Comments (8)
@areeh PR #407 has been opened which should support your case. The PR allows to read a local numpy through the same address space.
It may have limitations but if your process is local then performance might be good for large numpy array (as there is no serialization overhead before hand).
The dict/tuple of the features has been added as well.
from io.
@yongtang Arrow supports reading numpy arrays to record batches. So I don't think it would be much effort to add this, but there would be a limit on dimensionality - for now at least. Hopefully that will change in the future.
from io.
@BryanCutler I think there are two issues, one is the conversion between Apache Arrow and Numpy in memory, another is to read data from npy
or npz
file format. I haven't find out a way to open npy
or npz
from Arrow library. However, it seems like the npy
or npz
file format are very much straightforward, a simple parser should be enough. I will look into it and see if I could come up with something quick.
from io.
Actually the conversion from Numpy to Arrow is zero copy, so it wouldn't consume any more memory, but you're right, Arrow doesn't support reading these files. If you're able to read them directly in the op that would be cool!
from io.
Performant solutions using tf.data and disk are quite painful right now, so this would be a welcome addition. I just want to mention that it's particularly useful if you support a case where data looks like:
{ 'feature1': ([...], dtype=np.float32), 'feature2': ([...], dtype=np.int16), }
etc. Somewhat similar to the example from the guide in the split between feature and label, if I'm understanding correctly
from io.
@yongtang The PR looks great, it supports everything I had in mind when I wrote the comment. Thank you
from io.
@yongtang can this be closed?
from io.
@kvignesh1420 Ah yes thanks for the reminder 👍
from io.
Related Issues (20)
- S3 read throughput slow down after hit prefix limit HOT 1
- extra not provided
- Tensorflow version-pinning should be reflected in setup.py. HOT 1
- Bug in Reading Compressed String Column in Parquet Dataset HOT 2
- Unable to use tensorflow_io.audio.resample on Mac M1
- tensorflow-io-gcs-filesystem==0.36.0 not available via pip HOT 1
- Missing files for Python 3.12 HOT 2
- Tensorflow 2.16 support HOT 1
- Grail bazel toolchain changed/missing
- S3 Express One Zone support
- The wheel in tensorflow-io 0.37.0 is not compatible with Python 3.12 HOT 1
- Please Make TFIO work on Mac M1, I need to read mp3 files!!!!
- Can not build `tensorflow-io 0.37.0` from source on Apple silicon.
- AttributeError: module '77ab628d7ad4acaa62f6bde524b9d631895821c9' has no attribute 'io_audio_decode_mp3'
- build failure due to unreachable freetype download URL HOT 1
- AttributeError: module '724c78992deaa2284cfda7ef0ea82354ad2614a8' has no attribute 'io_read_fastq'
- GCS gfile operations fail in TF 2.17.0rc0 and 2.18 nightly outside of GCP HOT 1
- `SIGSEGV` (Address boundary error) in `tf.io.gfile` with `TF_USE_MODULAR_FILESYSTEM=1`
- Python-requires in package metadata is out of sync with actual binary packages distributed
- TF2.16.2 with tensorflow-io 0.37.1 shows pure virtual method called, terminate called without an active exception causing exit code 134 when working with s3 filesystem HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from io.