Comments (4)
The current binary file fomat is a provisional solution. It was not a requirement so far. For the python interface I am also using numpy binary files (npy, npz), which is also not the final solution, as we have to handle more complex data structures than tensors. Already the hexagonal layout is not directly provided.
Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.
Changes in the data format should only be done in a major release, once we have a stable specification.
from pink.
from pink.
Yes - the hexagonal scheme is a bit of a pain with the current infrastructure. I have taken to ignoring it completely. This makes my life a little simpler.
I did the same almost from the start. That makes life easier during debugging, easier during further visualization steps and easier when the SOM BMU's are folded into further ML techniques (think random forests or auto-encoders).
Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.
HDF5 is vastly superior to FITS (better python/pandas integration, faster I/O, smaller files) although both are better than XML and JSON.
Having the option to also store the WCS parameters for the cutouts in the HDF5 as suggested by the paper that Tim links to would be nice.
from pink.
Thanks for your valuable notes and suggestions. I have added some more details of the memory layout to https://github.com/HITS-AIN/PINK/blob/master/FILE_FORMATS.md#cartesian-layout. Indeed the hexagonal layout is a bit fiddly. Maybe the python interface using numpy can be helpful.
The suggested paper about HDFITS is very promising. I will think about it, how we can use it not only for the input files, but also for the result files (SOM and mapping).
from pink.
Related Issues (20)
- circular euclidean-distance-shape versus circular-masked data HOT 2
- Rotations seem to be non-deterministic and have unexpected values HOT 19
- Mapping broken in CPU version HOT 7
- Significant slowdown in mapping HOT 1
- Update pybind11 to version 2.5 HOT 1
- Very large data dimension size when initialising training HOT 8
- Compilation Error with Cuda 11.2 HOT 2
- Unable to run Astro-Pink on Mac HOT 2
- Add wheels to new python version 3.9 and 3.10
- Migrate to new wheel build container
- introduce openssf badge for pink
- Update Jenkins pipeline HOT 1
- Update conan to version 2 HOT 1
- Move from setup.py to pyproject.toml HOT 1
- Replacement of the binary file format by HDF5
- Replace setuptools by scikit-build-core
- Mixing PyBind11 and plain python module 'pink' HOT 1
- CUDA accelerator is not used in Python bindings
- Replace deprecated FindCUDA module by CMake's first-class CUDA language support HOT 1
- [Feature request] Periodic boundary condition HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pink.