Comments (4)
BerndDoser
commented on August 25, 2024
The current binary file fomat is a provisional solution. It was not a requirement so far. For the python interface I am also using numpy binary files (npy, npz), which is also not the final solution, as we have to handle more complex data structures than tensors. Already the hexagonal layout is not directly provided.
Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.
Changes in the data format should only be done in a major release, once we have a stable specification.
from pink.
tjgalvin
commented on August 25, 2024
The current binary file fomat is a provisional solution. It was not a requirement so far. For the python interface I am also using numpy binary files (npy, npz), which is also not the final solution, as we have to handle more complex data structures than tensors. Already the hexagonal layout is not directly provided.
Yes - the hexagonal scheme is a bit of a pain with the current infrastructure. I have taken to ignoring it completely. This makes my life a little simpler.
I’m happy to continue to use the current set of binary files until something is better developed, no problem at all. Its just a case that in order to correctly reshape the files into the corresponding tensors (in my case the cartesian type SOMs) I need to know in what order floats (or other data types) were serialised into the binary file. Is it simply the case that each of the arrays follow the the order of the dimensions in the header between fields like the `som_layout` and `neuron_layout`?
For the SOM file type it is easy enough to just plot and see the neurons reconstructed with correct proportions and layouts, but will become a lot harder for other file types (i.e. transform file) where I can’t simply eyeball a few neurons.
Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.
FITS it the obvious one, but HDF5 would also be good. I’m very much interested in things like this: https://arxiv.org/abs/1505.06421 <https://arxiv.org/abs/1505.06421> How widespread the libraries are is another question.
… On 3 Dec 2019, at 4:45 pm, Bernd Doser ***@***.***> wrote:
The current binary file fomat is a provisional solution. It was not a requirement so far. For the python interface I am also using numpy binary files (npy, npz), which is also not the final solution, as we have to handle more complex data structures than tensors. Already the hexagonal layout is not directly provided.
Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.
Changes in the data format should only be done in a major release, once we have a stable specification.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#43?email_source=notifications&email_token=ACQOAJSDN7SNYQ3UDXMLJTDQWYMEFA5CNFSM4JTPOKV2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFYR64I#issuecomment-561061745>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACQOAJUE6756272BRKQO273QWYMEFANCNFSM4JTPOKVQ>.
from pink.
RafaelMostert
commented on August 25, 2024
Yes - the hexagonal scheme is a bit of a pain with the current infrastructure. I have taken to ignoring it completely. This makes my life a little simpler.
I did the same almost from the start. That makes life easier during debugging, easier during further visualization steps and easier when the SOM BMU's are folded into further ML techniques (think random forests or auto-encoders).
Therefore, I would like to open the discussion to find a suitable data format. I would recommend XML, JSON, or HDF5. Maybe there is already a astro data format we can simple use or extend for our purpose.
HDF5 is vastly superior to FITS (better python/pandas integration, faster I/O, smaller files) although both are better than XML and JSON.
Having the option to also store the WCS parameters for the cutouts in the HDF5 as suggested by the paper that Tim links to would be nice.
from pink.
BerndDoser
commented on August 25, 2024
Thanks for your valuable notes and suggestions. I have added some more details of the memory layout to https://github.com/HITS-AIN/PINK/blob/master/FILE_FORMATS.md#cartesian-layout. Indeed the hexagonal layout is a bit fiddly. Maybe the python interface using numpy can be helpful.
The suggested paper about HDFITS is very promising. I will think about it, how we can use it not only for the input files, but also for the result files (SOM and mapping).
from pink.
Related Issues (20)
- circular euclidean-distance-shape versus circular-masked data HOT 2
- Rotations seem to be non-deterministic and have unexpected values HOT 19
- Mapping broken in CPU version HOT 7
- Significant slowdown in mapping HOT 1
- Update pybind11 to version 2.5 HOT 1
- Very large data dimension size when initialising training HOT 8
- Compilation Error with Cuda 11.2 HOT 2
- Unable to run Astro-Pink on Mac HOT 2
- Add wheels to new python version 3.9 and 3.10
- Migrate to new wheel build container
- introduce openssf badge for pink
- Update Jenkins pipeline HOT 1
- Update conan to version 2 HOT 1
- Move from setup.py to pyproject.toml HOT 1
- Replacement of the binary file format by HDF5
- Replace setuptools by scikit-build-core
- Mixing PyBind11 and plain python module 'pink' HOT 1
- CUDA accelerator is not used in Python bindings
- Replace deprecated FindCUDA module by CMake's first-class CUDA language support HOT 1
- [Feature request] Periodic boundary condition HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pink.