hits-ain / pink Goto Github PK
View Code? Open in Web Editor NEWParallelized rotation and flipping INvariant Kohonen maps
License: GNU General Public License v3.0
pink's People
Stargazers
Watchers
Forkers
ericahopkins gitter-badger bernddoser nchamba tjgalvin sirraherydya rafaelmostert csheneka mfkiwl hujh08 andifepink's Issues
Illegal memory access using multiple GPU devices
PINK 2.0-rc1 exit with an illegal memory access using multiple GPU devices:
GPUassert: an illegal memory access was encountered /home/doserbd/git/pink/src/CudaLib/../CudaLib/generate_euclidean_distance_matrix_first_step_multi_gpu.h 110
Randomized the order of images for training
A flag will be added to randomize the order of images for the training. Random number generator will be initialized with a seed number for reproducibility.
Significant slowdown in mapping
The 6bf8cc7 revision introduced a significant (10x) slowdown to mapping. I have only tested this for CPU mapping, since that is all I'm expecting to have access to for the near future. Is there any way to recover some of the original speed?
Here are the two relevant log files from my testing:
Before: map_f5378ab.log
After: map_6ebe6a6.log
--train result Pink v0.23 differs from Pink v2.2
Starting out with a 2x2 cartesian SOM, all four neurons containing only 1s.
I train with a single image containing only 1s and --dist-func gaussian 1 1.
I try the following setups:
setup 1: v2.2 default settings,
setup 2: v2.2 where --neuron-dimension == --euclidean-distance-dimension
setup 3: v0.23
After training, I expect the neurons in setup 1 to be similar to the neurons setup 2, except for at the edges as is to be expected from #15 (as I found out again in #37 ). That behaviour is true.
I also expect that the neurons in setup 2 are fully equal to the neurons in setup 3. That is not true.
The run command for setup 2:
CUDA_VISIBLE_DEVICES=3 Pink --dist-func gaussian 0.1 1 --cuda-off --input-shuffle-off --euclidean-distance-type float --som-width 2 --som-height 2 --neuron-dimension 70 --euclidean-distance-dimension 70 --train /data/test/single_image.bin /data/single_neuron_somv2.bin
and setup 3:
CUDA_VISIBLE_DEVICES=3 Pink --dist-func gaussian 0.1 1 --cuda-off --som-width 2 --som-height 2 --train /data/single_imagev1.bin /data/single_neuron_somv1.bin
The runtime output for setup 2:
*************************************************************************
* *
* PPPPP II NN NN KK KK *
* PP PP II NNN NN KK KK *
* PPPPP II NN NN NN KKKK *
* PP II NN NNN KK KK *
* PP II NN NN KK KK *
* *
* Parallelized rotation and flipping INvariant Kohonen maps *
* *
* Version 2.2 *
* Git revision: c55d0d8 *
* *
* Bernd Doser <[email protected]> *
* Kai Polsterer <[email protected]> *
* *
* Distributed under the GNU GPLv3 License. *
* See accompanying file LICENSE or *
* copy at http://www.gnu.org/licenses/gpl-3.0.html. *
* *
*************************************************************************
Data file = /data/single_image.bin
Result file = /data/single_neuron_somv2.bin
Number of data entries = 1
Data dimension = 100 x 100
SOM dimension (width x height x depth) = 2x2x1
SOM size = 4
Number of iterations = 1
Neuron dimension = 70x70
Euclidean distance dimension = 70x70
Maximal number of progress information prints = 10
Intermediate storage of SOM = off
Layout = cartesian
Initialization type = zero
Interpolation type = bilinear
Seed = 1234
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 40
Use CUDA = 0
Distribution function for SOM update = gaussian
Sigma = 0.1
Damping factor = 1
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Random shuffle data input = 0
[======================================================================] 100 % 0.101 s
Write final SOM to /data/single_neuron_somv2.bin ... done.
Total time (hh:mm:ss): 00:00:00.200 (0 s)
Successfully finished. Have a nice day.
and for setup 3:
************************************************************************
* Parallel orientation Invariant Non-parametric Kohonen-map (PINK) *
* *
* Version 0.23 *
* *
* Kai Polsterer, Bernd Doser, HITS gGmbH *
************************************************************************
Image file = /data/single_imagev1.bin
Result file = /data/test/single_neuron_somv1.bin
Number of images = 1
Number of channels = 1
Image dimension = 100x100
SOM dimension (width x height x depth) = 2x2x1
SOM size = 4
Number of iterations = 1
Neuron dimension = 70x70
Progress = 0.1
Intermediate storage of SOM = off
Layout = quadratic
Initialization type = zero
Interpolation type = bilinear
Seed = 1234
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 40
Use CUDA = 0
Use multiple GPUs = 1
Distribution function for SOM update = gaussian
Sigma = 0.1
Damping factor = 1
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Starting C version of training.
Progress: 1 updates, 100 % (0 s)
Total time (hh:mm:ss): 00:00:00 (= 0s)
Successfully finished. Have a nice day.
I tried with and without CUDA, with many or just 1 threads, with and without flipping and rotations. Every time the absolute neuron weights from setup 2 differ from those in setup 3.
Wrong source of jupyter notebooks in docker containers
Instead of using the notebooks of the cloned git repository within the docker container, the notebooks will be copied from the repository outside the docker container, which will only work building within the jupyter directory and contains the wrong version of notebooks.
See https://github.com/HITS-AIN/PINK/blob/master/jupyter/dockerfile-gpu#L46
Arch flags for CUDA compiler for other GPUs in CMakeLists.txt
I got the following error trying to run Pink:
cudaErrorNoKernelImageForDevice: no kernel image is available for execution on the device
Through this issue answer I got the notion that this error is related to the CUDA compiler arch flag.
From here and here I gathered this flag is GPU dependent, so I changed the CMakeLists.txt to suit my Tesla K80 GPU accordingly:
find_package(CUDA)
if(CUDA_FOUND)
set(PINK_USE_CUDA true)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DPINK_USE_CUDA")
# Set flags for CUDA compiler
list(APPEND CUDA_NVCC_FLAGS "
--expt-extended-lambda
--expt-relaxed-constexpr
--default-stream per-thread
-arch=sm_30
-gencode=arch=compute_30,code=sm_30
-gencode=arch=compute_50,code=sm_50
-gencode=arch=compute_52,code=sm_52
-gencode=arch=compute_60,code=sm_60
-gencode=arch=compute_61,code=sm_61
-gencode=arch=compute_70,code=sm_70
-gencode=arch=compute_70,code=compute_70
")
endif()
Followed by
cmake3 -DCMAKE_C_COMPILER=/usr/local/bin/gcc -DCMAKE_CXX_COMPILER=/usr/local/bin/g++ -DCMAKE_INSTALL_PREFIX=/data/mostertrij/PINK_git2 .
and make install
This solved the CUDA error.
Now I am left with a 13833 Floating point exception. I am trying to find out was causes that, maybe I made a mistake in translating one of the binaries from v1 to v2.
Line in binary data file can wrongly interpreted as header
The first character of the data section in a binary file can be a '#'-character, which is interpreted as a header comment line, introduced at #1.
To fix this, I would suggest to end the header section with the line:
# END OF HEADER
It will be still backward compatible, as the line is not needed if no header is used.
Missing shared libraries in PyPI wheels
PyPI wheels show missing shared libraries:
ImportError: libPythonBindingLib.so: cannot open shared object file: No such file or directory
Can be reproduced by:
docker run -it python:3.6 bash
pip install astro-pink
python -c 'import pink'
The issue is likely caused by 75a6854
Maybe can be fixed by pybind/pybind11#1430 (comment)
--inter-store keep not working
Training with Cart3d through the binary file interface fails to calculate ED region
A rather small bug introduced with the fix to allow training with 3D (ZYX) datasets.
Using the binary file interface (i.e. the same way with pink v1) lets assume data of two channels and 100x100 pixels.
When I make the header and data take the shape (2, 100, 100) I get a PINK exception: Images must be quadratic. Program was aborted. The neuron sizes and euclidean distance regions that are printed in the pink output are calculated correctly.
When I make the header and data take the shape (100, 100, 2) Pink works, buts the euclidean distance region is 1x1 and the neuron dimension is 3x3.
Update binary file format
The metadata section of all binary files should be updated by following:
- Version of the file format
- Type of data (SOM, heatmap, training or mapping data)
- Type of single data entry (float, double, integer)
- Layout
- Dimensions
Please see also https://github.com/HITS-AIN/PINK/blob/master/FILE_FORMATS.md for details.
SOM initialization for hexagonal and multidimensional cartesian layout
Missing implementation at SelfOrganizingMapLib/SOM.cpp line 72 ff.
Related to #22
Add a readable header to binary files
The binary file formats described at https://github.com/HITS-AIN/PINK/wiki/Description-of-the-binary-file-formats will be expanded by a readable header for data information and comments. For separating the readable text from binary data the ASCII NUL character '\0' will be used. If a SOM is initialized with a SOM-file, the header will be copied to the resulting SOM-file.
Strange training / data iteration behavior
I am currently using the git build Version 2.4.1 Git revision: 9346dd6 and noticed some strange behavior that I am a little unclear on. I thought I would report it here to get some thoughts and keep track.
I have an image binary with about 21,437 images of the shape (2, 150, 150). Against this image binary file I am training four SOMs with Cartesian layouts of 8x8, 15x15, 25x25 and 35x35. Training is taking place across three rounds with the unitygaussian neighbourhood function and the circular euclidean distance statistic regions. The first two training rounds have 180 rotations and 5 iterations across the data, while the third training stage has 360 rotation and 10 iterations across the data (flipping enabled for all).
Across the four SOMs the first two training stages finish without issue. But for the SOMs with sizes 15x15 and 25x25 SOM the training finished early. The 15x15 SOM finished at 80% and the 25x25 at 40%.
Data file = EMU_WISE_E95E05_Aegean_Components_Complex_islandNorm_Log_Reprojected.bin
Result file = SOM_B3Circular_h15_w15_emu.bin
Number of data entries = 21437
Data dimension = 2 x 150 x 150
SOM dimension (width x height x depth) = 15x15x1
SOM size = 225
Number of iterations = 10
Neuron dimension = 213x213
Euclidean distance dimension = 150
Data type for euclidean distance calculation = uint8
Shape of euclidean distance region = circular
Maximal number of progress information prints = 10
Intermediate storage of SOM = keep
Layout = cartesian
Initialization type = file_init
SOM initialization file = SOM_B2Circular_h15_w15_emu.bin
Interpolation type = bilinear
Seed = 1234
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 4
Use CUDA = 1
Distribution function for SOM update = unitygaussian
Sigma = 0.7
Damping factor = 0.05
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Random shuffle data input = 1
CUDA Device Query...
There are 2 CUDA devices.
CUDA Device #0
Major revision number: 6
Minor revision number: 0
Name: Tesla P100-SXM2-16GB
Total global memory: 17071734784
Total shared memory per block: 49152
Total registers per block: 65536
Warp size: 32
Maximum memory pitch: 2147483647
Maximum threads per block: 1024
Maximum dimension 0 of block: 1024
Maximum dimension 1 of block: 1024
Maximum dimension 2 of block: 64
Maximum dimension 0 of grid: 2147483647
Maximum dimension 1 of grid: 65535
Maximum dimension 2 of grid: 65535
Clock rate: 1480500
Total constant memory: 65536
Texture alignment: 512
Concurrent copy and execution: Yes
Number of multiprocessors: 56
Kernel execution timeout: No
CUDA Device #1
Major revision number: 6
Minor revision number: 0
Name: Tesla P100-SXM2-16GB
Total global memory: 17071734784
Total shared memory per block: 49152
Total registers per block: 65536
Warp size: 32
Maximum memory pitch: 2147483647
Maximum threads per block: 1024
Maximum dimension 0 of block: 1024
Maximum dimension 1 of block: 1024
Maximum dimension 2 of block: 64
Maximum dimension 0 of grid: 2147483647
Maximum dimension 1 of grid: 65535
Maximum dimension 2 of grid: 65535
Clock rate: 1480500
Total constant memory: 65536
Texture alignment: 512
Concurrent copy and execution: Yes
Number of multiprocessors: 56
Kernel execution timeout: No
[=======> ] 10 % 2013.26 s
[==============> ] 20 % 4026.46 s
[=====================> ] 30 % 6041.26 s
[============================> ] 40 % 8057.02 s
[===================================> ] 50 % 10086.3 s
[==========================================> ] 60 % 12121.4 s
[=================================================> ] 70 % 14162.4 s
[========================================================> ] 80 % 16208.1 s
Write final SOM to SOM_B3Circular_h15_w15_emu.bin ... done.
Total time (hh:mm:ss): 04:30:09.776 (16209 s)
Successfully finished. Have a nice day.
Whats strange for both is that they neither were killed nor experienced a fatal error. They just ended and successfully wrote out the SOM binary file. Inspecting both of these SOMs there are a set of neurons that are just blank, appearing to be completely zero.
Whats even stranger is that the SOMs from the second training stage (those that seeded these failed SOMs) appears perfectly fine.
I don't think the process of slurm job were canceled because in the same slurm script the mapping stage (subsequent to this failed training round) completed without issue.
When building the image binary file I exclude all images that fail np.sum(~np.isfinite(img)) > 0, so all should have valid pixels.
Really unsure what to make of this. Oh, the 8x8 and 35x35 SOM seem to be fine with no issues. Any ideas or initial thoughts?
Mapping produces weird summed euclidean distances
It is harder for me to get the SOMs to converge during training.
Especially since my diagnostics don't work as the mapping procedure leads to unexpected high numbers.
Did something change in the calculation of the summed euclidean distance between image and neuron? (apart from pull request 15)
To see where I go wrong in the mapping process, I made a small test case below:
# Create sample SOM
# Map a list containing a single test image to a sample SOM
import numpy as np
import struct
def create_test_som(som_path, som_data):
# Show single neuron
print('som_data shape:', np.shape(som_data))
print(som_data)
# Write SOM binary file
assert som_path.endswith('.bin')
with open(som_path,'wb') as file:
# <file format version> 1 <data-type> <som layout> <neuron layout> <data>
file.write(struct.pack('i' * 11, 2, #<file format version>
1, #<file type>
0, #<data type>
0, #<som layout>
2, 2,2, #<som dimensionality, width and height>
0, #<neuron layout>
2, np.shape(som_data)[1], np.shape(som_data)[2])) #<neuron dimensionality and width>
file.write(struct.pack('f' * som_data.size, *som_data.flatten()))
def create_test_data(data_path, image_data):
# Show single image (identical to the single SOM neuron)
print('image_data shape:', np.shape(image_data))
print(image_data)
# Write SOM binary file
assert data_path.endswith('.bin')
with open(data_path,'wb') as file:
# <file format version> 0 <data-type> <number of entries> <data layout> <data>
file.write(struct.pack('i' * 8, 2, 0, 0, 1, 0, 2, np.shape(image_data)[1], np.shape(image_data)[2]))
file.write(struct.pack('f' * image_data.size, *image_data.flatten()))
def map_test_data_to_test_som(gpu_id, som_width, som_height, data_path, som_path, map_path):
print(f"CUDA_VISIBLE_DEVICES={gpu_id} Pink -p 1 --som-width {som_width} --som-height {som_height}"
f" --map {data_path} {map_path} {som_path}")
def load_som_mapping(map_path, verbose=True):
# Create list of indexes to retrieve coordinates of the cut-outs
cut_out_index = []
with open(map_path, 'rb') as file:
# <file format version> 2 <data-type> <number of entries> <som layout> <data>
version, file_type, data_type, numberOfImages, som_layout, som_dimensionality = struct.unpack(
'i' * 6, file.read(4 * 6))
som_dimensions = struct.unpack('i' * som_dimensionality, file.read(4 *
som_dimensionality))
if verbose:
print('version:', version)
print('file_type:', file_type)
print('data_type:', data_type)
print('numberOfImages:', numberOfImages)
print('som_layout:', som_layout)
print('som dimensionality:', som_dimensionality)
print('som dimensions:', som_dimensions)
som_width = som_dimensions[0]
som_height = som_dimensions[1] if som_dimensionality > 1 else 1
som_depth = som_dimensions[2] if som_dimensionality > 2 else 1
maps = []
assert file_type == 2 # 2 equals mapping
map_size = som_width * som_height * som_depth
data_map = np.ones((numberOfImages, map_size))
for i in range(numberOfImages):
for t in range(map_size):
try:
data_map[i,t] = struct.unpack_from("f", file.read(4))[0]
if t == 0:
cut_out_index.append(i) # add index
except:
pass
data_map = data_map[:len(cut_out_index)]
print('\nMap result:')
print(data_map, numberOfImages, som_width, som_height, som_depth,'\n')
return data_map, numberOfImages, som_width, som_height, som_depth
gpu_id = 1
som_width=2
som_height=2
image_width=3
neuron_width = int(np.ceil(image_width/np.sqrt(2)*2))
som_path = '/data/single_neuron_som.bin'
data_path = '/data/single_image.bin'
map_path = '/data/map_single_image_to_single_neuron.bin'
create_test_som(som_path,
np.array([[np.ones([neuron_width,neuron_width])
for j in range(som_width)]
for i in range(som_height)]))
create_test_data(data_path, np.array([np.zeros([image_width,image_width])]))
map_test_data_to_test_som(gpu_id, som_width,som_height,data_path, som_path, map_path)
This results in the following output:
som_data shape: (2, 2, 5, 5)
[[[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]
[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]]
[[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]
[[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]
[1. 1. 1. 1. 1.]]]]
image_data shape: (1, 3, 3)
[[[0. 0. 0.]
[0. 0. 0.]
[0. 0. 0.]]]
CUDA_VISIBLE_DEVICES=1 Pink -p 1 --som-width 2 --som-height 2 --map /data/single_image.bin /data/map_single_image_to_single_neuron.bin /data/single_neuron_som.bin
Executing the last line gives the following output, which seem fine (dimensions seem right):
Data file = /data/single_image.bin
Result file = /data/map_single_image_to_single_neuron.bin
SOM file = /data/single_neuron_som.bin
Number of data entries = 1
Data dimension = 3 x 3
SOM dimension (width x height x depth) = 2x2x1
SOM size = 4
Number of iterations = 1
Neuron dimension = 5x5
Euclidean distance dimension = 2x2
Number of progress information prints = 1
Intermediate storage of SOM = off
Layout = cartesian
Initialization type = file_init
SOM initialization file = /data/mostertrij/pink-basics/output_LoTSS_DR1/single_neuron_som.bin
Interpolation type = bilinear
Seed = 1234
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 40
Use CUDA = 1
Distribution function for SOM update = gaussian
Sigma = 1.1
Damping factor = 0.2
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Store best rotation and flipping parameters = 0
[======================================================================] 100 % 0 s
Total time (hh:mm:ss): 00:00:02.087 (2 s)
Successfully finished. Have a nice day.
Now loading the som using load_som_mapping(map_path, verbose=True) results in the following (wrong) result:
version: 2
file_type: 2
data_type: 0
numberOfImages: 1
som_layout: 0
som dimensionality: 2
som dimensions: (2, 2)
Map result:
[[0. 0. 0. 0.]] 1 2 2 1
So the euclidean distance reported is 0,0,0,0 while it should be 4,4,4,4.
Repeating this proces for neurons and a single image all randomly sampled between 0 and 1 gives rise to the way to high numbers:
create_test_som(som_path2, np.array([[np.random.uniform(0,1,[neuron_width,neuron_width])
for j in range(som_width)]
for i in range(som_height)]))
create_test_data(data_path2, np.array([np.random.uniform(0,1,[image_width,image_width])]))
map_test_data_to_test_som(gpu_id, som_width,som_height, data_path2, som_path2, map_path2)
Produces:
som_data shape: (2, 2, 5, 5)
[[[[0.99971204 0.9649601 0.13961416 0.5277925 0.69232026]
[0.30195558 0.95923107 0.33910395 0.79623785 0.41668984]
[0.2981384 0.37406349 0.1064496 0.81195063 0.89717293]
[0.57093443 0.45074527 0.9573843 0.7156923 0.19820787]
[0.19550703 0.21997696 0.85291868 0.37672202 0.5505964 ]]
[[0.15731968 0.17053242 0.71608575 0.56746897 0.66786446]
[0.58221214 0.72226106 0.74181827 0.32114466 0.92622553]
[0.17948228 0.9350424 0.95751021 0.80663197 0.9809329 ]
[0.7201585 0.76561495 0.00955184 0.50519959 0.96674996]
[0.25371928 0.71401906 0.49218354 0.11929986 0.95515874]]]
[[[0.89801856 0.02230665 0.30972996 0.4629857 0.89403677]
[0.86597969 0.93576617 0.1125303 0.68526227 0.35861138]
[0.69389897 0.07442639 0.1937746 0.60510161 0.52935543]
[0.77156129 0.93942363 0.71076085 0.41047543 0.64435329]
[0.89491456 0.79810935 0.89999227 0.808086 0.59111661]]
[[0.2528187 0.42403009 0.36152541 0.84878211 0.43277483]
[0.85728984 0.16706833 0.81121466 0.94590693 0.52366488]
[0.384645 0.04069272 0.3186065 0.68401763 0.58106253]
[0.97269966 0.80036697 0.55934164 0.71930493 0.27985202]
[0.5184996 0.79485326 0.78941677 0.31408703 0.37666354]]]]
image_data shape: (1, 3, 3)
[[[0.28589343 0.06328625 0.6278434 ]
[0.82636295 0.64509669 0.04756091]
[0.28352401 0.23569587 0.96175407]]]
CUDA_VISIBLE_DEVICES=1 Pink -p 1 --som-width 2 --som-height 2 --map /data/single_image2.bin /data/map_single_image_to_single_neuron2.bin /data/single_neuron_som2.bin
version: 2
file_type: 2
data_type: 0
numberOfImages: 1
som_layout: 0
som dimensionality: 2
som dimensions: (2, 2)
Mapping result is:
Map result:
[[ 2001. 11979. 20095. 41844.]] 1 2 2 1
While it should be four times a value between 0 and 4.
It would be great if you could spot what I do wrong (in creating the binaries, at runtime or while loading the mapping).
PyPI astro-pink will not be built by sdist
Python 3.6, 3.7, 3.8 is available as wheel package as manylinux2010_x86_64 architecture. Other architectures should be built using the source distribution. However, the extension C++ source code is not available in the sdist package.
Wrong number of iterations in ProgressBar
Minor bug encountered after updating to the last git version.
I get the following error produced by the progressbar:
PINK exception: ProgressBar: total must be larger than 0
For any value of -p, 0, 1, 10, 100. And 4 data entries.
PINK/src/UtilitiesLib/ProgressBar.h
Lines 20 to 21 in c139c19
The value of total is invoked here:
Line 59 in 239c712
Probably InputData.numIter is unintialized or zero.
Removing total(total < 1 ? throw pink::exception("ProgressBar: total must be larger than 0") : total), , rebuilding PINK and setting -p 1 avoids the issue and allows me to see the Successfully finished. Have a nice day. message. The progressbar is not present though and only the header of the mapping file gets written...
Originally posted by @RafaelMostert in #30 (comment)
Ordering in mapping binary differs from ordering in data binary
On a per image basis, the mapping to the SOM works fine.
But the ordering returned by the mapping binary seems different from the ordering in the data binary.
As if there is some shuffle taking place of the images or the mapping results, at some point in the PINK code.
Here is the test:
This time I have a 2x2 (=4 neurons) test SOM.
A neuron consisting of only 1s in the top-left, a neuron with only 2s in the top-right,
a neuron with only 3s in the bottom-left and a neuron with only 4s in the bottom-right.
The data binary consists of 4 images, all ones, except the first image which contains all 100s.
But in the mapping binary it returns as second item in the list..
Here is the code:
# Create sample SOM
# Map a the same image to this sample SOM
# see if the result is 0 (or very close to zero indeed)
def create_test_som(som_path, som_data):
"""A test SOM, cartesian layout, neuron dimensionality = 2"""
# Show single neuron
print('som_data shape:', np.shape(som_data))
print(som_data)
# Write SOM binary file
assert som_path.endswith('.bin')
with open(som_path,'wb') as file:
# <file format version> 1 <data-type> <som layout> <neuron layout> <data>
file.write(struct.pack('i' * 11, 2, #<file format version>
1, #<file type>
0, #<data type>
0, #<som layout>
2, 2,2, #<som dimensionality, width and height>
0, #<neuron layout>
2, np.shape(som_data)[1], np.shape(som_data)[2])) #<neuron dimensionality and width>
file.write(struct.pack('f' * som_data.size, *som_data.flatten()))
def create_test_data(data_path, image_data):
"""A set of test images, cartesian layout, dimensionality = 2"""
# Show single image (identical to the single SOM neuron)
print('image_data shape:', np.shape(image_data))
print(image_data)
# Write SOM binary file
assert data_path.endswith('.bin')
with open(data_path,'wb') as file:
# <file format version> 0 <data-type> <number of entries> <data layout> <data>
file.write(struct.pack('i' * 8, 2, 0, 0, len(image_data), 0, 2, np.shape(image_data)[1], np.shape(image_data)[2]))
file.write(struct.pack('f' * image_data.size, *image_data.flatten()))
def map_test_data_to_test_som(gpu_id, som_width, som_height, data_path, som_path, map_path):
print(f"CUDA_VISIBLE_DEVICES={gpu_id} Pink -p 1 --euclidean-distance-type float --som-width {som_width} --som-height {som_height}"
f" --map {data_path} {map_path} {som_path}")
def load_som_mapping(map_path, verbose=True):
# Create list of indexes to retrieve coordinates of the cut-outs
cut_out_index = []
with open(map_path, 'rb') as file:
# <file format version> 2 <data-type> <number of entries> <som layout> <data>
version, file_type, data_type, numberOfImages, som_layout, som_dimensionality = struct.unpack(
'i' * 6, file.read(4 * 6))
som_dimensions = struct.unpack('i' * som_dimensionality, file.read(4 *
som_dimensionality))
if verbose:
print('version:', version)
print('file_type:', file_type)
print('data_type:', data_type)
print('numberOfImages:', numberOfImages)
print('som_layout:', som_layout)
print('som dimensionality:', som_dimensionality)
print('som dimensions:', som_dimensions)
som_width = som_dimensions[0]
som_height = som_dimensions[1] if som_dimensionality > 1 else 1
som_depth = som_dimensions[2] if som_dimensionality > 2 else 1
maps = []
assert file_type == 2 # 2 equals mapping
map_size = som_width * som_height * som_depth
data_map = np.ones((numberOfImages, map_size))
for i in range(numberOfImages):
for t in range(map_size):
data_map[i,t] = struct.unpack_from("f", file.read(4))[0]
print('\nMap result:')
print(data_map, numberOfImages, som_width, som_height, som_depth,'\n')
return data_map, numberOfImages, som_width, som_height, som_depth
def show_mapping(data_map):
for im_map in data_map:
print(np.argsort(np.argsort(im_map)))#.reshape([som_width,som_width]))
gpu_id = 1
som_width=2
som_height=2
image_width=2
np.random.seed=42
som_width=2
neuron_width = int(np.ceil(image_width/np.sqrt(2)*2))
som_path = '/data/test_som.bin'
data_path = '/data/test_images.bin'
map_path = '/data/map_images_to_test_som.bin'
# creating the neurons
t = 0
neurons = [[[] for j in range(som_width)] for i in range(som_width)]
for j in range(som_width):
for i in range(som_width):
t +=1
neurons[j][i] = t*np.ones([neuron_width,neuron_width])
neurons = np.array(neurons)
image = np.array([100*np.ones([image_width,image_width]),np.ones([image_width,image_width]),
np.ones([image_width,image_width]),np.ones([image_width,image_width])])
create_test_som(som_path, neurons)
create_test_data(data_path, image)
map_test_data_to_test_som(gpu_id, som_width,som_height,data_path, som_path, map_path)
Creating this input:
som_data shape: (2, 2, 3, 3)
[[[[1. 1. 1.]
[1. 1. 1.]
[1. 1. 1.]]
[[2. 2. 2.]
[2. 2. 2.]
[2. 2. 2.]]]
[[[3. 3. 3.]
[3. 3. 3.]
[3. 3. 3.]]
[[4. 4. 4.]
[4. 4. 4.]
[4. 4. 4.]]]]
image_data shape: (4, 2, 2)
[[[100. 100.]
[100. 100.]]
[[ 1. 1.]
[ 1. 1.]]
[[ 1. 1.]
[ 1. 1.]]
[[ 1. 1.]
[ 1. 1.]]]
Running Pink:
Number of data entries = 4
Data dimension = 2 x 2
SOM dimension (width x height x depth) = 2x2x1
SOM size = 4
Number of iterations = 1
Neuron dimension = 3x3
Euclidean distance dimension = 1x1
Number of progress information prints = 1
Intermediate storage of SOM = off
Layout = cartesian
Initialization type = file_init
SOM initialization file = /data/test_som.bin
Interpolation type = bilinear
Seed = 1234
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 40
Use CUDA = 1
Distribution function for SOM update = gaussian
Sigma = 1.1
Damping factor = 0.2
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Store best rotation and flipping parameters = 0
Loading the mapping:
data_map, numberOfImages, som_width, som_height, som_depth = load_som_mapping(map_path, verbose=True);
show_mapping(data_map)
Result:
version: 2
file_type: 2
data_type: 0
numberOfImages: 4
som_layout: 0
som dimensionality: 2
som dimensions: (2, 2)
Map result:
[[0.000e+00 1.000e+00 4.000e+00 9.000e+00]
[9.801e+03 9.604e+03 9.409e+03 9.216e+03]
[0.000e+00 1.000e+00 4.000e+00 9.000e+00]
[0.000e+00 1.000e+00 4.000e+00 9.000e+00]] 4 2 2 1
[0 1 2 3]
[3 2 1 0]
[0 1 2 3]
[0 1 2 3]
While I expect:
[3 2 1 0]
[0 1 2 3]
[0 1 2 3]
[0 1 2 3]
I tried mapping different sets of images and it is hard to find a pattern, thus I suspect a random shuffle somewhere...
Maybe it is due to the shuffle in PINK/src/SelfOrganizingMapLib/DataIterator.h ?
Dynamic interface to python bindings
An additional abstraction layer will be added between the python binding (pybind11) and the static C++ templated objects to allow user-friendly dynamic settings.
Favored interface:
som = pink.SOM(np_som, som_layout="cartesian-2d", neuron_layout="cartesian-2d")
trainer = pink.Trainer(som, use_gpu=True)
file INSTALL cannot find PINK/lib/libCudaLib.so
This time I tried installing with gcc 7.3:
-DCMAKE_INSTALL_PREFIX=/data/mostertrij/PINK .
-- The C compiler identification is GNU 7.3.0
-- The CXX compiler identification is GNU 7.3.0
-- Check for working C compiler: /usr/local/bin/gcc
-- Check for working C compiler: /usr/local/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/local/bin/g++
-- Check for working CXX compiler: /usr/local/bin/g++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Pink version 2.1
-- CMake build type: Release
-- Found OpenMP_C: -fopenmp (found version "4.5")
-- Found OpenMP_CXX: -fopenmp (found version "4.5")
-- Found OpenMP: TRUE (found version "4.5")
-- Found Git: /usr/bin/git (found version "1.8.3.1")
-- Git revision: 4afa711
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "10.1")
-- Could NOT find pybind11
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY)
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Configuring done
-- Generating done
-- Build files have been written to: /data/mostertrij/PINK
Make install fails on not finding PINK/lib/libCudaLib.so:
Scanning dependencies of target UtilitiesLib
[ 8%] Building CXX object src/UtilitiesLib/CMakeFiles/UtilitiesLib.dir/CheckArrays.cpp.o
[ 16%] Building CXX object src/UtilitiesLib/CMakeFiles/UtilitiesLib.dir/get_file_header.cpp.o
[ 25%] Building CXX object src/UtilitiesLib/CMakeFiles/UtilitiesLib.dir/InputData.cpp.o
[ 33%] Building CXX object src/UtilitiesLib/CMakeFiles/UtilitiesLib.dir/Point.cpp.o
[ 41%] Linking CXX static library ../../lib/libUtilitiesLib.a
[ 41%] Built target UtilitiesLib
[ 50%] Building NVCC (Device) object src/CudaLib/CMakeFiles/CudaLib.dir/CudaLib_generated_main_gpu.cu.o
[ 58%] Building NVCC (Device) object src/CudaLib/CMakeFiles/CudaLib.dir/CudaLib_generated_CudaLib.cu.o
[ 66%] Building NVCC (Device) object src/CudaLib/CMakeFiles/CudaLib.dir/CudaLib_generated_dot_dp4a.cu.o
Scanning dependencies of target CudaLib
[ 75%] Linking CXX shared library ../../lib/libCudaLib.so
[ 75%] Built target CudaLib
Scanning dependencies of target Pink
[ 83%] Building CXX object src/Pink/CMakeFiles/Pink.dir/main.cpp.o
[ 91%] Building CXX object src/Pink/CMakeFiles/Pink.dir/main_cpu.cpp.o
[100%] Linking CXX executable ../../bin/Pink
[100%] Built target Pink
Install the project...
-- Install configuration: "Release"
-- Set runtime path of "/data/mostertrij/PINK/bin/Pink" to "/data/mostertrij/PINK/lib"
CMake Error at src/CudaLib/cmake_install.cmake:47 (file):
file INSTALL cannot find "/data/mostertrij/PINK/lib/libCudaLib.so".
Call Stack (most recent call first):
src/cmake_install.cmake:44 (include)
cmake_install.cmake:46 (include)
make: *** [install] Error 1The error points to cmake_install.cmake:44 and46
if(NOT CMAKE_INSTALL_LOCAL_ONLY)
# Include the install script for each subdirectory.
include("/data/mostertrij/PINK/src/cmake_install.cmake")
So I thought it might be circumvented by making locally:
-- Pink version 2.1
-- CMake build type: Release
-- Found OpenMP_C: -fopenmp
-- Found OpenMP_CXX: -fopenmp
-- Found OpenMP: TRUE
-- Git revision: 4afa711
-- Could NOT find pybind11
-- Could NOT find GTest (missing: GTEST_LIBRARY GTEST_INCLUDE_DIR GTEST_MAIN_LIBRARY)
-- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE)
-- Configuring done
-- Generating done
-- Build files have been written to: /data/mostertrij/PINK
[ 41%] Built target UtilitiesLib
[ 50%] Linking CXX shared library ../../lib/libCudaLib.so
[ 75%] Built target CudaLib
[ 83%] Linking CXX executable ../../bin/Pink
[100%] Built target Pink
Installing only the local directory...
-- Install configuration: "Release"Which seems to work, but when attempting to run PINK it turns out not to work:
bin/Pink
bin/Pink: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by bin/Pink)
bin/Pink: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by bin/Pink)
bin/Pink: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by bin/Pink)
bin/Pink: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /data/mostertrij/PINK/lib/libCudaLib.so)
bin/Pink: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /data/mostertrij/PINK/lib/libCudaLib.so)
bin/Pink: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /data/mostertrij/PINK/lib/libCudaLib.so)
bin/Pink: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /data/mostertrij/PINK/lib/libCudaLib.so)
C++ exceptions will not be forwarded to python
Using the option euclidean_distance_type for the GPU trainer should throw an exception, but is running further and always updates the first neuron.
As described in pybind11 (https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html) the C++ std::exception should be automatically converted into a Pyhton Exception.
memory error for converting binaries to version 2
The python script to convert binaries from version one to version two can produce a memory allocation error for large binaries. The current implementation will try to read into memory the entire contents of a file. For 200GB+ files this is not feasible.
I'd suggest iterating over the read/write similar to
size = nb_channels * width * height
for _ in range(nb_images):
output.write(input.read(size*4))
It might lose time on the iteration but it won't crash anymore
Add option to store the the best matching rotation and flipping parameters of mapping
Add option to mapping procedure:
--store-rot-flip
which stores an additional file with best matching rotation angle and if the image was flipped.
(1)(integer) number_of_images
(1)(integer) SOM_width
(1)(integer) SOM_height
(1)(integer) SOM_depth
(number_of_images, SOM_width, SOM_height, SOM_depth)(bool) flipped
(number_of_images, SOM_width, SOM_height, SOM_depth)(float) rotation_angle
Compilation error using clang 6.0
Beside many compiler warnings the error is:
/work/conan-clang-6/.conan/data/pybind11/2.2.2/conan/stable/package/5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9/include/pybind11/cast.h:896:73: error: use 'template' keyword to treat 'type_caster' as a dependent template name
operator std::reference_wrapper< type> () { return ((subcaster).detail::type_caster< intrinsic_t< type> > ::operator subcaster_cast_op_type &()); }
^
template
It looks like an inconsistency between clang and pybind11.
--train updates neurons inconsistently
Starting out with a 2x2 cartesian SOM, all four neurons containing only 1s.
I train with a single image containing only 1s and --dist-func gaussian 1 1.
After training, I expect the neurons to be altered, 1 is the winner, 2 are equally far away from the winner and the last node is furthest away. That behaviour is true.
I also expect that the weights within each neuron to be all the same, but that does not happen.
The run command:
CUDA_VISIBLE_DEVICES=3 Pink --dist-func gaussian 1 1 --euclidean-distance-type float --som-width 2 --som-height 2 --train /data/single_image.bin /data/single_neuron_som.bin
The run output:
*************************************************************************
* *
* PPPPP II NN NN KK KK *
* PP PP II NNN NN KK KK *
* PPPPP II NN NN NN KKKK *
* PP II NN NNN KK KK *
* PP II NN NN KK KK *
* *
* Parallelized rotation and flipping INvariant Kohonen maps *
* *
* Version 2.2 *
* Git revision: c55d0d8 *
* *
* Bernd Doser <[email protected]> *
* Kai Polsterer <[email protected]> *
* *
* Distributed under the GNU GPLv3 License. *
* See accompanying file LICENSE or *
* copy at http://www.gnu.org/licenses/gpl-3.0.html. *
* *
*************************************************************************
Data file = /data/single_image.bin
Result file = /data/single_neuron_som.bin
Number of data entries = 1
Data dimension = 4 x 4
SOM dimension (width x height x depth) = 2x2x1
SOM size = 4
Number of iterations = 1
Neuron dimension = 6x6
Euclidean distance dimension = 2x2
Maximal number of progress information prints = 10
Intermediate storage of SOM = off
Layout = cartesian
Initialization type = zero
Interpolation type = bilinear
Seed = 1234
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 40
Use CUDA = 1
Distribution function for SOM update = gaussian
Sigma = 1
Damping factor = 1
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Random shuffle data input = 1
[======================================================================] 100 % 0 s
Write final SOM to /data/single_neuron_som.bin ... done.
Total time (hh:mm:ss): 00:00:00.952 (0 s)
Successfully finished. Have a nice day.
The retrieved neuron weights (rounded to one decimal):
[[0. 0. 0. 0. 0. 0. 0. 0.6 0.6 0.6 0.6 0. 0. 0.6 0.6 0.6 0.6 0.
0. 0.6 0.6 0.6 0.6 0. 0. 0.6 0.6 0.6 0.6 0. 0. 0. 0. 0. 0. 0. ]
[0. 0. 0. 0. 0. 0. 0. 0.2 0.4 0.2 0.2 0. 0. 0.2 0.4 0.4 0.4 0.
0. 0.4 0.4 0.4 0.2 0. 0. 0.2 0.2 0.4 0.2 0. 0. 0. 0. 0. 0. 0. ]
[0. 0. 0. 0. 0. 0. 0. 0.2 0.4 0.2 0.2 0. 0. 0.2 0.4 0.4 0.4 0.
0. 0.4 0.4 0.4 0.2 0. 0. 0.2 0.2 0.4 0.2 0. 0. 0. 0. 0. 0. 0. ]
[0. 0. 0. 0. 0. 0. 0. 0.1 0.3 0.1 0.1 0. 0. 0.1 0.3 0.3 0.3 0.
0. 0.3 0.3 0.3 0.1 0. 0. 0.1 0.1 0.3 0.1 0. 0. 0. 0. 0. 0. 0. ]]
--numrot 1 fails if neuron dimension equals Euclidean distance dimension
In pink Version 2.2 Git revision: 1113763,
if Neuron dimension equals Euclidean distance dimension,
running with --numrot 1 flag fails with error message:
PINK exception: euclidean distance dimension must be equal or smaller than neuron dimension.
Jupyter notebook within a docker container
Having the new Python interface (#7), it would be a very convenient way to deploy a ready-to-go docker container, where PINK is fully installed and can be used web-based via jupyter.
Et voila - here we are! 😆
After having docker installed, you can try it by typing:
docker run -it -p 8888:8888 astroinformatix/pink-jupyter
and open the link given in the docker log. You will find a tutorial "Training of a self-organizing map" waiting for you.
Empty SOM using no rotation on GPU
The trained SOM is empty using --numrot 1 and run the GPU version. The CPU path is correct.
CMake fails: Could NOT find OpenMP_CXX
I just tried to install from source with
cmake3 -DCMAKE_INSTALL_PREFIX=/data/mostertrij/PINK .
Eventhough OpenMP is found, it fails:
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Pink version 2.0
-- CMake build type: Release
-- Found OpenMP_C: -fopenmp (found version "3.1")
CMake Error at /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES)
Call Stack (most recent call first):
/usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake3/Modules/FindOpenMP.cmake:473 (find_package_handle_standard_args)
CMakeLists.txt:19 (find_package)
-- Configuring incomplete, errors occurred!
See also "/data/mostertrij/PINK/CMakeFiles/CMakeOutput.log".
See also "/data/mostertrij/PINK/CMakeFiles/CMakeError.log".
CMakeError.log
CMakeOutput.log
I tried removing the '-std=c++14' flag from the CMakeLists.txt file as the CMakeError.log states that this command option is unrecognized.
Rerunning without this option still fails, although this time, only a CMakeOutput.log is created:
CMakeOutput.log
Any clues on how to resolve this?
Periodic boundary condition option seems deprecated
The periodic boundary condition flag --pbc gets displayed both in the help message and at runtime Use periodic boundary conditions = 1.
Unlike in Pink version 0.23, running with or without the pbc flag makes no difference. The flag does not seem to propagate in the code beyond src/UtilitiesLib/InputData.h, is it deprecated?
Map initialisation with random noise and prefered direction
I am using an image binary file with about 200k images, with each image having a dimension of (150,150,2).
It seems that when using --init random_with_preferred_direction to create the initial map state there is a segmentation fault. Switching this to just random seems to work fine.
I am guessing that it has got to do with the third dimension of each image. I have no dug through the code to isolate much more than this at the moment.
Replace outdated InputData by modern data structure
The InputData can only be constructed by the argument list, which makes it hard to use it for unit tests. Using a modern data structure opens the way to transmit a settings object instead of single settings, which would be much more maintainable.
Estimated time: 48 h
Improve command arguments of SOM layout
The SOM layout will be defined with --layout, --som-width, --som-height, and --som-depth. To be prepared for future more complex layouts, e.g. growing self-organizing maps, the definition of the sizes should be part of the layout option, so that the new definition would be:
--layout cartesian 5 5 for a 2-dimensional quadratic map
--layout cartesian 10 10 10 for a 3-dimensional quadratic map
--layout hexagonal 11 11 for a hexagonal map
Rotations seem to be non-deterministic and have unexpected values
(Using PINK Version 2.5 Git revision: feac562)
Using real data, mapping about 2000 sources to a 10x10 SOM I plotted the stored flips and rotations.
As expected, the flips are mostly random:
The rotations are unexpectedly peaked at 4 different places, something that Tim also observed earlier.
Apart from this global effect, the reported rotations did not seem to have any relation to the individual inputted images.
Thus I tested a small SOM with mock data:
I created a 2x2 SOM with neurons that are all zeros except for the first neuron, which has a stripe pointing towards 12-oclock.
I created five images containing all zeros.
Except that the last four images have a stripe pointing towards 12, 3, 6 and 9 o-clock respectively.
I then map these five images to the 2x2 SOM and store the flips and rotations.
See attached ipynb:
test_rotations_pink.zip
Everytime I rerun the mapping command (without changing the input),
Pink --cuda-off --store-rot-flip test_rotations.bin --euclidean-distance-type float --euclidean-distance-shape circular --som-width 2 --som-height 2 --map test_data.bin test_map.bin test_som.bin
I get reasonable and consistent values for the mapping distances from each image to each neuron.
However, the rotation angles and flip-bools are different everytime and they are never what I expect them to be.
The non-deterministic behaviour for the flips and rotations happen without cuda.
With cuda, I only get 0s for the rotation-angles in this test-case.
--num-iter option ignored
I am using the latest release of pink v2.4 git revision f385715.
It seems that whatever value I pass to --num-iter is ignored. The progress bar will include it in its computation but not when it comes to iterating over data items.
Data file = TEST.bin
Result file = SOM_B2.bin
Number of data entries = 24853
Data dimension = 100 x 100
SOM dimension (width x height x depth) = 10x5x1
SOM size = 50
Number of iterations = 3
Neuron dimension = 142x142
Euclidean distance dimension = 70x70
Data type for euclidean distance calculation = uint8
Maximal number of progress information prints = 10
Intermediate storage of SOM = off
Layout = cartesian
Initialization type = file_init
SOM initialization file = SOM_B1.bin
Interpolation type = bilinear
Seed = 1234
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 1
Use CUDA = 1
Distribution function for SOM update = gaussian
Sigma = 1
Damping factor = 0.05
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Random shuffle data input = 0
CUDA Device Query...
There are 1 CUDA devices.
CUDA Device #0
Major revision number: 6
Minor revision number: 0
Name: Tesla P100-SXM2-16GB
Total global memory: 17071734784
Total shared memory per block: 49152
Total registers per block: 65536
Warp size: 32
Maximum memory pitch: 2147483647
Maximum threads per block: 1024
Maximum dimension 0 of block: 1024
Maximum dimension 1 of block: 1024
Maximum dimension 2 of block: 64
Maximum dimension 0 of grid: 2147483647
Maximum dimension 1 of grid: 65535
Maximum dimension 2 of grid: 65535
Clock rate: 1480500
Total constant memory: 65536
Texture alignment: 512
Concurrent copy and execution: Yes
Number of multiprocessors: 56
Kernel execution timeout: No
[=======> ] 10 % 13.222 s
[=============> ] 19 % 26.454 s
[====================> ] 29 % 39.68 s
Write final SOM to SOM_B2.bin ... done.
Total time (hh:mm:ss): 00:00:44.277
Refactoring quadratic layout to cartesian layout
Typo in the help message:
Running Pink -h gives:
--layout, -l <string> Layout of SOM (quadratic = default, hexagonal).
But the keyword 'quadratic' is not accepted anymore and has apparently been replaced by 'cartesian'.
Rotation transform binary file incorrectly written?
I am having a little weird behavior that I can't quiet nail down.
I'd like to use the transformation information outputted by pink with the --store-rot-flip option.
As I understand from pink version 1 and some example scripts in this repo, the binary file is written as a struct for each neuron of the form (unit8, float32).
The SOM I have is 45 x 45 x 1 neurons and I have 28220 images. The file size of this binary is 490MB, which implies that the rotation angle is written as a float64.
45*45*(8+64)*28220 / 1024 / 1024 / 8 = 490.49
Reading the data in as a (unit8, float64) leads to very erroneous angles that are certainly not correct (1e306). I would expect that the rotation angles for images to the BMUs should follow a uniform distribution between 0 to 2pi (point sources at the image center can make this a little biased I have filtered most of these out). Here I plot the angles when data is loaded as both float32 and float64
Clearly there is structure when the angles are treated as float32, with four peaks. I'm guessing this is related to the four different ways you can rotate an image just by accessing the memory in different directions.
I think the problem is in the main_generic.h file when writing the transform file. Looking at the order that the images are generated they appear to be ang, ang+90, ang+180, ang+270, but the loop writing out the information treats the angle as a j * angle_step_radians. There might have to be a + results[i] / number_of_rotations * angle_step_radians somewhere in here?
I'm a little unsure about
float angle_step_radians = static_cast<float>(0.5 * M_PI) / input_data.m_number_of_rotations / 4;
I think the the 0.5 * M_PI has already handled the trailing / 4 when calculating the radian rotation increment. I don't see where this factor of 4 has been included in the input_data.m_number_of_rotations attribute - which appears to account for the four easy ways of rotating an image.
Perhaps there is some magic between the two previous points.
I guess my questions are
(1) what data type are the rotations actually written out as. The file size implies float64 but the figure and example scripts imply float32
(2) is my analysis about the rotations and the mismatch between the order they are generated and written correct?
Use circular rotation region for the euclidean distance
Using the rotational invariant region for the euclidean distance would be much better than only the square inside that region. It will also solve some inconsistencies with the normalization of the input data.
circular euclidean-distance-shape versus circular-masked data
Hi Bernd,
I have a question about the behaviour of the circular euclidean-distance-shape.
Given a square cutout of the radio sky, I rescale the intensity before feeding it to the SOM.
It might happen that unrelated bright emission in one of the corners messes up this scaling.
This is one of the reasons why I apply a circular mask to my cutouts before rescaling the intensities and feeding them to the SOM. (where apply a circular mask means setting all values outside the mask to zero.)
Another advantage is that this saves memory as circularly masked cutouts enable me to use neurons with the same dimensions as the data dimension.
The setup in this case:
Data dimension = 50 x 50
Neuron dimension = 50x50
Euclidean distance dimension = 50
What changes in this case if I choose a quadratic or circular euclidean-distance-shape?
(keeping the Euclidean distance dimension fixed at 50)
The resulting mapping values differ slightly (<0.5%) but that might just be rounding errors?
What benefits does one get from not masking the data during pre-processing and using this setup instead?
Data dimension = 50 x 50
Neuron dimension = 71x71
Euclidean distance dimension = 50
Shape of euclidean distance region = circular
Is there a difference between the PINK v0.23 and v2 neuron update rule?
I am trying to replicate results that I get with PINK v0.23.
After training with the same parameters, I get a different result with PINK v2.
It seems harder to get a low AQE and TE with the same parameters.
Did something (even some constant like the sqrt after mapping) change for the update rule from v0.23 to v2.1?
Something in the neighbourhood function (maybe it got normalized in v2?) or something in the learning rate parameter?
The only notable difference is that the neuron size is now bigger, but the euclidean distance dimension of v2 is equal to the euclidean distance dimension used in v0.23.
v0.23:
Number of images = 3123
Number of channels = 1
Image dimension = 165x165
SOM dimension (width x height x depth) = 10x10x1
SOM size = 100
Number of iterations = 1
Neuron dimension = 116x116
Layout = quadratic
Initialization type = random
Interpolation type = bilinear
Seed = 42
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 1
Use CUDA = 1
Use multiple GPUs = 1
Distribution function for SOM update = gaussian
Sigma = 5
Damping factor = 12.5331
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
v2.1
Number of data entries = 3123
Data dimension = 165 x 165
SOM dimension (width x height x depth) = 10x10x1
SOM size = 100
Number of iterations = 1
Neuron dimension = 234x234
Euclidean distance dimension = 116x116
Layout = cartesian
Initialization type = random
Interpolation type = bilinear
Seed = 42
Number of rotations = 360
Use mirrored image = 1
Number of CPU threads = 40
Use CUDA = 1
Distribution function for SOM update = gaussian
Sigma = 5
Damping factor = 12.5331
Maximum distance for SOM update = -1
Use periodic boundary conditions = 0
Store best rotation and flipping parameters = 0
After a few rounds with ever decreasing GAUSS and SIGMA I get the following values:
v0.23 AQEs: [91.0, 102.0, 89.0, 70.0, 64.0, 59.0, 55.0, 51.0, 48.0, 46.0, 44.0, 42.0, 40.0, 39.0, 38.0, 37.0, 37.0, 36.0, 36.0, 36.0, 36.0, 35.0, 35.0, 35.0, 35.0, 35.0, 35.0]
v2.1 AQEs: [95.0, 96.0, 80.0, 79.0, 76.0, 73.0, 74.0, 70.0, 69.0, 67.0, 66.0, 66.0, 65.0, 64.0, 64.0, 63.0, 63.0, 63.0, 63.0, 63.0, 62.0, 62.0, 62.0, 62.0, 62.0, 62.0, 62.0]
v0.23 TEs: [7, 27, 23, 11, 8, 4, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2]
v2.1 TEs: [36, 49, 37, 32, 24, 28, 18, 17, 12, 11, 11, 10, 10, 9, 9, 9, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9]
or as picture:
(the value used to divide the AQEs in the picture is the same for both: the euclidean distance dimension.)
Error state when std::getline finds no header
This was introduced in #17 .
I've found that when attempting to train a SOM initialized with the a previous SOM output file that there will be readSOM errors raised. Testing the code out I think that the std::getline is getting into an error state as it unsuccessfully searches for the # END OF HEADER string in a file without one. If a file does not contain the # END OF HEADER it will eventually get to the end of a file and attempt to read in a new line, at which point an error is encountered and subsequent seekg will no longer work.
This behavior can be seen by inspecting the is.eof() bit before and after the scan of the header is performed.
Fixing it requires a is.clear() to be added. I am not sure if the problem persists in other areas of the code.
Inconsistency in file format caused by Fix #17
It seems that changes introduced in Fix #17 have created an inconsistency in the header parsing.
All files must currently have a mandatory header ending with "# END OF HEADER". Therefore the data format description has to change.
- In UtilitiesLib/InputData.cpp line 367 the break criterion has to change and the clause has to be rewritten to seek correctly for the pattern.
- In SelfOrganizingMapLib/FileIO.h line 28 when writing a SOM it has to be ensured that at least an empty header is written even, if the header was empty before or the SOM was newly created
- In SelfOrganizingMapLib/SOM.cpp line 72 ff. when reading a SOM the hexagonal format is not supported and multidimensional neurons do not work.
perhaps all operations concerning header should be move to a single dedicated class handling all header operation, so future changes can be realized easier.
Currently the initialization with a SOM is broken for this particular reason, above. When printing the parameters of PINK while starting, it would help to print out the filename used for initialisation, too.
Floating point exception if number of progress information prints < number of images
The progress print is controlled by
if (ticks % number_of_ticks_in_section == 0)
where
number_of_ticks_in_section = number_of_images / number_of_progress_prints
Division by zero if number of progress information prints < number of images.
Provide a C++/Python interface
Since the usage of PINK via command line is very limited, we will provide a C++/Python interface to call PINK functions directly within Python. For the implementation the framework PyBind11 will be used, which should be able to provide a seamless integration to the new generic, template-based C++ code of PINK (#6). The data exchange between C++ and Python will be over numpy arrays.
Please find an experimental implementation at https://github.com/HITS-AIN/PINK/blob/redesign/src/PythonBinding/pink.cpp and an example of using for the SOM training at https://github.com/HITS-AIN/PINK/blob/redesign/scripts/train.py.
At the moment the interface is using a 2-dimensional cartesian SOM and data layout on the CPU. The missing step is now to add the other static types and to insert an abstraction to use them in a dynamic way.
Update pybind11 to version 2.5
PyBind11 package cannot be found updating to version 2.5.0.
Documentation on binary file serialisation
I see in the FILE_FORMATS.md file that there are descriptions of the header information of each type of binary file.
Can we expand this to clearly specify the order in which the corresponding arrays (SOMs, neurons, transforms etc) are written to this files? Although that are example codes that show examples of reading these they can be a little difficult to follow and transplant into other codes.
pybind11 dependency is mandatory instead of optional
If conan and pybind11 is not available cmake exit with following error:
CMake Warning at CMakeLists.txt:101 (find_package):
By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "pybind11",
but CMake did not find one.
Could not find a package configuration file provided by "pybind11" with any
of the following names:
pybind11Config.cmake
pybind11-config.cmake
Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
"pybind11_DIR" to a directory containing one of the above files. If
"pybind11" provides a separate development package or SDK, be sure it has
been installed.
Mapping broken in CPU version
Redesign C++ code to be more generic
The first version of PINK was written basically in C-style and we have duplicated code for CPU and GPU execution. Therefore, for PINK version 2 we would like to rewrite the code to get a more generic structure to avoid the CPU/GPU code duplication and to enable the implementation of an python interface, so that the C++ functions can be called directly in python.
The development will be performed in the branch https://github.com/HITS-AIN/PINK/tree/redesign.
The basic idea is to separate data and algorithm classes and to formulate the SOM, neurons and training objects as LayoutTypes. We start with the two LayoutTypes Cartesian<dim, T> and Hexagonal<T>, where dim is the dimension of the cartesian grid and T the data type, e.g. float. A image has then the type Cartesian<2, float>. The type of a 2-dimensional, cartesian SOM would be Cartesian<2, Cartesian<2, float>> or Hexagonal<Cartesian<2, float>> for a hexagonal SOM.
The SOM training and mapping algorithms are formulated as Functors, where the call operator gets the SOM and the training data object:
struct Trainer {
template <typename SOMType>
void operator () (SOMType& som, typename SOMType::NeuronType const& image) const;
};The new design makes it also easier to add new SOM types and training algorithms.
Discussions and suggestions are always welcome.
Very large data dimension size when initialising training
Hi,
I'm trying to train an SOM (just to get used to the way PINK works, hence the tiny file numbers and parameters below) but when I set the training run going, my output log file is growing in size at a rate of ~1Gb every 30 seconds, being filled with imformation like that shown in the quote below.
Number of data entries = 1096810496
Data dimension = 1097859072 x 1100480512 x 1102053376 x 1101004800 x 1099956224 x 1099431936 x 1101004800 x 1097859072 x 1098907648 x 1101004800 x 1101004800 x 1103101952 x 1104674816 x 1103101952 x 1105199104 x 1102577664 x 1101529088 x 1101004800 x 1098907648 x 1100480512 x 1101529088 x 1102053376 x 1094713344 x 1093664768 x 1103626240 x 1099956224 x 1088421888 x 1101004800 x 1099431936 x 1099956224 x 1104150528 x 1095761920 x 1102577664 x 1101529088 x 1101004800 x 1099956224 x 1099431936 x 1101529088 x 1103626240 x 1101004800 x 1101004800 x 1102577664 x 1097859072 x 1098907648 x 1101529088 x 1101004800 x 1097859072 x 1102053376 x 1099956224 x 1096810496 x 1101004800 x 1098907648 x 1097859072 x 1100480512 x 1100480512 x 1098907648 x 1101529088 x 1096810496 x 1097859072 x 1095761920 x 1099956224 x 1091567616 x 1098907648 x 1101529088 x 1099431936 x 1092616192 x 1096810496 x 1101529088 x 1099431936 x 1096810496 x 1100480512 x 1106247680 x 1101529088 x 1103101952 x 1101529088 x 1102053376 x 1102053376 x 1097859072 x 1101004800 x 1095761920 x 1096810496 x 1104674816 x 1097859072 x 1099956224 x 1102053376 x 1097859072 x 1099956224 x 1104674816 x 1099956224 x 1109917696 x 1094713344 x 1097859072 x 1111490560 x 1100480512 x 1102577664 x 1113325568 x 1099431936 x 1103626240 x 1113325568 x 1101529088 x 1103626240 x 1113849856 x 1105199104 x 1100480512 x 1111752704 x 1098907648 x 1097859072 x 1108606976 x 1096810496 x 1096810496 x 1104674816 x 1102053376 x 1097859072 x 1106771968 x 1101529088 x 1099956224 x 1102577664 x 1101004800 x 1096810496 x 1100480512 x 1097859072 x 1096810496 x 1103626240 x 1096810496 x 1100480512 x 1092616192 x 1099431936 x 1102577664 x 1091567616 x 1097859072 x 1098907648 x 1097859072 x 1098907648 x 1098907648 x 1094713344 x 1096810496 x 1099956224 x 1099956224 x 1098907648 x 1099431936 x 1105723392 x 1101529088 x 1099431936 x 1108344832 x 1101529088 x 1100480512 x 1105723392 x 1098907648 x 1098907648 x 1104150528 x 1103101952 x 1103101952 x 1107296256 x 1099431936 x 1098907648 x 1104150528 x 1098907648 x 1100480512 x 1104150528 x 1103626240 x 1099956224 x 1103626240 x 1097859072 x 1102053376 x 1099956224 x 1100480512 x
I set the training run going using the following:
$Pink --train _scripts/test.bin _pink_out/som.bin --som-width 10 --som-height 10 --num-iter 1 --numrot 4
So as you can see I'm keeping this little training run simple with a small gridsize as well as only 4 rotations per image.
This is for a training run where I'm just using 3 images of size 128x128, in float 32 format. So I'm confused as to why the number of data entries is so high in the first line of the quote above?
I'd really appreciate any help you can give on this!
Extra info
My images are stored in test.bin using the following numpy raw binary example:
filename = "test.bin"
fileobj = open(filename, mode='wb')
stacked_images.tofile(fileobj)
fileobj.close()
... where stacked_images has shape (128, 128, 3) or (3, 128, 128). (I've tried both ways to see if that was the problem but it still causes the issue above)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.