Giter Club home page Giter Club logo

Comments (5)

virscience001 avatar virscience001 commented on August 16, 2024 2

If you are using python, you can use below code snippet to create hdf5 file.


import sys, h5py
import cv2, glob
from multiprocessing import Pool
import numpy as np

hf = h5py.File('data4torch.h5', "w")
train_paths = glob.glob('data/class/*.jpeg')

def process_image(impath):
    im = cv2.imread(impath)
    im = cv2.resize(im, (55,55))
    im = im.transpose()
    return im

# Class dictionary
label_dict = {'A': 1, 'B': 2, 'C': 3, 'D': 4, 'E': 5, 'F': 6}
def get_labels(impath):
    label = impath.split('/')[2]
    return label_dict[label]

p = Pool(4) # set this to number of cores you have
data = np.array(p.map(process_image, train_paths))
labels = np.array(p.map(get_labels, train_paths))

hf.create_dataset('data', data=data)
hf.create_dataset('labels', data=labels)
hf.close()

from jule.torch.

dcharua avatar dcharua commented on August 16, 2024

Thank you so much for your answer, it worked perfectly
I now have other problem :(
I resized the images to 32 x 32 and used the FRGC structure as suggested in another Issue, but got the same problem

/home/lifelogging/torch/install/bin/luajit: bad argument #2 to '?' (out of range at /home/lifelogging/torch/pkg/torch/generic/Tensor.c:913)
stack traceback:
[C]: at 0x7f1af1a2bb60
[C]: in function '__index'
train.lua:368: in function 'organize_samples'
train.lua:424: in function 'opfunc'
/home/lifelogging/torch/install/share/lua/5.1/optim/sgd.lua:44: in function 'sgd'
train.lua:438: in function 'updateCNN'
train.lua:489: in main chunk
[C]: in function 'dofile'
...ging/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
[C]: at 0x00405d50

Any idea what could be causing it ?
Thank you very much

from jule.torch.

dcharua avatar dcharua commented on August 16, 2024

I got it, the problem was the datatype, it needs to be a 32 float, I added the cast to your code
data = np.array(p.map(process_image, train_paths)).astype(np.float32)
labels = np.array(p.map(get_labels, train_paths)).astype(np.float32)

from jule.torch.

jwyang avatar jwyang commented on August 16, 2024

@dcharua great!

from jule.torch.

amir-sha avatar amir-sha commented on August 16, 2024

Do you have any solution for creating HDF5 from a set of images?
as mentioned in the readme
"Create a HDF5 file with the size of NxCxHxW, where N is the total number of images, C is the number of channels, H is the height of the image, and W the width of the image. Then move it to datasets/dataset_name/data4torch.h5"
I'm using c++ for creating HDF5 file but I'm not able to understand the procedure, I downloaded the datasets/CMU-PIE/data4torch and open the file but the size doesn't match the description for creating HDF5 it has only two tables "data[28552] and labels[28552]" (and if its an unsupervised method why we have labels ?!) would you please help me to figure out how to create HDF5 from my dataset ?

here is the code I've written so far ...

void CreateHDF5(fs::path dataSetPath)
{

    vector<fs::path> imagesPath;
    getfiles(dataSetPath,".bmp",imagesPath);
    //copy all files to new folder
    fs::path dsPath=curr/"HDF5/dataset";
    fs::path hd5file=curr/"HDF5/Dataset.h5";

   const H5std_string	FILE_NAME(hd5file.string().c_str());
//    const H5std_string	DATASET_NAME(dsPath.string().c_str());
     const H5std_string	DATASET_NAME("dset");
    int dsize=4;
    hsize_t dims[dsize];               // dataset dimensions
    const int	 RANK = 2;


    fs::create_directories (dsPath);
    ProgressBar progressbar1(imagesPath.size(),"Copy Dataset          ");
    for (int i=0;i<imagesPath.size();i++)
    {
        boost::filesystem::copy_file(imagesPath[i] ,dsPath/imagesPath[i].filename(),fs::copy_option::overwrite_if_exists);
        progressbar1.Progressed(i);
    }
    progressbar1.Progressed(imagesPath.size());
    cerr<<endl;


    // Create a new file using the default property lists.
    H5File dsfile(FILE_NAME, H5F_ACC_TRUNC);

    dims[0] = imagesPath.size();
    dims[1] =getimage(imagesPath[0].string().c_str()).channels() ;
    dims[2] = getimage(imagesPath[0].string().c_str()).rows;
    dims[3] = getimage(imagesPath[0].string().c_str()).cols;
    DataSpace dataspace(RANK, dims);
    // Create the dataset.
    DataSet dataset = dsfile.createDataSet(DATASET_NAME, PredType::STD_I32BE, dataspace);

}

from jule.torch.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.