Giter Club home page Giter Club logo

qlty's Introduction

qlty

qlty is a Python library designed to handle large 2D or 3D tensors efficiently by splitting them into smaller, manageable chunks. This library is particularly useful for processing large datasets that do not fit into memory. It includes methods for stitching the smaller chunks back together and computing the mean and standard deviation of the reconstructed data.

Modules and Features

  • qlty2D : in-memory functionality, requires things to fit into CPU memory.

  • qlty2DLarge: Similar functionality as qlty2D, but uses on-disc caching. If you ever think you want to graduate to the 3D case, this is what to use.

  • qlty3Dlarge: Use this for 3D datasets, has on-disc caching of intermediate results.

Tasks:

  • unstich: Split large tensors into smaller chunks for efficient processing.
  • stitch: Reassemble the smaller chunks into the original tensor, with potentially overlaps taken into account
  • Border Handling: Manage border pixels to minimize artifacts during stitching and training in segmentation.

Installation

To install the required dependencies, you can use pip:

pip install qlty torch zarr numpy einops dask

qlty's People

Contributors

phzwart avatar giselleu avatar

Watchers

 avatar

qlty's Issues

Running QLTY on GPU

@AlbertVong has run cropping with QLTY on GPU with some modifications to the source code. This issue aims to start a conversation and review of these changes so we can potentially merge them to the current source code.

@AlbertVong if you can please share your changes in a pull request, so @phzwart can take a look

Retrieving patch boundary locations

To visualize the patches in a webUI, we are currently copying the following lines of code from QLTY source code:

for yy in range(num_steps[0]):
        for xx in range(num_steps[1]):
            start_y = min(yy * step[0], img_height - window[0])
            start_x = min(xx * step[1], img_width - window[1])
            stop_y = start_y + window[0]
            stop_x = start_x + window[1]

To prevent code duplication, I have suggested @Danapolinar to send a PR to QLTY, such that we can retrieve the boundary locations of the patches. In l111, make the following change:

class NCYXQuilt(object):
    .
    .
    def unstitch(self, tensor):
            """
            Unstich a single tensor.
    
            Parameters
            ----------
            tensor
    
            Returns
            -------
            A patched tensor
            """
            N, C, Y, X = tensor.shape
            result = []
            list_of_boundaries = self.get_patches_boundaries()
            num_patches = len(list_of_boundaries)
    
            for n in range(N):
                tmp = tensor[n, ...]
                for ii in num_patches:
                        start_x, start_y, stop_x, stop_y = list_of_boundaries[ii]
                        patch = tmp[:, start_y:stop_y, start_x:stop_x]
                        result.append(patch)
            result = einops.rearrange(result, "M C Y X -> M C Y X")
            return result
    
    def get_patch_boundaries(self, xx, yy):
            """
            Get the path boundaries based on the location of the patch 
    
            Parameters
            ----------
            List
    
            Returns
            -------
            List of boundary locations of patch
            """
            start_y = min(yy * self.step[0], self.Y - self.window[0])
            start_x = min(xx * self.step[1], self.X - self.window[1])
            stop_y = start_y + self.window[0]
            stop_x = start_x + self.window[1]
            return start_x, start_y, stop_x, stop_y
    
    def get_patches_boundaries(self):
            """
            Should make use of get_patch_boundaries to return a list of boundaries
            """
    .....

We drafted this rather quickly, so we are open to alternative suggestions. Please let us know what you think @phzwart

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.