oliver-loeffler / image-registration Goto Github PK

Java API to perform for image registration using control points (landmarks). Supports rigid transforms and affine transforms. Decouples matrix computation from API, so use of different Java matrix computation libraries is possible.

Home Page: https://oliver-loeffler.github.io/image-registration/

License: Apache License 2.0

Java 100.00%

rigid-transformations affine-transformation image-registration jama la4j ejml apache-math-commons photomask photomask-image-placement control-points-registration

image-registration's Introduction

Image Registration using Control Points

Image Registration API:

Solver-API:

TL;DR;

Image registration is the process of finding the transform to match a given image with a desired reference. This library supports rigid body transforms (translation and rotation) as well as affine transforms (anisotropic scaling, anisotropic rotation/shear aka. non-orthogonality). Currently non-linear transforms are not supported.

Up to version 0.0.4, this API only worked together with NIST JAMA. Since version 0.0.5, different linear algebra frameworks can be used in image-registration using the Solver-API. Here it is now possible to choose either gov.nist.math.jama, org.la4j, org.ejml, org.apache.commons.math3 or even org.jblas. The library now consists of an API and a core JAR and a group of JARs providing a solver implementation for the linear algebra framework of choice. The solver is selected using the Java SPI (Service Provider Interface) mechanism. With version 0.0.7, selection of a solver implementation is mandatory as the main artifact no longer includes a default solver implementation.

Versions up to and including 0.0.5 run with Java-8
Version 0.0.5 will support different linear algebra libraries (will make use of service provider API)
Version 0.0.6 will support ~~Java-8 and~~ Java-11 (~~utilize multi-release JARs~~ support for modules will be introduced)
Version 0.0.7 will no longer provide a bundle version, the core is now the image-registration API library. It will be mandatory to add the required solver as needed.
Version 0.0.8 will support Java-17 with records (JEP 359)
Later versions will support higher order calculations (first: up to 3rd order, 20 coefficient model)

These methods are used e.g. in photomask manufacturing, medical imaging or geospatial applications. Control point or feature based methods have only limited scope of use in medical imaging, there intensity based or voxel based methods are preferred due to the natural structure of medical image data. These additional intensity and voxel based methods are not supported by this library.

Photomask related:http://www.lithoguru.com/scientist/glossary/R.html#registration, http://www.lithoguru.com/scientist/glossary/O.html#overlay, https://www.nist.gov/system/files/documents/pml/div683/conference/Hughes.pdf
Medical imaging: http://www.iro.umontreal.ca/~sherknie/articles/medImageRegAnOverview/brussel_bvz.pdf
Geo-Registration: http://www.georeference.org/doc/georegistration.htm

How to start?

The SNAPSHOT-API documentation is available on: https://www.raumzeitfalle.net/image-registration/reg/ The javadoc for the Solver-API: https://www.raumzeitfalle.net/image-registration/solver/

Version 0.0.7 is available on Maven Central using following snippet:

Maven Dependency for a project with JAMA backend (using `gov.nist.math:jama:1.0.3`):

<dependency>
  <groupId>net.raumzeitfalle.registration</groupId>
  <artifactId>image-registration</artifactId>
  <version>0.0.7</version>
</dependency>
<dependency>
  <groupId>net.raumzeitfalle.registration</groupId>
  <artifactId>jama-solver</artifactId>
  <version>0.0.7</version>
</dependency>

For a modular project, the projects module descriptor must be updated accordingly:

module yourmodule {
	requires net.raumzeitfalle.registration.solver;
	requires net.raumzeitfalle.registration.core;
	uses net.raumzeitfalle.registration.solver.SolverProvider;
}

Modular project with any custom backend:

<dependency>
  <groupId>net.raumzeitfalle.registration</groupId>
  <artifactId>solver-api</artifactId>
  <version>0.0.7</version>
</dependency>
<dependency>
  <groupId>net.raumzeitfalle.registration</groupId>
  <artifactId>la4j-solver</artifactId>
  <version>0.0.7</version>
</dependency>

The artifact image-registration provides the actual API for image registration using control points. The solver-api artifact defines the service provider interface which is required to integrate different externa linear algebra (LA) libraries. The third artifact in this example, jama-solver provides an implementation to the solver-api based on NIST JAMA library.

When the consumer project requires a different LA library, the appropriate implementation provided by this project can be used. In case a custom implementaton is required, this must be created based on solver-api.

New API structure

Module	Purpose
`image-registration`	Image Registration API without any external solver binding
`solver-api`	API to utilize different linear algebra frameworks for calculation
`solver-test`	Non-API, project to test numerics and service discovery of all available solvers
`example-non-modular`	Non-API, A Gradle example project on how to use this library in classical classpath based Java project aka. non-modular
`example-modular`	Non-API, The same example project but implemented using the module path aka. a modular project.
`jama`	Non-API, Fork of NIST JAMA: Java Matrix Package, based on version 1.0.3

Following solver implementations will be available

Solver Module	Dependency to:	Library version:
`apache-math3-solver`	`org.apache.commons:commons-math3`	3.6.1
`ejml-solver`	`org.ejml:ejml-simple`	0.41
`jama-solver`	`gov.nist.math:jama`	1.0.3
`jblas-solver`	`org.jblas:jblas`	1.2.5
`la4j-solver`	`org.la4j:la4j`	0.6.0
`jama`	no external dependency, functionality is equivalent to `gov.nist.math:jama:1.0.3`	0.6.0

More Linear Algebra libraries:

Goals

Todos

Handle 1D cases (handling of individual missing points already works)
Define high level API (ideally fluent or builder based) to compose evaluations.
Fluent-API for selecting specific displacements for alignment, positional calculation, first order calculation
Implement higher order polynomial model with configurable coefficients (polynomial model should work up to 9th order, coefficients to be used shall be configurable independently for x and y)
Sum up learnings and reshape structure of alignment/correction classes, possibly add higher level functions to do the all-in-one-job as its done in the demos.
Add graphical examples of first order distortions.
Functionality to determine which kind of correction / model can be used on a given dataset

How does is work?

Processing steps

Concepts

Image registration
Displacement
Rigid Body Transform
Affine Transform
Alignment and First Order Distortions
FirstOrderSetup
FirstOrderCorrection
FirstOrderResult

Domain types

Displacement
Category
SiteSelection
- AlignmentSelection
- CalculationSelection
Transform (high level)
- RigidTransform
- AffineTransform
- SimilarityTransform
- NonUniformSimilarityTransform
- HighOrderTransform (not yet implemented)
TransformCorrection
- SkipTransform
- SkippableOperation
Transform (low level)
- ModelEquation
- TransformModel
- Dimension
- Orientation
- SpatialDistribution
- Distribution

Interfaces

Transforms

Transforms are used to manipulate displacement elements. Depending on the model used, there may exist different transform types and implementations. Each transform is a Function which, applied to a Displacement, will return a Displacement. There are currently two base transforms, the RigidTransform and the AffineTransform. When transforms are parameterized in a way that there will be actually no change to the displacement data, then in some cases a SkipTransform might be created and used. A SkipTransform must not perform any calculations, instead it must pass through the given data without any modifications.

Transform Models

One idea and goal is, to make the underlying transformation models exchangeable. Therefore the RigidBodyModel and the AffineModel interfaces exist. The idea is to make the actual calculation algorithm exchangeable whereas the code used in an applications shall remain unchanged. The models are technically bi-functions which consume a Collection of equation objects (<T extends Orientable>) and a Dimension instance. The Dimension instance holds the information how many X,Y locations exist and how the data is oriented (e.g. 1D one-dimensional, either X or Y or 2D two-dimensional X and Y). Using the equations and the dimensional information any kind or matrix based system can be configured.

Examples (see example-modular)

There are some demos available, how this library is supposed to be used:

net/raumzeitfalle/registration/examples/modular/DemoFourpointsOnlyWithMissingMeas.java
net/raumzeitfalle/registration/examples/modular/DemoFourpointsScanner.java
net/raumzeitfalle/registration/examples/modular/DemoFourpointsStandard.java
net/raumzeitfalle/registration/examples/modular/DemoMultipoint.java
net/raumzeitfalle/registration/examples/modular/DemoMultipointMagnification.java
net/raumzeitfalle/registration/examples/modular/DemoMultipointOneDimensional.java (not yet validated)
net/raumzeitfalle/registration/examples/modular/DemoMultipointResidual.java

The class net.raumzeitfalle.registration.examples.modular.Demo is used to define all examples. This template allows it, to configure and parameterize the evaluation process as needed.

The following example code shows, how alignment on 4 selected locations works, with info only locations being removed. First order (scale/ortho) will be calculated on all locations. To calculate first order on selected locations, Alignments.SCANNER_SELECTED must be used.

    
    /* STEP 1, load displacements from file (or any other source)
     *
     * The CSV file may look like:
     * "refx","refy","posx","posy","diffx","diffy","type"
     * 10980.0,9012.5,10980.014331404400,9012.481628831100,0.014331404400,-0.018371168900,"ALIGN"
     * 10980.0,73512.5,10980.009388937700,73512.484174799200,0.009388937700,-0.015825200800,"REG_MARK"
     * 10980.0,143387.5,10980.004598393400,143387.490394277000,0.004598393400,-0.009605722700,"ALIGN"
     * ....
     * 146975.0,14925.0,146975.030283248000,NaN,0.030283248900,NaN,"INFO_ONLY"
     * 
     * All values are stored with the same unit, in this example all columns consist
     * of values in microns. The values in columns (diffx,diffy) are not used.
     * In case of one-dimensionality or missing values, NaN can be used.
     * 
     */

    List<Displacement> displacements = new FileLoader().load(Paths.get("Demo-4Point.csv"));
        
    /*
     * 
     * Each Displacement consists of a design location (x,y) and the actual displaced
     * location (xd,yd). By default, each Displacement is of type REG, but depending
     * on declaration in CSV file, a different DisplacementClass can be assigned.
     * The DisplacementClass allows to create simple predicates to select Displacmemt  
     * instances for different operations.
     *
     */
    

    // STEP 2, perform site selection
    Predicate<Displacement> allSites = d->true;
     
    SiteSelection selection = SiteSelection
                        .forAlignment(d -> d.isOfType(DisplacementClass.ALIGN))
                        .forCalculation(allSites)
                        .forRemoval(d->d.isOfType(DisplacementClass.INFO_ONLY));
    
    // STEP 3, parametrize evaluation model 
    FirstOrderSetup setup = FirstOrderSetup
                        .usingAlignment(Alignments.SELECTED)
                        .withCompensations(Compensations.SCALE, Compensations.ORTHO)
                        .withSiteSelection(selection);

    // STEP 4, perform correction and calculate results
    FirstOrderResult result = FirstOrderCorrection.using(displacements, setup);
    Collection<Displacement> results = result.getDisplacements();

    /* 
     *  Alternatively:
     *
     *  FirstOrderCorrection correction = new FirstOrderCorrection();
     *  results = correction.apply(displacements, setup).getDisplacements();
     * 
     *  As FirstOrderCorrection is a function, .andThen(...) and .compose(...) 
     *  can be used as well. 
     *
     */  
    
    // STEP 5, print results
        
    // Now print results before correction
    DisplacementSummary uncorrectedSummary = Displacement.summarize(displacements, selection.getCalculation());
    System.out.println(uncorrectedSummary);
        
    // after correction
    DisplacementSummary correctedSummary = Displacement.summarize(results, selection.getCalculation());
    System.out.println(correctedSummary);
        
    // now also print residual first order and alignment
    RigidTransform correctedAlignment = result.getAlignment();
    System.out.println(correctedAlignment);
        
    AffineTransform correctedFirstOrder = result.getFirstOrder();
    System.out.println(correctedFirstOrder);

Transform types

Symbol	Description
+	is directly calculated by model
o	can be approximated from results
sx/sy	anisotropic scaling (scale x/y)
mag	magnification, isotropic scaling (mag = average(sx,sy))
ox/oy	anisotropic rotation (shearing or non-orthogonality, ortho x/y)
ortho	isotropic non-orthogonality (ortho = oy - ox)
rot	isotropic rotation
tx/ty	translation in x/y direction

Transform	tx	ty	sx	sy	mag	ox	oy	ortho	rot
Rigid	+	+							+
Affine	+	+	+	+	o	+	+	+	o
Similarity	+	+			+			o	+
Non-Uniform Similarity	+	+	+	+	o			o	+

Cases

#1 common case, references and readings in 2D (X,Y)

Direction	Refs	Readings
X (count)	> 1	> 1
Y (count)	> 1	> 1

Input: X,Y,Xd,Yd
Result: tx,ty,sx,sy,ox,oy

#2 special case, references 2D, readings 1D (X)

Direction	Refs	Readings
X (count)	> 1	> 1
Y (count)	> 1	= 0

Input: X,Y,Xd
Result: tx,sx,ox

#3 special case, references 2D, readings 1D (Y)

Direction	Refs	Readings
X (count)	> 1	= 0
Y (count)	> 1	> 1

Input: X,Y,Yd
Result: ty,sy,oy

#4 common case, references in 2D, most readings (X,Y), some readings (X or Y)

Direction	Refs	Readings
X (count)	> 1	> 1
Y (count)	> 1	> 1

Input: X,Y,Xd,Yd; some readings have either Xd or Yd only
Result: tx,ty,sx,sy,ox,oy

#5 special case, references along horizontal line, readings in 2D (X,Y)

Direction	Refs	Readings
X (count)	> 1	> 1
Y (count)	= 1	> 1

Input: X,Y,Xd,Yd
Result: tx,ty,sx,ox

#6 special case, references along vertical line, readings in 2D (X,Y)

Direction	Refs	Readings
X (count)	= 1	> 1
Y (count)	> 1	> 1

Input: X,Y,Xd,Yd
Result: tx,ty,sy,oy

#7 special case, references along horizontal line, readings in 1D (X)

Direction	Refs	Readings
X (count)	> 1	> 1
Y (count)	= 1	= 0

Input: X,Y,Xd
Result: tx,sx

#8 special case, references along horizontal line, readings in 1D (Y)

Direction	Refs	Readings
X (count)	> 1	= 0
Y (count)	= 1	> 1

Input: X,Y,Yd
Result: ty,ox

#9 special case, references along vertical line, readings in 1D (X)

#10 special case, references along vertical line, readings in 1D (Y)

#11 special case, only one reference position, readings in (X and Y) or (X or Y)

References for further reading

License

Apache License, Version 2.0

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Attributions, external libraries used with their licenses

This project makes use of NIST.gov JAMA library. JAMA is public domain, see: https://math.nist.gov/javanumerics/jama/#license.

Also, this project uses LA4J library, which also follows Apache 2.0 license. See http://la4j.org for details, sources can be found at https://github.com/vkostyukov/la4j.

Badges have been created with:

image-registration's People

Contributors

Stargazers

Watchers

image-registration's Issues

Decouple matrix computation (equation solving) from high level transform code

Underlying matrix libraries shall be exchangeable. There are different libraries available, such as La4J, Ejml, JAMA or Apache Math Commons. All of these provide means to solve systems of linear equations.

It would be of advantage, if the user would be able to opt-in for the library of choice.

Consider a more generic approach to decouple matrix computation from image-registration API and abstracttion.

Add nice image for social preview

Written posts, blog entries or tweets will look more appealing when there is a nice preview image.

Create multiple artifacts - core and solver implementations

Idea is to break the project into multiple artifacts to slim down the dependencies.
Currently 4 matrix libraries are references. This will hold back users to use image-registration.

Instead, provide one image-registration JAR which contains the API and all domain types. Then, provide separate artifacts as per solver implementation. So the user can take the main-JAR and add the desired solver dependency. On other hand, one could add the main-JAR and implement its own solver.

Following approach is thinkable:

Create multi-build project
sub project: solver-api (not a public jar)
sub project: image-registration (depends on solver-api, public jar)
implement classes for solver service discovery
implement (a) lazy service discovery (first come, first serve) and (b) service discovery using class name. It should be possible to use different solver implementations per test
sub project: JAMA solver (depends on solver-api, public jar)
sub project: LA4J solver (depends on solver-api, public jar)
sub project: EJML solver (depends on solver-api, public jar)
sub project: Apache Math Commons 3 solver (depends on solver-api, public jar)

Update project documentation

Update README.md with class and interface hierarchy
Update how data processing works for FirstOrder
Complete JAVADOC with examples
Update code examples in README.md

HighOrder Distortions

Add support for image distortions of higher (polynomial) order.

Requirements:

support for 2nd order including cross-terms kn*X*Y, 2nd order means up to km*X^2*Y^2
model shall be parametrical, so that each coefficient can be toggled independently (e.g. equation such as t + a*X + b*Y + c*X^2 + d*X*Y + e*X^2*Y + f * X*Y^2 + g*X^2 * Y^2 etc.). Each coefficient in {a,b,c,d,e,f,g} should be independently configured to be enabled/disabled.
Ideally a builder is there to create the configuration.
The model should determine best fit conditions independently depending on data Orientation (Dimension) and SpatialDistribution.
Ideally, model should be expandable to higher orders beyond 2nd order - but right now I have no good idea how to do that.

Add non-linear rigid transform.

Add a non-linear alignment function and compare linear-alignment vs. non-linear alignment in terms of precision and performance.

Similarity transform models

Either finish implementation or throw code away. Technically the similarity transform can be implemented on-top of the AffineTransform with appropriate limitations set.

AffineTransform should not center data upfront

The AffineTransformCalculation function centers the given data before doing any calculations. There are cases, where this centering is not wanted. To cover those cases, try following approach:

remove the centering code from AffineTransformCalculation and instead compose it with a variant of net.raumzeitfalle.registration.alignment.TranslateToCenter.
A new version of TranslateToCenter is needed, as the center calculation should be controlled by a Predicate<Displacement>.

Implement handling for AffineTransform edge cases

multiple displacements along a straight line in either X or Y direction
multiple displacements at a single point

These cases usually end up with singular matrix exceptions or errors but basically a handling is possible.

Ideally this condition is detected before an exception occurs, so that the error handler in AffineTransformCalculation can be eliminated later on.

Unify how transform classes are built and how they should be used

All transforms in alignment package take a collection of displacements and return a collection of displacements. Unlike thise in distortions package. Here each transform takes a single displacement and returns a single displacement.

This does not feel good and straight forward. Ideally transforms in both package would be structured in same way. For distortions, there is a class AffineTransformCorrection which is a bi-function which takes a predicate to Displacement and a collection of displacements.
A similar way could be used for alignments OR distortion transforms should work like alignment functions.

The question is:

either Function<Collection,Collection>
or Function<Displacement,Displacement>

Example from SimpleRigidTransform.java:

@Override
public Collection<Displacement> apply(Collection<Displacement> d) {
	return d.stream()
		 .map(mappper())
		 .collect(Collectors.toList());
}
	
private Function<Displacement,Displacement> mappper() {
	return source -> Displacement.from(source, 
			source.getXd() - this.getTranslationX() + source.getY() * this.getRotation(), 
			source.getYd() - this.getTranslationY() - source.getX() * this.getRotation());

Example in AffineTransformCorrection.java:

public class AffineTransformCorrection implements 
    BiFunction<AffineTransform, Collection<Displacement>, Collection<Displacement>> {
  
    @Override
    public Collection<Displacement> apply(AffineTransform t, Collection<Displacement> u) {
       return u.stream()
                    .map(t)
                    .collect(Collectors.toList());
    }
}

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.