An overview of a typical place recognition pipeline. At first, the input data is encoded into a query descriptor. Then, a K-nearest neighbors search is performed between the query and the database. Finally, the position of the closest database descriptor found is considered as the answer.

Detailed description of featured library modules can be found in the docs/modules.md document.

1. PlaceRecognitionPipeline
2. SequencePointcloudRegistrationPipeline
3. PlaceRecognitionPipeline with semantics
4. ArucoLocalizationPipeline
5. LocalizationPipeline without dynamic objects
6. Localization by specific scene elements (Semantic Object Context (SOC) module)
7. Module for generating global vector representations of multimodal outdoor data
8. MultimodalPlaceRecognitionTrainer
9. TextLabelsPlaceRecognitionPipeline
10. DepthReconstruction
11. ITLPCampus

• The library requires PyTorch, MinkowskiEngine and (optionally) faiss libraries to be installed manually:

  • PyTorch Get Started
  • MinkowskiEngine repository
  • faiss repository

• Another option is to use the docker image. You can read detailed description in the docker/README.md. Quick-start commands to build, start and enter the container:

  # from repo root dir
  bash docker/build_devel.sh
  bash docker/start.sh [DATASETS_DIR]
  bash docker/into.sh

• After the pre-requisites are met, install the Open Place Recognition library with the following command:

  pip install -e .

• If you want to use the GeoTransformer model for pointcloud registration, you should install the package located in the third_party directory:

  # load submodules from git
  git submodule update --init
  
  # change dir
  cd third_party/GeoTransformer/
  
  # install the package
  bash setup.sh

You can download the weights from the public Google Drive folder.

dvc pull

We introduce the ITLP-Campus dataset. The dataset was recorded on the Husky wheeled robotic platform on the university campus and consists of tracks recorded at different times of day (day/dusk/night) and different seasons (winter/spring). You can find more detail in the VitalyyBezuglyj/ITLP-Campus repository.

Subpackage containing dataset classes and functions.

Usage example:

from opr.datasets import OxfordDataset

train_dataset = OxfordDataset(
    dataset_root="/home/docker_opr/Datasets/pnvlad_oxford_robotcar_full/",
    subset="train",
    data_to_load=["image_stereo_centre", "pointcloud_lidar"]
)

The iterator will return a dictionary with the following keys:

• "idx": index of the sample in the dataset, single number Tensor
• "utm": UTM coordinates of the sample, Tensor of shape (2)
• (optional) "image_stereo_centre": image Tensor of shape (C, H, W)
• (optional) "pointcloud_lidar_feats": point cloud features Tensor of shape (N, 1)
• (optional) "pointcloud_lidar_coords": point cloud coordinates Tensor of shape (N, 3)

More details can be found in the demo_datasets.ipynb notebook.

The opr.losses subpackage contains ready-to-use loss functions implemented in PyTorch, featuring a common interface.

Usage example:

from opr.losses import BatchHardTripletMarginLoss

loss_fn = BatchHardTripletMarginLoss(margin=0.2)

idxs = sample_batch["idxs"]
positives_mask = dataset.positives_mask[idxs][:, idxs]
negatives_mask = dataset.negatives_mask[idxs][:, idxs]

loss, stats = loss_fn(output["final_descriptor"], positives_mask, negatives_mask)

The loss functions introduce a unified interface:

• Input:
  • embeddings: descriptor Tensor of shape (B, D)
  • positives_mask: boolean mask Tensor of shape (B, B)
  • negatives_mask: boolean mask Tensor of shape (B, B)
• Output:
  • loss: loss value Tensor
  • stats: dictionary with additional statistics

More details can be found in the demo_losses.ipynb notebook.

The opr.models subpackage contains ready-to-use neural networks implemented in PyTorch, featuring a common interface.

Usage example:

from opr.models.place_recognition import MinkLoc3D

model = MinkLoc3D()

# forward pass
output = model(batch)

The models introduce unified input and output formats:

• Input: a batch dictionary with the following keys (all keys are optional, depending on the model and dataset):
  • "images_<camera_name>": images Tensor of shape (B, 3, H, W)
  • "masks_<camera_name>": semantic segmentation masks Tensor of shape (B, 1, H, W)
  • "pointclouds_lidar_coords": point cloud coordinates Tensor of shape (B * N_points, 4)
  • "pointclouds_lidar_feats": point cloud features Tensor of shape (B * N_points, C)
• Output: a dictionary with the requiered key "final_descriptor" and optional keys for intermediate descriptors:
  • "final_descriptor": final descriptor Tensor of shape (B, D)

More details can be found in the demo_models.ipynb notebook.

The opr.trainers subpackage contains ready-to-use training algorithms.

Usage example:

from opr.trainers.place_recognition import UnimodalPlaceRecognitionTrainer

trainer = UnimodalPlaceRecognitionTrainer(
    checkpoints_dir=checkpoints_dir,
    model=model,
    loss_fn=loss_fn,
    optimizer=optimizer,
    scheduler=scheduler,
    batch_expansion_threshold=cfg.batch_expansion_threshold,
    wandb_log=(not cfg.debug and not cfg.wandb.disabled),
    device=cfg.device,
)

trainer.train(
    epochs=cfg.epochs,
    train_dataloader=dataloaders["train"],
    val_dataloader=dataloaders["val"],
    test_dataloader=dataloaders["test"],
)

The opr.pipelines subpackage contains ready-to-use pipelines for model inference.

Usage example:

from opr.models.place_recognition import MinkLoc3Dv2
from opr.pipelines.place_recognition import PlaceRecognitionPipeline

pipe = **PlaceRecognitionPipeline**(
    database_dir="/home/docker_opr/Datasets/ITLP_Campus/ITLP_Campus_outdoor/databases/00",
    model=MinkLoc3Dv2(),
    model_weights_path=None,
    device="cuda",
)

out = pipe.infer(sample)

The pipeline introduces a unified interface for model inference:

• Input: a dictionary with the following keys (all keys are optional, depending on the model and dataset):
  • "image_<camera_name>": image Tensor of shape (3, H, W)
  • "mask_<camera_name>": semantic segmentation mask Tensor of shape (1, H, W)
  • "pointcloud_lidar_coords": point cloud coordinates Tensor of shape (N_points, 4)
  • "pointcloud_lidar_feats": point cloud features Tensor of shape (N_points, C)
• Output: a dictionary with keys:
  • "idx" for predicted index in the database,
  • "pose" for predicted pose in the format [tx, ty, tz, qx, qy, qz, qw],
  • "descriptor" for predicted descriptor.

More details can be found in the demo_pipelines.ipynb notebook.

• KirillMouraviev/simple_toposlam_model - An implementation of the Topological SLAM method that uses the OPR library.
• VitalyyBezuglyj/ITLP-Campus - ITLP-Campus dataset tools.

MIT License (the license is subject to change in future versions)