Rust implemenatation of a skip-gram gazetteer tagger

This project can be used to find large numbers of search terms in documents in linear time with respect to the document length.

Generally, the process is as follows.

1. Given an input tab-separated list of search-term-label-pairs, will construct a search tree of words.
2. Then, for each document, iterate over the words in the document and traverse the tree if a matching word is found.

To create the tree, the input lists are segmented using a pre-tokenizer from the tokenizers library.

The tree is backed by Rusts standard library HashMap.

The RESTful interface is implemented using rocket.

There are configuration options to enable the abbrevation of search terms or the creation of n-grams from search term segments, see config.toml.

It is important to note, that multiple occurrences of search terms in the input data will result in multiple labels on the leafs of the search tree. This is especially true if n-grams are generated. Each resulting match is returned alongside its match type, which may either be MatchType::Full, MatchType::Abbreviated or MatchType::NGram.

See below for an example tree given the input:

Puffinus          https://www.gbif.org/species/5229335
Puffinus puffinus https://www.gbif.org/species/5229380
Sula              https://www.gbif.org/species/2480966
Sula bassana      https://www.gbif.org/species/4352320
Sula leucogaster  https://www.gbif.org/species/2480975

Here, the orange nodes represent nodes within the search tree and purple nodes represent Full type matches with their corresponding labels.

See below for a small tree given the following list, with n-grams enabled:

Sula                          https://www.gbif.org/species/2480966
Sula bassana                  https://www.gbif.org/species/4352320
Sula leucogaster              https://www.gbif.org/species/2480975
Sula leucogaster leucogaster  https://www.gbif.org/species/7192777

Here, the light green node represents a NGram type match.

Supports the new TextImager interface v1. See:

• main.rs→v1_communication_layer() and main.rs→v1_process() for API methods.
• communication_layer.lua for the communication layer.
• See TTLab-UIMA for DUUI bindings and more variants.

You can also build the tool with --features gui to enable an additional user interface that allows tagging small texts or uploading small plaintext files for tagging. This requires an additional dependency: rocket_dyn_templates

• This project is currently still a work-in-progress.
• See biofid-gazetteer for a Java implementation of a similar tool that integrates with the TextImager pipeline.