@property
    def f_measure(self) -> float:
        """
        Based on the mathematical formula:
            f_measure = 2 * true positive / (2 * true positive + false positive + false negative)
        Calculate and return the f_measure

        Returns:
            f_measure (float)
        """
        return 1 / 2 * (self.precision + self.recall)

Shouldn't the f_measure be the harmonic mean of precision and recall? That is,

f_measure = 2 * (precision * recall) / (precision + recall)

Right now it just seems to return half of precision + recall

This issue is caused by incomplete serialization of Record class defined in __main__.

cloudpipe/cloudpickle#243

The experiment should use the datasets and metrics as defined in the project description.

6173ce7

Write script to compute pairs completeness of a blocking scheme.

It needs as input a ground truth dataset.

The smallest dataset should contain 10,000 records and the largest should the full datasets. The remaining 3 should be somewhere in the middle.

I was following a tutorial using rltk.CrfTokenizer() but in the latest version, the code seems to have migrated elsewhere. Could you help me figure out how to now instantiate a tokenizer, i.e., the new equivalent of the old:

Tagging recent contributors: @saggu

tokenizer = rltk.CrfTokenizer()

Installing the rltk dev branch requires many of the packages specified in the requirements.txt file to be already installed.

This is because setup.py gets __version__ from rltk package:

https://github.com/usc-isi-i2/rltk/blob/dev/setup.py#L3

And, the rltk package imports other rltk modules:

https://github.com/usc-isi-i2/rltk/blob/dev/rltk/__init__.py#L3

For Jaccard similarity, we can calculate union length / intersection length from De-Morgan's laws. There is no need to perform another set operation. This optimization saves around ~60s on 100K records computations.

Better to inspect other similarity metrics for other code optimizations.

RLTK returns incorrect score for Jaro Winkler similarity metric for the following pair: ("United Kingdom", "Sengenia (United Kingdom)").

Test:

import jaro
import rltk.similarity as sim
import py_stringmatching.similarity_measure.jaro_winkler
import py_stringmatching.similarity_measure.jaro
import rapidfuzz.distance

key = "United Kingdom"
query = "Sengenia (United Kingdom)"

print(sim.jaro_winkler_similarity(key, query, threshold=0.7, scaling_factor=0.1, prefix_len=4))
jw = py_stringmatching.similarity_measure.jaro_winkler.JaroWinkler(); print(jw.get_raw_score(key, query))
print(jaro.jaro_winkler_metric(key, query))
print(rapidfuzz.distance.JaroWinkler.similarity(key, query, score_cutoff=0.7, prefix_weight=0.1))

Output:

0.5054761904761905
0.7342857122421265
0.7342857142857143
0.7342857142857143

Similarly, the Jaro metric is also incorrect.

We need custom exceptions to handle different cases such as

• Malformed json / text
• Missing id

Currently, I cannot distinguish between the two in record indexing. Having well defined exception classes will help in finer control during indexing

I am trying to compute the needleman wunsch similarity using RLTK. But I am receiving an error. The description of the error can be seen in the attached screenshot.

The experiment should use the datasets and metrics as defined in the project description.

RLTK is missing a wrapper logger module when executing a workflow. The logger module should support log levels such as

• Error
• Warnings
• Info

Add to rltk a script to compute reduction ratio.

Consult with Yixiang whether this script exists already and where it should go.

Ideally, this script should work with the blocks of any blocking algorithm.