Implementation of this should be generic enough to allow used any other model, for example:

• We'd like to use vectors obtained from a Vector Space model (Raw Counts, Tf-idf, LSA)
• We also need to support using other types of word embeddings, such as GloVe or Word2Vec or embeddings obtained from BERT (for sentences), for example: https://huggingface.co/

Our representation should allow us to implement metrics such as:

• Average sentence similarity (e.g. cosine distance, euclidean distance)
• Other metrics based on sentence similarity (e.g. max distance between two sentences, average distance to the cluster center)
• Givenness using semantic spaces
• etc.

One design idea is having a callable that takes the text and returns de vectors as a numpy array. From the spacy dependency we should already have numpy in our dependencies, so no worries about that.

The, we will need to file issues to implement different metrics.

TTR module can be found here:

https://github.com/dpalmasan/TRUNAJOD2.0/blob/master/src/TRUNAJOD/ttr.py

For example, using type hints we could define:

def type_token_ratio(word_list):

as

def type_token_ratio(word_list: List[str]) -> float:

Connect Database with superset to do make visualizations and analyses, that could help in getting ideas/insights about how well is performing the complexity model and the app.

Since we are using python 3.6+ and not supporting 3.5 (deprecated), pre-commit hooks should use back instead of yapf and all the files should be reformatted. Hooks configuration should also be updated.

For some reason, when building locally, documentation is shown correctly. However, in readthedocs TTR documentation is missing.

Add type hints to discourse markers features. This can be found in the following source code:

https://github.com/dpalmasan/TRUNAJOD2.0/blob/master/src/TRUNAJOD/discourse_markers.py

For example, the function find_matches (which already has a docstring specifying types):

def find_matches(text, list):
    """Return matches of words in list in a target text.
    Given a text and a list of possible matches (in this module, discourse
    markers list), returns the number of matches found in text. This ignores
    case.
    .. hint:: For non-Spanish users
       You could use this function with your custom list of discourse markers
       in case you need to compute this metric. In that case, the way to call
       the funcion would be: ``find_matches(YOUR_TEXT, ["dm1", "dm2", etc])``
    :param text: Text to be processed
    :type text: string
    :param list: list of discourse markers
    :type list: Python list of strings
    :return: Number of ocurrences
    :rtype: int
    """
    counter = 0
    for w in list:
        results = re.findall(r"\b%s\b" % w, text, re.IGNORECASE)
        counter += len(results)
    return counter

Could be updated to:

def find_matches(text: str, list: List[str]) -> int:

Add type hints to src/surface_proxies.py

With the new release of stanza:

https://stanfordnlp.github.io/stanza/

Maybe it is a good opportunity to improve accuracy. The issue is about investigating if this could improve our accuracy and cost estimates of migration.

There are some grammar issues on the paper. For instance, using "have" instead of "has", missing articles (such as "the" text), missing hyphen (such as grid-based), among others. I suggest using Grammarly (https://app.grammarly.com/) to assist the authors on this task.

cc openjournals/joss-reviews#3153

This estimate, estimates lexical diversity using a non-linear model. it is computed by the following procedure:

1. Take a random sample of words from the text.
2. Calculate the TTR (type token ratio)
3. Find the value that bests fit the following equation:

It is allowed using numpy as spaCy is already set as a dependency for TRUNAJOD

In the documentation, there is only one example of usage. Since there is a lot of features, the authors should include at least two or three more examples, using other features to improve the understanding of the proposed package. Currently, the remaining functionality is described only at an API level, which seems insufficient.

cc openjournals/joss-reviews#3153

TRUNAJOD use spaCy but the setup.py file does not add it as a dependency. On the other hand, long_description uses an extra unnecessary rst file that does not render properly on pypi site. This issue aims to fix this.

Add type hints to src/lexico_semantic_norms.py

Hey 👋 ! This issue is part of the JOSS review (openjournals/joss-reviews#3153) for this project.

First of all, kudos on the project! As a Spanish-speaker NLP researcher myself, I honestly cannot commend you enough regarding the quality and breadth of this work. We need this kind of research in our community and I thank you on behalf of all my colleague researchers as well.

Checking over the CONTRIBUTING guidelines, it seems there are some TBD sections that could be completed, not necessarily in great detail, but at least to a point where they're useful. If you feel there is a section that is unnecessary you can just remove it (e.g., how to create a pull-request is a general Github-level issue that I don't necessarily think your guidelines should spend too much effort in explaining).

My suggestion is to review the CONTRIBUTING guidelines and either add some content in those TBD sections that you consider relevant or reformat the document to remove them, thus making the guidelines complete and welcoming to new contributors. Please let me know if you feel this is a big issue or if you want to further discuss it.

Add type hints to src/utils.py

The entity grid module can be found in the following source:

https://github.com/dpalmasan/TRUNAJOD2.0/blob/master/src/TRUNAJOD/entity_grid.py

For example, the function:

def get_local_coherence(egrid):

Could be defined as:

def get_local_coherence(egrid: EntityGrid) -> Tuple[float, float, float, float, float]:

Implement and evaluate the following coherence model:

Louis, A., & Nenkova, A. (2012, July). A coherence model based on syntactic patterns. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (pp. 1157-1168).

In terms of state of the field, the paper include a lot of discussions around text measurements. But how about competitor libraries? TRUNAJOD is the only library/package (including other programming languages) that serves this purpose? If it isn't the only one, how does it works compared to those implementations?

cc openjournals/joss-reviews#3153

Currently TRUNAJOD models live in a folder called models, and they are loaded using a snippet as follows:

from TRUNAJOD import surface_proxies
from TRUNAJOD.entity_grid import EntityGrid
from TRUNAJOD.lexico_semantic_norms import LexicoSemanticNorm
import pickle
import spacy
import tarfile


class ModelLoader(object):
    """Class to load model."""
    def __init__(self, model_file):
        tar = tarfile.open(model_file, "r:gz")
        self.crea_frequency = {}
        self.infinitive_map = {}
        self.lemmatizer = {}
        self.spanish_lexicosemantic_norms = {}
        self.stopwords = {}
        self.wordnet_noun_synsets = {}
        self.wordnet_verb_synsets = {}

        for member in tar.getmembers():
            f = tar.extractfile(member)
            if "crea_frequency" in member.name:
                self.crea_frequency = pickle.loads(f.read())
            if "infinitive_map" in member.name:
                self.infinitive_map = pickle.loads(f.read())
            if "lemmatizer" in member.name:
                self.lemmatizer = pickle.loads(f.read())
            if "spanish_lexicosemantic_norms" in member.name:
                self.spanish_lexicosemantic_norms = pickle.loads(f.read())
            if "stopwords" in member.name:
                self.stopwords = pickle.loads(f.read())
            if "wordnet_noun_synsets" in member.name:
                self.wordnet_noun_synsets = pickle.loads(f.read())
            if "wordnet_verb_synsets" in member.name:
                self.wordnet_verb_synsets = pickle.loads(f.read())


# Load TRUNAJOD models
model = ModelLoader("trunajod_models_v0.1.tar.gz")

However, other similar projects (e.g. spacy), have models in a different repo, and they are installed via python tooling (e.g. cmd line options). This ticket is to track the effort of implementing such mechanism.

This is a lexical diversity measurement that penalizes number of words. It is computed as:

Where is the number of distinct words in the text, and is the total number of words in the text.

Unit tests should be added as well.

Docs should be updated as well, adding the following reference:

@misc{herdan1961problemes,
  title={Probl{\`e}mes et m{\'e}thodes de la statistique linguistique},
  author={Herdan, Gustav},
  year={1961},
  publisher={JSTOR}
}

Documentation on readthedocs does contain the documentation generated from master branch, but not from stable version. Documentation should be set up properly to not mislead users.

Word variation index can be thought as "density of ideas" in the text. It can be computed as:

Currently, EntityGrid features only support a uniform model for entities, however, salient features might improve the performance of applications using the entity grid model. This issue is to extend the model to consider salience features with the approach stated in:

Barzilay, R., & Lapata, M. (2008). Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1), 1-34.

I've successfully installed TRUNAJOD and its dependencies. However, when I tried to run the example presented in the README, the following error occurred:

Traceback (most recent call last):
  File "main.py", line 44, in <module>
    nlp = spacy.load("es", disable=["ner", "textcat"])
  File "/home/matheus/.local/lib/python3.8/site-packages/spacy/__init__.py", line 47, in load
    return util.load_model(name, disable=disable, exclude=exclude, config=config)
  File "/home/matheus/.local/lib/python3.8/site-packages/spacy/util.py", line 328, in load_model
    raise IOError(Errors.E941.format(name=name, full=OLD_MODEL_SHORTCUTS[name]))
OSError: [E941] Can't find model 'es'. It looks like you're trying to load a model from a shortcut, which is deprecated as of spaCy v3.0. To load the model, use its full name instead:

nlp = spacy.load("es_core_news_sm")

For more details on the available models, see the models directory: https://spacy.io/models. If you want to create a blank model, use spacy.blank: nlp = spacy.blank("es")

The Yule's K is defined as:

With , and is the number of words occurring times in a text containing words.
Unit tests should be added as well.

Currently is_word function receives as an argument a pos_tag:

https://github.com/dpalmasan/TRUNAJOD2.0/blob/master/src/TRUNAJOD/utils.py#L241

Would be better for usability purposes that it receives a spacy Token instead.

Acceptance criteria

• Signature for is_word is updated and input is a spacy Token
• Type hints updated properly
• No regressions in test cases

Add type hints to src/emotions.py

Currently we support givenness metrics, based on pronoun patterns. An estimate that might help for some complexity assessment tasks is using givenness based on a vector space model, as depicted in the following paper:

https://escholarship.org/content/qt87n81224/qt87n81224_noSplash_9e003a08316eb6817c3aeda212d9ae27.pdf

Implement this feature.

Add type hints to src/semantic_measures.py

Travis CI is kind of slow, and now we have available github actions. This ticket aims to migrate from travis-ci to github actions.

Both the documentation page and the README file should contain clear guidelines for third parties who wish to contribute to the software, report issues, or seek support. Here is an example:

If you wish to contribute to the software, report issues or problems, or seek support, feel free to use the issue report of this repository or to contact us.

Author's 1 name - [email protected]
Author's 2 name - [email protected]

The supported python version should be explicitly specified in the README.

cc openjournals/joss-reviews#3153

Constant was set as 10^-4, but it is 10^4. Therefore, we need to change this and also update docstring.

SpaCy tokens can contain universal POS tags and detailed POS tags, both properties in a spacy Token are pos_ and tag_ respectively. Currently, the function pos_ratio is kind of misleading as in the docstring it is describing pos_ tags, but computing ratio using tag_. This ticket is to fix this and implement a similar function using universal POS tags.

https://github.com/dpalmasan/TRUNAJOD2.0/blob/master/src/TRUNAJOD/surface_proxies.py#L515

Acceptance Criteria

• Docstring is updated properly
• Functionality to compute pos_ratio based on universal POS tag is added.

There was an statistical analisys on what the factors are, and what are they factor loadings, in particular we consider the following factors:

• Lexical Similarity
• Connectivity
• Referential Cohesion
• Concreteness
• Narrativity

Since we already can compute these factors, we need to implement the computation bits. This issue is to track that.

This is defined as the number of words occurring once in the text. Might be a good estimate when comparing texts of similar length.

Several functions of our complexity text assessor rely on external databases (such as giveness with the MCR (multilingual central repository), stopwords, lemmatizer, etc. We would like to provide an easy API for testing TRUNAJOD lib, such as the one spacy provdes with the nlp method. This issue is about investigating an alternative to achieve this.

Add type hints to src/syllabizer.py

Add type hints to src/givenness.py

Currently the codebase is a mess, with no unit tests and no documentation. This issue is to track efforts of adding unit tests to the current codebase, and do some refactors to improve maintainability.