saltudelft / libsa4py Goto Github PK

In the pipeline, an optional argument can be added to type-check processed source code file.
Here are the required changes:

• Add the field tc to modules in JSON files. Has either True or False value.
• Add the argument --tc to the process command.
• Add helper classes to a new module, type-check.py.
• Update the Pipeline class to run the type-checker (e.g. mypy)

The Visitor class should also include self variables in multiple assignments in the variables of a class method. For instance:

self.a, self.b = 45, 10

In the Pipeline class, skip empty files when processing. Also, a CLI arg might be useful for this.

Currently, type names are extracted from its slot without considering the module from which it's imported. For instance:

from typing import List

x: List[int] = []

The type of x should be extracted as typing.List[int].

Currently, the walrus operator (:=) can only be extracted and tested if Python 3.8+ is present in a user's system.
A workaround is needed to at least skip the unit test for the walrus operator if the Python version is lower than 3.8.

Line and column numbers should be added to the final JSONOutput in the field fn_lc.

To uniquely identify classes within a file, a qualified name is required.
To add this feature, a new field, i.e., q_name, will be added to both class representation and the final JSON output.

• Add type annotations coverage for each processed source code file (i.e. a module)
• Add type annotations coverage for each processed project, i.e. a JSON field type_annot_cove
• Update the docs in the JSONOutput.md file accordingly

It is expected to see all usages of a typed variable in a file to have its types. See the unit test test_propagated_types_file and the file exp_outputs/propagated_types.py in tests branch.

The TypeAdder class in cst_transformations.py should be improved.

Type aliases should be resolved when extracting type annotations. For instance:

Vector = list[float]

Currently, Vector is extracted, not the original type (i.e. list[float]).

• Move helper functions for merging projects into a merge.py module
• Save merged JSONs on the disk

The SpaceAdder CST transformer doesn't add an extra space in some rare cases like ~ operator.

The class NumberRemover in cst_transformers.py should replace imaginary numbers with the [number] special token.

The calculated type annotation coverage for both classes and modules is not the same as the manual calculation of these for a small code snippet.

MRE:

import textwrap
from pprint import pprint

import libcst
from libcst import metadata

from libsa4py.cst_extractor import Extractor
from libsa4py.cst_transformers import TypeAnnotationRemover, TypeApplier


mod = libcst.parse_module(
    textwrap.dedent(
        """
def f():
    x: int = 10

def g():
    x: str = "Hello World"
"""
    )
)
mod_annotations = Extractor.extract(mod.code).to_dict()
pprint(mod_annotations["funcs"])

annoless = mod.visit(TypeAnnotationRemover())
annotated = metadata.MetadataWrapper(annoless, unsafe_skip_copy=True).visit(
    TypeApplier(mod_annotations, apply_nlp=False)
)
new_annotations = Extractor.extract(annotated.code).to_dict()
pprint(new_annotations["funcs"])

Output:

[{'docstring': {'func': None, 'long_descr': None, 'ret': None},
  'fn_lc': ((2, 0), (3, 24)),
  'fn_var_ln': {'x': ((3, 4), (3, 5))},
  'fn_var_occur': {'x': [['x', 'builtins', 'int']]},
  'name': 'f',
  'params': {},
  'params_descr': {},
  'params_occur': {},
  'q_name': 'f',
  'ret_exprs': [],
  'ret_type': '',
  'variables': {'x': 'builtins.int'}},
 {'docstring': {'func': None, 'long_descr': None, 'ret': None},
  'fn_lc': ((5, 0), (6, 35)),
  'fn_var_ln': {'x': ((6, 4), (6, 5))},
  'fn_var_occur': {'x': [['x', 'builtins', 'str']]},
  'name': 'g',
  'params': {},
  'params_descr': {},
  'params_occur': {},
  'q_name': 'g',
  'ret_exprs': [],
  'ret_type': '',
  'variables': {'x': 'builtins.str'}}]
[{'docstring': {'func': None, 'long_descr': None, 'ret': None},
  'fn_lc': ((3, 0), (4, 24)),
  'fn_var_ln': {'x': ((4, 4), (4, 5))},
  'fn_var_occur': {'x': [['x', 'builtins', 'int']]},
  'name': 'f',
  'params': {},
  'params_descr': {},
  'params_occur': {},
  'q_name': 'f',
  'ret_exprs': [],
  'ret_type': '',
  'variables': {'x': 'builtins.int'}},
 {'docstring': {'func': None, 'long_descr': None, 'ret': None},
  'fn_lc': ((6, 0), (7, 21)),
  'fn_var_ln': {'x': ((7, 4), (7, 5))},
  'fn_var_occur': {'x': []},
  'name': 'g',
  'params': {},
  'params_descr': {},
  'params_occur': {},
  'q_name': 'g',
  'ret_exprs': [],
  'ret_type': '',
  'variables': {'x': ''}}]

Note that the second dict for the function g does not contain a type for the variable x, despite the original clearly finding the anntotation builtins.str

A tentative fix that I applied was to add the following to the leave_FunctionDef method of TypeApplier, but I am unsure of its correctness.

self.last_visited_assign_t_name = None
self.last_visited_assign_t_count = 0

Some resetting of self.last_visited_assign_t_name seems to be in order, or perhaps this should be assigned a fully qualified name instead of just the variable's name.

I hope this helps (or that I have made a mistake :) )