agamvrinos / liicd Goto Github PK

Replace print statements with file logging to be able to revisit the output of the detector for each commit that was processed, even after the termination of the app.

Currently the generate_config.py script, when parsing the commits of a repository, only saves the file changes that correspond to Modifications (M), Deletions (D) or Additions (A) statuses. However, when a file is moved from one place to another or when it gets renamed, git uses another status tag, in the format of R{number}, for this.

The problem that might occur is the following:

1. The detector checks out to HEAD~2 to handle the commit that corresponds to that project snapshot
2. The detector reads the file changes for that commit from the configuration file but since renames are not tracked, there are no related changes, hence the index does not get updated with the new location or filename
3. The detector checks out to HEAD~1 to handle the commit that corresponds to that project snapshot
4. The commit includes changes that refer to the now renamed/moved file.
5. The detector tries to find the file in the index but since the index is not updated, it fails.

When reading a codebase to create the initial Clone Index, the application skips binary and invalid (non-unicode) files. This happens on the basis of a list that includes the extensions that should be skipped. This list is not exhaustive though, in the sense that there might be binary file extensions that have not been included.

In such a case, the issue would be the following:

1. The application would ignore the file due to the inability to read it (an exception is thrown but the application continues by ignoring the file)
2. A commit includes changes that affect the specific file. For instance, a .jpg file (assuming .jpg is not handled, which is) was renamed.
3. The application tries to find the file in the index, but since it was not processed when the codebase was read, it fails.

Possible Solutions

1. Prior to reading it, try to do an initial check to see if a file is binary or invalid. For binary files, there are libraries that do this but they do so probabilistically, so these are still not suitable.
2. Update the application to only consider file extensions that form the majority of the codebase. For example for a Java-based project, only consider .java extensions or maybe a combination of extensions in case the project uses multiple languages.