platisd / duplicate-code-detection-tool Goto Github PK

I am running on a legacy repo and it has lots of duplication. So much in fact that the output file is larger than Github allows to be posted to a comment.

Posting results to GitHub failed with code: 422
{"message":"Validation Failed","errors":[{"resource":"IssueComment","code":"unprocessable","field":"data","message":"Body is too long (maximum is 65536 characters)"}],"documentation_url":"https://docs.github.com/rest/reference/issues#create-an-issue-comment"}

Can the output be clamped in size to avoid this error?

My temporary workaround is to increase the ignore_below value.

I'm getting high similarity scores, even though when I analyze the files with grep -Fxf FILE1 FILE2 it's just comments. It would be nice to ignore comments.

I get this error when trying to run the action on a private repo (when I set the repo to public it works just fine)
fatal: could not read Password for 'https://***@github.com': No such device or address

I've looked around and it seems like it's the same issue as this one

Here is a log of my run:

Run platisd/[email protected]
  with:
    github_token: ***
    project_root_dir: projects
    directories: .
    fail_above: 70
    ignore_below: 0
    file_extensions: h, hpp, c, cpp, cc, java, py, cs
    warn_above: 100
/usr/bin/docker run --name a95ec8fa99b7c7894e8abe71ec[2](https://github.com/itaykraise-vayyar/testing-git-hub-actions/runs/5480820825?check_suite_focus=true#step:4:2)eb60de7cf_d2f098 --label 29a95e --workdir /github/workspace --rm -e INPUT_GITHUB_TOKEN -e INPUT_PROJECT_ROOT_DIR -e INPUT_DIRECTORIES -e INPUT_FAIL_ABOVE -e INPUT_IGNORE_BELOW -e INPUT_IGNORE_DIRECTORIES -e INPUT_FILE_EXTENSIONS -e INPUT_WARN_ABOVE -e HOME -e GITHUB_JOB -e GITHUB_REF -e GITHUB_SHA -e GITHUB_REPOSITORY -e GITHUB_REPOSITORY_OWNER -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RETENTION_DAYS -e GITHUB_RUN_ATTEMPT -e GITHUB_ACTOR -e GITHUB_WORKFLOW -e GITHUB_HEAD_REF -e GITHUB_BASE_REF -e GITHUB_EVENT_NAME -e GITHUB_SERVER_URL -e GITHUB_API_URL -e GITHUB_GRAPHQL_URL -e GITHUB_REF_NAME -e GITHUB_REF_PROTECTED -e GITHUB_REF_TYPE -e GITHUB_WORKSPACE -e GITHUB_ACTION -e GITHUB_EVENT_PATH -e GITHUB_ACTION_REPOSITORY -e GITHUB_ACTION_REF -e GITHUB_PATH -e GITHUB_ENV -e GITHUB_STEP_SUMMARY -e RUNNER_OS -e RUNNER_ARCH -e RUNNER_NAME -e RUNNER_TOOL_CACHE -e RUNNER_TEMP -e RUNNER_WORKSPACE -e ACTIONS_RUNTIME_URL -e ACTIONS_RUNTIME_TOKEN -e ACTIONS_CACHE_URL -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/testing-git-hub-actions/testing-git-hub-actions":"/github/workspace" 29a95e:c8fa99b7c789[4](https://github.com/itaykraise-vayyar/testing-git-hub-actions/runs/5480820825?check_suite_focus=true#step:4:4)e8abe71ec2eb[6](https://github.com/itaykraise-vayyar/testing-git-hub-actions/runs/5480820825?check_suite_focus=true#step:4:6)0de[7](https://github.com/itaykraise-vayyar/testing-git-hub-actions/runs/5480820825?check_suite_focus=true#step:4:7)cf
Cloning into 'itaykraise-vayyar/testing-git-hub-actions'...
fatal: could not read Password for 'https://***@github.com': No such device or address

What do you think?

from the source code, I see gensim be used here. gensim be used as text similarity check is cool, but whether be suitable to code similarity check? I mean, for example I want to compare two Unity projects, all scripts must follow C# syntax, many C# fixed words, and to use Unity Engine, all same framework sentences be used such as 'using UnityEngine'. Can gensim ignore these? no mis-check?

What the title says, let's fix it

@platisd

Is it possible for your to provide output markdown file which i can use for publishing as checkrun using another Github Action. Or alternatevely if you can publish the same markdown table as checkrun thats is also good enough for me.

It better to reuse (overwrite) existing comment (which starts with Duplicate code detection tool report) to avoid being overloaded by bot comments.

Hi, I've been using this GH Action and set ignore_below: 20, tool anyway shows up, even when there's no similarities found whatsoever

Example:

I might be doing something wrong, but I've literally copied over from the example provided

Thanks

Thinking about being able to compare SQL used by data analysts and present similarity results in a familiar form e.g. a table.

python duplicate_code_detection.py -d smartcar_shield/src
Traceback (most recent call last):
File "E:\nbu\similar\code\duplicate-code-detection-tool\duplicate_code_detection.py", line 10, in
import gensim
ModuleNotFoundError: No module named 'gensim'

Hello,
I'm using:
> duplicate_code_detection.py --ignore-directories general/util/async/file/test -d general
in gitbash terminal on Windows, but the files in general/util/async/file/test are not ignored.

Value example of involved variables:

ignore_directories = ['general/util/async/file/test']
files_to_ignore = ['general/util/async/file/test\\util_async_file_test\\File_test.cpp', ...]
source_code_files = [..., 'general\\util\\async\\file\\test\\util_async_file_test\\File_test.cpp',...]

I fixed it, do you want me to submit a PR ? :-)

It would be nice to have an option to do not run the comparison on all of the files within the directory, but just on a filtered subset of them.
Idea to implement: An input argument that could be a regular expression that would be executed on the list of files.

If you don't mind, I would volunteer to implement it too.

Hello,
It would be nice to help the results analysis to add an option --with-loc that add the lines of code count for each file.
The ouputs will be:

• text

Code duplication probability for general\bar\EventLoop.cpp,351
--------------------------------------------------------------
                   File,#LoC                     Similarity (%)
--------------------------------------------------------------
general\util\async\EventLoop.cpp,351                 100.00
general\foo\EventLoop.cpp,351                        100.00

• csv

File A,#LoC A,File B,#LoC B,Similarity
general\bar\EventLoop.cpp,351,general\foo\EventLoop.cpp,351,100.0
general\bar\EventLoop.cpp,351,general\util\async\EventLoop.cpp,351,100.0

• json

{
    "general\\bar\\EventLoop.cpp": {
        "#LoC": 351,
        "general\\foo\\EventLoop.cpp": {
            "#LoC": 351,
            "similarity": 100.0
        },
        "general\\util\\async\\EventLoop.cpp": {
            "#LoC": 351,
            "similarity": 100.0
        }
    }
}

The main business logic under duplicate_code_detection.py is lacking unit tests to be automatically run on CI.

Let's introduce some! 🎈

I dont know how to use it, I have django project run by docker on my local development, and I need to run this tool within it, How ?