glato / emerge Goto Github PK

Getting the following error when trying to start emerge:

(app) ➜  app git:(master) emerge
Traceback (most recent call last):
  File "/Users/dillon.jones/.pyenv/versions/app/bin/emerge", line 33, in <module>
    sys.exit(load_entry_point('emerge-viz==1.1.0', 'console_scripts', 'emerge')())
  File "/Users/dillon.jones/.pyenv/versions/app/bin/emerge", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/Users/dillon.jones/.pyenv/versions/3.7.10/envs/app/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 167, in load
    module = import_module(match.group('module'))
  File "/Users/dillon.jones/.pyenv/versions/3.7.10/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/dillon.jones/.pyenv/versions/3.7.10/envs/app/lib/python3.7/site-packages/emerge/main.py", line 8, in <module>
    from emerge.appear import Emerge
  File "/Users/dillon.jones/.pyenv/versions/3.7.10/envs/app/lib/python3.7/site-packages/emerge/appear.py", line 14, in <module>
    from emerge.languages.javaparser import JavaParser
  File "<fstring>", line 1
    (result=)
           ^
SyntaxError: invalid syntax

Seems like python doesn't like this line?

Describe your environment

Python 3.7.10
MacOS 11.6.2, intel cpu
emerge-viz 1.1.0

To Reproduce
Steps to reproduce the behavior:
Install emerge. I had to brew install graphviz and then run the following to avoid a wheel issue

pip install emerge-viz --global-option=build_ext --global-option="-I/usr/local/Cellar/graphviz/2.50.0/include/" --global-option="-L/usr/local/Cellar/graphviz/2.50.0/lib/" pygraphviz

run emerge

Expected behavior
emerge starts

Describe the bug

Analysis is taking a long time with no indication of progress.

emerge -a CPP
edited paths in cpp-template.yml

emerge -c cpp-template.yml
...
2023-02-21 17:12:41 analysis I ✅ created the filesystem graph
2023-02-21 17:12:41 analysis I 👉 starting file result creation in cpp check
...
came back to check 4 hours later no progress reported.
A python process is still running using 1Gb RAM
Overall CPU usage is at 12%, MEM usage at 68% (27Gb of 40Gb)

Describe your environment

• Windows 11
• Python 3.9
• 12th Gen Intel(R) Core(TM) i7-12700H 2.30 GHz , 40Gb
• Graphviz 7.10

emerge-viz installed using pip

The code base is on the medium to large side (14055 fliles of which 7674 are headers, 65440 lines).
I should probably have tried emerge out on something smaller first
and read the manual to see if there are any slow analysis steps worth switching off on a first pass.

Describe the bug
I'm trying to exclude test files and other kinds of files from the output graph, but they are always present.

Describe your environment

• Are you using the tool on macOS or linux? macOS
• Which Python version are you using? Python 3.9.6
• Which browser (with version) are you using? Chrome 121.0.6167.184
• Are you scanning a public source code repository that you can provide? I tried this in a private repo and in a public repo. The results were the same. This is the public repo I used for testing: https://github.com/resend/react-email
• If not - can you provide code snippets where or further information on the data that might be relevant for the bug?

To Reproduce
Steps to reproduce the behavior:

1. Create a config like

project_name: react-email-analysis
loglevel: info
analyses:
- analysis_name: react email analysis
  source_directory:  /tmp/emerge/packages
  only_permit_languages:
  - typescript
  only_permit_file_extensions:
  - .ts
  - .tsx
  ignore_files_containing:
  - .spec.tsx
  ignore_dependencies_matching:
  - (.*)\.spec\.tsx
  file_scan:
  - number_of_methods
  - source_lines_of_code
  - dependency_graph
  - fan_in_out
  - louvain_modularity
  - tfidf
  export:
  - directory:  /tmp/emerge/export
  - graphml
  - json
  - tabular_file
  - tabular_console_overall
  - d3

3. Run the tool

docker run --rm -v <full path to the root of the project cloned on machine>:/tmp/emerge achtelik/emerge:2.0.0 /tmp/emerge/config.yml

5. Once emerge is done, open the emerge.html page generated
6. See that test files (.spect.tsx) are in the graph

Expected behavior
Any file filtered by the config options ignore_files_containing and ignore_dependencies_matching is not present in the output.

Screenshots

Additional context
Add any other context about the problem here.

Thanks!

Hiiii again,
Louvain method considers the edge weights in its formula, I wanted to know if there is a weight for the edges in emerge graph code? It makes sense that this weight should be equal to the number of calls.

Also, what is meant by an entity? If we want to consider both public and private classes, can't we get a more logical metric?

Hi, I got this error when I used d3 in YAML config on my java code.

with open(target_export_file_path, 'w') as file:
FileNotFoundError: [Errno 2] No such file or directory: '/home/amir/emerge_test/export/force-graph-html/resources/js/graph_representations_d3_force_graph.js'

Thanks.

Should the emerge.py and my java code be in the same directory? I can have other output formats when i call emerge.py from another location (its source) but it seems it doesn't work for d3 format when they are not in the same directory.

The problem:
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Currently the d3 web frontend only offers a very simple search that highlights a given subset of nodes, based on a single search term. Sometimes it might be useful to only select/highlight a specific subset of nodes that a not based on simple string search criteria. This feature would increase usability, since it would facilitate more explorative approaches and visualization.

A solution:
A user should be able to put together a specific subset of highlighted nodes (optically with the same result as performing a simple node string search in the search box) by simply hovering over a given node, pressing a given key (e.g. space) and then adding/removing the hovered node to a current node highlight selection. The currently implemented search should be compatible with this multiselect feature, i.e. If I search for a given term, I still can add more nodes the the highlighted selection by simply hovering and selecting/deselecting them by pressing a given key.

All highlight logic currently holding for the search should be valid also the the hover/multiselect feature, e.g. if I highlight two connected nodes, the connecting edge should also be highlighted).

Is your feature request related to a problem? Please describe.
First, thanks to this fantastic initiative, I was looking for a tool like this one.
I'd like to scan only one file instead of a directory.

Describe the solution you'd like
It could be great to have an option to scan only a file instead of a whole directory

Optional: Describe alternatives you've considered
I'm attempting to scan all the module, but it's big.

Optional: Additional context
I'd like to see the depencies of a file in python, but I got this error:

NotADirectoryError(f'error in analysis {analysis.analysis_name}: source directory not found/ accessible:

We are currently using the "Clean Architecture" (https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) in our projects, combined with the idea of a "Modulith" (multiple packages on the top level for different domains/features).

It would be fantastic to have a graph that provides an overview of this structure and shows the dependencies between the modules, including any circular dependencies.

I believe a graph that combines the file system view with the dependency information would be ideal for this use case.

To give you an idea of what we are looking for, it should look something like this:

What are your thoughts on this? I will try to create a prototype, but perhaps for someone else, this could be an easy task. :D

Is your feature request related to a problem? Please describe.
It's hard not possible to exclude built-in Python modules with short names like os, re, etc using ignore_dependencies_containing without implicitly excluding other dependencies. For example, position (contains "os") and create_user (contains "re") will be unwittingly excluded as well.

Describe the solution you'd like
It would be nice to either allow regular or glob-style expressions in ignore_dependencies_containing or add a new parameter, such as ignore_dependencies_matching.

Optional: Describe alternatives you've considered
No alternatives for this as far as I'm aware.

Optional: Additional context
I'm new to emerge, so please let me know in case I missed anything or if this is actually possible ouf of the box.

This is a very exciting tool.
I try to use it to generate the FILE_RESULT_DEPENDENCY_GRAPH of Java project, and then found a problem.
For each Java file, the tool will generate two nodes: 1) file node (with absolute_name) and 2) class node (only with display name)

for example, node<D:\idea_workspace\ACPG4J\src\main\java\analyser4J\graph\AbstractVertex.java> and node<analyser4J.graph.AbstractVertex>

In my opinion, they should be regarded as one single node.
Wondering if there is a solution.

Again, this project is pretty amazing. Thanks!

Is your feature request related to a problem? Please describe.
I just created the first graph of one of the projects I work in. It looks AMAZING!!! First seeing this graph however it is kind of hard to draw concrete conclusions from it. It would be awesome to have some opinionated examples (f.e. I could imagine nicely seperated clusters like in the firework-like graphs in the README point to good interfaces and well made modules whereas tightly entangled ones smell like a need for interface-lift...) in order to speed up the process of learning to interpret those.

Describe the solution you'd like
Would be cool to have some more examples as in the readme that do not only show how it looks like but point to possible interpretations.

Describe the bug
When installing from pip3 on ubuntu deps

apt-get install graphviz graphviz-dev

are needed, could you add this to the how to install from pip section of the README?

Describe your environment

• ubuntu 20.04
• Python 3.8.10, pip 22.0.3

I noticed something while generating graphs for JS projects I'm working with:

These 2 files are the same, but they are imported from 2 distinct files relatively:

I think a solution would be to resolve the path of imported files relative to the root of the project. What do you think? Is there a similar problem in other languages or is it specific to the JS parser?

Windows 10 AMD64 20H2
Python

Running "C:\Users\User\AppData\Local\Temp\pyrun.bat.9590\pyrun.bat" :
cd=h:\_TST\wm_lng_python__emerge
PIP_PROX=--proxy="http://gateway.schneider.zscaler.net:80"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2518k  100 2518k    0     0   753k      0  0:00:03  0:00:03 --:--:--  754k
Installing "https://bootstrap.pypa.io/get-pip.py"...
Collecting pip
  Using cached pip-23.1.2-py3-none-any.whl (2.1 MB)
Collecting setuptools
  Using cached setuptools-67.7.2-py3-none-any.whl (1.1 MB)
Collecting wheel
  Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
Installing collected packages: wheel, setuptools, pip
Successfully installed pip-23.1.2 setuptools-67.7.2 wheel-0.40.0

 - - - Installing "pip" - - - - - - - - - - - - - - - - - - - - - - - -


 - - - Installing "git+https://github.com/glato/emerge.git" - - - - - - - - - - - - - - - - - - - - - - - -

Collecting git+https://github.com/glato/emerge.git
  Running command git clone --filter=blob:none --quiet https://github.com/glato/emerge.git 'C:\Users\User\AppData\Local\Temp\pip-req-build-9c3tn3wg'
  Cloning https://github.com/glato/emerge.git to c:\users\user\appdata\local\temp\pip-req-build-9c3tn3wg
  Resolved https://github.com/glato/emerge.git to commit 9af6d63e278e7c405180322794e459588d353cde
  Preparing metadata (setup.py): started
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\User\AppData\Local\Temp\pip-req-build-9c3tn3wg\setup.py", line 13, in <module>
          README = (HERE / "README.md").read_text()
        File "h:\_TST\wm_lng_python__emerge\Python\Lib\pathlib.py", line 1135, in read_text
          return f.read()
        File "h:\_TST\wm_lng_python__emerge\Python\Lib\encodings\cp1252.py", line 23, in decode
          return codecs.charmap_decode(input,self.errors,decoding_table)[0]
      UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2196: character maps to <undefined>
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
  Preparing metadata (setup.py): finished with status 'error'

Installation done...
Running default "pyrun.py"...
h:\_TST\wm_lng_python__emerge\Python\python: can't open file 'h:\\_TST\\wm_lng_python__emerge\\pyrun.py': [Errno 2] No such file or directory



h:\_TST\wm_lng_python__emerge>python --version
Python 3.10.8

h:\_TST\wm_lng_python__emerge>

Works with regular pip install emerge-viz but is a bit "old".

I have installed emerge following the instruction in the readme, but when I try to start it throws error message.

ModuleNotFoundError: No module named 'sklearn'

But when I run pip3 install sklearn the message is that requirement is already satisfied. I am running on macOS Ventura 13.0.1.

1. Some Swift language entities and properties are missing

• Swift entities are the following: class, struct, protocol, enum

• Swift properties can be: var, let

• Missing implementation: protocol, enum, let

2. Swift Entities dependencies are not evaluated when used as static

example:

class A {
 static let shared = A()
 
 func doSomething() {
  print("A")
 }
}

class B {
 func doSomethingWithA() {
  A.shared.doSomething()
 }
}

The class B depends from A, but the relation is not implemented in the output.

Proposed output

Consider the following valid swift example:

protocol Naming {
    var name: String { get set }
}

protocol Describing {
    var describing: String { get }
}

class ClassA: Naming & Describing {
    
    static let shared = ClassA()
    
    var name: String = "A"
    let id: UUID = UUID()
    var describing: String = "Class A"
}

class ClassB: ClassA {
    func show() {
        print(name)
    }
}

enum EnumA {
    case one
    case two(EnumB)
}

enum EnumB: String, Describing {
    case one
    case two
    
    var describing: String {
        self.rawValue
    }
}

struct StructC<T: Naming>: Describing {
    var name: String = "C"
    let state: EnumA = .one
    var generic: T
    
    var describing: String = "Class C"
    
    init(generic: T) {
        self.generic = generic
    }
}

public struct EmergeSwiftTest {
    public private(set) var text = "Hello, World!"

    var b = ClassB()
    let eB: EnumB = .one
    let eA = EnumA.two(.one)
    
    func show() {
        b.show()
        print(ClassA.shared.name)
    }
}

actual output:

desired output:

• Naming doesn't depend from other entities
• Describing doesn't depend from other entities
• ClassA depends on Naming, Describing
• ClassB depends on ClassA
• EnumA depends on EnumB
• EnumB depends on Describing
• StructC depends on Naming, Describing, EnumA
• EmergeSwiftTest depends on ClassB, EnumB, EnumA, ClassA

note:

• String and UUID are ignored in this context as they should appear as dependency on the Foundation framework

I've prepared a version solving the point 1. Happy to do a PR if you are interested. Not sure how to sort out point 2 though.

Use os.path.realpath to follow link.
In case the link failed, use os.path.exists to check the file is exists.

For example:

import ast.Visitor;

public class Normalizer {
    visitors = new ArrayList<>();
    visitors.add(new VisitorA ());
    public class VisitorA extends Visitor{
    }
}

In this case, VisitorA will not generate a link with Normalizer;
At the same time, imported ast.Visitor identified as node<ast.Visitor>in dependency_graph, and identified as node <visitor> in inheritance_graph. This causes the repeatedly appearing node to in complete_graph.

I haven`t tested in other language.

Is your feature request related to a problem? Please describe.
It seems emerge does not support the popular programming language Golang

Describe the solution you'd like
Support golang

Hey @glato 👋

I did more tests to identify edge cases that would not be covered yet with emerge. I'll report them here as I find these. If you don't have time to fix them, I'll have a look some time myself.

1. Import local index

This is a valid import:

import { GenericId, Age, Country, Amount } from ".";

"." refers to the current folder. In JS, it will pick the index.{js,ts} file.

Today, this doesn't seem to be resolved. This structure should highlight the issue:

src/
|__ index.js 
|__ math.js    # import { someFunction } from "." 

This would generate 2 orphan nodes. math.js should depend on index.js though.

Is your feature request related to a problem? Please describe.

I'm trying to run emerge against a big React app codebase.
python emerge.py -c frontend.yml

---
project_name: frontend
loglevel: info
analyses:
- analysis_name: check_tsx_files
  source_directory: <some-project-path>/frontend/src
  only_permit_languages:
  - javascript
  only_permit_file_extensions:
  - .tsx
  - .ts
  ignore_dependencies_containing:
  - node_modules
  ignore_files_containing:
  - spec.ts
  - story.ts
  - mock.ts
  - .snap
  - .gql
  file_scan:
  - number_of_methods
  - source_lines_of_code
  - dependency_graph
  - louvain_modularity
  - fan_in_out
  export:
  - directory: <some-project-path>/frontend
  - graphml
  - dot
  - json
  - tabular_file
  - tabular_console_overall
  - d3

I'm getting tons and tons of messages:

.tsx is an unknown extension, ignoring `/<edited>/src/Issues/Issues/Issues.tsx`

Pretty much the whole app gets skipped and not mapped.

Describe the solution you'd like
I'd like to be able to analyze a codebase that contains tsx which are essentially jsx with Typescript.

Given the size of our app, the ability to visualise it and check in an interactive way, would be an immense help.

Describe the bug
Getting a "Permission denied" with .git/config.lock error when trying to generate git-based.

2023-08-04 23:24:50   analysis I ⏩ calculating code metric results for: git metrics
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 941, in _obtain_lock_or_raise
    fd = os.open(lock_file, flags, 0)
PermissionError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/emerge", line 8, in <module>
    sys.exit(run())
  File "/usr/local/lib/python3.10/dist-packages/emerge/main.py", line 13, in run
    emerge.start()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 91, in start
    self.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/appear.py", line 112, in start_analyzing
    analyzer.start_analyzing()
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 51, in start_analyzing
    self.start_scanning(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 90, in start_scanning
    self._calculate_code_metric_results(analysis)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analyzer.py", line 196, in _calculate_code_metric_results
    analysis.calculate_metric(metric)
  File "/usr/local/lib/python3.10/dist-packages/emerge/analysis.py", line 265, in calculate_metric
    metric.calculate_from_results(filtered_results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 86, in calculate_from_results
    self._calculate_git_metrics(results)
  File "/usr/local/lib/python3.10/dist-packages/emerge/metrics/git/git.py", line 113, in _calculate_git_metrics
    for commit in repository.traverse_commits():
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 215, in traverse_commits
    with self._prep_repo(path_repo=path_repo) as git:
  File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/repository.py", line 186, in _prep_repo
    self.git = Git(local_path_repo, self._conf)
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 61, in __init__
    self._open_repository()
  File "/usr/local/lib/python3.10/dist-packages/pydriller/git.py", line 88, in _open_repository
    self._repo.config_writer().set_value("blame", "markUnblamableLines", "true").release()
  File "/usr/local/lib/python3.10/dist-packages/git/repo/base.py", line 596, in config_writer
    return GitConfigParser(self._get_config_path(config_level), read_only=False, repo=self)
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 366, in __init__
    self._acquire_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/config.py", line 385, in _acquire_lock
    self._lock._obtain_lock()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 951, in _obtain_lock
    return self._obtain_lock_or_raise()
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 944, in _obtain_lock_or_raise
    raise IOError(str(e)) from e
OSError: [Errno 13] Permission denied: '/tmp/emerge/source/.git/config.lock'

Describe your environment

• Are you using the tool on macOS or linux? - macos
• Which Python version are you using? - 3.9
• Which browser (with version) are you using? - Not relevant
• Are you scanning a public source code repository that you can provide? - no
• If not - can you provide code snippets where or further information on the data that might be relevant for the bug?
  My config.yml

---
project_name: swift_example_project
loglevel: info
analyses:
- analysis_name: full swift check
  source_directory: /tmp/emerge/source
  git_directory: /tmp/emerge/source/.git
  git_commit_limit: 500
  only_permit_languages:
  - swift
  only_permit_file_extensions:
  - .swift
  file_scan:
  - number_of_methods
  - source_lines_of_code
  - dependency_graph
  - fan_in_out
  - louvain_modularity
  - tfidf
  - git_metrics
  entity_scan:
  - dependency_graph
  - inheritance_graph
  - complete_graph
  - source_lines_of_code
  - number_of_methods
  - fan_in_out
  - louvain_modularity
  - tfidf
  export:
  - directory: /tmp/emerge/export
  - graphml
  - json
  - tabular_file
  - tabular_console_overall
  - d3

To Reproduce
Steps to reproduce the behavior:

1. Create a config like the above
2. Run the tool using docker docker run --rm -v /Users/<my_path>/emerge:/tmp/emerge achtelik/emerge:2.0.0 /tmp/emerge/config.yml

Expected behavior
Git metrics to be produced

Additional context
Running this on a private git repo, but one that I have access to pull etc. If I cd into source and do a git pull it updates the branch.

Is your feature request related to a problem? Please describe.
I'm having issue setting up Emerge to analyze Zephyr RTOS, a rather large C project.

Describe the solution you'd like
I would like Emerge to be able to read compile_commands.json files, that contain a list of all C directories, files, and compiler arguments.

Hello Dr. glato,
I'm very impressed by your work. I've been used it for a year to help finish some of my research.
The D3Exporter is so amazing. Now I'm trying to export AST and CPG like your work. I wonder if you can provide a standalone D3Exporter which can generate html from networkx (or something else).
No matter how, thank you for your wonderful work! Looking forward to your reply.

Hickey

Describe the bug

pip install fails due to error The 'sklearn' PyPI package is deprecated, use 'scikit-learn' rather than 'sklearn' for pip commands.

⇒  pip install emerge-viz

..snip..

Collecting sklearn
  Downloading sklearn-0.0.post1.tar.gz (3.6 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      The 'sklearn' PyPI package is deprecated, use 'scikit-learn'
      rather than 'sklearn' for pip commands.

      Here is how to fix this error in the main use cases:
      - use 'pip install scikit-learn' rather than 'pip install sklearn'
      - replace 'sklearn' by 'scikit-learn' in your pip requirements files
        (requirements.txt, setup.py, setup.cfg, Pipfile, etc ...)
      - if the 'sklearn' package is used by one of your dependencies,
        it would be great if you take some time to track which package uses
        'sklearn' instead of 'scikit-learn' and report it to their issue tracker
      - as a last resort, set the environment variable
        SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True to avoid this error

      More information is available at
      https://github.com/scikit-learn/sklearn-pypi-package

      If the previous advice does not cover your use case, feel free to report it at
      https://github.com/scikit-learn/sklearn-pypi-package/issues/new
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Describe your environment

• Are you using the tool on macOS or linux? macOS
• Which Python version are you using?
• Which browser (with version) are you using? N/A
• Are you scanning a public source code repository that you can provide? N/A
• If not - can you provide code snippets where or further information on the data that might be relevant for the bug? N/A

To Reproduce

pip install emerge-viz

Expected behavior

The tool installs correctly.

Screenshots
N/A

Additional context
N/A

Describe the bug
Emerge uses outdated click version 7. Could you update to >8 to avoid conflicts with other packages?

Describe your environment

• Linux 20.04
• python 3.8.10
• pip 22.0.04

To Reproduce
Steps to reproduce the behavior:

1. install f.e. pip install dunk
2. observe conflict message

Expected behavior
Install without conflict

Describe the bug
It seems that the behavior of determining dependency names differs based on the import style used.

Example project:

# foobar/services/stuff1.py
import io
io.foo()

# foobar/services/stuff2.py
from io import StringIO
StringIO.bar()

This yields four nodes with the following dependencies (using graphviz syntax for demo purposes here):

digraph {
  stuff1 -> io
  stuff2 -> foobar/io
}

Logs:

2022-10-09 20:02:01     parser D ⏩ generating file results...
2022-10-09 20:02:01     parser D ⏩ extracting imports from file result stuff2.py...
2022-10-09 20:02:01     parser D ⏩ adding import: foobar/io
2022-10-09 20:02:01     parser D ⏩ generating file results...
2022-10-09 20:02:01     parser D ⏩ extracting imports from file result stuff1.py...
2022-10-09 20:02:01     parser D ⏩ adding import: io

I did some introspection on pyparser.py and it seems that:

• from io import StringIO form sets global_import to True (due to the from keyword being used)
• import io sets global_import to False
  This leads to inconsistency of final names because the condition on line 260 requires global_import to be False.

Describe your environment

• Are you using the tool on macOS or linux? - Linux
• Which Python version are you using? - Python 3.9.13
• Which browser (with version) are you using? - N/A

To Reproduce
Steps to reproduce the behavior:

1. Create project "foobar"
2. Import module xxx in two different files using import xxx and from xxx import yyy foms
3. Run the tool
4. See xxx and foobar/xxx listed as two separate dependencies

Expected behavior
In both cases, only xxx should be shown.

Screenshots

Describe the bug
I'm trying to exclude test files and other kinds of files from the output graph, but they are always present.

Describe your environment

• Are you using the tool on macOS or linux? macOS
• Which Python version are you using? Python 3.9.6
• Which browser (with version) are you using? Chrome 121.0.6167.184
• Are you scanning a public source code repository that you can provide? I tried this in a private repo and in a public repo. The results were the same. This is the public repo I used for testing: https://github.com/resend/react-email
• If not - can you provide code snippets where or further information on the data that might be relevant for the bug?

To Reproduce
Steps to reproduce the behavior:

1. Create a config like

project_name: react-email-analysis
loglevel: info
analyses:
- analysis_name: react email analysis
  source_directory:  /tmp/emerge/packages
  only_permit_languages:
  - typescript
  only_permit_file_extensions:
  - .ts
  - .tsx
  ignore_files_containing:
  - .spec.tsx
  ignore_dependencies_matching:
  - (.*)\.spec\.tsx
  file_scan:
  - number_of_methods
  - source_lines_of_code
  - dependency_graph
  - fan_in_out
  - louvain_modularity
  - tfidf
  export:
  - directory:  /tmp/emerge/export
  - graphml
  - json
  - tabular_file
  - tabular_console_overall
  - d3

3. Run the tool

docker run --rm -v <full path to the root of the project cloned on machine>:/tmp/emerge achtelik/emerge:2.0.0 /tmp/emerge/config.yml

5. Once emerge is done, open the emerge.html page generated
6. See that test files (.spect.tsx) are in the graph

Expected behavior
Any file filtered by the config options ignore_files_containing and ignore_dependencies_matching is not present in the output.

Screenshots

Additional context
Add any other context about the problem here.

Thanks!

Is your feature request related to a problem? Please describe.
Currently emerge has no support for .nix files.

Describe the solution you'd like
Nix and NixOS is become the new defacto software supply chain management, configuration, deployment toolbox.

It would be amazing to have .nix files supported in this tool.
Additionally it would be great to also have a nixpkg of emerge.

Optional: Describe alternatives you've considered
Currently there is no .nix support from emerge but there is a wider appeal rising towards Nix based approaches.

Optional: Additional context
Being able to embed this tool into your own IDE or even VSCodeServer or OpenVSCodeServer would be a great way to boost visibility /engagement for the project as well.

Is your feature request related to a problem? Please describe.
PHP Support

Optional: Additional context
scan file composer.json or composer.lock

Describe the bug
In a Go codebase, the file result dependency graph shows each .go file as depending on all .go files inside each of its imported modules, instead of only the ones with exported members used in the given file.

Describe your environment

• Ubuntu 23.04?
• Python 3.11.4
• Google Chrome Version 115.0.5790.170 (Official Build) (64-bit)
• Private repo

Here is an example output with of the program with debug (-d) flag enabled:

...
2023-08-10 11:03:26     parser D ⏩ adding import: project/pkg/somepkg/gofile1.go
2023-08-10 11:03:26     parser D ⏩ adding import: project/pkg/somepkg/gofile2_test.go
2023-08-10 11:03:26     parser D ⏩ adding import: project/pkg/somepkg/gofile3.go
2023-08-10 11:03:26     parser D ⏩ adding import: project/pkg/somepkg/gofile1_test.go
2023-08-10 11:03:26     parser D ⏩ adding import: project/pkg/somepkg/gofile2.go
...

Only project/pkg/somepkg/gofile1.go contains exported members that are used in the file being processed, but all files under project/pkg/somepkg are added as dependencies.

To Reproduce
Steps to reproduce the behavior:

1. Create a config with at least dependency_graph and louvain_modularity scans enabled
2. Run the tool, go to the generated HTML
3. Open the file result dependency graph
4. See nonexistent dependencies between some nodes

Expected behavior
I'd expect only project/pkg/somepkg/gofile1.go to be added as a dependency and to see this reflected in the visualization.

Additional context
This issue is more evident in projects with many .go files under the same packages.
Given in Go you import by package, and not by file, it may be tricky to accurately represent dependencies between files. The Go standard library provides the go/parser and go/token packages, which can be used to parse Go files and analyze their abstract syntax trees (AST). That could be helpful, but I don't know how could it fit in this Python tool.

Failed tests
I have failed tests on step 5.

Describe your environment

• MacOS 12.0.1 Monterrey
• Python 3.10.2
• Google Chrome 94.0.4606.81
• Branch dev/1.1.0

To Reproduce

1. $git clone https://github.com/glato/emerge.git
2. $brew install graphviz
3. $cd emerge
4. $pip3 install virtualenv
5. $virtualenv -p python3 venv
6. $source venv/bin/activate
7. $pip install -r requirements.txt
8. $python emerge/tests/run_tests.py

Expected behavior
Successful tests

Reality

Traceback (most recent call last):
  File "/Users/akrotov/Downloads/emerge/emerge/tests/run_tests.py", line 9, in <module>
    from interrogate import coverage
  File "/Users/akrotov/Downloads/emerge/venv/lib/python3.10/site-packages/interrogate/coverage.py", line 10, in <module>
    import tabulate
  File "/Users/akrotov/Downloads/emerge/venv/lib/python3.10/site-packages/tabulate.py", line 16, in <module>
    from collections import Iterable
ImportError: cannot import name 'Iterable' from 'collections' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/collections/__init__.py)