miki725 / importanize Goto Github PK

I have a use case where i would like importanize (ideally) to run on a file but ignore a specific region. I can of course exclude the file, but that is not the goal.

if i have for example a os.environ or sys.path call where i add something needed for the import after it i would like to surround it with #importanize off and #importanize on or the like.

Something like this but with multiline support (unless i just comment the sys.path and import and it ignores them).

What do you think?

Instead of relying on config file to determine local libraries, try to determine that automatically.
Perhaps the heuristic could be something like:

is_local = not stdlib()
           and importable(with sys.path stripped to only allow local imports)

I found this project via this StackOverflow question. I was able to use the code snippet @miki725 posted there to quickly identify which imports in my project are from the Python Standard Library and which are not. It would be a nice little feature if importanize had a command-line option to do this.

General idea for the interface:
$ importanize --print-stdlib-pkgs path/to/python/files

Possible output example:

stdlib packages:
* abc
* collections
* gzip

non-standard packages:
* docopt
* humanize

Instead of exiting, try fixing multiple imports on a single line

Hi,

I am getting an error that i'm not sure is pre-commit related or importanize (although i suspect both). When i add the following to my pre-commit config i suddenly cant run pre-commit as i get the below error. Only when i add language_version: python3.8 to the importanize hook does it work. I suspect it is related to the hook you have in the repo that sets the langauge to python3. Any idea?

  - repo: https://github.com/miki725/importanize/
    rev: '0.7'
    hooks:
    - id: importanize
      args: [--config=setup.cfg]
      exclude: ^launcher/requirements/

The below is from the pre-commit log already pre-formatted.

version information

pre-commit version: 1.20.0
sys.version:
    3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:21:23) [MSC v.1916 32 bit (Intel)]
sys.executable: c:\users\user\appdata\local\programs\python\python38-32\python.exe
os.name: nt
sys.platform: win32

error information

An unexpected error has occurred: CalledProcessError: command: ('c:\\users\\user\\appdata\\local\\programs\\python\\python38-32\\python.exe', '-mvirtualenv', 'C:\\Users\\user\\.cache\\pre-commit\\repo0fr10xh9\\py_env-python3.6', '-p', 'C:\\python36\\python.exe')
return code: 3
expected return code: 0
stdout:
    The path C:\python36\python.exe (from --python=C:\python36\python.exe) does not exist
    
stderr: (none)

Traceback (most recent call last):
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\error_handler.py", line 72, in error_handler
    yield
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\main.py", line 344, in main
    return run(args.config, store, args)
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\commands\run.py", line 292, in run
    install_hook_envs(hooks, store)
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\repository.py", line 220, in install_hook_envs
    hook.install()
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\repository.py", line 97, in install
    lang.install_environment(
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\languages\python.py", line 169, in install_environment
    _make_venv(env_dir, python)
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\languages\python.py", line 181, in make_venv
    cmd_output_b(*cmd, env=env, cwd='/')
  File "c:\users\user\appdata\local\programs\python\python38-32\lib\site-packages\pre_commit\util.py", line 141, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('c:\\users\\user\\appdata\\local\\programs\\python\\python38-32\\python.exe', '-mvirtualenv', 'C:\\Users\\user\\.cache\\pre-commit\\repo0fr10xh9\\py_env-python3.6', '-p', 'C:\\python36\\python.exe')
return code: 3
expected return code: 0
stdout:
    The path C:\python36\python.exe (from --python=C:\python36\python.exe) does not exist
    
stderr: (none)

Note that although i have python3.8 installed and it runs it, it tries to make a virtual env with python3.6 due to importanize's language version.

Imports could be ordered either:

import a
import z
from a import b
from z import y

or

import a
from a import b
import z
from z import y

There should be a config to determine style. Default should be imports first (my preference).

The help message states that grouped and inline-grouped are the only available options, however the documentation in the README.md file implies that you can also use lines. I was only able to get lines to work properly when adding

{
  "formatter": "lines"
}

to an importanize.json file

I suppose you can either add lines as part of the CLI or update the README.md file to state that lines is not available via the CLI

Hi,

I installed importanize with pip and it seems like I am not grouping stdlib imports properly. Do you have any insights on the issue? I am using version 0.7.0
$ importanize --list python.py

stdlib
------

remainder
---------
from __future__ import absolute_import, print_function, unicode_literals
import structlog

🎉

you might also find https://github.com/asottile/reorder_python_imports helpful. I had the same frustrations with isort and it looks like we went down similar routes 😆 (my testsuite might be of help to you -- plus the testsuite for https://github.com/asottile/aspy.refactor_imports)

different options:

from package.subpackage.module.submodule import (
    CONSTANT,
    Klass,
    bar,
    foo,
    rainbows,
)

from package.subpackage.module.submodule import (
    CONSTANT, Klass, bar,
    foo, rainbows,
)

from package.subpackage.module.submodule import (CONSTANT,
                                                 Klass,
                                                 bar,
                                                 foo,
                                                 rainbows)

from package.subpackage import (CONSTANT, Klass, bar,
                                foo, rainbows)

also maybe all of the above except using \ expect of ()

In some scenarios, before an import is made, it requires another import which also needs to execute a regular Python statement:

import a
a.foo()
import b # requires both statements above

There should be ability to skip such import sections:

<imports>

# block importanize                         |    
import a                                    | will
a.foo()                                     | not be
import b # requires both statements above   | organized
# endblock importanize                      |

<more imports>

Preserver original newline characters whether they are \n or \r\n

First, thanks for the great library! It's a great feeling to leave import hygiene to an automated process. Let me know what other details I can provide.

Summary

importanize 0.6.2 hangs indefinitely for relative directories. The same command succeeds with 0.6.1.

Repro Steps

Successful commit using 0.6.1: numberoverzero/oga@88193a4

I've reproduced this with multiple projects, but if it matters at all you can test with this project:

git clone https://github.com/numberoverzero/oga.git
cd oga
python3 -m venv --copies .venv
source .venv/bin/activate
pip install importanize==0.6.2
importanize -vvv oga/

^--- hangs forever

roll back and importanize succeeds:

pip install --force-reinstall importanize==0.6.1
importanize -vvv oga/

add some description to readme

Right now If I have imported something like this

import asyncio as aio

Importanize would wrongfully group it into local imports. I've somewhat traced this issue.

1. is_stdlib wronfully return False for this module. And the module argument is 'asyncio as aio'
2. The module argument is wrong because parse_import_statement incorrectly parses as import.

Few other notes

It would've been easier probably just to use existing python parser like jedi etc. But if you still want to use your own, then I should say that names like stem and leafs is somewhat misleading, because for AST as for any other tree it's usually called node and children.

That's maybe (I might be wrong) the root of the problem. You probably only have ImportNode and ImportLeaf but the truth is that import foo.bar as baz should parse to something like

Node ImportStatement
    Node BinaryOperator "AS"
        Node Module "foo.bar", Node Label "baz"

This is related to #18 where there is a proposal to add a way to skip certain parts of a file.

It might be nice to have a way to ignore a file completely, similar to .gitignore and such.

This can also be included in the currently implemented .importanizerc as an 'ignore' group, however that may get large over time and the benefits of a key : value notation are not really required.

My personal preference for blanket ignores is similar to what .editorconfig uses

from a import b, c  # noqa

is transformed to:

from a import (
    b,
    c  # noqa,
)

which is of course invalid syntax

from foo import \
    (class1, class2, class3,
     class4 , class5, class6,
     class7)

Does not translate correctly

Currently importanize does not correctly parse:

import bar
# comment
import foo

it ignores the comment in between the imports. all comments in between imports should be left attached to the import statement

Hi,

I am running this with pre-commit (in the process of setting it up) and encountering issues where the file i'm running it on has a "# -- coding: utf-8 --" up the top, but in the importanize script it uses with 'ource.read_text() in run_importanize_on_file which results in the platform default encoding used and a UnicodeDecodeError.

I take it this isn't completely unicode safe yet?

Instead of using one long line (> 80 chars), when multiple leafs are present, import should be reformatted over multiple lines:

from a import (
    b,
    c,
)

Unfortunately there are no tools to reliably organize Python imports to match PEP8 suggestions

is not strictly true due to https://github.com/timothycrosley/isort

when I created lib, was not aware so mentioned that but now that I know about isort, it should be mentioned

exit with proper exit code

I have an __init__.py file with the following content:

from .first import Second.

After running importanize --config=setup.cfg src/dir/__init__.py --print -v i get this output

=============================

src\dir_init_.py
=============================

where it deletes all my imports in the file (this was a trivial example, but in general all imports are deleted resulting in other usages breaking).

Is this by design?

Hi,

When i run importanize and yapf together with pre-commit i get the following circular fix issue:
importanize changes the following code to have only one line after the import (when setting after_imports_new_lines=1 to achieve compatibility with yapf's 1 line style) as you would expect but yapf also considers (as pep8 does to the best of my knowledge) that before classes/functions there should be 2 lines. So yapf would leave this 2 lines while importanize changes it to 1.

Any possibility of adding a flag that allows for lines before class (that overrides after imports)?

import requests


def a():

Not sure that this should be default, but that should be possible, because this is the unix way and there is no reason your program couldn't handle it.

Hi Miki, just now I have found some bugs about add_imports.

demo code like the following:

# coding: utf-8


class Foo(object):
    ...


def hello():
    print("world")

importanize config like following:

{
    "after_imports_new_lines": 2,
    "add_imports": [
        "from __future__ import absolute_import"
    ],
    "formatter": "grouped",
    "groups": [
        {
            "type": "stdlib"
        },
        {
            "type": "sitepackages"
        },
        {
            "type": "remainder"
        },
        {
            "type": "packages",
            "packages": ["test"]
        },
        {
            "type": "local"
        }
    ]
}

it's all seem good. but when I optimize import it by importanize, output like the following:

from __future__ import absolute_import


# coding: utf-8


class Foo(object):
    ...


def hello():
    print("world")

it seems bugs here. I guess you miss the situation that has no import statements? or importanize just tread # coding: utf-8 like a normal comment? and I add some import statements to that python file, it works fine. Have any idea about this? how can fix it gracefully?

Space added after empty comment line that doesn't allow for using pre-commit trim_trailing_whitespace. Each one makes a change each time and fails the run.

Would it be possible to add an option to disable adding whitespace as below:

# [whitespace added here when line previously ended with #]
# This file is part of pysqlite.
# [whitespace added here when line previously ended with #]

Instead of printing out imports, actually replace imports in given file

Running importanize on a file that has Unix style endings on Windows will change all line endings to CRLF. It should leave line ending alone and respect source file formatting.

Note: the line ending change only if there is something for importanize to do.

Black used to have this issue too but they fixed it psf/black#258

exit if import a, b are found since these are not valid Pep8

assuming foo.py:

__all__ = ['foo']

foo = 'foo'
bar = 'bar'

sometimes the following is desired:

from foo import *
from foo import bar

importanize combines above imports which is incorrect. it should only combine non-star imports.

I installed importanize using the instructions on your PyPi homepage, but I can't seem to get the cli interface to work from the command line. Chances are that I am using this feature wrongthough.

Running 'importanize test_file.py --config .importanizerc` on Windows produces

"""
test file
"""
from datetime import datetime, timedelta

import copy
import simplejson as json
import uuid
from dateutil import parser
from decimal import Decimal
from something.common.bla import constants

running same command on Linux produces

"""
test file
"""
import copy
import uuid
from datetime import datetime, timedelta
from decimal import Decimal

import simplejson as json
from dateutil import parser
from something.common.bla import constants

Windows env: Python 2.7.9 (default, Dec 10 2014, 12:24:55)

flake8==2.4.0
future==0.14.3
importanize==0.3
ipython==2.3.1
mccabe==0.3
pep8==1.5.7
pyflakes==0.8.1
pyreadline==2.0
six==1.9.0
virtualenv==1.9.1
virtualenvwrapper-win==1.1.5

.importanizerc contents

{
    "groups": [
        {
            "type": "stdlib"
        },
        {
            "type": "remainder"
        },
        {
            "type": "local"
        }
    ]
}

Order leafs within an import by their priority:

1. Constants (all caps)
2. Classes (CamelCase or First upper char)
3. Functions (lowercase)

from a import (
    CONSTANT,
    Class,
    function_name,
)

Hi. According to the suggestion of PEP8,

Imports should be grouped in the following order:
Standard library imports.
Related third party imports.
Local application/library specific imports.
You should put a blank line between each group of imports.

But during my use. my code:

from flask import g, request

from cmylocalpackage.some.model import func
from cmylocalpageage.another.model import helper

after importanize:

from cmylocalpackage.some.model import func
from cmylocalpageage.another.model import helper
from flask import g, request

That into a dictionary order. is this a bug?

Hi again,

I updated everything to the latest version to take advantage of the latest fixes and get the following issue. When i run:

importanize src/Packages/sqlite/pysqlite2/test/__init__.py -vv --config=setup.cfg

i get the following:

$ importanize src/Packages/sqlite/pysqlite2/test/__init__.py -vv --config=setup.cfg
Running importanize with RuntimeConfig(_paths=(), path_names=('src/Packages/sqlite/pysqlite2/test/__init__.py',), formatter_name=None, length=None, should_add_last_line=True, _config=None, root_config=[importanize]
path=setup.cfg
after_imports_new_lines=1
length=120
formatter=lines
groups=
  stdlib
  sitepackages
  remainder
  local
  packages:db,gui
exclude=
  *launcher\requirements*
add_imports=
allow_plugins=False
plugins=
  unused_imports, config_path='setup.cfg', is_subconfig_allowed=True, are_plugins_allowed=None, should_deactivate_piped_plugins=None, found_configs={}, verbosity=2, is_version_mode=False, is_list_mode=False, is_ci_mode=False, show_diff=False, is_print_mode=False, show_header=True, is_in_piped=False, is_out_piped=False, stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>, stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>)
Running with python c:\users\user\appdata\local\programs\python\python38-32\python.exe
Installed plugins: separate_libs, unused_imports
About to importanize src\Packages\sqlite\pysqlite2\test\__init__.py
Found 9 imports in src\Packages\sqlite\pysqlite2\test\__init__.py
Successfully importanized src\Packages\sqlite\pysqlite2\test\__init__.py
Nothing to do src\Packages\sqlite\pysqlite2\test\__init__.py

This is as should be.

When i run pre-commit with the one file i get this:

$ pre-commit run importanize
[WARNING] Unstaged files detected.
importanize..............................................................Failed
- hook id: importanize
- exit code: 1

Running importanize with RuntimeConfig(_paths=(), path_names=('src/Packages/sqlite/pysqlite2/test/__init__.py',), formatter_name=None, length=None, should_add_last_line=True, _config=None, root_config=[importanize]
path=setup.cfg
after_imports_new_lines=1
length=120
formatter=lines
groups=
  stdlib
  sitepackages
  remainder
  local
  packages:db,gui
exclude=
  *launcher\requirements*
add_imports=
allow_plugins=False
plugins=
  unused_imports, config_path='setup.cfg', is_subconfig_allowed=True, are_plugins_allowed=None, should_deactivate_piped_plugins=None, found_configs={}, verbosity=2, is_version_mode=False, is_list_mode=False, is_ci_mode=False, show_diff=False, is_print_mode=True, show_header=True, is_in_piped=True, is_out_piped=True, stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='cp1255'>, stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1255'>)
Running with python c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\scripts\python.exe
Installed plugins: separate_libs, unused_imports
About to importanize src\Packages\sqlite\pysqlite2\test\__init__.py
Found 9 imports in src\Packages\sqlite\pysqlite2\test\__init__.py
Successfully importanized src\Packages\sqlite\pysqlite2\test\__init__.py
==============================================
src\Packages\sqlite\pysqlite2\test\__init__.py
==============================================
Traceback (most recent call last):
  File "c:\users\user\appdata\local\programs\python\python38-32\Lib\runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\user\appdata\local\programs\python\python38-32\Lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\Scripts\importanize.exe\__main__.py", line 7, in <module>
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\importanize\main.py", line 174, in cli
    main(
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\importanize\main.py", line 227, in main
    return runtime_config.aggregator()
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\importanize\importanize.py", line 376, in __call__
    self.update(result)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\importanize\importanize.py", line 452, in update
    click.echo(result.organized, file=self.runtime_config.stdout)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\site-packages\click\utils.py", line 260, in echo
    file.write(message)
  File "c:\users\user\.cache\pre-commit\repo1c2ulo2f\py_env-python3\lib\encodings\cp1255.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\xe4' in position 133: character maps to <undefined>

[INFO] Restored changes from C:\Users\user/.cache\pre-commit\patch1577976846.

I'll draw attention to the following differences:

is_print_mode=False
is_in_piped=False,
is_out_piped=False
stdin/out has encoding encoding='utf-8'.

I am running in cygwin in both examples, but even if i run from cmd it gets utf8 as its encoding when run directly.

Is the difference (printing and different encoding) because of the way pre-commit runs importanize or due to some issue with click?

Simplify imports with same stems by combining them:

# from
from a import b
from a import c

# to
from a import b, c

Hello. Could you please suggest for some reasons why this config in pre-commit-config.yaml does not work:

repos:
  - repo: https://github.com/miki725/importanize/
    rev: 'master'
    hooks:
      - id: importanize
        name: importanize
        entry: python -m importanize
        language: python
        language_version: python3
        types: [python]
        args: [-v, --no-subconfig]
        additional_dependencies: [click, pluggy, pyflakes]

I always get 'Passed' from importanize hook, but no changes in files

I tried excluding a directory and it was still processed. I then tried a specific file and the same occurred. How can i exclude an entire directory?

c:\CodeShare\dir>importanize --config=setup.cfg launcher/requirements/bcrypt/__init__.py -vv

Running importanize with RuntimeConfig(_paths=(), path_names=('launcher/requirements/bcrypt/__init__.py',), formatter_name=None, length=None, should_add_last_line=True, _config=None, root_config=[importanize] path=setup.cfg after_imports_new_lines=2 length=120 formatter=lines groups= stdlib sitepackages remainder local packages:db,gui exclude= launcher/requirements/bcrypt/*.py add_imports= allow_plugins=True plugins= unused_imports, config_path='setup.cfg', is_subconfig_allowed=True, are_plugins_allowed=None, should_deactivate_piped_plugins=None, found_configs={}, verbosity=2, is_version_mode=False, is_list_mode=False, is_ci_mode=False, show_diff=False, is_print_mode=False, show_header=True, is_in_piped=False, is_out_piped=False, stdin=<_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>, stdout=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>) Running with python c:\users\user\appdata\local\programs\python\python38-32\python.exe Installed plugins: separate_libs, unused_imports About to importanize launcher\requirements\bcrypt\__init__.py Found 15 imports in launcher\requirements\bcrypt\__init__.py Successfully importanized launcher\requirements\bcrypt\__init__.py Importanized launcher\requirements\bcrypt\__init__.py

I tried the following

launcher/requirements/*

launcher/requirements/

launcher/requirements/*.py

launcher/requirements/bcrypt/*.py

in the config file for exclude.