Giter Club home page Giter Club logo

flake8-nb's Introduction

flake8-nb

PyPi Version Conda Version Supported Python Versions Pre-commit License

Actions Status Documentation Status Testing Coverage Documentation Coverage

Codacy Badge All Contributors Code style Python: black Binder

flake8 checking for jupyter notebooks.

This tool is mainly aimed towards writing tutorials/lecture material, where one might also want to show off bad practices and/or errors, while still keeping the rest of the code clean and without adding the complexity of tooling to the readers (see docs on cell tags).

Basically this is a hack on the flake8's Application class, which adds parsing and a cell based formatter for *.ipynb files.

This is NOT A PLUGIN but a stand alone CLI tool/pre-commit hook to be used instead of the flake8 command/hook.

Features

  • flake8 CLI tests for jupyter notebooks
  • Full base functionality of flake8 and its plugins
  • Input cell based error formatting (Execution count/code cell count/total cellcount)
  • Report fine tuning with cell-tags (flake8-noqa-tags see usage)
  • pre-commit hook

Examples

Default reporting

If you had a notebook with name example_notebook.ipynb, where the code cell which was executed as 34th cell (In[34]) had the following code:

bad_formatted_dict = {"missing":"space"}

running flake8_nb would result in the following output.

Execution count

$ flake8_nb example_notebook.ipynb
example_notebook.ipynb#In[34]:1:31: E231 missing whitespace after ':'

Custom reporting

If you prefer the reports to show the cell number rather then the execution count you can use the --notebook-cell-format option, given that the cell is the 5th code cell and 10th total cell (taking raw and markdown cells into account), you will get the following output.

Code cell count

$ flake8_nb --notebook-cell-format '{nb_path}:code_cell#{code_cell_count}' example_notebook.ipynb
example_notebook.ipynb:code_cell#5:1:31: E231 missing whitespace after ':'

Total cell count

$ flake8_nb --notebook-cell-format '{nb_path}:cell#{total_cell_count}' example_notebook.ipynb
example_notebook.ipynb:cell#10:1:31: E231 missing whitespace after ':'

Similar projects

  • nbQA: Run isort, pyupgrade, mypy, pylint, flake8, mdformat, black, blacken-docs, and more on Jupyter Notebooks

Contributors ✨

Thanks goes to these wonderful people (emoji key):

Sebastian Weigand
Sebastian Weigand

πŸ’» πŸ€” 🚧 πŸ“† πŸš‡ ⚠️ πŸ“–
Jt Miclat
Jt Miclat

πŸ›
Philipp Eisenhauer
Philipp Eisenhauer

πŸ›
Shoma Okamoto
Shoma Okamoto

⚠️
Marco Gorelli
Marco Gorelli

πŸ”§ πŸ“–
Tony Hirst
Tony Hirst

πŸ€”
Dobatymo
Dobatymo

πŸ›
Alp ArΔ±bal
Alp ArΔ±bal

πŸ›
1kastner
1kastner

πŸ›
Dominique Sydow
Dominique Sydow

πŸ›
Liam Keegan
Liam Keegan

πŸ› πŸ’»

This project follows the all-contributors specification. Contributions of any kind welcome!

flake8-nb's People

Contributors

allcontributors[bot] avatar deepsourcebot avatar dependabot-preview[bot] avatar dependabot[bot] avatar lgtm-com[bot] avatar lkeegan avatar marcogorelli avatar pyup-bot avatar s-weigand avatar shmokmt avatar sourcery-ai[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

flake8-nb's Issues

[BUG] pre-commit.ci `Read-only file system` error for v0.5.2

Describe the bug

v0.5.2 gives an error when ran on pre-commit.ci:

https://results.pre-commit.ci/run/github/515183454/1679944727.JClYHld9R86c8Z37AT7JVQ

flake8-nb................................................................Failed
- hook id: flake8-nb
- exit code: 1

Traceback (most recent call last):
  File "/pc/clone/YcfFcd-3Q7uLovJnWJL3mg/py_env-python3/bin/flake8-nb", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/pc/clone/YcfFcd-3Q7uLovJnWJL3mg/py_env-python3/lib/python3.11/site-packages/flake8_nb/__main__.py", line 21, in main
    app = Flake8NbApplication()
          ^^^^^^^^^^^^^^^^^^^^^
  File "/pc/clone/YcfFcd-3Q7uLovJnWJL3mg/py_env-python3/lib/python3.11/site-packages/flake8_nb/flake8_integration/cli.py", line 197, in __init__
    hack_config_module()
  File "/pc/clone/YcfFcd-3Q7uLovJnWJL3mg/py_env-python3/lib/python3.11/site-packages/flake8_nb/flake8_integration/cli.py", line 158, in hack_config_module
    hacked_config_path.write_text(hacked_config_source)
  File "/usr/lib/python3.11/pathlib.py", line 1078, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 1044, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 30] Read-only file system: '/pc/clone/YcfFcd-3Q7uLovJnWJL3mg/py_env-python3/lib/python3.11/site-packages/flake8_nb/flake8_integration/hacked_config.py'

I think this is due to #251 writing the hacked config to disk, and pre-commit.ci running on a read-only filesystem

Clarification regarding "hack"

Hello, this project looks really interesting, but to me personally the use of "hack" in the README puts me off.

Basically this is a hack on the flake8's Application class

I propose to have this rewritten to

Basically this subclasses flake8's Application class

If you agree, I'm happy to create a PR. :)

Edit: I see there are a lot of references to "hacks" in the source code. Still, the badges are quite convincing, so maybe "hack" is an understatement.

image

[Security] Workflow test.yml is using vulnerable action s-weigand/setup-conda

The workflow test.yml is referencing action s-weigand/setup-conda using references v1. However this reference is missing the commit a30654e576ab9e21a25825bf7a5d5f2a9b95b202 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

[Security] Workflow test.yml is using vulnerable action s-weigand/setup-conda

The workflow test.yml is referencing action s-weigand/setup-conda using references v1. However this reference is missing the commit a30654e576ab9e21a25825bf7a5d5f2a9b95b202 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

[BUG] New version v0.5.0 ignores `setup.cfg`?

Thanks a lot for this tool! We use it across several of our projects in the @volkamerlab and @wolberlab organizations.

Describe the bug

Is it possible that flake8-nb configurations from setup.cfg are ignored since the version update from v0.4.0 to v0.5.0?
Do I have to make any updates to our procedure as described below?

To Reproduce

In the CI for our dynophores package, we test notebooks with

flake8-nb --config setup.cfg docs/tutorials/*.ipynb

The flake8-nb configuration looks like this in setup.cfg:

[flake8_nb]
max-line-length = 99
ignore = E402,E703,W503
exclude = 
    docs/.ipynb_checkpoints

Last week, the CI passed; this week, we see the following output:

$ flake8-nb --config setup.cfg docs/tutorials/*.ipynb
docs/tutorials/explore_view3d.ipynb#In[2]:1:1: E402 module level import not at top of file
...
docs/tutorials/explore_view3d.ipynb#In[8]:1:57: E703 statement ends with a semicolon
...

Expected behavior
flake8-nb ignores E402 and E703 errors as defined in the config file.

Desktop (please complete the following information):
Reproduced on macOS 12.3 and GitHub Actions server

Identifying code cells in un-run notebooks

If the flake8-nb tool is run over a notebook with unrun cells, all the cell references seem to be returned with an an empty [ ] cell reference.

It would be useful if there were an option to at least add an accession number relating to each reported code cell, or to be able to override perhaps override cell run numbers with the code cell accession number?

[BUG] Not reading `flake8` config from `tox.ini`

Describe the bug

I would expect flake8-nb to honor the flake8 configuration set in tox.ini.

When having a mix of Python (.py) and Notebook (.ipynb) files in a project/repo, it is nice to use the same style/rules in all the code. 😊

As an examle, black[jupyter] honors the default Black configuration file if it is present.

To Reproduce

Write a tox.ini file with this content:

[flake8]
max-line-length = 20

Then write this code in a notebook cell:

a_long_line = "............................"

Run flake8-nb and see no error.

Expected behavior

E501 line too long (44 > 20 characters)

--exclude flag

  • flake8-nb version: 0.1.4 (mccabe: 0.6.1, pycodestyle: 2.5.0, pyflakes: 2.1.1) CPython 3.7.3 on Linux

  • Python version: 3.7.6

  • Operating System: Ubuntu

Description

We are trying to exclude subdirectories from the checks.

What I Did

First of all, thank you very much for providing such a useful tool! We are trying to integrate it into our groups's research workflow and are struggling to exclude subdirectories from the checks. See below.

(base) peisenha@heracles:~/Desktop/test$ flake8-nb --exclude=subdir
main.ipynb#In[1]:1:1: F401 'numpy as np' imported but unused
subdir/subdir.ipynb#In[ ]:1:1: F401 'numpy as np' imported but unused

Any guidance is highly appreciated. Thanks, @peisenha @rafaelsuchy

Generate annotated notebook with embedded flake8 reports

To make it easier to check notebooks, it would be useful to generate notebooks annotated with flake8 reports at the cell level, either as a tagged report cell immediately after each code cell, or by clearing the notebook of all code cell outputs and then injecting the flake8 report into appropriate code cell outputs.

If reports are added to code cell outputs, it would be useful if they were styled with pink warning style such as used in the stale/deprecated pycodestyle_magic:

image

`.flake8_nb` does not work in certain cases

I'm using flake8_nb version 0.4.0.

I have a subdirectory in my project called ./docs/ which contains Jupyter notebooks that fail the flaking, e.g. ./docs/fail_flake8.ipynb.

My .flake8_nb configuration file includes docs/*:

[flake8_nb]
exclude =
    .*,
    _*,
    docs/*

Nonetheless, flake8_nb reports failures on the ./docs/fail_flake8.ipynb notebook.

[BUG] configuration files ignored?

Describe the bug
I have a setup.cfg file in my project that seems to be ignored by flake8_nb (but not by flake8).

To Reproduce
Steps to reproduce the behavior:

Create a setup.cfg file containing

[flake8]
ignore=
    E501 # line too long 
  • run flake8 on a *.py file with a line that is too long
  • fun flake8_nb on a *.ipynb file with a line that is too long

Expected behavior
No E501 error.

Desktop (please complete the following information):

  • OS: linux
flake8                    3.8.4                      py_0    conda-forge
flake8-nb                 0.3.1              pyhd8ed1ab_0    conda-forge

Additional context
Add any other context about the problem here.

[BUG] .flake8 file is ignored

Describe the bug
The configuration set inside the .flake8 file is being ignored.

To Reproduce
Create test.py

def this_is_a_super_long_method_name_for_a_function_this_is_a_super_long_method():
    pass

.flake8

[flake8]
ignore =
    E501
max-line-length = 88

Expected behavior
The error E501 should not be raised.

Desktop (please complete the following information):

  • OS: ubuntu
  • Version 22.04 LTS

Additional context
Running flake8 works as intended.
Running flake8-nb --config .flake8 or flake8-nb --append-config .flake8 wont not work either.

[BUG] ModuleNotFoundError for ipython_genutils

Describe the bug
After the recent ipython version bump, our flake8-nb precommit hooks started failing with the error: ModuleNotFoundError: No module named 'ipython_genutils'.

To Reproduce
Steps to reproduce the behavior:

conda create -n test python=3.8
conda activate test
pip install flake8-nb
python
>>> import flake8_nb

Expected behavior
flake8_nb should be imported without any errors.

Screenshots
N/A

Desktop (please complete the following information):

  • OS: Ubuntu
  • Version 20.04

Additional context
This might be due to ipykernel not stating its dependencies properly, see here.

ENH: pre-commit hook?

Would it be welcome if I added a .pre-commit-hook.yaml file, so this tool can be used as a pre-commit hook?

Improve explanation of how to ignore per file if the file name contains spaces

This discussion started at PyCQA/flake8#1519.

I have a file called 01 Einfuehrung in Python.ipynb and I would like to ignore specific issues here. In the file, I harm some design principles purposefully and show the user how this creates an issue lateron. However, my per-file-ignores are not considered. I am not sure how exactly the glob pattern is working. There are a few examples, e.g. at https://flake8.pycqa.org/en/latest/user/options.html?highlight=glob%20#cmdoption-flake8-filename but they are not exhausting enough for me.

Extract of my config:

per-file-ignores =
    "01 Einfuehrung in Python.ipynb": F403, F405, E116, E731

With or without hyphens around the file name, in both cases the extract of the console output looks like this:

01-einfuehrung-in-toolbox\01 einfuehrung in python.ipynb#In[14]:1:19: E226 missing whitespace around arithmetic operator
01-einfuehrung-in-toolbox\01 einfuehrung in python.ipynb#In[14]:3:1: E731 do not assign a lambda expression, use a def
01-einfuehrung-in-toolbox\01 einfuehrung in python.ipynb#In[23]:2:23: E226 missing whitespace around arithmetic operator
01-einfuehrung-in-toolbox\01 einfuehrung in python.ipynb#In[23]:4:1: F403 'from math import *' used; unable to detect undefined names
01-einfuehrung-in-toolbox\01 einfuehrung in python.ipynb#In[23]:8:7: F405 'sin' may be undefined, or defined from star imports: math
01-einfuehrung-in-toolbox\01 einfuehrung in python.ipynb#In[23]:8:11: F405 'pi' may be undefined, or defined from star imports: math
01-einfuehrung-in-toolbox\01 einfuehrung in python.ipynb#In[23]:8:13: E226 missing whitespace around arithmetic operator

So I would wish to be able how exactly the glob patterns work but in the documentation I could not find sufficient information how to deal with spaces in file names. Thanks a lot for your assistance!

[BUG] Cells which start with `%%html` lead to SyntaxError

Describe the bug
%%html at the start of a cell can be used to output raw html. flake8-nb fails with E999 SyntaxError: invalid syntax.

To Reproduce
Have a cell like

%%html
<a href="#">asd</a>

in the notebook and run flake8-nb

Expected behavior
These cells should be ignored like other non-code cells.

Desktop (please complete the following information):

  • OS: Windows 10 x64
  • Version: Python 3.8.10 with flake8-nb==0.3.1

[BUG] Excluded directory is checked.

Describe the bug
Directories that I want to exclude and that were previously excluded are not scanned by flake8_nb.

To Reproduce
Steps to reproduce the behavior:

  1. Create a .flake8_nb file with the following content:
exclude =
    *.git*
    *venv*
    *.ipynb_checkpoints*
    *.virtual_documents*
    *build*
    *.py
  1. Run flake8_nb from the CI
  2. Observe how it enters the directory .virtual_documents where it should not end up!

Expected behavior
The excluded directories, such as .virtual_documents , should not be checked.

Desktop (please complete the following information):

  • OS: Windows 10
  • Version: flake8_nb-0.5.1

Additional context
This was working with flake8-nb-0.4.0

[BUG] Freshly installed flake8_nb won't properly start

Describe the bug

I just run flake8_nb from the CLI and the following error message shows up:

Traceback (most recent call last):   File "/home/runner/.local/bin/flake8_nb", line 8, in <module>     sys.exit(main())   File "/home/runner/.local/lib/python3.8/site-packages/flake8_nb/__main__.py", line 30, in main     app = Flake8NbApplication()   File "/home/runner/.local/lib/python3.8/site-packages/flake8_nb/flake8_integration/cli.py", line 78, in __init__     super().__init__(program, version) TypeError: __init__() takes 1 positional argument but 3 were given

There seems to be an issue with the dependencies, because this is what pip showed me:

2022-08-01T09:21:01.3609612Z INFO: pip is looking at multiple versions of flake8-nb to determine which version is compatible with other requirements. This could take a while.
2022-08-01T09:21:01.3669041Z Collecting flake8_nb
2022-08-01T09:21:01.3761741Z   Downloading flake8_nb-0.3.1-py3-none-any.whl (22 kB)
2022-08-01T09:21:01.4176481Z   Downloading flake8_nb-0.3.0-py3-none-any.whl (22 kB)
2022-08-01T09:21:01.4629941Z   Downloading flake8_nb-0.2.7-py3-none-any.whl (21 kB)
2022-08-01T09:21:01.5077060Z   Downloading flake8_nb-0.2.6-py3-none-any.whl (22 kB)
2022-08-01T09:21:01.5519271Z   Downloading flake8_nb-0.2.5-py3-none-any.whl (22 kB)
2022-08-01T09:21:01.5935728Z   Downloading flake8_nb-0.2.4-py3-none-any.whl (22 kB)
2022-08-01T09:21:01.6288193Z   Downloading flake8_nb-0.2.3-py3-none-any.whl (22 kB)
2022-08-01T09:21:01.6570612Z INFO: pip is looking at multiple versions of flake8-nb to determine which version is compatible with other requirements. This could take a while.
2022-08-01T09:21:01.6749410Z   Downloading flake8_nb-0.2.1-py3-none-any.whl (21 kB)
2022-08-01T09:21:01.7085630Z   Downloading flake8_nb-0.2.0-py3-none-any.whl (21 kB)
2022-08-01T09:21:01.7450504Z   Downloading flake8_nb-0.1.8-py3-none-any.whl (21 kB)
2022-08-01T09:21:01.7788708Z   Downloading flake8_nb-0.1.7-py3-none-any.whl (21 kB)
2022-08-01T09:21:01.8233164Z   Downloading flake8_nb-0.1.6-py3-none-any.whl (21 kB)
2022-08-01T09:21:01.8482448Z INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.

Are you aware of any dependency issue that might cause this behavior?

To Reproduce
Steps to reproduce the behavior:

  1. Setup github workflow without a fixed version
  2. Run flake8_nb from CLI
  3. See error

Expected behavior
flake8_nb should run like it has done in the past and pip should install the latest version. Thanks for the project, it does help me a lot!

flake8-nb is reading `.flake8_nb` instead of reading `.flake8`

  • flake8-nb version: 0.1.4 .
  • Python version: 3.7.7
  • Operating System: Ubuntu 18.04 LTS

Description

HI thanks for a cool package! I was looking at the docs it mentions to configure flake8_nb you can use .flake8 https://github.com/s-weigand/flake8-nb/blob/master/docs/usage.rst. After experimenting and debugging it seems like it will not read .flake8 but instead reads .flake8_nb

What I Did

Generate a notebook with more than 79 char lines.
Create a .flake8 containing the following

[flake8_nb]
ignore = E501

Run the flake8-nb command

flake8_nb notebook.ipynb

This generates the following warnings

notebook1ipynb#In[1]:1:80: E501 line too long (84 > 79 characters)

Move .flake8 to .flake8_nb to fix.

Not 100% if this a docs issue or an actual bug

[BUG] Certain cells that start with %%bash lead to Syntax error, even with 'flake8-noqa-cell' tag

Describe the bug
Certain cells that start with %%bash lead to Syntax error, even with 'flake8-noqa-cell' tag.

To Reproduce
Have a cell like:

%%bash 
cd /home/myhome

This works, however:

%%bash 
cd /home/myhome/

leads to syntax error. Another example:

%%bash
for type in ct mr
do
    echo $type
done

Expected behavior
No syntax error when 'flake8-noqa-cell' tag is used.

Desktop

  • OS: Rocky Linux 8.5
  • Python: 3.11.3
  • flake8-nb: 0.5.3

Temporary Quickfix
The problem disappeared after I marked my bash cells as raw. However, this is not desirable.

[Security] Workflow test.yml is using vulnerable action pre-commit/action

The workflow test.yml is referencing action pre-commit/action using references v2.0.3. However this reference is missing the commit 80db042ff08cdddbbbfb6f89c06a6bfc4dddf0b7 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    πŸ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. πŸ“ŠπŸ“ˆπŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❀️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.