Giter Club home page Giter Club logo

docopt-ng's Introduction

docopt-ng creates beautiful command-line interfaces

Test codecov image Jazzband Ruff

docopt-ng is a fork of the original docopt, now maintained by the jazzband project. Now with maintenance, typehints, and complete test coverage!

docopt-ng helps you create beautiful command-line interfaces:

"""Naval Fate.

Usage:
  naval_fate.py ship new <name>...
  naval_fate.py ship <name> move <x> <y> [--speed=<kn>]
  naval_fate.py ship shoot <x> <y>
  naval_fate.py mine (set|remove) <x> <y> [--moored | --drifting]
  naval_fate.py (-h | --help)
  naval_fate.py --version

Options:
  -h --help     Show this screen.
  --version     Show version.
  --speed=<kn>  Speed in knots [default: 10].
  --moored      Moored (anchored) mine.
  --drifting    Drifting mine.

"""
from docopt import docopt

if __name__ == "__main__":
    argv = ["ship", "Guardian", "move", "100", "150", "--speed=15"]
    arguments = docopt(__doc__, argv)
    print(arguments)

results in:

{'--drifting': False,
 '--help': False,
 '--moored': False,
 '--speed': '15',
 '--version': False,
 '<name>': ['Guardian'],
 '<x>': '100',
 '<y>': '150',
 'mine': False,
 'move': True,
 'new': False,
 'remove': False,
 'set': False,
 'ship': True,
 'shoot': False}

Beat that! The option parser is generated based on the docstring above that is passed to docopt function. docopt parses the usage pattern ("Usage: ...") and option descriptions (lines starting with dash "-") and ensures that the program invocation matches the usage pattern; it parses options, arguments and commands based on that. The basic idea is that a good help message has all necessary information in it to make a parser.

Also, PEP 257 recommends putting help message in the module docstrings.

Installation

Use pip:

python -m pip install docopt-ng

docopt-ng is tested with Python 3.7+.

API

def docopt(
    docstring: str,
    argv: list[str] | str | None = None,
    default_help: bool = True,
    version: Any = None,
    options_first: bool = False,
) -> ParsedOptions:

docopt takes a docstring, and 4 optional arguments:

  • docstring is a string that contains a help message that will be used to create the option parser. The simple rules of how to write such a help message are given in next sections. Typically you would just use __doc__.

  • argv is an optional argument vector; by default docopt uses the argument vector passed to your program (sys.argv[1:]). Alternatively you can supply a list of strings like ["--verbose", "-o", "hai.txt"], or a single string that will be split on spaces like "--verbose -o hai.txt".

  • default_help, by default True, specifies whether the parser should automatically print the help message (supplied as doc) and terminate, in case -h or --help option is encountered (options should exist in usage pattern, more on that below). If you want to handle -h or --help options manually (as other options), set help=False.

  • version, by default None, is an optional argument that specifies the version of your program. If supplied, then, (assuming --version option is mentioned in usage pattern) when parser encounters the --version option, it will print the supplied version and terminate. version could be any printable object, but most likely a string, e.g. "2.1.0rc1".

    Note, when docopt is set to automatically handle -h, --help and --version options, you still need to mention them in usage pattern for this to work. Also, for your users to know about them.

  • options_first, by default False. If set to True will disallow mixing options and positional argument. I.e. after first positional argument, all arguments will be interpreted as positional even if the look like options. This can be used for strict compatibility with POSIX, or if you want to dispatch your arguments to other programs.

The return value is a simple dictionary with options, arguments and commands as keys, spelled exactly like in your help message. Long versions of options are given priority. Furthermore, dot notation is supported, with preceeding dashes (-) and surrounding brackets (<>) ignored, for example arguments.drifting or arguments.x.

Help message format

Help message consists of 2 parts:

  • Usage pattern, e.g.:

    Usage: my_program.py [-hso FILE] [--quiet | --verbose] [INPUT ...]
    
  • Option descriptions, e.g.:

    -h --help    show this
    -s --sorted  sorted output
    -o FILE      specify output file [default: ./test.txt]
    --quiet      print less text
    --verbose    print more text
    

Their format is described below; other text is ignored.

Usage pattern format

Usage pattern is a substring of doc that starts with usage: (case insensitive) and ends with a visibly empty line. Minimum example:

"""Usage: my_program.py

"""

The first word after usage: is interpreted as your program's name. You can specify your program's name several times to signify several exclusive patterns:

"""Usage: my_program.py FILE
          my_program.py COUNT FILE

"""

Each pattern can consist of the following elements:

  • <arguments>, ARGUMENTS. Arguments are specified as either upper-case words, e.g. my_program.py CONTENT-PATH or words surrounded by angular brackets: my_program.py <content-path>.
  • --options. Options are words started with dash (-), e.g. --output, -o. You can "stack" several of one-letter options, e.g. -oiv which will be the same as -o -i -v. The options can have arguments, e.g. --input=FILE or -i FILE or even -iFILE. However it is important that you specify option descriptions if you want your option to have an argument, a default value, or specify synonymous short/long versions of the option (see next section on option descriptions).
  • commands are words that do not follow the described above conventions of --options or <arguments> or ARGUMENTS, plus two special commands: dash "-" and double dash "--" (see below).

Use the following constructs to specify patterns:

  • [ ] (brackets) optional elements. e.g.: my_program.py [-hvqo FILE]
  • ( ) (parens) required elements. All elements that are not put in [ ] are also required, e.g.: my_program.py --path=<path> <file>... is the same as my_program.py (--path=<path> <file>...). (Note, "required options" might be not a good idea for your users).
  • | (pipe) mutually exclusive elements. Group them using ( ) if one of the mutually exclusive elements is required: my_program.py (--clockwise | --counter-clockwise) TIME. Group them using [ ] if none of the mutually-exclusive elements are required: my_program.py [--left | --right].
  • ... (ellipsis) one or more elements. To specify that arbitrary number of repeating elements could be accepted, use ellipsis (...), e.g. my_program.py FILE ... means one or more FILE-s are accepted. If you want to accept zero or more elements, use brackets, e.g.: my_program.py [FILE ...]. Ellipsis works as a unary operator on the expression to the left.
  • [options] (case sensitive) shortcut for any options. You can use it if you want to specify that the usage pattern could be provided with any options defined below in the option-descriptions and do not want to enumerate them all in usage-pattern.
  • "[--]". Double dash "--" is used by convention to separate positional arguments that can be mistaken for options. In order to support this convention add "[--]" to your usage patterns.
  • "[-]". Single dash "-" is used by convention to signify that stdin is used instead of a file. To support this add "[-]" to your usage patterns. "-" acts as a normal command.

If your pattern allows to match argument-less option (a flag) several times:

Usage: my_program.py [-v | -vv | -vvv]

then number of occurrences of the option will be counted. I.e. args["-v"] will be 2 if program was invoked as my_program -vv. Same works for commands.

If your usage patterns allows to match same-named option with argument or positional argument several times, the matched arguments will be collected into a list:

Usage: my_program.py <file> <file> --path=<path>...

I.e. invoked with my_program.py file1 file2 --path=./here --path=./there the returned dict will contain args["<file>"] == ["file1", "file2"] and args["--path"] == ["./here", "./there"].

Option descriptions format

Option descriptions consist of a list of options that you put below your usage patterns.

It is necessary to list option descriptions in order to specify:

  • synonymous short and long options,
  • if an option has an argument,
  • if option's argument has a default value.

The rules are as follows:

  • Every line in doc that starts with - or -- (not counting spaces) is treated as an option description, e.g.:

    Options:
      --verbose   # GOOD
      -o FILE     # GOOD
    Other: --bad  # BAD, line does not start with dash "-"
    
  • To specify that option has an argument, put a word describing that argument after space (or equals "=" sign) as shown below. Follow either <angular-brackets> or UPPER-CASE convention for options' arguments. You can use comma if you want to separate options. In the example below, both lines are valid, however you are recommended to stick to a single style.:

    -o FILE --output=FILE       # without comma, with "=" sign
    -i <file>, --input <file>   # with comma, without "=" sign
    
  • Use two spaces to separate options with their informal description:

    --verbose More text.   # BAD, will be treated as if verbose option had
                           # an argument "More", so use 2 spaces instead
    -q        Quit.        # GOOD
    -o FILE   Output file. # GOOD
    --stdout  Use stdout.  # GOOD, 2 spaces
    
  • If you want to set a default value for an option with an argument, put it into the option-description, in form [default: <my-default-value>]:

    --coefficient=K  The K coefficient [default: 2.95]
    --output=FILE    Output file [default: test.txt]
    --directory=DIR  Some directory [default: ./]
    
  • If the option is not repeatable, the value inside [default: ...] will be interpreted as string. If it is repeatable, it will be splited into a list on whitespace:

    Usage: my_program.py [--repeatable=<arg> --repeatable=<arg>]
                         [--another-repeatable=<arg>]...
                         [--not-repeatable=<arg>]
    
    # will be ["./here", "./there"]
    --repeatable=<arg>          [default: ./here ./there]
    
    # will be ["./here"]
    --another-repeatable=<arg>  [default: ./here]
    
    # will be "./here ./there", because it is not repeatable
    --not-repeatable=<arg>      [default: ./here ./there]
    

Examples

We have an extensive list of examples which cover every aspect of functionality of docopt-ng. Try them out, read the source if in doubt.

Development

We would love to hear what you think about docopt-ng on our issues page. Make pull requests, report bugs, and suggest ideas.

To setup your dev environment, fork this repo and clone it locally. We use pdm to manage the project, so install that first.

Then create a virtual env, install dev requirements and the package itself as editable, then install the pre-commit hooks:

pdm sync --dev --group dev
pdm run pre-commit install

Useful testing, linting, and formatting commands:

pdm run pytest
pdm run ruff check .
pdm run ruff format .

docopt-ng's People

Contributors

andreypopp avatar cjwelborn avatar graingert avatar h4l avatar itdaniher avatar jazzband-bot avatar jeffrimko avatar jezdez avatar johari avatar johnscillieri avatar jonlundy avatar keleshev avatar ladsgroup avatar laurentb avatar lenesk avatar mboersma avatar met48 avatar nickcrews avatar nyurik avatar oconnor663 avatar pre-commit-ci[bot] avatar renefritze avatar renesac avatar schmir avatar shabbyrobe avatar sturmianseq avatar thewawar avatar thosrtanner avatar tubaman avatar vlad-shcherbina avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

docopt-ng's Issues

Integrate docopt-dispatch in this project

I have started using docopt-dispatch and I find it very intriguing. It makes docopt CLI code super-concise and readable.

Unfortunately, that package both has docopt as a hard dependency and is unmaintained for 7+ years.

If we integrated the dispatching project in docopt-ng some CLI code could look like this, preserving the compatibility with the original docopt spec and staying aligned with the syntax of the original dispatch project:

"""Run something in development or production mode.

Usage: run.py --development <host> <port>
       run.py --production <host> <port>
       run.py remote add <item>
       run.py remote delete <item>
"""
from docopt import dispatch


@dispatch.on('--development')
def development(host, port, **kwargs):
    print('in *development* mode')


@dispatch.on('--production')
def development(host, port, **kwargs):
    print('in *production* mode')


if __name__ == 'main':
    dispatch(__doc__)

Would you accept a PR that adds the dispatching feature to docopt-ng?

docopt-ng fails to parse usage string that worked with docopt

I've got an old CLI program which uses OG docopt. I'm giving it a bit of minor TLC to refresh the tooling, and I tried switching to docopt-ng, but -ng fails to parse my usage string for some reason:

vscode@46fada5e45f4 /w/rnginline ((bef5200c…)) [127]> poetry run ipython
Python 3.10.5 (main, Jun  6 2022, 12:05:50) [GCC 9.5.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import docopt

In [2]: docopt.__version__
Out[2]: '0.8.1'

In [3]: from rnginline import cmdline

In [4]: docopt.docopt(cmdline.__doc__, argv=['rnginline', '--no-libxml2-compat', '/some/file'])
An exception has occurred, use %tb to see the full traceback.

DocoptExit: Warning: found unmatched (duplicate?) arguments [Option(None, '--no-libxml2-compat', 0, True)]
usage: rnginline [options] <rng-src> [<rng-output>]
       rnginline [options] --stdin [<rng-output>]

/home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3406: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

In [5]:                                                                                                                                                                                                            
Do you really want to exit ([y]/n)? y
vscode@46fada5e45f4 /w/rnginline ((bef5200c…))> poetry run pip uninstall docopt-ng
Found existing installation: docopt-ng 0.8.1
Uninstalling docopt-ng-0.8.1:
  Would remove:
    /home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/docopt/*
    /home/vscode/.cache/pypoetry/virtualenvs/rnginline--qKLlanv-py3.10/lib/python3.10/site-packages/docopt_ng-0.8.1.dist-info/*
Proceed (Y/n)? y
  Successfully uninstalled docopt-ng-0.8.1
vscode@46fada5e45f4 /w/rnginline ((bef5200c…))> poetry run pip install docopt
Collecting docopt
  Using cached docopt-0.6.2-py2.py3-none-any.whl
Installing collected packages: docopt
Successfully installed docopt-0.6.2

vscode@46fada5e45f4 /w/rnginline ((bef5200c…))> poetry run ipython
Python 3.10.5 (main, Jun  6 2022, 12:05:50) [GCC 9.5.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import docopt

In [2]: docopt.__version__
Out[2]: '0.6.2'

In [3]: from rnginline import cmdline

In [4]: docopt.docopt(cmdline.__doc__, argv=['rnginline', '--no-libxml2-compat', '/some/file'])
Out[4]: 
{'-': None,
 '--base-uri': None,
 '--default-base-uri': None,
 '--help': False,
 '--no-libxml2-compat': True,
 '--stdin': False,
 '--traceback': False,
 '--version': False,
 '<rng-output>': '/some/file',
 '<rng-src>': 'rnginline'}

This is the usage string: https://github.com/h4l/rnginline/blob/b1d1c8cda2a17d46627309950f2442021749c07e/rnginline/cmdline.py#L14

I really appreciate your efforts in keeping docopt going, It's a great library.

error messages and autocorrect?

regarding the changelog, there should be smarter error messages and an autocorrect function, but still when iam using more_magic=True I will get only the usage message returned and a meaningless message already reported in #5

fix pytest 5.4+ deprecation error

Deprecation warning reported by pytest during testing: direct construction is deprecated as of pytest 5.4 and will be removed in the future. Use of the [Node].from_parent constructor is recommended."

example:

$ py setup test

conftest.py:14
  .../docopt/conftest.py:14: PytestDeprecationWarning: direct construction of DocoptTestFile has been deprecated, please use DocoptTestFile.from_parent

    return DocoptTestFile(path, parent)

conftest.py:44: 175 tests with warnings
  .../docopt/conftest.py:44: PytestDeprecationWarning: direct construction of DocoptTestItem has been deprecated, please use

DocoptTestItem.from_parent
    yield DocoptTestItem("%s(%d)" % (name, index), self, doc, case)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

When docopt and docopt-ng are in the same venv, docopt takes precedence

If you install
pipenv install pipreqs pip-upgrader docopt-ng
Which sometimes happens, you can't put everything into system or pipx.

Then apps use the wrong docopt.

$ python
>>> import docopt
>>> docopt.__file__
'C:\\Users\\matth\\.virtualenvs\\demo_doc-j-5bRj6B\\lib\\site-packages\\docopt.py'
>>> exit()

$ pip install docopt-ng
Requirement already satisfied: docopt-ng in c:\users\matth\.virtualenvs\demo_doc-j-5brj6b\lib\site-packages (0.7.2)

In C# there was a way to handle this and ensure that two similarly named dlls weren't confused with each other, I don't know what the python way is to handle this scenario.

Claim `docopt` name on pypi (PEP-541)

If this project is a more maintained version of the original docopt (now unmaintained), I think that this project should claim the pypi docopt name.

fork

the idea of forking docopt because it's unmaintained it's cool but the idea of docopt is to be available in tons of languages, while docopt-ng only forked the python implementation.
Do you plan to fork all others https://github.com/docopt?

Spaces in default value

I would like to set a default value containing spaces like in - . It does work:

--name-sep=<sep>  name field separator [default:  - ]   

But it is not so pleasing to the eye and prone to bad interpretation. I don't know what is the standard here, but I would imagine accepting a quoted default value, as in:

--name-sep=<sep>  name field separator [default: " - "]   

Could be one way to go. Or am I missing something ?

backwards incompatible assertion

After migrating from docopt 0.6.2 to docopt-ng 0.7.2, this help caused an assertion at https://github.com/bazaar-projects/docopt-ng/blob/bbed40a2335686d2e14ac0e6c3188374dc4784da/docopt.py#L616

"""
Evaluate tile generation performance against a PostgreSQL database.

Usage:
  test-perf <tileset> [-t <set>] [-h <host>]

  <tileset>             Tileset definition yaml file

Options:
  -t --test=<set>       Which predefined test to run.  [default: us-across]

PostgreSQL Connection Options:
  -h --pghost=<host>    Postgres hostname.
"""

Allow dumping of parse tree, to make it easier to check what docopt will actually parse

As mentioned in #54 by @NickCrews and @Sylvain303 , I'm splitting of the issue to ensure it doesn't get lost.

Original issue

Having used the original docopt - so excuse me if you already support this.

I would really love for a special argument that just dumps a structured (json / yaml) representation of the argument parser, that easily allows me to check what is parsed and wether changes I did might have changed the way args will be parsed.

This could also allow diffing between the way stuff is parsed in versions (as a kind of acceptance tests).

That is something I always missed when using the original docopt, that I always just used very rarely and was having a lot of trouble remembering the intricacies of the docopt configuration language.

@NickCrews answer

@dwt that feature makes sense, and should be theoretically be possible.

If you want to discuss this further, please make a new feature request issue, so this thread doesn't go off topic. But in short:
I am not going to implement that (don't have the time time or interest), but I would consider merging a PR if

  • I didn't have to do huge amounts of edits/review on it
  • no additional dependencies, or very lightweight ones
  • didn't look like a huge maintenance burden. I'm skeptical this would be possible
  • didn't break existing users

I don't know of any common json schema that would work for this. I would love to not reinvent the wheel here, so would be worth it to explore what other. It also seems to me like "checking for a change in parsing behavior" would not be that useful on its own. If the behavior started off incorrect, you aren't actually catching that. It seems more useful to actually test assert my_parse("myprogram new --name foo") == , even if it might be a bit more verbose. Also would serve as documentation.

@Sylvain303 answer

Hello, I'm not very active yet on docopts coding (golang version for bash) watch-out the extra 'S'. But I
try to implement such thing: dumping what was actually parsed by docopt parser.

Could you create an new issue with some sample of what you would love to see outputted? That would be very useful. 🤩

Some ideas about the expected output

To be honest I do not have a precise idea what the correct output format would be. My initial idea was, that all implementations should be able to dump that exact same parse tree, to then be able to compare them to each other and have them act as a regression test suite for each other.

However that use case would be mostly about machines being able to compare this, not necessarily humans. I guess some json/yaml, that just documents all the options would be mostly fine here? The format should be as simple as possible to ease interoperability.

The other use case I had for myself is that it can be hard for a casual user to actually write the correct docopt syntax, as it has it's intricacies that can be hard to get right.

To help with that, I'm not quite sure what would help best? To me generating the source for an equivalent argparse based parser would be quite helpful, but having something declarative gives me a better belly feeling.

To be honest, the first thing I would do if this was my project is probably to just dump the tree of the internal data structure that docopt-ng uses to then parse command lines, then take a look at that and see what can be achieved from there. This would already give you internal comparability between different versions of the parser and would enable regression testing with easy regeneration of the fixtures. Then tweak from there to make it useful for users without big knowledge of the internals and propose that to other implementations to make them comparable.

Does that make sense?

Some thoughts about what this dump format should contain

docopt_format: "Naval Fate …" # what was parsed
program_name: naval_fate
alternatives:
- marker: ship new
  positional:
  - name: name
    repeats: infinite
- marker: ship
  positional:
  - name: name
  rest_alternatives:
  - marker: move
    positional:
    - name: x
    - name: y
    named:
    - flag_long: speed
      name: kn

Obviously this format idea is very incomplete and probably unusable for several reasons, but that's what is floating in my mind.

DISCUSS: future of docopt-ng

This is a broad topic that I've been thinking about recently, that has come up in several places, such as #4 and #45

What does the community want for the future of docopt-ng?

  1. A maintained and type-hinted port of the original docopt
  2. that, plus more features
  3. something else?

I have so far been pursuing 1. This was because

  • I was estimating that most users are just porting over from the old docopt to here.
  • There are now many more CLI libraries than when docopt was released, so I think any new applications are more likely to use one of the more modern frameworks, that provides type hints, input validation etc. Docopt inherently is limited in those two aspects because it operates on plain docstrings, so those would have to be done as a post-processing step, leading to code duplication. Perhaps we should add this to the README so that new users know the limitations coming in.
  • I liked the scope that @keleshev started with, a function that takes a usage message and sys.argv, and returns a dict. It's well defined, has a narrow interface but a complex implementation, and composes well. In release 0.9.0 I even removed the magic stuff in pursuit of this (see the changelog for rationale)
  • So far, additional features have been able to be implemented as extensions, rather than needing to change this repo
    • the magic stuff could have been an extension
    • docopt-dispatch could be implemented as an extension
  • I have limited time and already am not giving this all the attention that it should probably be getting, so a smaller scope is more tenable for me.

I would love to hear people's opinions here. I don't have a lot of open source maintainership experience, so I apologize for any frustration that is caused by that.

Cheers,
Nick

Performance penalty: combinatorial explosion in `transform`

Description

Using docopt for building utilities with relatively wide range of options is pretty limited because of a huge performance penalty.
Namely, a combinatorial explosion may happen in the transform function: the pattern expansion (like ((-a | -b) (-c | -d)) => (-a -c | -a -d | -b -c | -b -d)) has unacceptable computational complexity.

A good example would be the GNU ls utility. See the sample below.

To Reproduce

The script below takes almost 3 seconds to run which is terribly slow for just to parse CLI arguments.

"""
ls with a subset of GNU options (that's not even all of them!)

Usage:
    ls [-a|-A] [--hide=PATTERN] [-I=PATTERN] [-dLR]
       [--color] [-h|--si] [--indicator-style=WORD|-p|-F|--file-type]
       [--format=WORD|-x|-m|-x|-l|-1|-C|-g|-n|-o] [-Giks]
       [--sort=WORD|-f|-U|-S|-t|-v|-X] [--group-directories-first] [-r]
       [--time=WORD|-u|-c] [--time-style=TIME_STYLE]
       [FILES ...]
    ls --help
    ls --version

Arguments:
    FILES
        list of files
"""
from docopt import docopt

args = docopt()

Is docopt-ng dead?

This question was previously asked about docopt, but now docopt-ng seems to have ended up in the same situation - unmerged/unanswered PRs and issues.
It would be really great to have a 'live' docopt for Python.

Written a docopt grammar, interested ?

Not seeing any references to an actual docopt formal grammar yet, I am writing a PEG grammar for use in Python with Tatsu (https://tatsu.readthedocs.io). I am hopeful that it will be usable with other PEG parser generators but am not well enough versed in the area to know by how much they differ in syntax.

I would like to work with others interested in developing a formal grammar for docopt. I'm not fixed on Tatsu or even PEG though I suspect PEG grammars will provide more accurate results for the corner cases.

I appended a sketch of the steps I believe necessary to develop the grammar. I've written a grammar for the usage examples section. Now I need to take a step back and write the frame work to snag each major section. Divide and conquer seemed like the best approach. In a day or so, I could post the grammar and test cases to github.

So, any information in this area or would anyone like to collaborate ?

Philip


Unix Utility Usage Grammar

Or, a formal grammar for the docopy language.

First identify which elements must be parsed and which (if any) may be discarded (or perhaps remain free form text).

Characterize the delineations of each section sufficiently to write regular expression matchers for their start and end, or span.

Write a grammar to parse to just the sections out as blocks of free form text. This forms the framework within which the section parsers will operate.

For each section independently write a grammar to parse the it.

One by one, incorporate the section parsers into the framework.

command line parsing is storing some options multiple times

This little program here:

#!/usr/bin/env python

"""
Usage:
    test.py [--to=SITE]... [--] FILE...
    test.py [--to=SITE]... --config CONFIG [[--] FILE...]

Options:
    --config CONFIG     Configuration file.
    --to=SITE           Target site

"""

import docopt

options = docopt.docopt(__doc__)
print(options)

If I run test.py --to a --to b c, I get this output:

{'--': False,
 '--config': None,
 '--to': ['a', 'b', 'b'],
 'FILE': ['c']}

Notice the double occurrence of 'b'

Failing to put two spaces in description sometimes results in very obscure error.

I just began using docopt-ng. I was creating my interface from scratch (I currently use a functional but spaghetti-like script), and my docstring read something like:

"""Performous macOS Bundler

Usage:
    macos_bundler.py
    macos_bundler.py [options]
    macos_bundler.py (-h | --help)

Options:
...
"""

But I kept getting the following error and couldn't figure out why.

    raise tokens.error("unmatched '%s'" % token)
docopt.DocoptLanguageError: unmatched '('

I double checked brackets and there were definitely no unmatched ones. I then double-checked the documentation to make sure I hadn't made a mistake with the syntax, and I couldn't find one.

I had read sometimes docopt (I think the original one?) had issues with pipes, so even though that exact syntax appeared in the documentation, I tried to remove the macos_bundler.py (-h | --help) line and, the error went away; but I couldn't use either -h nor --help, because it kept telling me those optons needed an argument.

Upon closer inspection, I noticed I had missed a space between -h --help and the option description.

So, it seems missing writing something such as

"""Command
Usage:
    cli_tool.py (-s | --long)

Options:
    -s --long Some description for this option.
"""

Will produce such an error.

Of course, there is an error there, and one the documentation specifically warns against. However, in this particular scenario, the error message was very obscure.

If there is unused data in the command line, the thrown exception doesn't include the collected and left items

https://github.com/jazzband/docopt-ng/blob/master/docopt/__init__.py#L920-L922

Because the message produced if you get to this point is extremely unhelpful, I wanted to inspect what had/hadn't been parsed and produce a more meaningful exception.

But because left and collected aren't passed, I've had to do a very nasty bit of code inspecting the stack to find out what they should have been.

Could they be passed in that particular instance so that the client has a chance of producing a more reasonable error (and it might be worth subclassing DocoptError so that it's clear to clients whether docopt has given up as opposed to finding a specific error).

assert instr.opname.startswith("CALL_") throws on Python 3.11

Hello!
Similar to this StackOverflow message I've got the same error while trying to (indirectly) use docopt-ng with Python 3.11.

I've added print(instr.opname) right after the line 941 and the resulting list of instructions indeed doesn't have anything stareted with CALL_ (see file atached). On the contrary here is a move shorter list of instructions when I run the same with Python 3.10:

LOAD_GLOBAL
LOAD_GLOBAL
LOAD_CONST
LOAD_CONST
CALL_FUNCTION_KW

If I remove the following block completely along with output_value_assigned variable then everything seems to work just fine with Python 3.11:

if more_magic and parent_frame:
import dis
instrs = dis.get_instructions(parent_frame.f_code)
for instr in instrs:
if instr.offset == parent_frame.f_lasti:
break
assert instr.opname.startswith("CALL_")
MAYBE_STORE = next(instrs)
if MAYBE_STORE and (
MAYBE_STORE.opname.startswith("STORE")
or MAYBE_STORE.opname.startswith("RETURN")
):
output_value_assigned = True

Support envvar token

Hello,

Firstly thank you very much for giving this project a breath of life!

This PR in the original docopt supported the envvar parameter. It never made it in.

Being able to know which environment variables have been specified in the documentation (like variables), would be very useful for the various projects I use docopt for.

It would be great if this could be added into docopt-ng repo

Provide option metadata

For the following option definition

-c --count=<int>    Option for a count of some kind [default: 42] 

Is there a way to retrieve the default value ( 42) and the value of the value designation (<int>) programatically? I.e. is the a way to get from the resulting args after docopt-ng is done parsing? I have an oustanding PR in original docopt for that sort of change but, as well all know, that's not going to be merged.

1.0 Release?

After a few days and maybe dropping in docopt-ng for some popular projects that use docopt, I'm hoping to do a 1.0 release - slightly breaking semver, but with 100% coverage I think the project deserves to be pulled out of "beta".

TODO: port spellchecker to suggest, rather than fix

I removed the public interface to the spellchecking in https://github.com/jazzband/docopt-ng/releases/tag/0.9.0, but that spellchecking code still exists.

I think that code is still useful, but it should be pivoted. Instead of auto-correcting mispelled args (which might be dangerous, much better to be strict), the parse should still fail. BUT, we can still extract useful help messages from this, and say "perhaps you meant XXXX".

This issue tracks that progess

Implement Jazzband guidelines for docopt-ng

This issue tracks the implementation of the Jazzband guidelines for the project docopt-ng

It was initiated by @itdaniher who was automatically assigned in addition to the Jazzband roadies.

See the TODO list below for the generally required tasks, but feel free to update it in case the project requires it.

Feel free to ping a Jazzband roadie if you have any question.

TODOs

Project details

Description Humane command line arguments parser. Now with maintenance, typehints, and complete test coverage.
Homepage
Stargazers 36
Open issues 9
Forks 6
Default branch master
Is a fork False
Has Wiki False
Has Pages False

Warning message incomprehensible for end-user

Wenn I run my script without a necessary subcommand, docopt-ng issues a warning - shouldn't it be an error? - which in my opinion is not suitable for the enduser (found unmatched (duplicate?) arguments [Argument(None, '...')].

Even I as the script writer, don't know what that exactly means. It looks to me like docopt-ng exposes internals from the parsing process which should not be given to the script user.

> python interpreter.py '.\10. PowerShell.ps1'
Warning: found unmatched (duplicate?) arguments [Argument(None, '.\\10. PowerShell.ps1')]
Usage:
 interpreter.py run <script> [-- <script_option> ...]
 interpreter.py doc [-s <text>] <script>

docopt.DocoptLanguageError: unmatched '['

Hello

This docstring was working before release docopt-ng==0.9.0

Note only the Usage: + up to the empty line should be parsed by docopt language. Unless you added new section name handling?

By the way the error message doesn't help to figure out what is wrong.

sample code

#!/usr/bin/env python
"""
prepare an upgrade archive for upgrade_instance.py

Usage:
    ./prepare_upgrade.py create UPGRADE_FOLDER [--new]
    ./prepare_upgrade.py generate_sql DATE --sql-repos=INPUT_DIR_SQL [UPGRADE_DIR]
    ./prepare_upgrade.py fetch_releases RELEASE_PIPELINE [UPGRADE_DIR]
    ./prepare_upgrade.py show [UPGRADE_DIR]
    ./prepare_upgrade.py build [--upgrade-folder=UPGRADE_DIR] [--dest-zip=UPGRADE_ZIP]
    ./prepare_upgrade.py add (--sql|--repair) SQL_FILE [UPGRADE_DIR]
    ./prepare_upgrade.py convert_to_dep_txt [--old] [UPGRADE_DIR]
    ./prepare_upgrade.py tree [UPGRADE_DIR]
    ./prepare_upgrade.py install_helpers [--clean] [UPGRADE_DIR] [--ext_dir=INFRASTRUCTURE_DIR]

Env var:
    UPGRADE_DIR    if set, replace the UPGRADE_DIR parameter, so it can be ommited. If
                   both are present, parameter will be chosen.

Actions:
    create               Create an empty folder in UPGRADE_FOLDER (parent) that will contains
                         the upgrade tree Output the foldername suitable for UPGRADE_DIR env var
                         assignment.
                         --new will instruct to create a new_one if the folder exists with
                         the current date: upgrade_YYYY-MM-DD.2 (the last digit will
                         increase etc.)
    generate_sql         Take the SQL file in INPUT_DIR_SQL and filter from DATE included
                         in order to generate an sql folder tree into UPGRADE_DIR
    fetch_releases       Connect to VSTS to fetch the releases in prod based on RELEASE_PIPELINE.
                         This is a slow operation that take ~40s to complete. The outout yaml file
                         will be saved in UPGRADE_DIR.
    show                 Explain what is in UPGRADE_DIR and validate.
    build                Create the zip archive UPGRADE_ZIP based on the content of the
                         UPGRADE_DIR.  If UPGRADE_ZIP is not given the zip will be generated
                         in to the parent forled (aka: UPGRADE_FOLDER)
    add                  Add a new sql file manually to the UPGRADE_DIR. In sql/ dir if --sql
                         at the end with continuous order. In repair/ if --repair, with no
                         specific order.
    convert_to_dep_txt   For internal transitional conversion dep_version.txt format from releases.yaml.
                         Outputs to stdout.
    tree                 Display UPGRADE_DIR as a tree view.
    install_helpers      Copy helpers into UPGRADE_DIR. If --clean is present remove all helper first.
"""

from docopt import docopt

args = docopt(docstring=__doc__)

print(args)

stacktrace

python bug_docopt-ng.py create prepare_upgrade 2>&1 | sed -e 's/cloudiway/mycode/'
Traceback (most recent call last):
  File "/home/sylvain/code/mycode/upgrade_instance/bug_docopt-ng.py", line 47, in <module>
    args = docopt(docstring=__doc__)
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 913, in docopt
    pattern = parse_pattern(formal_usage(sections.usage_body), options)
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 608, in parse_pattern
    result = parse_expr(tokens, options)
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 617, in parse_expr
    seq_0: list[Pattern] = parse_seq(tokens, options)
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 638, in parse_seq
    atom = parse_atom(tokens, options)
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 657, in parse_atom
    matched_pattern = pattern(*parse_expr(tokens, options))
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 617, in parse_expr
    seq_0: list[Pattern] = parse_seq(tokens, options)
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 638, in parse_seq
    atom = parse_atom(tokens, options)
  File "/home/sylvain/.local/lib/python3.10/site-packages/docopt/__init__.py", line 659, in parse_atom
    raise tokens.error("unmatched '%s'" % token)
docopt.DocoptLanguageError: unmatched '['

Question: auto formatting & sorting imports

@NickCrews would you be OK with auto sorting/formatting imports (probably with isort)?

When I was rebasing the error message PR just now, and a few other branches I've got, probably 3/4 of my merge conflicts were just imports. I'm thinking it would help to use isort with the single line option, so that instead of:

from typing import Any, Callable, Tuple, Type, Union, cast

we get:

from typing import Any
from typing import Callable
from typing import Tuple
from typing import Type
from typing import Union
from typing import cast

Which doesn't look as nice, but means import changes shouldn't cause merge conflicts.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.