countvajhula / cli Goto Github PK

These are some examples of the syntax for flags at the moment:

(flag (verbose)
  ("-v" "--verbose" "Show detailed messages.")
  (verbose #t))

(flag (attempts n)
  ("-a" "--attempts" "Number of attempts to make")
  (attempts (string->number n)))
 
(flag (transaction n timeout)
  ("-t" "--transaction" "Size of transaction, and timeout value")
  (transaction (map string->number (list n timeout))))
 
(flag (links #:param [links null] link)
  ("-l" "--link" "Links to validate")
  (links (cons link (links))))

It seems that the goal of a flag function is typically to set the value of the corresponding parameter. So, should the syntax instead encode this transparently instead of requiring the user to do it explicitly? That is, we could treat the return value of the function as the value that the parameter should be set to, and do this behind the scenes. Otherwise, the syntax seems to have some redundancy, and in the most common cases (e.g. the first three examples above), the user would probably not even care to mention the parameter. The new versions would look like this:

(flag (verbose)
  ("-v" "--verbose" "Show detailed messages.")
  #t)

(flag (attempts n)
  ("-a" "--attempts" "Number of attempts to make")
  (string->number n))
 
(flag (transaction n timeout)
  ("-t" "--transaction" "Size of transaction, and timeout value")
  (map string->number (list n timeout)))
 
(flag (links #:param [links null] link)
  ("-l" "--link" "Links to validate")
  (cons link (links)))

This change would likely be backwards incompatible, and should probably be done in connection with #5 for that reason (as that is also backwards incompatible).

Ordinary Racket functions accept individually named arguments, but also packed "rest" arguments using the dot notation:

(define (func . args)
   body ...)

With #lang cli, we'd like to support something similar:

(program (prog . args)
   body ...)

and

(program (prog . [args "Files to process"])
   body ...)

... would be ideal. But I seem to recall that it's tricky (impossible?) to match the literal dot in macros. So it may be necessary to use a custom syntax to indicate that arguments should be collected into a list. Example:

(program (prog (rest args))
   body ...)

and

(program (prog . (rest [args "Files to process"]))
   body ...)

Note that Racket's built-in command-line does support this kind of argument packing, and it too is implemented on top of parse-command-line, as #lang cli is. So it should be possible to leverage the built-in facilities for this.

Workaround: At the moment, we can use the multi constraint to pass multiple arguments via a flag.

Arguments received at the command line are always strings which usually need to be parsed into the appropriate types for use in the command line script. At the moment, every script would need to do this parsing (for instance, parsing a value as a number or as a date) independently, in an ad hoc way.

We could define a schema layer where, for instance, numeric flags needn't specify an implementation for the flag function, and instead use a (schema number) subform in place of the body, which would set the flag parameter to the value received from the command line cast as a number.

Similarly, other schemas to handle boilerplate parsing could be isodate (for dates provided in ISO format), currency (for e.g. dollar amounts that should use precision to 2 decimal places), vector, and other Racket types.

It should also be possible to specify a transformer for positional arguments, and ideally it should be possible to use schemas here as well. E.g. (program ([n "Number of entries" string->number]) ...) or (program ([n "number of entries" (schema number)]) ...)

It would also be good to support user-defined schemas for custom types.

At the moment, using #lang cli in a source file allows us to define a command line interface using all of the forms of the cli language, including specifying flags and help text and so on. In addition to defining such commands at the top level, we can also define nested commands within this file if we enclose these commands within submodules, like this:

(module another-command cli
  (help (usage "help for a nested command"))
  (program (another-command)
    (displayln "hiya"))
  (provide another-command))

We can then run this command by adding, at the top level:
(run another-command)

Note that when the source file is executed, it doesn't run anything by default unless such a run declaration is present, since we have the ability to execute either the top level command, or a nested command, or any number of them in sequence. This is why we must indicate explicitly what we mean to run.

But having to expose the Racket submodule machinery in order to define these nested commands isn't ideal. We'd prefer to be able to do something like this instead:

(command another-command
  (help (usage "help for a nested command"))
  (program (another-command)
    (displayln "hiya")))

... and it should take care of the submodule definition and provide boilerplate.

Notes
Initial attempts ran aground of "unbound identifier" and "no #%app syntax transformer bound" issues - it may be that defining a submodule via a macro needs special handling to ensure the submodule is defined in the calling scope in the right way. There were also some issues with injecting a module and having it reflect at the top level, since modules are "allowed only at top level."

Summary

When using cli as the module language for a main submodule, defining a macro within the submodule causes the following error (scroll to the bottom for a workaround):

; /Users/siddhartha/work/lisp/racket/sandbox/cli/bugs/module-macro/bug.rkt:22:5: opt: module mismatch;
;  attempted to use a module that is not available
;   possible cause:
;    using (dynamic-require .... #f)
;    but need (dynamic-require .... 0)
;   module: #<module-path-index:(submod #<path:/Users/siddhartha/work/lisp/racket/sandbox/cli/bugs/module-macro/bug.rkt> main)>
;   phase: 0
;   in: opt
; Context (plain; to see better errortrace context, re-run with C-u prefix):
;   /Users/siddhartha/.emacs.d/straight/build/racket-mode/racket/syntax.rkt:66:0
;   /Users/siddhartha/work/lisp/racket/cli/expander.rkt:146:0 read-spec
;   /Users/siddhartha/work/lisp/racket/cli/expander.rkt:181:0 read-specs
;   /Users/siddhartha/work/lisp/racket/cli/expander.rkt:207:7 prog

Observations

The error only happens if all of the following are true:

• cli is the module language for a main submodule
• at least one flag is defined (via flag)
• a macro is defined somewhere within the main submodule

Minimal example

(module* main cli

  ;; either commenting out this macro, or putting it outside the present
  ;; module (above it) and then requiring it via (require (submod "..")),
  ;; causes the problem to disappear
  (define-syntax-rule (blah thing)
    (thing thing))

  ;; alternatively, removing this flag causes the
  ;; problem to disappear as well
  (flag (opt name)
    ("-o" "--option" "An option.")
    (opt name))

  (program (prog)
    (displayln "Hello! The option is:")
    (displayln (opt)))

  (run prog))

Control examples

These examples are identical in functionality to the example above, but don't exhibit the error.

1. Using the cli as a top level #lang doesn't have this problem:

#lang cli

(define-syntax-rule (blah thing)
  (thing thing))

(flag (opt name)
  ("-o" "--option" "An option.")
  (opt name))

(program (prog)
  (displayln "Hello! The option is:")
  (displayln (opt)))

(run prog)

2. For a main submodule using racket/base as the module language, using parse-command-line directly doesn't have the problem either:

#lang racket/base

(module* main racket/base
  (require racket/cmdline)

  (define-syntax-rule (blah thing)
    (thing thing))

  (define opt (make-parameter #f))

  (parse-command-line
   "prog"
   (current-command-line-arguments)
   `((once-each
      [("-o" "--option")
       ,(lambda (flg ~opt)
          (opt ~opt))
       ("An option." "option")]))
   (λ (flags-accum)
     (displayln "Hello! The option is:")
     (displayln (opt)))
   '()))

Workarounds

1. Define the macro outside the main submodule and require it within the main submodule.

It could be defined in the containing module:

(define-syntax-rule (blah thing)
  (thing thing))

(provide blah)

(module* main cli
  (require (submod ".."))
  ...)

... or in any other module and just required, as usual:

(module* main cli
  (require "macro-containing-module.rkt")
  ...)

Fix

I haven't had a chance to investigate. The workaround above is reasonably straightforward so it doesn't seem vital to address this bug right away. I would, of course, love for it to be fixed.

Dear Reader: if you're up for a bug hunting challenge, I invite you to take a crack at finding the cause. The above examples show clear cases where it works and doesn't work, and the changes that cause it to go from working to not-working. It's a good start for a seasoned bug hunter like yourself, and could be a fun one to track down. If you decide to try, please share any findings in the comments (or a PR), and your efforts would be greatly appreciated.

If no one gets to it, I'll plan to look at it the next time I'm dedicating time to this package.

Flags are provided via the flag form, which supports basic flags, mutually exclusive ones (one-of), those that can be provided multiple times (multi), and terminating flags (final). All of these distinct types of flags have something in common -- they define functions that should be called with the arguments received from the command line. It would be good if we could change the flag syntax to directly correspond to this functional nature, so that we could define the flags as functions via a common form like so:

(flag (verbose v)
  (("-v" "--verbose") "Use verbose mode")
  (verbose v))

(flag (brief v)
  (("-b" "--brief") "Use brief mode")
  (brief v))

(flag (links v)
  (("-l" "--link") "Links to use")
  (cons v (links))

And then independently define higher-level constraints on the flags, something like:

(constraint (one-of brief verbose))
(constraint (multi links))

Advantages of this include (1) flexibility -- not specifying any constraints on the flags will expect each flag at most once, and then specifying these constraints can be done at a later stage of development when the constraints are apparent. (2) uniformity / minimalism-- there is less diversity in the core syntax, all the distinct flag types have the same (rather than subtly variant) core specification.

At the moment, the cli forms make extensive use of unhygienic macros as well as mutation of module scope variables to track configuration changes. This is an artefact of the need to keep the configuration forms separate rather than unify them into one giant macro (which is what Racket's command-line already is). But it makes me a little suspicious just on principle that there could be cases where unexpected things may happen, in particular, in multithreaded settings. Obviously, if cases are encountered in practice that reveal such problems, they can be investigated then, so there's no need to jump the gun and fix vague hypothetical problems that may never arise. But at such a time, here are some possible alternatives to consider:

• instead of module-level variables keeping track of configuration, we could define a single struct for this purpose
• each form could simply result in an identifier (the form name) being assigned the result of the form
• we could, prior to running the command, use another form like (config param ...) which accepts each individually named aspect of the configuration (e.g. the flags and constraints) and yields a named configuration object, an instance of the configuration struct type as output
• define a new command form that takes a function specified using program and a configuration object, to yield the actual program that could be executed using run

Assessment:
This would make the implementation functional / immutable, but would still rely on unhygienic identifiers (which may be fine -- the only thing this could affect is composability rather than robustness), and not sure what this would mean for the parameters that get defined for each flag. After all, two distinct configuration objects could refer to the same parameters which are dynamic and stateful. But maybe that's OK too. Finally, such an implementation would be syntactically less lightweight (and backwards-incompatible), but potentially more flexible since it could make composability (see #3) easier by allowing independent definitions of composition for the programs and the configuration (e.g. "run program p3 composed from p1 and p2 (run consecutively), and parametrized by this configuration object that is composed from c1 and c2 using union composition / override composition"). It also avoids having to rely on submodules to be able to define commands with distinct or conflicting configurations.

At the moment this is the expected syntax of the program form:

(program my-command ([arg desc] ...)
    body ...)

To even more closely mimic the syntax of regular functions, it would be nice to support making the descriptions optional, so that:

(program my-command (arg ...)
    body ...)

works. This should treat the descriptions as if they were "". It should do this on a per-argument basis so that some arguments could have descriptions even while others don't.

Since this macro uses unhygienic identifiers, rewriting the latter to the former without losing the calling scope may present some challenges.

Defining commands in #lang cli has similar syntax to defining functions in Racket, and in fact, commands compile down to Racket functions.

Since commands are just functions, it should be relatively straightforward to define some form of composition on them. For instance, we may want to define a host of commands for various administrative tasks that we run individually, but which we'd also like to be able to compose so that, in order to run some combination of them, we needn't define an entirely new command from scratch, duplicating all of the flag specification and other boilerplate already encapsulated in the constituent commands.

To accomplish this, we'd need to define a notion, or several notions, of "command composition," which allows us to compose commands and have them share or divide up parameters supplied on the command line, and so that help text and other metadata also compose naturally in the resulting command.

For inspiration: the Click library for python supports composable commands.

Given the following program --help generates help text that doesn't seem to support --parameter=text format, but --parameter text

#lang cli

(flag (file  in-file-path)
      ("-f" "--file" "filepath")
      (file in-file-path))

(program (hello)
  (displayln (file))
)

(run hello)

CLI output:

PS ... > D:\Racket\Racket.exe .\tmp.rkt --help
usage: hello [ <option> ... ]


<option> is one of


  -f <in-file-path>, --file <in-file-path>
     filepath to write the changelog to. Can NOTE use --file=foo must use --file foo
  --help, -h
     Show this help
  --
     Do not treat any remaining argument as a switch (at this level)

 *   Asterisks indicate options allowed multiple times.

 Multiple single-letter switches can be combined after
 one `-`. For example, `-h-` is the same as `-h --`.

PS ...> D:\Racket\Racket.exe .\tmp.rkt --file hello
hello
PS ...> D:\Racket\Racket.exe .\tmp.rkt --file=hello
hello: unknown switch: --file=hello
PS C:\Users\aias\Documents\Development\simple_changelog>

(but thank you very much for making this little language!)

At the moment, constraints validate some attributes of the input, e.g. that flags are provided a certain number of times, and so on. It would be nice to broaden the scope to ensure conditions that should hold across all provided inputs -- over multiple flags, for instance. Something like:

(constraint (> width height))

... which would ensure that the value received for the width flag is greater than that received for the height flag. The predicate here could be any function, so it could ensure arbitrary conditions on the provided flags -- conditions which could not be asserted within the handler functions for the individual flags.

Another thing is that since both flags as well as the core program are simply functions, it may be possible to leverage Racket's contract system to provide some guarantees. But since CLI scripts are unlikely to return values, and since the arguments and flags are already parsed from strings in each handler function, where appropriate errors may be raised, it seems doubtful that contracts can provide much additional value here.