lfe-deprecated / lfetool Goto Github PK

Depends upon:

• lfe-deprecated/lfe-utils#6
• https://github.com/lfe/roz/issues/10

When given a custom install dir like so:

lfetool install expm ./bin

expm does not show up there when the command finishes, even though lfetool says the install was successful.

I heard something recently about unit testing in Bash. Need to check that out and update the Makefile to run some unit tests that Travis can do something useful with.

The things that need to be tested are pretty basic:

• run new script and check for the expected output when the script is run.
• run new library and do some checks on each of the files generated, ensuring that expected content is there.
• run new service and do the same thing as above.

Note that when both a new library and a new service are created, they run intentionally-failing unit tests, so doing a test of successful creation in Bash might be a little tricky. Dunno yet.

This will require the Travis CI .yml file as well as a check target that doesn't use Rebar (rebar + LFE fails on Travis CI servers).

I'd like to remove the lfetool binary from the repo. This, however, would impact users, so there needs to be some sort of transition...

Ideally, we could do something like:

• update lfetool to do updates against tags and not master
• release a couple of versions like that, with a warning about deprecating updates from the master branch
• remove lfetool from master, clean up history, and force push. Yeah, I said it. Force. Push.

This should be something like what rebar does. Robert doesn't use rebar and doesn't want to have to depend upon it.

First iteration: use the rebar.config to pull dependencies into ./deps.

Use the rebar.config to define where the deps are.

Support an alternative file (and syntax/structure?) to rebar.config -- perhaps something shaped by our users preferences.

This functionality would essentially identify the abstraction of a "package" (per discussions with Robert on IRC).

This depends upon the progress made in this ticket:

• oubiwann/jlfe#3

Add a command for downloading dependencies. This can wrap git clone initially (use checksum support). This should also support using curl or even an LFE/Erlang HTTP download function in a module.

The development workflow outlined here is rather cumbersome. Once we move to lfetool written in LFE, this should get much simpler. We should design, from the start, with a simple process in mind.

Probably need custom rebar config files for this. Also, will need to tell eunit to look in the following dirs (instead of just test):

• test/unit
• test/integration
• test/system

We will need to introduce a new LFE standard for this as well: unit--tests.lfe, integration--tests.lfe, system--tests.lfe. This also needs to take into account the current implicit standard of test/testing-.lfe being used for modules needed only during testing of a project (fixtures, general utilities, etc.).

In fact, this ticket shouldn't be closed until the EUnit chapter of the new User Guide is updated with all of this, so LFE standard is explicitly documented.

For reference, see the following:

• http://en.wikipedia.org/wiki/Unit_testing
• http://en.wikipedia.org/wiki/Integration_testing
• http://en.wikipedia.org/wiki/System_testing

The initial plan is to:

1. have a fairly small (relatively) lfetool shell script which
2. calls to lfe which
3. gets passed parameters which represent commands, subcommands, options, etc., and
4. these are mapped to modules/functions which, in turn, get called

This needs to be explored with:

• a basic shell script and
• some sample modules

In addition to this flow, another one needs to be supported: bootstrapping. The plan here is to:

1. check for a local lfetool install
2. if it doesn't exist, clone to ~/.lfetool/lib (note that the directory ./lfetool itself will be used for configuration files, user overrides, etc.)
3. run all commands with ERL_LIBS=~/.lfetool/lib:$ERL_LIBS to pick up lfetool modules

Currently lfetool calls erl using lfetool mods/funcs. It needs to start using Robert's new lfe executable instead.

Use slate by default ;-)

Should probably take optional parameter of kerl Erlang release string (which would map to a directory)...

When checking Travis CI job output, it's very handy to have each target print out its status.

With one of the latest changes to lfetool, the colors are no longer rendering - just getting color command characters displayed in terminal output.

Note: this was originally posted to the LFE mail list

Background

Current Flow

I've been making progress prototyping the new lfetool. I've got a very basic system in place that exercises the following flow:

  command from system shell -> eval fn -> parse fn -> dispatch fn

where the "dispatch function" will make a call to a function in the plugin module, as identified during the parse phase.

Dispatching and Plugin Discovery

Next I'm going to experiment with the dispatching, with an eye towards making it as easy as possible for new plugins to be created by contributors. lfetool will search for plugins in a standard dir (part of the lfetool source), but will also search in ~/.lfetool/plugins so that users can create their own plugins without submitting them for inclusion in lfetool proper, if they so choose.

Plugin System Internals

This has, however, brought up a bit of an interesting problem to solve, with regard to the lfetool "architecture", and it hinges upon making things as simple as possible for plugin developers. As we all know, when one attempts to make things simple for users, software gets very complicated ;-)

Sought simplicity:

• users should be able to create a plugin module and any associated
  template files in one location
• users should be able to have commands for their plugins accessible via
  lfetool
  • for example, I create a plugin that adds support for generating Garrett
    Smith's e2 service
  • lfetool new should have a new subcommand called e2 (or whatever I name
    it)
• the bash auto-complete should be easily/automatically updated with this
  addition
• lfetool help should automatically list all official plugins as well as
  ones added by users
• lfetool should support getting help/docs for individual plugins
• lfetool code should be able to introspect each plugin and get plugin
  name, version info, supported commands, help, etc.

Subsequent complexity:

• the lfetool project should create and document a behaviour for plugin
  modules
• in order for all the introspection to work, lfetool will need to be
  "aware" of all plugins and all commands which they support
• as such, I'm planning on creating a process which does the following:
  • gets a list of all plugins, official and user-created
  • builds a data structure for each plugin (possibly storing in ETS)
  • listens for a message, which will dispatch to a plugin function
• this same infrastructure will also be able to auto-generate shell auto-
  complete files

Let me know if there are better ways to do this sort of thing using more standard Erlang/OTP idioms ...

Epic: Version 2 Project Tracking

This ticket gives an overview of the work to be done in Version 2 of lfetool. Though this ticket represents the develop branch in the repo, actual coding will not happen against this ticket. Rather, new ones will be opened with their branches merged to develop when complete.

Once all features have landed, been merged to develop, and lfetool behaves as expected, having been fully tested, we will merge develop to master.

Overview of Dev Plan

In a conversation on IRC, I managed to talk myself (pretty easily/quickly, tbh) into a Version 2 implementation of lfetool. It will follow more closely the lein tool which inspired it in the first place:

• there will be a shell script named lfetool which will
  • bootstrap itself when first run
  • provide a compatibility layer for different shells and OSes
• most of the functionality of lfetool will be moved into LFE library files
• there will be an entry-point (probably written in LFE) which will dispatch to lfetool plugins (library files)
• the entry-point will be called from the lfetool shell script (probably using @rvirding's new lfe script + parameter handling)

Outstanding Questions

• What about bootstrapping for other things? For instance, installing things like Erlang (via kerl) or rebar?
  • Robert has a potential answer for this: include lfetool with lfe and use lfetool for everything by default (downloading, compiling, dependency configs, etc.).

Resources

• Inspiration will be taken from lein: https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein
• Work on this will be done in the develop branch: https://github.com/lfe/lfetool/tree/develop
• Full dev plan notes (including the IRC conversation which inspired this work) are kept here: https://github.com/lfe/lfetool/blob/develop/doc/notes/version2-dev-plan.md

Sub-Tickets

Sub-tickets (stories) for this epic are listed below in the comments (to take advantage of active checkboxes in Github markdown).

Bootstrapping lfetool

LFE projects are created with the understanding that lfetool is installed and available. If, however, we were to make lfetool a project that had dependencies such as erl and LFE (and other deps in a rebar.config), we have to consider things more carefully.

Here is a list of lfetool commands that can be executed without Erlang installed:

• lfetool install lfetool
• lfetool install expm
• lfetool install kerl
• lfetool install erlang
• lfetool install erjang

The following need at least Erlang (since they install into a directory obtained by making erl calls):

• lfetool install rebar
• lfetool install relx
• lfetool info *

The following require lfetool to be installed:

• lfetool update *
• lfetool repl *

The following require LFE:

• lfetool new *
• lfetool tests *

Bootstrapping Other Projects

• What about installing Erlang itself (via kerl)?
• What about rebar?

Running the command

    lfetool new library my-test-lib

(per the quickstart guide) results in the following error message:

    Setting up starter library project my-first-project ...
    Initialized empty Git repository in /home/[redacted]/src/my-first-project/.git/
    Getting dependencies ...
    ERROR: Failed to load /home/[redacted]/src/my-first-project/rebar.config: {error,
                                                                        {16,
                                                                         erl_parse,
                                                                         ["syntax error before: ",
                                                                          "']'"]}}
    make: *** [get-deps] Error 1

For reference, I'm using Erlang R17 and rebar from ESL's repo on Debian 7.0.

The rebar.config that resides in the my-first-project directory looks like this:

{erl_opts, [debug_info, {src_dirs, ["test/unit",
                                    "test/integration",
                                    "test/system"]}]}.
{lfe_first_files, []}.
{deps_dir, ["deps"]}.
{eunit_compile_opts, [
   {src_dirs, ["test/unit",
               "test/integration",
               "test/system",
               "src"]}
  ]}.
{deps, [
   {lfe, ".*", {git, "git://github.com/rvirding/lfe.git", "develop"}},
   {'lfe-utils', ".*", {git, "https://github.com/lfe/lfe-utils.git", "master"}},
   {lfeunit, ".*", {git, "git://github.com/lfe/lfeunit.git", "master"}},
  ]}.

Using kerl.

I tried running the repl in a new service project, and I get the following error.
Any clue would be really helpful.

> make shell                                                                                                    ruby-2.1.0 [54/1393]
Getting dependencies ...
==> lfe (get-deps)
==> lfeunit (get-deps)
==> lfe-utils (get-deps)
==> rebar (get-deps)
==> lfe-sample-rebar-plugin (get-deps)
==> my-service (get-deps)
Updating dependencies ...

        Updating ./deps/lfe ...

Already up-to-date.

        Updating ./deps/lfe-sample-rebar-plugin ...

Already up-to-date.

        Updating ./deps/lfeunit ...

Already up-to-date.

        Updating ./deps/lfe-utils ...

Already up-to-date.

        Updating ./deps/rebar ...

Already up-to-date.
Cleaning ebin dir ...
Compiling project code and dependencies ...
==> lfe (compile)
==> lfeunit (compile)
==> lfe-utils (compile)
==> rebar (compile)
==> lfe-sample-rebar-plugin (compile)
==> my-service (compile)
Compiled src/my-service-app.lfe
Compiled src/my-service-sup.lfe
Compiled src/my-service-server.lfe
Starting shell ...

Unknown context: 'repl'

Script: lfetool, v

Usage: /home/darth10/bin/lfetool <command> <context> [<arg> | <context command> <arg>] | <options>

-D is not supported in base64 8.20:

./lfetool help

base64: invalid option -- 'D'
Try 'base64 --help' for more information.

base64 --help
Usage: base64 [OPTION]... [FILE]
Base64 encode or decode FILE, or standard input, to standard output.

  -d, --decode          decode data
  -i, --ignore-garbage  when decoding, ignore non-alphabet characters
  -w, --wrap=COLS       wrap encoded lines after COLS character (default 76).
                          Use 0 to disable line wrapping

      --help     display this help and exit
      --version  output version information and exit
....

base64 --version
base64 (GNU coreutils) 8.20

Rebar should be installed to the bin dir for whichever Erlang build is active.

lfetool needs to:

• identify the current active Erlang
• find its bin dir
• compile rebar with the active erl
• copy the built rebar to the active Erlang's bin dir

When unit tests fail, they should return 1.

@rvirding has updated the LFE Makefile to use a more succinct method of reading version info from the .app.src file. We should use this as well. His change is the first link and the second shows the current code:

• lfe/lfe@3642ee5
• https://github.com/rvirding/lfe/blob/develop/Makefile#L87

This can probably be done by optionally passing a custom parameter to erl/lfe when starting lfetool. Probably just -debug. A debug?/0 utility function should be suitable for checking.

This can probably be done by optionally passing a custom parameter to erl/lfe when starting lfetool. Probably just -debug. A debug?/0 utility function should be suitable for checking.

Originally proposed by Josh Schairbaum:

"if a DEBUG env var is set, have it spit out the commands it's going to run before it runs them."

Once the common.mk is created, split out the OTP-specific stuff.

$ lfetool update erjang

Should do the following:

• find the Erjang install
• cd to it
• git pull origin master
• ant alljar

Also, add comment for: if one needs to install to a particular path, the context is required. E.g., one cannot do lfetool install /usr/local/bin for installing lfetool, but must use lfetool install lfetool /usr/local/bin instead.

This needs to be checked.

Other installation types should be checked as well, and ensure that they each work with custom install dirs.

Just like it says...

Envisioned:

$ lfetool install erlang R16B03

This would do the following:

• download kerl (if not on $PATH)
• install it in /usr/local/bin
• run kerl build <release arg> <release arg>
• run kerl install <release arg> /opt/erlang/<release arg>

With pull request #26, I started getting unit test runs that look like this:

------------------
Running unit tests ...
------------------
expr: not a decimal number: '      56'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      46'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      51'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      60'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      32'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      62'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      35'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      33'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      56'
bash: line 548: [: : integer expression expected
expr: not a decimal number: '      38'
bash: line 548: [: : integer expression expected
======================== EUnit ========================

Subcommands would specify what to compile:

• lfetool compile src - compile .lfe and .erl files in ./src
• lfetool compile deps - compile .lfe and .erl files in ./deps/*/src
• lfetool compile plugins - compile .lfe and .erl files in ./plugins and ./~.lfetool/plugins
• lfetool compile all - all of the above
• lfetool compile - alias for lfetool compile src? Or lfetool compile all?

Used in conjunction with

1. a dependency/package definition and
2. download capabilities,
   lfetool could compile these to a standard location.

Again, this would mean people wouldn't have to depend upon rebar; they'd only need one tool.

We could, of course, also provide support for rebar, we just wouldn't have to require it.

This should use the lfe-reveal-js repo.

lfetool should support the use case where a user would want to execute many commands in a row. In other words, starting up an lfeshell, executing the desired commands in it, and then shutting down the shell.

This could be used for:

• build deps (this could also do a rebar get-deps)
• build tests
• build project
• build all

Also, if build was supported, other commands could use them. For example, the REPL commands could take additional options:

• lfetool repl lfe build
• lfetool repl lfe build deps
• lfetool repl lfe build projects
• lfetool repl lfe build all

This would provide users support for many different workflows.

When a module only has one unit test, a formatting pattern/regex in lfetool is not met and the ellipses are not long enough.

Here's an example (see unit-misc-utils-tests: uuid4-test below):

------------------
Running unit tests ...
------------------

======================== EUnit ========================
module 'unit-files-utils-tests'
  is-home-dir? ......................... [0.002 s] [ok]
  expand-home-dir ................................ [ok]
  Total module test time: 0.008 s
module 'unit-math-utils-tests'
  fast-floor ..................................... [ok]
  round .......................................... [ok]
  dot-product .................................... [ok]
  scale .......................................... [ok]
  unit-scale ..................................... [ok]
  color-scale .................................... [ok]
  odd? ........................................... [ok]
  even? .......................................... [ok]
  Total module test time: 0.024 s
unit-misc-utils-tests: uuid4-test ...[0.008 s] [ok]
module 'unit-types-utils-tests'
  add-tuples ..................................... [ok]
  partition-list ................................. [ok]
  pair-dict ...................................... [ok]
  list->tuple .......................... [0.008 s] [ok]
  atom-cat ....................................... [ok]
  strip ................................ [0.002 s] [ok]
  stinrg? ........................................ [ok]
  list? .......................................... [ok]
  tuple? ......................................... [ok]
  atom? .......................................... [ok]
  Total module test time: 0.039 s
=======================================================
  All 21 tests passed.

Maybe common.mk, yaws.mk, etc.?

When installing to a non-existent directory, lfetool will install as the intended directory name instead of lfetool itself. For instance:

$ ls /opt/scripts/

$ bash lfetool install lfetool /opt/scripts/bin
chmod: /opt/scripts/bin/lfetool: Not a directory

Installed lfetool to /opt/scripts/bin.
$ ls -al /opt/scripts/bin
-rwxr-xr-x  1 oubiwann  wheel  356468 May  1 09:41 /opt/scripts/bin

Users have to manually update the Makefile whenever they add new dependencies. Can we do this automatically?