erlang / rebar3 Goto Github PK

$ rebar3 compile
===> Fetching erequest_id
(tchevalier@tchevalier-ltm)
[Tue Jan 06~18:08:39] ~/heroku/hundur $ echo $?
1

The compile command failed, but didn't print a message indicating the failure. If I do this:

$ DEBUG=1 rebar3 compile

I get more info:

[snip]
===> Fetching erequest_id
===> sh info:
    cwd: "/Users/tchevalier/heroku/hundur"
    cmd: git clone https://github.com/heroku/erequest_id.git .tmp_dir193467206960 -b 0.2.1 --single-branch

===>    opts: [{cd,"/tmp"}]

===> Port Cmd: "git clone https://github.com/heroku/erequest_id.git .tmp_dir193467206960 -b 0.2.1 --single-branch"
Port Opts: [{cd,"/tmp"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide]

===> sh(git clone https://github.com/heroku/erequest_id.git .tmp_dir193467206960 -b 0.2.1 --single-branch)
failed with return code 128 and the following output:
Cloning into '.tmp_dir193467206960'...
fatal: could not read Username for 'https://github.com': Device not configured

Not sure what's going on here, but in any case, rebar3 should be reporting the failure without having to use DEBUG=1.

i'm working on restoring cover functionality to rebar and i've run into an obstacle

ideally, i'd like to expose cover functionality in a general manner that any provider can benefit from. when cover is present in the list of providers to be run (either as a dependency defined in another provider or as an argument to the do provider) all preceeding providers should be aware that cover is in the task list. that allows this:

$ rebar cover
# does nothing
$ rebar do eunit, ct, cover
# runs eunit and ct with any compiled files being compiled with cover support and then performs
#  cover analysis

however, there's no mechanism for providers to communicate information backwards to let other tasks know cover will be run. this can still be supported but it requires all compiled code is compiled with cover support regardless of whether the cover task is to be run. that's a lot of overhead and is potentially problematic

alternatively, a cover task can be written in a way that tasks are explicitly declared as tasks to be run during cover analysis. however, this presents it's own challenge as command line arguments to dependent tasks will not be supported

$ rebar cover
# does coverage of some default task list (probably eunit and ct)
$ rebar cover --tasks=eunit,cover,erlang.test
# does coverage of eunit, cover and erlang.test

the last option is to add explicit cover support to each provider but that makes it difficult (possibly impossible without changes to rebar's execution model) to provide aggregated coverage metrics

possible solutions:

instead of the "parse one task, execute task, parse next task" loop rebar currently does construct the entire list of tasks to be executed ahead of time and store it in the state record for providers to inspect

lazily add only providers that will actually be invoked allowing providers to advertise their existence in the task list to other tasks to be run at the time they are created

As like as type(), location(), ref() which it uses. Should they be removed?

seems that the return value of dialyzer doesn't get passed back to the shell correctly:

===> Doing success typing analysis...
===> Analyzing 16 files with "/Users/evan/src/xxxxxxx/_build/.rebar3.otp-17.4.plt"... 
xxxxxxx_app.erl:24: Function start/2 has no local return
$ echo $?
0

I had mistakenly generated dumb demo apps in the same directory:

 → DEBUG=1 rebar3 compile
===> Consult config file "rebar.config"
===> Consult config file "rebar.lock"
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {'EXIT',
                             {{case_clause,
                               ["/private/tmp/squirrels/src/dummy_proj.app.src",
                                "/private/tmp/squirrels/src/squirrels.app.src"]},

When rebar3 encounters many of these apps, it exits with an uncaught error. Maybe we should display a friendlier error message.

@talentdeficit and @ferd I don't think I made it clear what I was thinking the eunit (and another other) provider could do to set {d, 'TEST'}:

ConfigProfiles = rebar_state:get(State, profiles, []),
TestProfile = proplists:get_value(ConfigProfiles, test, []),
ErlOpts = proplists:get_value(TestProfile, erl_opts, []),
ErlOpts1 = lists:keystore(erl_opts, 1, TestProfile, {erl_opts, [{d, 'TEST'} | ErlOpts]}),
ConfigProfiles1 = lists:keystore(test, 1, ConfigProfiles, {test, ErlOpts1}),
State2 = rebar_state:set(State, profiles, ConfigProfiles1),

{ok, State1}.

This is something I can add to rebar_state so it simply takes the state, profile name and option:

rebar_state:update_profile(State, test, [{erl_opts, [{d, 'TEST'}]}])

Thoughts?

Plan TBD

It would be cool to just define a github dependency like this:

{git, "okeuday/uuid", {branch, "master"}}

This is a low-priority, feature request that I could likely submit a pull-request for myself.

Here's a fun test to replicate the bug:

1. Make a directory for a bunch of projects called projects/
2. Checkout a copy of rebar2 in that project, under any directory name.
3. Go back to the projects/
4. Call rebar3 new to list templates.

Rebar3 should crash with:

→ rebar3 new
===> Consult config file "rebar.config"
===> Consult config file "rebar.lock"
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {'EXIT',
                             {undef,
                              [{rebar_templater,new,
                                ["release",
...

Rebar3 seems to pick up the rebar_templater module from old rebar and load it locally while it runs (even if its escript is based on a totally different location).

Rebar3 is therefore at risk of crashing when run from an environment that contains unrelated (but similarly named) modules, which is a weird edge case, but could lead to really confusing bug reports.

I haven't checked for how deep rebar3 is going to go look under that hierarchy.

JSX is a good example. When I build it outside, it all works correctly.

I want to get all the current stakeholders involved in this discussion. @ferd @omarkj @talentdeficit @oubiwann @marianoguerra

In working on supporting subcommands for lfe and using erlydtl as my basis for it I'm thinking it is best supported by pushing more of the command and task handling into the providers app.

This would basically gut rebar_core, moving process_command and do to the providers module. It'll also require making a providers_state behaviour for rebar_state because functions to set state like command_args, command_parsed_args and apply_profiles are needed in process_command.

The main change to providers themselves would be init no longer returning {ok, State} but instead either {ok, providers:t() | list(), State} or {ok, providers:t() | list()}. Depending on if init should be able to update the state and if it should return a created provider or just the proplist that is used to create the provider.

I'm getting the following error when compiling merl:

===> Compiling merl
===> Compiling /home/nuex/_dev/mezzanine/_deps/merl/src/merl_transform.erl failed:
/home/nuex/_dev/mezzanine/_deps/merl/src/merl_transform.erl:none: undefined parse transform 'merl_transform'

Any ideas? rebar2 works for me.

Use case: app1 has dependency app2. app2 uses apps structure:

app2/apps/
         | - app2a
         | - app2b
         ...

Expected: app2a and app2b are available in app1.

So building a plugin doesn't require depending on rebar3 we've moved rebar.hrl to src/ to make it private. We need to still provide the functionality a user would want from rebar.hrl in their plugin and document important public functions.

rebar is unique as a build tool for writing all over and not into a single directory. mix uses _build, lein uses target, cabal uses dist, ocamlbuild uses _build and sinan used _build.

So I suggest a structure like:

_build
├── dev
│   ├── lib
│   └── releases
├── plugins
│   └── lib
├── prod
│   ├── lib
│   └── releases
└── test
    └── lib

We could support this being optional and outputting as we are currently if not selected. But since where things are outputted doesn't change actual build compatibility with rebar2 (I hope...), maybe that is a waste of effort.

Obviously we'll also need to make some optional changes to rebar.config as well to allow for these different build profiles (dev, prod, etc.) But my idea for that is to simply have anything at the top level be dev or default and you can then define a context like {prod, [....]}. where any usual rebar.config term can be within that list.

The escriptize command while mentioned on the README does not show up in the rebar3 binary nor does calling it seem to have a effect.

To replicate:

λ a → echo "{deps, [{c, ".*", {git, "https://[email protected]/ferd/c.git", "master"}}]}." > rebar.config
λ a → DEBUG=1 ~/code/self/rebar3/rebar3 compile
...
===> Consult config file "_build/default/lib/b/rebar.config"
===> Consult config file "_build/default/lib/c/rebar.config"
===> Consult config file "rebar.lock"
===> Consult config file "_build/default/lib/c/rebar.config"
===> Consult config file "_build/default/lib/b/rebar.config"
===> Consult config file "rebar.lock"
===> Consult config file "_build/default/lib/c/rebar.config"
===> Consult config file "rebar.lock"
===> Consult config file "_build/default/lib/c/rebar.config"
===> Consult config file "_build/default/lib/b/rebar.config"
===> Consult config file "rebar.lock"
===> Consult config file "_build/default/lib/b/rebar.config"
===> Consult config file "_build/default/lib/c/rebar.config"
===> Consult config file "rebar.lock"
===> Consult config file "_build/default/lib/c/rebar.config"
===> Consult config file "_build/default/lib/c/rebar.config"
===> Consult config file "rebar.lock"
^C

The issue appears to be that the transient dependencies, even if they exist and were fetched properly, will keep looking at each other forever somehow.

The two dependencies are for this reproduction are:

• https://bitbucket.org/ferd/b
• https://bitbucket.org/ferd/c

Right now hooks in rebar.config can only be commands to run with os:cmd. We want to also support providers. I think the format should be:

-type namespace() :: atom().
-type provider() :: atom() | {namespace(), atom()}.

{hooks, [{provider(), {[provider()], [provider()]}}]}. 

rebar3 ct currently uses the auto_compile feature of ct to build tests before running any suites. this means any erl_opts a project may rely on are not used in compilation

i think it probably should, right?

Running rebar3 do lock, lock will leave duplicate lock entries in the lock file.

I haven't tested it (because I'm working on something else and don't know where to add tests yet), but this patch should resolve it, in rebar_prv_install_deps:maybe_lock:

maybe_lock(Profile, AppInfo, Seen, State, Level) ->
    Name = rebar_app_info:name(AppInfo),
    case Profile of
        default ->
            case sets:is_element(Name, Seen) of
                false ->
                    AppName = rebar_app_info:name(AppInfo),
                    Locks = rebar_state:lock(State),
                    case lists:any(fun(App) -> rebar_app_info:name(AppInfo) =:= AppName end,Locks) of
                        true ->  {sets:add_element(Name, Seen), State};
                        false -> {sets:add_element(Name, Seen),
                                  rebar_state:lock(State, rebar_app_info:dep_level(AppInfo, Level))}
                    end;
                true ->
                    {Seen, State}
            end;
        _ ->
            {Seen, State}
    end.

With code coverage enabled, the build times are greatly increased. Currently, the only way (that I know of) to enable/disable code coverage on demand is to have an alternative configuration file, which then needs to be kept in sync with the original.

I'd like some way to be able to override certain config file settings. Potentially via environment variables, extra command line arguments, or by specifying more than one configuration file.

For instance, running this:

$ DEBUG=1 ./rebar3 help shell

Gives this:

===> Evaluating config script "rebar.config.script"
===> Consult config file "rebar.lock"
Start a shell with project and deps preloaded similar to
'erl -pa ebin -pa deps/*/ebin'.


===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {'EXIT',
                                {{badmatch,[]},
                                 [{getopt,usage_cmd_line,3,
                                      [{file,"src/getopt.erl"},{line,588}]},
                                  {getopt,usage,5,
                                      [{file,"src/getopt.erl"},{line,570}]},
                                  {rebar_prv_help,do,1,
                                      [{file,"src/rebar_prv_help.erl"},
                                       {line,48}]},
                                  {providers,run_all,2,
                                      [{file,"src/providers.erl"},{line,82}]},
                                  {rebar_core,do,2,
                                      [{file,"src/rebar_core.erl"},{line,64}]},
                                  {rebar3,main,1,
                                      [{file,"src/rebar3.erl"},{line,46}]},
                                  {escript,run,2,
                                      [{file,"escript.erl"},{line,752}]},
                                  {escript,start,1,
                                      [{file,"escript.erl"},{line,276}]}]}}

I mis-spent a lot of time trying to track this down, thinking it was a bug in my own plugin/providers code.

I'm on my way to a hack-around, but it would be nice if we could save the next contributor an our or so of confusion ;-)

these ones are my fault i think, my bad

An example is common_test provider which still runs even if the compile provider finds errors.

It seems that rebar3 suffers from this old bug from rebar https://github.com/rebar/rebar/issues/367

» ./rebar3 eunit
 ===> Compiling my_app
===> sh(cp -R /path/with spaces/to/my_app/test/my_app_tests.erl ".eunit")
failed with return code 2 and the following output:
sh: -c: line 0: syntax error near unexpected token `('
sh: -c: line 0: `exec cp -R /path/with spaces/to/my_app/test/my_app_tests.erl ".eunit"'

Currently app validation is a mix of rebar2 and rebar3 code. This need to be cleaned up.

It manifests itself in at least one bug, attempting to validate an app that was fetched as a binary package will blow up.

Besides that bug it is just ugly and will lead to more bugs in the future.

since there's no way to install templates from within rebar I add a chain of commands to make it easy for users to start, something like:

mkdir -p ~/.rebar3/templates && git clone https://github.com/efene/rebar3_efene_app_template.git ~/.rebar3/templates/ && rm -rf ~/.rebar3/templates/.git

the last part is because if someone else does a checkout like I do the .git from the previous checkout will be there and may cause problems.

it would be nice to do instead:

mkdir -p ~/.rebar3/templates && git clone https://github.com/efene/rebar3_efene_app_template.git ~/.rebar3/templates/fn_app

or even better ;)

rebar3 template install https://github.com/efene/rebar3_efene_app_template.git

Currently only when a profile is used by the provider setting it as the required profile to run under, like rebar_prv_common_test does with the test profile, do the deps get installed to the profile's subdirectory of _build instead of default.

erlydtl relies on merl which apparently needs to be raw:

{merl, ".*",
   {git, "git://github.com/erlydtl/merl.git", "28e5b3829168199e8475fa91b997e0c03b90d280"},
   [raw]}

Which I think means "don't compile me or do anything else like I was a real dependency".

I don't want to support non-OTP apps.. but maybe supporting deps that are basically just "download me" is ok.

I just had a thought. appups are artifacts that should be distributed with the app, meaning they should be first class with the app development tool, no the release development tool.

I'm proposing appups being treated as something every app should have if it is created with rebar3. Not enforced, but possibly created by default in the template (empty of course), and providing basic functionality for creating and updating.

Currently namespaces aren't included in the help output so a command like erlydtl compile ends up as:

$ ../rebar3/rebar3 help
....
compile         Compile apps .app.src and .erl files.
compile         Compile erlydtl templates.
....

If I run:

DEBUG=1 rebar3 new lib squirrels

in any non-empty directory, I get this output:

===> Consult config file "rebar.config"
===> Consult config file "rebar.lock"

and then it hangs forever. If I make a new directory and cd into it first, it succeeds.

Example: rebar3 do dialyzer, ct

Should it simply stop if dialyzer has errors? Should it continue but just make sure to still return 1 as the status code if only dialyzer had errors?

should be a noop, but still it's a waste of time.

I would like to be able to use suite/group/case filtering with rebar3 ct, e.g.:

rebar3 ct --suite=app_SUITE --group=[top_group1,top_group1_sub_group],top_group2

It seems many other arguments don't work as documented either.

Fetching dependencies takes a long while, and on larger projects, just running nothing (compiling a compiled project) can take upwards of 10 seconds if the project is large enough.

We should optimize for that case and reduce how long it takes to verify everything.

git url of "Working on an Existing Rebar3 Project" in http://www.rebar3.org/static/guide.html should be git://github.com/tsloughter/minasan.git

Currently each dependency is built with a new rebar_state. Meaning any options set in the top level project's rebar.config are lost, purposely.

The question is, would it be useful to allow settings dependency config variables? It would almost be most useful to be able to do it per dependency.

I hit this tonight when wanting to build jsx with {d, maps_always} in erl_opts. There is currently no way to do this without forking.

On the first rebar3 ct the test profile deps are fetched and compiled. However when common_test tries to compile the files in the test/ directory a make_failed error occurs on first run if they included a *.hrl from a test dep. If rebar3 ct is called a second time the test files are compiled by common_test successfully.

An example is available at https://github.com/fishcakez/squeue/blob/rebar3/rebar.config

A test and feature we need to add is around not having plugins and their deps in the code path when compiling.

We can test this with multiple versions of a test app used as dep and dep of a plugin included but that is missing an include in the plugin version that exists in the version expected by the app we are building.

Rationale

Maintaining features you don't use, don't have examples for, and have no interest in results in shoddy maintenance and slow, bad, development. It is often better to find maintainers for specific features or parts of code, and delegate responsibility.

Unfortunately, build tools are often boring and uninteresting, and it's far easier to make feature requests, dump code, and forget about it than it is to maintain it and keep it useful in backwards compatible manners.

The more of that unused code there is, the harder it is to find maintainers for a project, and the cost of change becomes in the code base.

Proposal

Rebar3 should be focused on a solid, shared core for features mostly everyone could use:

• Fetching dependencies on common protocols (git, maybe hg and tarballs)
• Fetching rebar3 packages
• Handle a common directory structure (apps vs. releases)
• Templates
• Plugins
• Building releases
• Running tests (CT/EUnit)
• Dialyzer
• Shell
• Archives (tar)
• Profile management
• Clean
• help

And other related features. Some features we're still not sure of may include C compiling and whatnot.

Any other feature should be delegated to the community with plugins:

• protobuffs
• support for other languages (LFE, Elixir, asn1, etc.)
• support for templating languages
• support for alternative source formats
• support for alternative package formats
• support whatever else you can think of

However, we realize the usefulness of a feature-complete tools that go beyond the basic tasks everyone uses.

As such, rebar3 should be able to be built with the 'core' features, and 'non-core' features. The non-core features would be plugins fetched and installed at compile time for rebar3, global to the install and included as if they were core.

What's the difference? These have different maintainers, timelines, and could use different release cycles along with different backwards compatibility guarantees.

I would also like to see the inclusion of a wtf command, allowing to report an issue. For example, calling rebar3 wtf ct would lead the user to https://github.com/rebar/rebar3/issues/new. On the other hand, calling rebar3 wtf lfe could lead to a repo such as https://github.com/rebar/rebar3-lfe/issues/new (or maybe their README?), where a group of dedicated maintainers could handle questions, issues, and so on, apart from core rebar3 features.

The end objective is to allow specialization on some features, along with paths to deprecate and replace non-core work without ever risking to majorly break core features.

Jiffy currently doesn't build with rebar3.

$ rebar3 eunit
...
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {'EXIT',
                             {undef,
                              [{rebar_utils,cleanup_code_path,
...

Then, with grep:

$ grep -r cleanup_code_path /tmp/rebar3/
/tmp/rebar3/src/rebar_prv_eunit.erl:            true = rebar_utils:cleanup_code_path(CodePath),
/tmp/rebar3/src/rebar_prv_eunit.erl:    true = rebar_utils:cleanup_code_path(CodePath),

Hi friends!

I'm trying to include Erlydtl in my project, but I'm hitting crashes in rebar core however I try to do it.

If I try the package manager dependency (with DEBUG=1):

in rebar.config --- {deps, [{erlydtl, "0.9.2"}]}
$ rebar3 compile
... %% Many lines ellided
===> Consult config file "/home/pablo/projects/myproject/_build/lib/ssl_verify_hostname/rebar.config"
===> Consult config file "/home/pablo/projects/myproject/_build/lib/ssl_verify_hostname/rebar.lock"
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {error,function_clause}

If I try a source dependency:

in rebar.config --- {deps, [{erlydtl, {git, "https://github.com/erlydtl/erlydtl.git", {tag, "0.10.0"}}}]}
$ rebar3 compile
... %% Many lines ellided
===> Consult config file "/home/pablo/projects/myproject/_build/lib/erlydtl/rebar.config"
===> Consult config file "/home/pablo/projects/myproject/_build/lib/erlydtl/rebar.lock"
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {error,function_clause}

I'm also open to using the plugin, but had some trouble knowing how to include it in the compile cycle (do I add it to {plugins, ...}? Will this change soon?) and wasn't sure how to add erlydtl to my kerl-managed Erlang's libraries anyways.

I'll try diving into the source for an answer, but posting here for since it's likely something you will have more insight on :)

Thanks for all the work on a great tool, and let me know if there's anything I can do to help! :D

I'm trying out rebar3 with a newly created app and adding in cowboy and I'm getting an error when I try and compile. Here is the error:

===> Consult config file "rebar.config"
===> Consult config file "rebar.lock"
===> Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace
===> Uncaught error: {'EXIT',
                             {undef,
                              [{crypto,start,[],[]},
                               {rebar3,run_aux,2,
                                [{file,"src/rebar3.erl"},{line,100}]},
                               {rebar3,main,1,
                                [{file,"src/rebar3.erl"},{line,46}]},
                               {escript,run,2,
                                [{file,"escript.erl"},{line,741}]},
                               {escript,start,1,
                                [{file,"escript.erl"},{line,277}]},
                               {init,start_it,1,[]},
                               {init,start_em,1,[]}]}}

Any ideas as to why this is happening? I'm trying this with the newest rebar3 and I'm on R15B03-1.

Hello,

When I try to build minasan a the following error appears

Uncaught error in rebar_core. Run with DEBUG=1 to see stacktrace

After running rebar3 with debug

===> Uncaught error: badmatch {ok,
                                  {{"HTTP/1.1",403,"Forbidden"},
                                   [{"date",.....

With rebar3 upgraded to latest build the error persists.

Regards.

Having these options present in a project appear to make it so the whole project will compile and fetch dependencies when running rebar3 compile, but ebin/ and the .beam files themselves will never be compiled.

Removing the options from the rebar config solves the problem.

• Source Deps fetching
• Package Deps fetching
• Compiling
• Deps command (partial tests)
• Source Deps updating
• Package Deps updating
• Templates application
• Profiles and merging
• Plugin fetching and running
• Dialyzer command
• CT
• Eunit
• Cover
• Clean
• Help
• Release building
• Tar
• do
• pre/post compile hooks
• namespaces
• xref
• erl_first_files
• locks

Omitted for hard to test without human interaction:

• Shell
• Version
• Pkgs

Anything else?

Rebar returns successfully if it fails to fetch deps appropriately during the compile command.

λ c → cat rebar.config
{deps, [{bad, ".*", {git, "[email protected]:ferd/fake-repo.git", "master"}}]}.
λ c → DEBUG=1 ~/code/self/rebar3/rebar3 compile
===> Consult config file "rebar.config"
===> Consult config file "rebar.lock"
===> Consult config file "/tmp/c/rebar.config"
===> Consult config file "rebar.lock"
===> Consult config file "/tmp/c/rebar.config"
===> Fetching bad
===> WARNING: It is recommended to use {branch, Name}, {tag, Tag} or {ref, Ref}, otherwise updating the dep may not work as expected.
===> sh info:
        cwd: "/tmp/c"
        cmd: git clone -n [email protected]:ferd/fake-repo.git .tmp_dir857810636347

===>    opts: [{cd,"/tmp"}]

===> Port Cmd: "git clone -n [email protected]:ferd/fake-repo.git .tmp_dir857810636347"
Port Opts: [{cd,"/tmp"},
            exit_status,
            {line,16384},
            use_stdio,stderr_to_stdout,hide]

===> sh(git clone -n [email protected]:ferd/fake-repo.git .tmp_dir857810636347)
failed with return code 128 and the following output:
Cloning into '.tmp_dir857810636347'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

λ c → echo $!
0

The status code remains 0 whether it runs as install_deps or compile.

And the resulting tree:

λ c → tree _build
_build
└── default
    └── lib
        └── bad