Environment <div class="highlight highlight-source-shell notranslate position-re

See <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id

My project can build successfully after checkout <a class="issue-link js-issue-link" d

limit the number of parallel build process about rebar3 HOT 3 CLOSED

shionryuu commented on May 31, 2024

limit the number of parallel build process

from rebar3.

Comments (3)

ferd commented on May 31, 2024

Hm, that number should already be limited, at least in the actual compilation.
The limit is set at

rebar3/apps/rebar/src/rebar_parallel.erl

Lines 13 to 19 in 69a5898

 queue(Tasks, WorkF, WArgs, Handler, HArgs) -> 

 Parent = self(), 

 Worker = fun() -> worker(Parent, WorkF, WArgs) end, 

 Jobs = min(length(Tasks), erlang:system_info(schedulers)), 

 ?DIAGNOSTIC("Starting ~B worker(s)", [Jobs]), 

 Pids = [spawn_monitor(Worker) || _ <- lists:seq(1, Jobs)], 

 parallel_dispatch(Tasks, Pids, Handler, HArgs).

The queue of worker is set up at:

rebar3/apps/rebar/src/rebar_compiler.erl

Lines 316 to 322 in 69a5898

 compile_parallel(Targets, Opts, BaseOpts, Mappings, CompilerMod) -> 

 Tracking = erlang:function_exported(CompilerMod, compile_and_track, 4), 

 rebar_parallel:queue( 

 Targets, 

 fun compile_worker/2, [Opts, BaseOpts, Mappings, CompilerMod], 

 fun compile_handler/2, [BaseOpts, Tracking] 

 ).

This should give you no more compile units than there are schedulers on your VM for normal Erlang code builds.

However the issue you show comes from the code server in the analysis step... It is possible that the issue comes from the OTP-provided epp module that we use to access the code's abstract files:

rebar3/apps/rebar/src/rebar_compiler_epp.erl

Lines 35 to 38 in 69a5898

 deps(File, Opts) -> 

 {EppOpts, ExtraOpts} = split_opts(Opts), 

 {ok, Forms} = epp:parse_file(File, EppOpts), 

 normalize(handle_forms(Forms, default_attrs(), ExtraOpts)).

What I've found is that we had a patch in the DAG analysis to speed it up that does call for more concurrency, unbounded one at that:

rebar3/apps/rebar/src/rebar_compiler_dag.erl

Lines 449 to 466 in 69a5898

 %% Add dependencies of a given file to the DAG. If the file is not found yet, 

 %% mark its timestamp to 0, which means we have no info on it. 

 %% Source files will be covered at a later point in their own scan, and 

 %% non-source files are going to be covered by `populate_deps/3'. 

 prepopulate_deps(Compiler, InDirs, Source, DepOpts, Control) -> 

 {Worker, _MRef} = spawn_monitor( 

 fun () -> 

 SourceDir = filename:dirname(Source), 

 AbsIncls = case erlang:function_exported(Compiler, dependencies, 4) of 

 false -> 

 Compiler:dependencies(Source, SourceDir, InDirs); 

 true -> 

 Compiler:dependencies(Source, SourceDir, InDirs, DepOpts) 

 end, 

 Control ! {deps, self(), AbsIncls} 

 end 

 ), 

 Worker.

That was added in a performance drive in 2020: #2322

Chances are that the only needed fix is to transform the DAG analysis to use a worker queue to be faster. I can try experimenting with that, it should be workable.

from rebar3.

ferd commented on May 31, 2024

See #2768 for a potential fix to this. It seems fragile, but may allow checking the early approach.

from rebar3.

shionryuu commented on May 31, 2024

My project can build successfully after checkout #2768. Thank you.

from rebar3.

limit the number of parallel build process about rebar3 HOT 3 CLOSED

Comments (3)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	queue(Tasks, WorkF, WArgs, Handler, HArgs) ->
	Parent = self(),
	Worker = fun() -> worker(Parent, WorkF, WArgs) end,
	Jobs = min(length(Tasks), erlang:system_info(schedulers)),
	?DIAGNOSTIC("Starting ~B worker(s)", [Jobs]),
	Pids = [spawn_monitor(Worker) \|\| _ <- lists:seq(1, Jobs)],
	parallel_dispatch(Tasks, Pids, Handler, HArgs).

	compile_parallel(Targets, Opts, BaseOpts, Mappings, CompilerMod) ->
	Tracking = erlang:function_exported(CompilerMod, compile_and_track, 4),
	rebar_parallel:queue(
	Targets,
	fun compile_worker/2, [Opts, BaseOpts, Mappings, CompilerMod],
	fun compile_handler/2, [BaseOpts, Tracking]
	).

	deps(File, Opts) ->
	{EppOpts, ExtraOpts} = split_opts(Opts),
	{ok, Forms} = epp:parse_file(File, EppOpts),
	normalize(handle_forms(Forms, default_attrs(), ExtraOpts)).

	%% Add dependencies of a given file to the DAG. If the file is not found yet,
	%% mark its timestamp to 0, which means we have no info on it.
	%% Source files will be covered at a later point in their own scan, and
	%% non-source files are going to be covered by `populate_deps/3'.
	prepopulate_deps(Compiler, InDirs, Source, DepOpts, Control) ->
	{Worker, _MRef} = spawn_monitor(
	fun () ->
	SourceDir = filename:dirname(Source),
	AbsIncls = case erlang:function_exported(Compiler, dependencies, 4) of
	false ->
	Compiler:dependencies(Source, SourceDir, InDirs);
	true ->
	Compiler:dependencies(Source, SourceDir, InDirs, DepOpts)
	end,
	Control ! {deps, self(), AbsIncls}
	end
	),
	Worker.