Giter Club home page Giter Club logo

horus's Introduction

Horus: anonymous function to standalone module

Hex.pm Test Codecov

Horus is a library that extracts an anonymous function's code as well as the code of the all the functions it calls, and creates a standalone version of it in a new module at runtime.

The goal is to have a storable and transferable function which does not depend on the availability of the modules that defined it or were called.

How does it work?

To achieve that goal, Horus extracts the assembly code of the anonymous function, watches all calls it does and recursively extracts the assembly code of other called functions. When complete, it creates a standalone Erlang module based on it. This module can be stored, transfered to another Erlang node and executed anywhere without the presence of the initial anonymous function's module.

If the extracted function calls directly or indirectly modules from the erts, kernel or stdlib applications, the called functions are not extracted. That's ok because the behavior of Erlang/OTP modules rarely changes and they will be available. Therefore, there is little value in extracting that code.

While processing the assembly instructions and watching function calls, Horus can use callbacks provided by the caller to determine if instructions and calls are allowed or denied.

Project maturity

Horus is still under active development and should be considered Alpha at this stage.

Documentation

Getting started

Add as a dependency

Add Horus as a dependency of your project:

Using Rebar:

%% In rebar.config
{deps, [{horus, "0.2.5"}]}.

Using Erlang.mk:

# In your Makefile
DEPS += horus
dep_horus = hex 0.2.5

Using Mix:

# In mix.exs
defp deps do
  [
    {:horus, "0.2.5"}
  ]
end

Extract an anonymous function

To extract an anonymous function, use horus:to_standalone_fun/1:

Fun = fun() ->
          do_something_fancy()
      end,

StandaloneFun = horus:to_standalone_fun(Fun).

It works with references to regular functions are well:

Log = fun logger:info/2,

StandaloneLog = horus:to_standalone_fun(Log).

Execute a standalone function

Once extracted, the function can be stored as an Erlang binary, or transfered to a remote Erlang node. You then use horus:exec/2 to execute it:

receive
    {standalone_fun, StandaloneLog} ->
        horus:exec(
          StandaloneLog,
          ["~p received and executed function", [self()]])
end.

How to build

Build

rebar3 compile

Build documentation

rebar3 edoc

Test

rebar3 xref
rebar3 eunit
rebar3 ct --sname ct
rebar3 as test dialyzer

Copyright and License

© 2021-2024 Broadcom. All Rights Reserved. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries.

This work is dual-licensed under the Apache License 2.0 and the Mozilla Public License 2.0. You can choose between one of them if you use this work.

SPDX-License-Identifier: Apache-2.0 OR MPL-2.0

horus's People

Contributors

dcorbacho avatar dumbbell avatar janpieper avatar kianmeng avatar michaelklishin avatar the-mikedavis avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

janpieper

horus's Issues

Support cover-compiled modules

Why

Currently, when an anonymous function is extracted from a cover-compiled module, we loose the cover-compilation because Horus takes the assembly from the original module, not the cover-compiled copy.

How

It should be possible to use the cover-compiled module assembly instead to keep cover support.

Use a custom disassembler based on `beam_disasm`

Is your feature request related to a problem? Please describe.

Horus currently uses the beam_disasm module from the compiler application in Erlang/OTP to disassemble (convert BEAM chunks into instructions). beam_disasm had some bugs involving mistranslated instructions (for example erlang/otp#6447) which are papered over by Horus (for example see pass1_process_instructions/3). These mistranslations are fixed upstream but we must wait until Erlang/OTP 26 to take full advantage of the upstream fixes and drop compatibility code in Horus.

Also, beam_disasm in Erlang/OTP 26 cannot decode modules compiled with the Erlang/OTP 25 compiler. Erlang/OTP 26 introduced a new format for the Type BEAM chunk and also dropped all code for serializing and deserializing the version 1 type chunk from Erlang/OTP 25. This could happen again to this particular chunk or to any other BEAM chunk. A custom disassembler could support multiple BEAM chunk versions.

Describe the solution you'd like

Horus could implement a custom disassembler based on beam_disasm. This would allow us to fix the mistranslations in-tree and drop the compatibility code now. We could also support multiple BEAM chunk versions and have fine control over which Erlang/OTP versions are supported.

The downside to a custom disassembler is that it's more work to maintain. beam_disasm in Erlang/OTP provides a good guide but sometimes has bugs as mentioned. This part of Horus would need some work for every Erlang/OTP major release that introduces new instructions or changes the format of BEAM chunks. (Every major release since Erlang/OTP 20 has added new instructions.)

Describe alternatives you've considered

We could restrict the versions recommended for running Rabbit releases that include Horus and Khepri (see https://www.rabbitmq.com/which-erlang.html) to work around the version compatibility problem. The compatibility code for mistranslations could stay in-tree until Horus depends on Erlang/OTP 26 as a minimum.

Additional context

No response

Could exceptions have stacktraces originating from the initial anonymous function?

Why

Currently, if an extracted standalone function throws an exception, the stacktrace will point to the generated module. Therefore, the stacktrace will have very little sense and help the developer to locate the real line of code which failed.

How

It would be nice if the generated module could rewrite the exception's stacktrace to point to the original source code.

Support `maybe ... end` expressions

Why

I guess the maybe expressions come with new Beam instructions and we don't handle them right now.

How

We at least need to add a testcase to make sure horus can extract them, and add support for the related instructions if there are.

OTP 27 will break coverage support

Before OTP 27, cover worked by inserting code to do counters:add(CRef, LineIndex, 1) around any significant code. The CRef is a counter for the original module's coverage, so Horus only needed to preserve those instructions in the standalone module in order to have the function's coverage count for the original module.

OTP 27 introduces "native" coverage (introduced in erlang/otp#7856). The compiler now emits executable_line instructions when passed the force_line_counters (or line_coverage option, depending on the system's code:get_coverage_mode/0). The JIT turns that that instruction into an atomic add or literal mov that updates a section of the BeamCodeHeader's coverage buffer (see x86, arm). BeamCodeHeader is a member of each module instance (erl_module_instance) and the BIF that cover uses to read coverage information when native coverage is enabled reads from this new buffer from the current module. (As a consequence, old coverage is now no longer accessible after upgrading/reloading a module.)

The old cover code is still around for architectures that use the emulator but we can't turn off native coverage on a JIT-enabled build of OTP.

Since coverage information is now attached to module instances, I don't believe we can trick cover into counting executions of horus functions anymore.

Add a helper to execute an anonymous function remotely

It's possible to use erpc:call/2 or erpc:multicall/2 to execute an anonymous function remotely. However, it relies on the fact that the module is available on the remote node.

It would be nice to cover the use case where the remote node does not have the module where the anonymous function is defined. Horus could extract the function and load it on the remote node.

The API could look like:

-spec exec(Node | Nodes, StandaloneFun, Args) -> Ret when
      Node :: node(),
      Nodes :: [node()],
      StandaloneFun :: horus_fun(),
      Args :: [any()]).

Or remote_exec/3 or rpc/3 if exec/3 is too confusing vs. exec/2.

Add `erts` as an explicit dependency

Why

Horus relies on the availability of erts to initialize the list of modules to skip when it needs to decide if a call should be followed to extract a function or if the call should be preserved as is.

It already depends on kernel and stdlib for instance. I assumed that erts was implicit as I don't see how Erlang can work without it. It looks like if erts is not included explicitly in an Erlang release file or in the applications it bundles, it is excluded from the release (the erts Erlang application, not the actual runtime). This breaks Horus because it won't know about e.g. the erlang module and will try and fail to extract functions from that module.

See rabbitmq/khepri#212 and rabbitmq/khepri#222 for users who encountered this problem. A function extraction will fail with {horus_ex, module_not_found, #{module => erlang}} if erts is missing from a release.

How

It looks like a bug in the way releases are created because, to me, it makes no sense to not always include erts (like kernel and stdlib).

As a workaround, we should include erts in the dependencies of Horus.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.