revng / orchestra Goto Github PK

rev.ng configuration for orchestra. The starting point for any development.

License: MIT License

Dockerfile 0.31% Shell 32.85% Python 50.61% C 14.42% JavaScript 1.82%

orchestra's Introduction

The rev.ng binary analysis framework and decompiler

Welcome to the rev.ng project!

rev.ng is an open source binary analysis framework based on LLVM and QEMU. It features a full-fledged decompiler. It's developed by rev.ng Labs in the open.

This repository contains the source code of the rev.ng infrastructure and of the lifter. You can find the source of the decompiler in the revng-c repository. Eventually, these two repositories will be merged.

Getting started

Head to the online documentation to get started (source).

Getting in touch

You can get in touch with us through the following means:

X/Twitter, for the latest news about the project;
Discord, a place for chatting for users and developers;
Discourse, for user discussions;
GitHub, for development issues and PRs;
E-mail, for anything else;

License

The rev.ng binary analysis framework and decompiler is Free Software.

rev.ng depends on certain on GPLv2 components (in particular, QEMU), so the project is GPLv2-licensed as a whole. However, the individual files are licensed under the MIT license.

orchestra's People

Contributors

Stargazers

Watchers

Forkers

carpikes pfez fcremo tszdanger ztehryan cvelth tvandera aleclearmind mrjackv dwarfyresearch amesianx 00mjk depau-forks addaddgo andrealinux1 gmh5225 zommiommy

orchestra's Issues

macOS support

Are there any plans to support running on MacOS？

Restore macOS and mingw toolchains

In order to improve and test our support for Mach-O and PE/COFF binary formats we need working toolchains so that we can extend revng-qa to include tests for those.

How do I use it?

I'm a new hand and after following the steps in the readme, I find the following question.

I just rename the global options to the user_options.yml.

/home/revng/orchestra]# orc components
[+] ERROR - Subprocess failed with return code 1
[+] ERROR - The command was:
['/root/.local/lib/python3.8/site-packages/orchestra/model/configuration/../../support/ytt', '--dangerous-allow-all-symlink-destinations', '-f', '/home/revng/orchestra/.orchestra/config']
[+] ERROR - Stdout was:
[+] ERROR - Stderr was:
ytt: Error: Overlaying data values (in following order: global_options.yml, user_options.yml):
Document on line user_options.yml:19:
Map item (key 'branches') on line user_options.yml:35:
Array item on line user_options.yml:36:
Expected array item to have 'overlay/match' annotation
Traceback (most recent call last):
File "/root/.local/bin/orc", line 8, in
sys.exit(main())
File "/root/.local/lib/python3.8/site-packages/orchestra/init.py", line 46, in main
return _main(sys.argv[1:])
File "/root/.local/lib/python3.8/site-packages/orchestra/init.py", line 39, in _main
return_code = cmd_parser.handler(args)
File "/root/.local/lib/python3.8/site-packages/orchestra/cmds/components.py", line 41, in handle_components
config = Configuration(use_config_cache=args.config_cache)
File "/root/.local/lib/python3.8/site-packages/orchestra/model/configuration/configuration.py", line 58, in init
self.parsed_yaml = generate_yaml_configuration(self.orchestra_dotdir, use_cache=use_config_cache)
File "/root/.local/lib/python3.8/site-packages/orchestra/model/configuration/_generate.py", line 27, in generate_yaml_configuration
expanded_yaml = get_subprocess_output(
File "/root/.local/lib/python3.8/site-packages/orchestra/actions/util/init.py", line 118, in get_subprocess_output
return _get_subprocess_output(
File "/root/.local/lib/python3.8/site-packages/orchestra/actions/util/impl.py", line 245, in _get_subprocess_output
raise OrchestraException(err_msg)
orchestra.util.OrchestraException: Subprocess failed with return code 1

So I wonder how to install it as the https://rev.ng/getting-started.html says.

Thx a lot!

orc fails with strange error - "'NoneType' object has no attribute 'encode'"

Hi,

I installed orchestra as mentioned in the README.md file:

pip3 install --user --force-reinstall https://github.com/revng/revng-orchestra/archive/master.zip

... but I am getting a strange error with orchestra commands, e.g. (with Python 3.7 on Debian 10):

$ git clone https://github.com/revng/orchestra
$ cd orchestra
$ orc components
[+] INFO - This is the first time you run orchestra, welcome!
[+] INFO - Creating default user options in .orchestra/config/user_options.yml
[+] INFO - Populating default remotes for repositories and binary archives
[+] INFO - Remember to run `orc update` next
[+] INFO - Checking for redirects in https://github.com/revng/binary-archives
[+] INFO - Redirecting to https://rev.ng/gitlab/revng/binary-archives.git
Traceback (most recent call last):
  File "/home/edrdo/.local/bin/orc", line 10, in <module>
    sys.exit(main())
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/__init__.py", line 28, in main
    return cmd_parser.handler(args)
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/cmds/components.py", line 35, in handle_components
    config = Configuration(use_config_cache=args.config_cache)
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/model/configuration.py", line 110, in __init__
    self._parse_components()
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/model/configuration.py", line 247, in _parse_components
    set_self_hash(component, component_yaml)
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/model/configuration.py", line 434, in set_self_hash
    to_hash += component.commit().encode("utf-8")
AttributeError: 'NoneType' object has no attribute 'encode'

Another example:

$ orc environment
Traceback (most recent call last):
  File "/home/edrdo/.local/bin/orc", line 10, in <module>
    sys.exit(main())
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/__init__.py", line 28, in main
    return cmd_parser.handler(args)
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/cmds/environment.py", line 13, in handle_environment
    config = Configuration(use_config_cache=args.config_cache)
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/model/configuration.py", line 110, in __init__
    self._parse_components()
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/model/configuration.py", line 247, in _parse_components
    set_self_hash(component, component_yaml)
  File "/home/edrdo/.local/lib/python3.7/site-packages/orchestra/model/configuration.py", line 434, in set_self_hash
    to_hash += component.commit().encode("utf-8")
AttributeError: 'NoneType' object has no attribute 'encode'

Any idea why this happens and how it can be fixed?

Thanks

Improve error message when local commit doesn't match binary archive

If you have a local clone of a component that you are not building from source, orchestra still assumes that you want the commit that's locally checked out. Hence, if your local repository is not checked-out at the latest commit of the default branch, even if it is not listed among the build_from_source components, installing it will fail with an error message like:

Binary archive <THE_REPO>/<HASH>.tar.xz for <COMPONENT> not found. Try `orc update`

Proposed Solution

Improve the error message with an indication of whether there is a mismatch between the local checkd-out commit (and possibly branch) and the binary archive hash, suggesting to switch to the default branch and orc update if that is not the case.

Investigate suggestions received from LLVM Distributors Conference

CMake option LLVM_APPEND_VC_REV=Off should be enough to exclude the git remote from llvm-config --version
investigate LLVM_ENABLE_ABI_BREAKING_CHECKS which might help with the public ABI changing depending whether NDEBUG is set

Failure when trying to reproduce the README instructions

Hi, I follow the instruction to install revng. But orchestra cannot work correctly to install rev.ng https://github.com/revng/orchestra/tree/master#building-from-source.

I type orc update and get

[+] WARNING - The remote HEADs cache does not exist, you should run `orchestra update`
[+] INFO - Updating orchestra configuration
[+] INFO - Updating binary archives
0archives [00:00, ?archives/s]
[+] INFO - Resetting ls-remote cached info
[+] INFO - Updating ls-remote cached info
[+] INFO -                                      
Could not find the following repositories in any remote:
  - vscode
  - llvm-project
  - revng-qa
  - revng
  - revng-c
  - revng-vscode-theme
  - llvmcpy
  - revng-ui
  - qemu

You will not be able to install components that depend on them.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 931.15repository/s]

I guess it does not matter because revng is built from source. So I ignore it and continue the next step.
I type orc install --test revng and get error

ytt: Error: 
- assert.fail: fail: You must provide a data value for remote_base_urls in datavalue lib/util.lib.yml:9 | #@       
- assert.fail("You must provide a data value for " + attrname) in <top-level> components.yml:41 | remote_base_urls: #@ datavalue("remote_base_urls")

Because build from source, I comment variable remote_base_urls and binary_archives and get return

ytt: Error: Overlaying (in following order: components/boost.yml, components/bzip2.yml, components/clang-release.yml, components/cmake.yml, components/cython.yml, components/early-gcc.yml, components/elfutils.yml, components/expat.yml, components/gcc-runtime.yml, components/glib.yml, components/glibc.yml, components/gmp.yml, components/icu.yml, components/ida.yml, components/libX.yml, components/libb2.yml, components/libedit.yml, components/libffi.yml, components/libgcc.yml, components/libltdl.yml, components/libunwind.yml, components/libxcrypt.yml, components/libyaml.yml, components/lit.yml, components/llvm-documentation.yml, components/llvm.yml, components/llvmcpy.yml, components/meson.yml, components/mpc.yml, components/mpfr.yml, components/ncurses.yml, components/ninja.yml, components/nodejs-support.yml, components/nodejs.yml, components/openssl.yml, components/python-base-packages.yml, components/python.yml, components/pyyaml.yml, components/qemu.yml, components/readline.yml, components/revng-c.yml, components/revng-development.yml, components/revng-distributable.yml, components/revng-js_prerequisites.yml, components/revng-python-dependencies.yml, components/revng-qa-python-dependencies.yml, components/revng-qa.yml, components/revng-test.yml, components/revng-theme.yml, components/revng-ui.yml, components/revng-vscode-web.yml, components/revng.yml, components/source-highlight.yml, components/sqlite.yml, components/tcl.yml, components/toolchain/arch/aarch64.yml, components/toolchain/arch/arm.yml, components/toolchain/arch/host.yml, components/toolchain/arch/i386.yml, components/toolchain/arch/mips.yml, components/toolchain/arch/mipsel.yml, components/toolchain/arch/s390x.yml, components/toolchain/arch/win32.yml, components/toolchain/arch/win64.yml, components/toolchain/arch/x86_64.yml, components/toolchain/host-toolchain.yml, components/toolchain/macos/clang.yml, components/toolchain/macos/ld64.yml, components/toolchain/macos/libtapi.yml, components/toolchain/macos/sdk.yml, components/toolchain/msvc.yml, components/vscode-web.yml, components/xz.yml, components/zlib.yml, lib/overlays/build-from_source.yml):
  Document on line components/boost.yml:55:
    Expected number of matched nodes to be 1, but was 2 (lines: components.yml:1, user_options.yml:1)

So, I delete components.yml and leave user_options.yml.
user_options.yml contains

#@overlay/replace    
build_from_source:   
      - revng

and get return

[+] ERROR - Invalid configuration. Got the following error at path $:
Additional properties are not allowed ('build_from_source' was unexpected)

Could you tell me how to install revng? I guess the guide of revng installation has something wrong.

Add tests and/or commit hooks

Opening this issue to keep track of this TODO.
I think we should discuss some tests that warn (our outright deny) when committing/pushing a configuration which, for example:

causes ytt to error
contains malformed components
does not contain a set of expected components

Provide an overview of the projects in the README.md

As per title. People is not aware of the fact that the UI is ui/cold-revng.

Purpose of clang-release / link-only

Hi,

what it the purpose of having a separate clang-release compiled? Is it related to the link-only installation?
Wouldn't it be better to install it in separate prefix?

I'm asking because I want llc with debug symbols, but this is part of clang-release.

Cheers,
Tom

cannot Initialise default configuration

Hey,

I am trying to install orchestra script on raspberry pi (ARM) but I have got an issue to initialise the default configuration.
I got the following error:
OSError: [Errno 8] Exec format error: '/home/pi/.local/lib/python3.7/site-packages/orchestra/model/configuration/../../support/ytt'

How can I solve it please ??

Thank you in advance

`Terminals database is inaccessible` when running `clear` from within `orc shell`

Problem

Executing clear from within orc shell returns a Terminals database is inaccessible error, while outside from the shell it works fine.

Temporary fix

By running strace clear from both within and outside the orc shell and comparing results I was able to observe that the error happens when trying to access terminfo. In particular, the outside clear command looks at the following locations:

stat("~/.terminfo", 0x56005bf064d0) = -1 ENOENT (No such file or directory)
stat("/etc/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/lib/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/share/terminfo", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

while orchestra/root/bin/clear looks just at

stat("~/.terminfo", 0x559ca76fe4c0) = -1 ENOENT (No such file or directory)
stat("/builds/gitlab/revng/orchestra/orchestra/root/share/terminfo", 0x559ca76fe4c0) = -1 ENOENT (No such file or directory)

and exits. The temporary fix for this is adding a symlink to the real terminfo file inside the user's home directory.

rev.ng/gitlab is down?

I'm following the directions for building with orchestra, but the checkout of https://rev.ng/gitlab/revng/binary-archives.git stalls. curling rev.ng/gitlab also doesn't respond. Is rev.ng/gitlab down?

I don't mind building completely from source, but the instructions are vague on how to do that. The Getting Started instructions (clearing BINARY_COMPONENTS) are out of date and the orchestra readme only says how to build revng itself from source, not its dependencies.

CI checks out `orchestra` `master` instead of `next-master` when pushing on `master` on some other repo

If I push the master branch of llvm-project, the CI is triggered with COMPONENT_TARGET_BRANCH=master.
This leads to check out the master branch of orchestra, which is undesired.

I suggest we blacklist master, develop, next-master and next-develop from being considered in COMPONENT_TARGET_BRANCH. I think the use case for COMPONENT_TARGET_BRANCH is just for feature/... branches.

@fcremo, what do you think?

clangd does not use flags provided through hard_compiler_wrapper.py

With the latest orchestra, the flags provided through hard_compiler_wrapper.py are passed directly to the compiler and they are not written in compile_commands.json, which is used by clangd to provide code completion.

At the moment, the only flag which is causing problems with ui/cold-revng is -fstdlib=libc++ which would be useful to write in some way in the compile_commands file.

GCC does not build on Archlinux

toolchain/host/gcc, toolchain/x86-64/gcc and probably others fail to build on Archlinux with errors like

/home/fcremo/projects/revng-orchestra/build/toolchain/host/gcc/stage2/./gcc/xgcc -B/home/fcremo/projects/revng-orchestra/build/toolchain/host/gcc/stage2/./gcc/ -B/home/fcremo/projects/revng-orchestra/root/x86_6
4-pc-linux-gnu/bin/ -B/home/fcremo/projects/revng-orchestra/root/x86_64-pc-linux-gnu/lib/ -isystem /home/fcremo/projects/revng-orchestra/root/x86_64-pc-linux-gnu/include -isystem /home/fcremo/projects/revng-orc
hestra/root/x86_64-pc-linux-gnu/sys-include    -I/usr/include/tirpc --sysroot /home/fcremo/projects/revng-orchestra/root/link-only -idirafter /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idiraft
er /usr/include -O2  -I/usr/include/tirpc --sysroot /home/fcremo/projects/revng-orchestra/root/link-only -idirafter /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include -DIN_GCC -
fPIC   -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wstrict-prototypes -Wmissing-prototypes -Wno-error=format-diag -Wold-style-definition  -isystem ./include  -fpic -mlong-double-
80 -DUSE_ELF_SYMVER  -g -DIN_LIBGCC2 -fbuilding-libgcc -fno-stack-protector  -fpic -mlong-double-80 -DUSE_ELF_SYMVER  -I. -I. -I../.././gcc -I/home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/lib
gcc -I/home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/. -I/home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/../gcc -I/home/fcremo/projects/revng-orchestra/sources/t
oolchain/host/gcc/libgcc/../include -I/home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/config/libbid -DENABLE_DECIMAL_BID_FORMAT -DHAVE_CC_TLS  -DUSE_TLS -o _gcov_ior_profiler.o -MT _gcov
_ior_profiler.o -MD -MP -MF _gcov_ior_profiler.dep -DL_gcov_ior_profiler -c /home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/libgcov-profiler.c
In file included from /home/fcremo/projects/revng-orchestra/build/toolchain/host/gcc/stage2/gcc/include-fixed/pthread.h:38,
                 from /home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/generic-morestack-thread.c:41:
/usr/include/bits/types/struct_timespec.h:10:8: error: redefinition of ‘struct timespec’
   10 | struct timespec
      |        ^~~~~~~~
In file included from /home/fcremo/projects/revng-orchestra/root/link-only/include/sys/select.h:44,
                 from /home/fcremo/projects/revng-orchestra/root/link-only/include/sys/types.h:220,
                 from /home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/../gcc/tsystem.h:90,
                 from /home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/generic-morestack-thread.c:27:
/home/fcremo/projects/revng-orchestra/root/link-only/include/time.h:120:8: note: originally defined here
  120 | struct timespec
      |        ^~~~~~~~
In file included from /home/fcremo/projects/revng-orchestra/build/toolchain/host/gcc/stage2/gcc/include-fixed/pthread.h:38,
                 from /home/fcremo/projects/revng-orchestra/sources/toolchain/host/gcc/libgcc/generic-morestack-thread.c:41:

The issue might be caused by where GCC's fixinc takes pthread.h from (the host filesystem instead of $ORCHESTRA_ROOT/link-only).

Docker reproducer (issue cmds in docker run -it --rm -v $ORCHESTRA:/orchestra archlinux /bin/bash):

cd /orchestra
.orchestra/support/revng-distributable/dockers/install-revng-dependencies
pacman -Syu gcc diffutils m4 make patch git 
curl https://bootstrap.pypa.io/get-pip.py | python # pip as shipped by archlinux is slightly broken
pip3 install --user --force-reinstall https://github.com/revng/revng-orchestra/archive/master.zip
export PATH=$HOME/.local/bin:$PATH
orc install -B toolchain/host/gcc

List dependencies?

Instead of us troddingly along and painstakingly trying to figure out dependencies for orchestra, perhaps a nice text file that "describes" the required dependencies.

Something like DEPENDENCIES.txt?

Orchestra fails to install anything on a `NTFS`-drive.

Command orc install <package> fails when it targets a drive formatted as NTFS.
Reported error is:

<timestamp> DEBUG - The following script is going to be executed:
cp -farl "$TMP_ROOT/$ORCHESTRA_ROOT/." "$ORCHESTRA_ROOT"
<timestamp> ERROR - Script failed with return code 1
<timestamp> ERROR - The script was: 
cp -farl "$TMP_ROOT/$ORCHESTRA_ROOT/." "$ORCHESTRA_ROOT"
<timestamp> ERROR - The output was: 
cp: cannot create hard link '/mnt/b/rn/orchestra/root/./lib' to '/mnt/b/rn/orchestra/.orchestra/tmproot/zlib_default/'
'/mnt/b/rn/orchestra/root/./lib': Is a directory

Installation on an ext4 formatted drive worked just fine.