Giter Club home page Giter Club logo

redub's Introduction

Redub - Dub Based Build System

Running redub without having it on path

  • Change directory to the project you want to build and enter in terminal dub run redub
  • You may also get help by running dub run redub -- --help

Building redub

  • Enter in terminal and execute dub
  • Highly recommended that you build it with dub build -b release-debug --compiler=ldc2 since this will also improve its speed on dependency resolution
  • I would also add redub/bin to the environment variables, with that, you'll be able to simply execute redub in the folder you're in and get your project built and running

Using its library API

The usage of the library APIispretty straightforward. You get mainly 2 functions

  1. resolveDependencies which will parse the project and its dependencies, after that, you got all the project information
  2. buildProject which will get the project information and build in parallel
import redub.api;
import redub.logging;

void main()
{
  import std.file;
  //Enables logging on redub
  setLogLevel(LogLevel.verbose);

  //Gets the project information
  ProjectDetails d = resolveDependencies(
    invalidateCache: false,
    std.system.os,
    CompilationDetails("dmd", "arch not yet implemented", "dmd v[2.105.0]"),
    ProjectToParse("configuration", getcwd(), "subPackage", "path/to/dub/recipe.json (optional)")
  );

  /** Optionally, you can change some project information by accessing the details.tree (a ProjectNode), from there, you can freely modify the BuildRequirements of the project
  * d.tree.requirements.cfg.outputDirectory = "some/path";
  * d.tree.requirements.cfg.dFlags~= "-gc";
  */

  //Execute the build process
  buildProject(d);
}

Project Meta

Making it faster

Have you ever wondered why dub was slow? I tried solving it, but its codebase was fairly unreadable. After building this project, I've implemented features that dub don't use

  • Lazy build project configuration evaluation
  • Parallelization on build sorted by dependency
  • Faster JSON parsing
  • Fully parallelized build when only link is waiting for dependencies

Philosophy

  • Separate build system from package manager.
  • Have total backward compatibility on dub for initial versions.
  • On initial versions, develop using phobos only
  • Make it less stateful.
  • Achieve at least 90% of what dub does.
  • Isolate each process. This will make easier for adding and future contributors

Achieving it

Legend

  • api -> Can be freely be imported from any module
  • module -> Needs to be isolated as much as possible from each module. If one needs to communicate with other, a bridge/api may be created after the initial idea

How it works

Here are described the modules which do most of the work if someone wants to contribute.

  • buildapi: Defines the contents of build configurations, tree of projects and commons for them
  • parsers.json: Parse dub.json into a build configuration
  • parsers.automatic: Parse using an automatic parser identification
  • cli.dub + app: Parse CLI to get the build root and an initial build configuration
  • parsers.environment: Merge environment variables into build configuration
  • tree_generators.dub: Output build dependency tree while merging their configurations
  • command_generator.automatic: Convert build configuration into compilation flags
  • buildapi + building.compile: Transform build tree into dependency order tree
  • building.compile: Spawn build processes for the dependencies until it links

Contributor Attracting

  • Isolate module as much as possible to attract contributors working on self contained modules which only gets input and returns an output

Starting small

  • No need to handle edge cases in the beginning. They may become even separate modules.

A week project

  • This project had a small start. I gave one week for doing it, but since it was very succesful on its achievements, I decided to extend a little the deadline for achieving support. Right now, it has been tested with

Working examples

Those projects were fairly tested while building this one

  • dub
  • glui
  • dplug
  • arsd-official
  • Hipreme Engine

Package manager

  • Will be implemented as a hook in the build system. I'm accepting PRs, but I'm not implementing myself.

redub's People

Contributors

mrcsnm avatar amlal-elmahrouss avatar jacob-carlborg avatar amlalelmahrouss avatar ichordev avatar

Stargazers

 avatar  avatar Jeremy Baxter avatar Victor Fernandez avatar Nicol  avatar Su avatar Lawrence Aberba avatar Bagomot avatar  avatar  avatar Paul Crane avatar  avatar ZyeByte avatar zoujiaqing avatar  avatar  avatar dd avatar Ezneh avatar  avatar Ki Rill avatar Serg Gini avatar  avatar  avatar  avatar  avatar Artha avatar

Watchers

 avatar  avatar  avatar

redub's Issues

Compilation error

error message:
hipjson/source/hipjson.d(175,50): Error: found T when expecting )
hipjson/source/hipjson.d(175,51): Error: found ) when expecting ;
hipjson/source/hipjson.d(175,52): Deprecation: use { } for an empty statement, not ;
Error ldc2 failed with exit code 1.

Branch:
main

Compiler:
LDC - the LLVM D compiler (1.30.0)
DMD64 D Compiler v2.102.2

Command:
dub build -b release-debug --compiler=ldc2

Can you please advice what is going on. Is there are any compiler version limitation?

Allow using local compiler installation.

dub can use a compiler executable from any directory, while redub can so far only handle global installs.

dub build --compiler=./ldc2-1.32.0-linux-x86_64/bin/ldc2
redub build --compiler=./ldc2-1.32.0-linux-x86_64/bin/ldc2
object.Error@(0): Unsupported compiler: ./ldc2-1.32.0-linux-x86_64/bin/ldc2
----------------
??:? immutable(char)[] building.cache.getCompilerVersion(immutable(char)[]) [0x5d2f9df8744f]
??:? immutable(char)[] building.cache.hashFrom(const(buildapi.BuildRequirements), immutable(char)[]) [0x5d2f9df87d69]
??:? building.cache.CompilationCache[] building.cache.cacheStatusForProject(buildapi.ProjectNode, immutable(char)[]) [0x5d2f9df87a66]
??:? void building.cache.invalidateCaches(buildapi.ProjectNode, immutable(char)[]) [0x5d2f9df87bcb]
??:? app.ProjectDetails app.resolveDependencies(immutable(char)[][]) [0x5d2f9df809d3]
??:? app.ProjectDetails app.buildBase(immutable(char)[][]) [0x5d2f9df80454]
??:? int app.buildMain(immutable(char)[][]) [0x5d2f9df8041c]
??:? _Dmain [0x5d2f9df7fe19

Crash building

Testing Environment

OS: ArchLinux & Windows 11
Arch: x86_64 (zen)
Compiler: ldc2-master (1.36.0+)

I tried to run redub during my tests for dub support. But I didn't expect it to fail because it doesn't happen with dub.

redub build
$> LD_LIBRARY_PATH=$PWD/zig-out/lib dub run redub -y -- :clear --compiler=ldc2
             Building package redub in /home/kassane/.dub/packages/redub/1.2.13/redub/
     Warning 
     Warning ## Warning for package redub ##
     Warning 
     Warning The following compiler flags have been specified in the package description
     Warning file. They are handled by DUB and direct use in packages is discouraged.
     Warning Alternatively, you can set the DFLAGS environment variable to pass custom flags
     Warning to the compiler, or use one of the suggestions below:
     Warning 
     Warning debugMode: Call DUB with --build=debug
     Warning debugInfo: Call DUB with --build=debug
     Warning 
    Starting Performing "debug" build using ldc2 for x86_64.
  Up-to-date colorize ~master: target for configuration [library] is up to date.
  Up-to-date semver ~master: target for configuration [library] is up to date.
  Up-to-date redub 1.2.13: target for configuration [cli] is up to date.
    Finished To force a rebuild of up-to-date targets, run again with --force
     Running ../.dub/packages/redub/1.2.13/redub/build/redub :clear --compiler=ldc2
Dependencies resolved in 22 ms.
Build Failure: 'clear  [] ' using flags
        'ldc2' --enable-color=true --d-version=Have_clear -I/home/kassane/sokol-d/src -c --od=/tmp/.redub --of=/home/kassane/sokol-d/build/clear.o /home/kassane/sokol-d/src/examples/clear.d /home/kassane/sokol-d/src/sokol/log.d /home/kassane/sokol-d/src/sokol/utils.d /home/kassane/sokol-d/src/sokol/audio.d /home/kassane/sokol-d/src/sokol/debugtext.d /home/kassane/sokol-d/src/sokol/gfx.d /home/kassane/sokol-d/src/sokol/gl.d /home/kassane/sokol-d/src/sokol/shape.d /home/kassane/sokol-d/src/sokol/time.d /home/kassane/sokol-d/src/sokol/glue.d /home/kassane/sokol-d/src/sokol/app.d /home/kassane/sokol-d/src/examples/clear.d /home/kassane/sokol-d/src/handmade/math.d /home/kassane/sokol-d/src/shaders/blend.d /home/kassane/sokol-d/src/shaders/bufferoffsets.d /home/kassane/sokol-d/src/shaders/cube.d /home/kassane/sokol-d/src/shaders/instancing.d /home/kassane/sokol-d/src/shaders/mrt.d /home/kassane/sokol-d/src/shaders/noninterleaved.d /home/kassane/sokol-d/src/shaders/offscreen.d /home/kassane/sokol-d/src/shaders/quad.d /home/kassane/sokol-d/src/shaders/shapes.d /home/kassane/sokol-d/src/shaders/texcube.d
Failed after 6041ms with message
        Error: module `examples.clear` from file /home/kassane/sokol-d/src/examples/clear.d is specified twice on the command line

object.Error@(0): Build failure
----------------
??:? [0x5ef5a3f0a13e]
??:? [0x5ef5a3f09db2]
??:? [0x5ef5a3f3043e]
??:? [0x5ef5a3f11a6f]
../.dub/packages/redub/1.2.13/redub/source/redub/api.d:84 [0x5ef5a3e8ea9d]
../.dub/packages/redub/1.2.13/redub/source/app.d:70 [0x5ef5a3e8f8cb]
../.dub/packages/redub/1.2.13/redub/source/app.d:63 [0x5ef5a3e8f746]
??:? [0x5ef5a3f1174c]
??:? [0x5ef5a3f11646]
??:? [0x5ef5a3f1149c]
/home/kassane/ldcup/ldc2-bb083913-linux-x86_64/bin/../import/core/internal/entrypoint.d:42 [0x5ef5a3e91851]
??:? [0x7679dd1f7ccf]
??:? __libc_start_main [0x7679dd1f7d89]
??:? [0x5ef5a3e35b04]
Error Program exited with code 1
dub build
$> LD_LIBRARY_PATH=$PWD/zig-out/lib dub :clear --compiler=ldc2 
    Starting Performing "debug" build using ldc2 for x86_64.
  Up-to-date sokol-d:clear 0.1.0: target for configuration [application] is up to date.
    Finished To force a rebuild of up-to-date targets, run again with --force
     Running build/sokol-d_clear 
Backend: Glcore33

Reference

Problems building and other things

I wanted to try redub, but I had some problems with it.
Some info about my setup:

  • OS: Ubuntu Studio 23.10
  • DMD64 D Compiler v2.107.0
  • LDC 1.30.0 based on DMD v2.100.1 and LLVM 14.0.6

So:

  1. I cloned the repo and run dub -b debug-release --compiler=ldc2 but I get this error message: Unknown build type for redub: 'debug-release'. Building with dub build seems to work.
  2. I can build redub with redub but I can't build my project with redub. I get this error message: core.exception.RangeError@/home/alex/Documents/redub/source/redub/parsers/environment.d(259): Range violation. My project depends on one library. I also put that library inside my project's source folder but I had the same error.

For some extra info about 2, the library that I use with my project is this one.

Support redub test

Doing redub test in, for instance, https://gitlab.com/per.nordlow/gmp-d fails as

Dependencies resolved in 6 ms for "debug" using dmd
Project gmp-d is fully parallelizable! Will build everything at the same time
Up-to-Date: gmp-d  []. Took 0ms
Up-to-Date: gmp-d, skipping linking
Built project in 0 ms.
Error Program exited with code 1

. Supporting this would be great.

Warning about compiler

I tried Redub on one of my projects and got this warning:

Warning: The compiler 'dmd' that was specified in your system wasn't found. Redub found ldc2 and it will use for this compilation.

How to use

The README does not show how to go about using this package

lflags don't get applied to libs

For whatever reason, redub doesn't seem to apply lflags for projects with targetType "library".

Here I attach a very simple project as an example:
nuklear_test.zip

For security reasons I did not attach the libmything.a library file, as it's a binary. To create the libmythin.a you can go to nuklear_test/mything/lib/ and run make in there, or just create your own ;) .

I get the following error when compiling.

[ vabenil@arch-pc: /nuklear_test ]$ redub -f -v
Parsing /nuklear_test/dub.json at /nuklear_test with : -c 
Added project nuklear_test to memory cache.
Parsing /nuklear_test/mything/dub.json at /nuklear_test/mything with : -c 
Dependencies resolved in 0 ms for "debug" using /usr/bin/dmd [linux-x86_64]
Project nuklear_test is fully parallelizable! Will build everything at the same time
Built: mything. Took 46ms

        /usr/bin/dmd '-color=on' -op -debug -g '-version=Have_mything' -I/nuklear_test/mything/source /nuklear_test/mything/source/mything.d -lib -od/tmp/.redub -of/nuklear_test/mything/libmything.a 

Built: nuklear_test. Took 51ms

        /usr/bin/dmd '-color=on' -op -debug -g '-version=Have_mything' '-version=Have_nuklear_test' -I/nuklear_test/source -I/nuklear_test/mything/source /nuklear_test/source/app.d -c -od/tmp/.redub -of/nuklear_test/nuklear_test.o 

Linking Error at "nuklear_test". 
        Redub v1.7.9
        dmd 2.109.1
        Failed with flags: 

        /usr/bin/dmd -of/nuklear_test/nuklear_test /nuklear_test/nuklear_test.o -g /nuklear_test/mything/libmything.a
                  :
        /usr/bin/ld: /nuklear_test/nuklear_test.o: in function `_Dmain':
/nuklear_test/source/app.d:5:(.text._Dmain[_Dmain]+0xa): undefined reference to `foo'
collect2: error: ld returned 1 exit status
Error: undefined reference to `foo`
       referenced from `_Dmain`
       perhaps a library needs to be added with the `-L` flag or `pragma(lib, ...)`
Error: linker exited with status 1
       cc /nuklear_test/nuklear_test.o -o /nuklear_test/nuklear_test -g -m64 -Xlinker --export-dynamic /nuklear_test/mything/libmything.a -L/usr/lib -Xlinker -Bstatic -lphobos2 -Xlinker -Bdynamic -lpthread -lm -lrt -ldl 

Warning: If you think this is a bug on redub, do test with dub, if it works, do file an issue at https://github.com/MrcSnm/redub/issues/new/choose
object.Exception@source/redub/api.d(136): Build failure
----------------
source/redub/api.d:136 [0x576eb1005f86]
source/app.d:76 [0x576eb10073c7]
source/app.d:69 [0x576eb1007115]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).runAll() [0x76203cfaa06c]
??:? _d_run_main2 [0x76203cfa9e86]
??:? _d_run_main [0x76203cfa9cdc]
??:? [0x76203cccfc87]
??:? __libc_start_main [0x76203cccfd4b]
??:? [0x576eb0fc3e34]

If you pay attention, the flag -lmything is nowhere.

This compiles normally via dub.

My setup is Arch Linux, and redub is version 1.7.9.

P.S. You probably want a test to catch this one

JSON parsing error building Lime

I tried to build one of my projects with Redub and got this error:

object.Exception@source/redub/parsers/json.d(86): preGenerateCommand 'n  "$${DUB_EXE}" \n  --arch="$$(echo "$$DUB_ARCH" | sed -E 's/ .+//g')" \n  --compiler="$${DC}" \n  --single tools/setup_plugins_overrides.dn ; exited with code 127
----------------
??:? object.Throwable.TraceInfo core.runtime.defaultTraceHandler(void*) [0x100352c6b]
??:? _d_run_main [0x10035b177]
??:? start [0x183dd20df]
??:? 0x0 [0x2c557fffffffffff]

The package I tried to build is: https://github.com/jacob-carlborg/lime. It contains some more advance use cases of preGenerateCommand, so it might be a good test bed.

[MAYBE MACHINE DEPENDANT] Segmentation fault

When trying to compile the project attached below. I get the following:

[ vabenil@arch-pc: sync/D/mblas ]$ redub -f
Dependencies resolved in 0 ms for "debug" using /usr/bin/dmd [linux-x86_64]
Project mblas is single dependency, performing single threaded build
Segmentation fault (core dumped)

This happens both in redub-1.7.8 and also in the latest git redub version.

Project files:
mblas.zip

When compiling normally I get the following, though I suspect this may be an error specific to my machine:

[ vabenil@arch-pc: sync/D/mblas ]$ dub
    Starting Performing "debug" build using /usr/bin/dmd for x86_64.
    Building mblas ~master: building configuration [application]
     Linking mblas
/usr/bin/ld: -f may not be used without -shared
collect2: error: ld returned 1 exit status
Error: linker exited with status 1
       cc /home/vabenil/.dub/cache/mblas/~master/build/application-debug-FP-wj8rju3IMj2OEdhNFiw/mblas.o -o /home/vabenil/.dub/cache/mblas/~master/build/application-debu
g-FP-wj8rju3IMj2OEdhNFiw/mblas -g -m64 -Xlinker --no-as-needed -Xlinker -fopenmp -Xlinker --export-dynamic -lopenblas -L/usr/lib -Xlinker -Bstatic -lphobos2 -Xlinker -B
dynamic -lpthread -lm -lrt -ldl 
Error /usr/bin/dmd failed with exit code 1.

Lastly, this maybe best in an a separate issue, but when trying to compile with redub the cblas project (https://github.com/DlangScience/cblas). I get the following error:

object.Exception@source/redub/buildapi.d(681): Project named 'cblas which is a sourceLibrary, can not depend on project mir-core since it can only depend on sourceLibrary. Dependency is a library
----------------
/usr/include/dlang/ldc/std/exception.d:522 pure @safe noreturn std.exception.bailOut!(Exception).bailOut(immutable(char)[], ulong, scope const(char)[]) [0x57cf98dd8617]
/usr/include/dlang/ldc/std/exception.d:442 [0x57cf98d94698]
source/redub/tree_generators/dub.d:127 [0x57cf98d9324e]
source/redub/tree_generators/dub.d:37 [0x57cf98d930f2]
source/redub/api.d:207 [0x57cf98dd5acc]
source/app.d:261 [0x57cf98dd7848]
source/app.d:76 [0x57cf98dd63dc]
??:? void rt.dmain2._d_run_main2(char[][], ulong, extern (C) int function(char[][])*).runAll() [0x75161157906c]
??:? _d_run_main2 [0x751611578e86]
??:? _d_run_main [0x751611578cdc]
??:? [0x751610d4ec87]
??:? __libc_start_main [0x751610d4ed4b]
??:? [0x57cf98d92e34]

Compiling this project with dub works normally.

Lastly I am currently using Arch Linux, dub version is 1.38.1 and dmd version is 2.109.1

Any plans for SDL support, or some kind of build script?

I did think about writing my own SDL parser, as I use that a lot in my projects, and I also have appropriated the components of the old std.experimental.xml (now newxml) project, so I can use its lexer and take influences from the design of the XML DOM (current SDL DOM is primitive).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.