craftr-build / craftr-build-4.x Goto Github PK
View Code? Open in Web Editor NEWFrontend for the Craftr build framework.
Home Page: https://craftr-build.github.io/craftr/
License: Other
Frontend for the Craftr build framework.
Home Page: https://craftr-build.github.io/craftr/
License: Other
Currently, extending a module works by adding a reference to the child module's DataEntity
that will be accessed alternatively if a member couldn't be found in the original DataEntity
. It would be cleaner in all aspects to execute the contents of the module that is being extended directly (maybe even again if the module that is being extended was already loaded) but in the namespace of the child module.
Makes more sense. :-)
objects = cc.compile(
sources = path.glob('src/*.c'),
include = path.local('include'), # Error! need a list, not a string
)
Results in an error message like this:
File "c:\users\niklas\repos\craftr-build\craftr\craftr\__init__.py", line 303, in merge
return self.options.get_merge(key)
File "c:\users\niklas\repos\craftr-build\craftr\craftr\__init__.py", line 432, in merge
'in framework {1!r}, got {0}'.format(key, fw.name, type(value).__name__))
TypeError: expected a non-string sequence for 'include' in framework '<module>', got include
The first step should be to replace <module>
with the module identifier of the module that called the rule function cc.compile()
.
It should be possible to pass options via the command-line, like
craftr -Ddebug=True
The value should be converted into the most applicable datatype. In the example above, it should result in a boolean True
value to be set, numbers should be converted to numbers and everything else should just be set as strings.
If you're declaring the same target with the same command over and over again, you will have the command duplicated every time in a new rule. We should split a Target
into Rule
and Build
, just like Ninja does it. The compiler interface could look like this instead.
objects = compiler.toolset.CxxCompiler().objects(include=[...], etc..)
objects.build(path.glob('src/*.cpp'))
Passing a list to prefix()
or suffix()
should apply the function on all items in the list.
Currently, msvc_deps_prefix
is set to Note: including file:
which is for the English version of MSVC. Some people might be using a different language, so it would be nice if it could be detected automatically.
The Module.target()
function accepts an optional keyword argument called target_class
that can be passed an alternative class to be used instead of craftr.runtime.Target
. This could be used as a "plugin hook".
Consider:
from craftr.utils.path import glob, join
C = load_module('compiler')
module.target('Objects',
target_class = C.ObjectsTarget,
inputs = glob(join(project_dir, 'source', '*.cpp')),
base_dir = project_dir,
build_dir = join(project_dir, 'build', 'obj'),
)
module.target('Program',
target_class = C.LinkTarget,
inputs = Objects.outputs,
output_file = join(project_dir, 'build', 'main'),
)
These custom targets could also implement special support for specific backends, for example Ninja. The custom target class would know how to properly implement auto dependencies for the used compiler in Ninja.
This is just a draft on how such a target subclass could look like:
from craftr.utils.path import move
import craftr
P = load_module('platform')
# ...
class ObjectsTarget(craftr.runtime.Target):
def __init__(self, module, name, inputs, base_dir, build_dir):
outputs = move(inputs, base_dir, build_dir, suffix=P.obj)
command = ['g++', '-c', '%%in', '-o', '%%out', '-MMD', '-MF', '%%out.d']
super().__init__(module, name, inputs, outputs, command=command)
def ninja_rule(self):
return {'deps': '$out.d'}
By default, the C/C++ targets of the compiler
modules should output (especially object-) output files to a sub directory in the build directory named by the modules identifier.
build/
some_lib/
obj/
tools.o
...
libsome_lib.a
app/
obj/
main.o
...
main
The setdefault()
function would be more useful if it would also accept dotted names and resolve it in the modules global variables. Also, a setoption()
function that would basically do the same but always set the value instead of only if it is not already present. This would make it easier to set options before a module is loaded. Compare:
some = get_namespace('some_fancy_module')
some.debug = True
load_module(some)
setoption('some_fancy_module.debug', True)
some = load_module('some_fancy_module')
Currently, there are two ways to build a project. You can derive the output build directory from code, but that can sometimes be rather complicated depending on your needs and dependent modules might not take this build directory into account.
The preferred way now is to use the current working directory as the build output directory, but each module should use a folder that is named after its module name (so that no conflicts in eg. object files can occur between modules).
You usually don't want to mess up your working tree with the stuff, so you instead create a build directory inside it and run craftr from there.
mkdir build && cd build
craftr -c .. export && ninja
A convenient subcommand should be added to merge these steps and that can be called from the working directory rather than the build directory.
craftr build
There should be options to specifiy the build directory name, and whether the ninja build file should be exported even if it already exists.
Craftr generates a default <target> [<target> [...]]
declaration at the end of a ninja build file. There should be an option exclude a target from this declaration, so it will only be built if another target depends on it or if it is explicitly targeted when calling craftr -b <target>
or ninja <target>
.
Currently, we can only pass inputs = ...
to a target()
definition. A new parameter requires = ...
should be added that can list up files that are required for the build, but are not inserted in place of the %%in
placeholder.
An attribute Target.pool
should be added (already done by @winksaville in PR #15). But also we should be able to define new pools in a module. These pool names would also be prefixed with the module namespace.
Ninja on Pools: https://martine.github.io/ninja/manual.html#ref_pool
Currently, we have to pass a task to be executed via -f <task_name>
. It would be nice if we only had to pass the <task_name>
without the -f
option. Craftr would automatically recognize whether a name refers to a task or a target. Also, we should re-add a -d
option that enables the dry-run (currently only possible by specifying the null
backend). That would greatly ease the execution of a task when no export is actually desired.
Although usually not necessary, Craftr is built with the possibility to create multiple Session
objects and execute Craftr modules encapsulated in that Session. We currently work on os.environ
directly if we want to read/write global option. Session.env
should be initialized with a copy of os.environ
and inside the Session context, os.environ
should be the very same dictionary.
That is pretty stupid :-/ Since craftr.path.normpath()
converts paths to lowercase, all filenames in the archive become lowercase, too. We should find a method to get the proper casing of a filename on Windows.
It's not the goal of Craftr to be a convenient interface to Ninja as it was with Creator, but it sure was convenient to be able to invoke Python functions from the command-line to execute custom tasks. It was even possible to require a target to be built before a task can be run.
In Craftr, we no longer need to decorate a Python function with a @task()
decorator since the function is automatically exposed in the module. In order to build targets from a Python task, we can extend the backend interface to provide a function build
that builds a number of targets.
from craftr.utils.shell import call
def run():
session.backend.build(Program)
binary = Program.outputs[0]
call(binary, session)
You can use OpenCL with craftr 0.20.0-dev on Windows using compiler.nvcc
. There should be support for other vendors (Intel, AMD) on Windows as well, and also support on Linux and OSX.
I still have to find out, but I think (and hope) that there is some pkg-config
magic or a standard way to find all required information to compile OpenCL on Unix platforms. On Windows, I have no idea how I should figure out which vendor must be used, so the only chance is currently to hardcode that into the Craftfile.
from craftr.ext.compiler.nvcc import NvccCompiler
nvcc = NvccCompiler()
opencl = nvcc.get_opencl()
print(opencl['libs'], opencl['libpath'], opencl['includes'])
Ideally, there should be something to automatically detect the applicable vendor and compiler for the current platform, eg.
from craftr import path
from craftr.ext.compiler import get_platform_toolset
tools = get_platform_toolset
cc = tools.CCompiler()
objects = cc.compile(
sources = path.glob('src/*.c'),
includes = tools.opencl.includes,
libs = tools.opencl.libs,
libpath = tools.opencl.libpath,
)
I'm not sure how to best describe it in the title... Consider the following two Craftfiles:
# craftr_module(project)
def baz():
print("hello!")
# craftr-module(project.eggs)
from craftr import session, path
session.path.append(path.local('..')) # To find 'project'
from craftr.ext.project import baz
Now when you execute the project.eggs
module without the project
module in the search path, a proxy module will be created for project
so that project.eggs
can be loaded. Importing baz
will fail because the proxy module does not contain it! The project
module can be found after appending the path to session.path
though, so craftr.ext.CraftrImporter
should be capable of load the file delayed.
This issue is regarding craftr 0.20.0-dev. The compiler.nvcc
module has been tested with Windows only, yet. It needs to be tested, and eventually adjusted, for Linux and OSX.
A rule to create a target that allows to be run via Ninja should be added. Eg.
objects = cxx.objects(sources = glob(join(project_dir, 'src/**/*.cpp')))
program = cxx.executable(filename = 'main', inputs = [objects])
run = rules.run(executable = program, pool = 'console')
I think we should add a new built-in Craftr module called rules
for it as a run()
rule is not directly related to C/C++ compiler.
set MYO_SDK_PATH=..\myo-sdk-win-0.9.0
craftr -eb
Since Craftr switches into the build directory, the path in the environment variable is no longer valid.
It should be possible to mark a target as the default target to be built in a module. This should result in a Ninja default
statement (see https://martine.github.io/ninja/manual.html#_default_target_statements). The default
statement that is being generated should only list the targets that are listed by the main module (aka. the Craftfile
of the current directory or the module specified via the -m
option).
Craftr is usually configured with the environment, just like most other build tools. The -Dkey=value
option also just appends to the process' environment. There might be cases where you want a single project or all projects you build to be configured with a specific environment always. Also, if your project requires platform dependent configuration, users should not need to modify the Craftfile
but instead add or modify the Craftr environment file.
I want to add support for a file that is also plain Python but loaded before any other modules. This file will be loaded from the current working directory of Craftr and from the user home directory. I would call it Craftenv
or .craftenv
or something like that. It could look like this:
from os import environ
environ['MYO_SDK_PATH'] = '/absolute/path/to/myo/sdk'
environ['CROSS_COMPILER_PREFIX'] = 'arm-eabi-'
# etc.
CC
, CXX
and LD
environment variablesEg. if you use $in
in a command, it is escaped to '$in'
by shlex.quote()
(called by craftr.shell.quote()
) which in turn causes multiple to $in
not be taken as multiple arguments but only as one.
The compiler modules are currently based on compiler.base.BaseCompiler
which implements the objects()
, executable()
, etc. rule functions and they use the command()
function which must be implemented by subclasses of BaseCompiler
. The design is very flawed and makes it hard to maintain the code and it is not elegant at all.
There should be a base class, maybe called Translator
or CommandBuilder
that implements and interface that fullfills the following constraints:
autodeps
could append -MMD -MF %%out.d
to the build command but as well add a meta variable depfile
to the result of the call that specifies %%out.d
. The caller would know that this meta variable might be set and can use it accordingly)Note: There is already a
craftr.utils.CommandBuilder
atm. but it isn't really what I am looking for either. It will be removed with the resolving of this issue.
An early draft of a Translator
subclass:
class CCompiler(Translator):
def _handle_autodeps(self, autodeps=True, depsfile='%%out.d'):
if autodeps:
self.result.meta['depsfile'] = depsfile
self.result.command.extend(['-MMD', '-MF', depsfile])
def _handle_includes(self, includes):
self.result.meta.setdefault('includes', []).extend(includes)
self.result.command.extend(['-I{}'.format(x) for x in includes])
def _handle_defines(self, defines):
self.result.meta.setdefault('defines', []).extend(defines)
self.result.command.extend(['-D{}'.format(x) for x in defines])
All functions that start with _handle
would be called for the options passed to Translator.translate(**options)
. In order to be able to accept only the necessary arguments, introspection would be applied to figure out the accepted argument names and call those functions that accept an argument of an option. A _handle...()
function would be invoked even if only one of its arguments are defined in the options. The Translator.result
would be set to a TranslatorResult
instance during the runtime of a Translator.translate()
call.
This would ease the eye on some syntactic constructs and would probably allow to make no difference between lists or tuples of filenames and targets. The autoexpand()
function should allow the expansion of Targets.
internals = cxx.objects(
inputs = glob(join(project_dir), 'src/internal/*.cpp')),
)
public = cxx.objects(
inputs = glob(join(project_dir), 'src/public/*.cpp')),
)
# Now we could use the targets directly instead of accessing their *outputs* attribute.
cxx.objcopy(
inputs = [internals, public],
outputs = addsuffix(rmvsuffix([internals, public]), '.bin'),
additional_flags = ['-O', 'binary'],
)
If you start a subprocess and the executable can not be found, you'll see something like this
FileNotFoundError: [WinError 2] The system could not find the specified file
But no clue about what file could not be found.
craftr.ext.platform
if sys.platform.startswith('win32'):
# ...
from craftr.ext.compiler import msvc as toolset
craftr.ext.compiler.msvc
from craftr.ext import platform
This results in an ImportError
. The from craftr.ext import platform
looks for an attribute platform
in the craftr.ext
module rather than �importing craftr.ext.platform
(this is only done when the module was not already imported).
In some cases, you might use a rule function to generate a target, but that rule function does not support an option for additional implicit dependencies. Doing target << [one, two, three]
should use craftr.expand_inputs()
and add the expanded filenames to Target.implicit_deps
.
sys.stdin
, etc. globally in craftr/daemon.py
?I want to be able to run a Python function defined in a Craftfile from ninja. This could be realized with sockets. Eg. the generated ninja rule for a function dance
in the Craftr module walz
could look like this:
rule walz.do_the_dance
command = craftr-com walz.dance localhost $CRAFTR_PORT -- $in -o $out
description = walz.dance()
The craftr-com
program would conntect to Craftr
via a socket and pass the program arguments. These arguments would be passed as a list to the Python function craftr.ext.walz.dance()
. Note that it could be possible that a delayed execution of the Craftr module could be necessary with this technique. (Craftr skips the execution of build definitions if it is only used to invoke ninja with the -b
option).
Inside of Craftr, it could look like this:
def dance(args):
parser = argument.ArgumentParser(prog='dance()')
parser.add_argument('infiles', args='*')
parser.add_argument('-o', default=None)
args = parser.parse_args()
print(args)
# xxx: do stuff
do_the_dance = rules.pycom(
function = dance,
inputs = path.glob('src/*.c'),
outputs = ['foo'],
)
While you should usually import stuff by the module name, it could sometimes be useful to import by filename. The import_file()
function would extract the craftr module identifier and then import it by that identifier. If a module of that name already exists and is not the exact same file, an ImportError
should be raised.
GCC and Clang support the -MMD -MF <filename>.d
option. This will be more complicated with MSVC as it can only output the included headers files to stdout using the /showIncludes
option. Ninja is capable of taking that into account as well, but we need to figure the include prefix from the current compiler, we can't assume everybody has an english version of the compiler installed.
More information here: Ninja auto dependencies
An argument called replace
rather than append
for the suffix()
function might be better suited and more self documenting.
A rule objcopy()
to invoke the GNU objcopy
tool should be added.
As far as I know, you usually use gcc
/g++
instead of ld
to link C object files into an executable or shared library as it does the job of invoking the linker for you. Craftr does currently not take the LD
variable into account and just uses whatever C compiler is configured as the linker.
The compiler
modules should get rule functions that will generate the correct commands to build object files, libraries and executables.
C = load_module('compiler')
build_dir = join(project_dir, 'build')
C.cc_objects('Objects',
sources = glob(join(project_dir, 'src/**/*.c')),
includes = [join(project_dir, 'include')],
)
C.cc_executable('Bin',
name = 'main',
inputs = Objects,
)
Derived from issue #3, it should be possible to pass a linker script to linkers for Linux & OSX. Something like
from craftr.ext.compilers import get_platform_toolset as tools
ld = tools().Ld()
ld.link(
inputs = objects,
output = 'main',
linker_script = path.local('link.ld'),
)
With other meta build systems, such as meson, you can have the "build" directory outside of the source tree. This means you never modify the source tree, which I find convenient.
So I'd like to be able to do the following:
$ ./gen_srcs craftr test 1 1
$ mkdir build
$ cd build
$ craftr export ../test
Another idiom is the "build" directory is inside the source tree but all artifacts are just inside that directory.
$ ./gen_srcs craftr test 1 1
$ cd test
$ mkdir build
$ cd build
$ craftr export ..
from craftr import magic
name = magic.get_assigned_name(magic.get_frame()) + '_foo'
assert name == 'name_foo'
That is how it is supposed to work, but it doesn't. Instead, you get
File "<stdin>", line 9, in <module>
name = magic.get_assigned_name(magic.get_frame()) + '_foo'
File "/home/niklas/Desktop/craftr/craftr/magic.py", line 116, in get_assigned_name
raise ValueError(message, op.opname)
ValueError: ('expected {LOAD_NAME, LOAD_FAST, STORE_NAME, STORE_FAST}', 'LOAD_CONST')
The instruction at the point where get_assigned_name()
is called is followed by additional instructions for '_foo'
(namely LOAD_CONST
). The current implementation however requires the assignment sequence to appear immediately after the last instruction.
A possible solution could be to reset if an unexpected instruction is found and continue like that until we find STORE_NAME
or STORE_ATTR
. At the time we found any of these two instructions, we should've captured the complete assigned name. If POP_TOP
is reached at any point, the result of the expression is not assigned and a ValueError
should be raised.
Calling this function from a Craftr module will stop the module execution and jump to the importing module.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.