Something like X:\etc will be ignored.
Reason for this is the way it does the traversal, as it intentionally doesn't pick up top-levels to not consider /etc .

The latter is only interesting for developers, who have a local checkout.
Considering the severity of this issue, I'd rather change it to be similar on both platforms.

   def _traverse_config_trees(self):
        """@return a list of configuration directories, based on our pre-configured configuration directory, 
        including the latter"""
        dirs = list()

        for path in self._trees:
            path = path.abspath() 
            # prevent to reach root, on linux we would get /etc, which we don't search for anything
            while path.dirname() != path:
                new_path = path / self.config_dir_name
                if new_path.isdir():
                    dirs.insert(0, new_path)
                # end keep existing
                path = path.dirname()
            # end less loop
        # end for each directory to traverse
        return dirs

Currently it's quite common in bcore to use delegation as a mechanism to delegate functionality to different implementations. These implementations are usually provided as plugin as part of the context.

Compared to signal-slot mechanisms, delegates may additionally provide return values, may interrupt calls through exceptions, and work without an intermediate queue. However, this must be the case only if the delegate call is expected to have a return value. In any other case, it's very similar to a signal which can be handled in any way under the hood.

The delegate mechanism also forces clients to implement an entire interface, even though they are only interested in a single signal. Nonetheless, through derivation from a standard implementation, one can easily bring down the boiler plate to a class with as much as a single implemented method.

Thanks to instantiation, delegates may have state, and implementors have more context as they see it as a whole.

What's an advantage can also be a disadvantage, and it's clear that individual signal-slot based events are very granular, and thus potentially more flexible to use.

Also, in the Signal-Slot case, the client chooses which signal to connect to, whereas in this case, control over which delegates are fed is purely in the hands of the implementor.

Possible Applications

• ProcessController
  • support for chained delegates, allowing to specify them where they belong
• sg-events
  • The entire daemon is essentially a dispatcher of calls, which can easily be implemented using such an event system as base.

Required Features

• A template interface should be given which allows to specify per-method (class/instance) definition of
  • how to aggregate return values
  • how to implement reduce-like behaviour, feeding output of one delegate to the input of another.
  • support for one or many delegates
  • whether or not only implemented methods should be called (as to prevent calling a base implementation)
  • How to handle errors (catch all and raise when all calls are done. Raise on first exception ? Log and ignore ?)
• It must be customizable how delegates are retrieved. Default should be to use the specified context. Should these be forced to be stateless ? Are their instances kept between calls ?
  • Order of delegate calls should be controllable that way
• (A way to identify a base implementation)
  • Otherwise, if chained calls are possible, the same base implementation would be called multiple times. Maybe this isn't required anyway, as such methods simply shouldn't be chained at all

Problem

In order to get #19 done, bprocess must be informed about the context location. program @context/location seems like a sweet syntax, which is equivalent to cd context/location && program .

Cleanup

While at it, it's required to cleanup bprocess argument parsing. From the past I learn that the delegate never does anything special with them, and thus shouldn't be seeing them at all. Therefore the ProcessController has to do all parsing and interpretation.

Escaping of Wrapper Arguments

It should also be able to escape wrapper arguments, for good measure, like so:

• to escape a context, place an additional @, like @@XZV, which becomes @XZV when passed to the program
• to escape wrapper arguments, put an additional -, which means that ----myone becomes ---myone when passed to the program.

Delegates are meant to read all their configuration from the kvstore, which can be overridden on the commandline already. If special arguments should be defined that would clash with the one interpreted by the ProcessController, they need to be escaped.

This ticket is four in one actually, but all related.

Problems

universal commandline tool (->UCT)

With bcmd it is already easy to create complex, plugin-based commands whose parsers can form arbitrary trees, and who are truly decoupled from each other thanks to argparse's awsomeness.

In order to prevent everyone to be forced to do her very own thing, it's vital to provide an official place into which to hook in functionality.

It must be highly configurable to allow it to work under any other name as well. Using the standard plugin system, it must be easy to extend it with subcommands.

The context if the command is extremely important, as it defines the startup environment of it. After all, it is likely to be installed in a central and 'blank' location that doesn't do much by itself. It must be extremely easy to adjust the context to the desired one.

As graphical environments are fundamental different in requirements from a standard commandline application, a separately configurable variant of the UCT must exist.

The default UCT should be launched through the standard python interpreter, yet it must be trivially easy to make it launch through another interpreter.

Proposed solutions

the 'be' commandline tool

The tool should be called be for no other reason as that it is easy to type, starts with 'b', and is written like it sounds. It shall be commandline only, without support for graphical interfaces.

Sample usage should be as simple as be [options] {subcommand}, where options are suited to override kvstore variables and set the location of the program.

Subcommands are registered as plugin, which is already easy using existing capabilities of the bprocess package settings.

Example:

# a full launcher invocation, 
be @/projects/a ---packages.python.version=2.6 launch

To aid setting the context, the '@dir' optional argument should change the CWD to the given directory for the entire invocation of be. It is just syntactic surgar for cd DIR && be cmd.
However, the @ Syntax aids platforms where it's difficult to create launchers that invoke a full shell, or that can't set the CWD.

simplified subcommand configuration

For subcommands, it should be easy to pull configuration, possibly from underneath the be commands own settings root. For that, some helpers should be provided.

commandline program launcher

The first be subcommand shall be launch, which allows to launch the given executable. It can list all available executable too. All excess arguments are passed to the launched program.

Usage could look like this

be launch {program,...} [args]

If no program is given, list all available ones.

Documentation

For all tools, make sure to keep an .md file around which describes why the tool exists, and why you want to use it.

To my mind, plenty of information from this issue can go there.

problem

bprocess is used everywhere to launch programs of all kinds. For that reason, it will need to be somewhat efficient. Sluggish performance becomes especially visible when commandline utilities are wrapped.

For example, pyside-rcc takes just 1ms to execute without arguments, but 210ms when the wrapper is involved (on osx). In this particular example, it takes about 173ms to wrap the program.

If python starts without doing any work, as in python -c "print 42", I have measured the following times.

• centos 6.5 VM: 0.11ms
• os 10.9: 0.22ms
• windows 7 VM: 0.62ms

This means that the wrapping costs 152ms on OSX, which by far is not the fastest contender when it comes to launching programs.

proposed solution I

Implement a boot-time profiling mechanism, much like this: wrapped-program ---with-boot-profile. This shall use the python profiler to profile everything from the first line of code to just before the program is launched. ---dry-run is recommended, but not required.

From there, try to find simple ways to improve performance.

Maintainability has priority.

reference

python launch details

os 10.9

steelbook:core byron$ time python -c "print 42"
42

real    0m0.022s
user    0m0.012s
sys 0m0.009s

centos-6.5 VM

[root@localhost bdevel]# time python -c "print 42"
42

real    0m0.011s
user    0m0.010s
sys 0m0.001s

windows 7 VM

# in msysgit
$ time python -c "print 42"
42

real    0m0.062s
user    0m0.000s
sys     0m0.015s

Currently, the context is just picked up from the initial location of the executable, and the final resolved location. However, it can be practical in some situations to be able to 'guide' the system along more than one symbolic links, and have it pick up context from there as well.

This must work consistently on windows as well, but shouldn't be a problem at all using the current system of placing symlink standins.

When package.alias is set to something unknown in the package database.

ERROR:be:An unhandled exception occurred
Traceback (most recent call last):
  File "[..]dependencies/lib/bcore/latest/noarch/src/python/bcmd/base.py", line 315, in parse_and_execute
    return self.execute(parsed_args, remaining_args)
  File "[..]dependencies/lib/bcore/latest/noarch/src/python/bcmd/base.py", line 195, in execute
    return cmd.execute(args, remaining_args)
  File "[..]dependencies/lib/bcore/latest/noarch/src/python/bprocess/plugins/be_go.py", line 92, in execute
    programs = self._executable_package_names()
  File "[..]dependencies/lib/bcore/latest/noarch/src/python/bprocess/plugins/be_go.py", line 80, in _executable_package_names
    package = ProcessController._resolve_package_alias(package, lambda n: self._to_package(n, packages[n]))
  File "[..]dependencies/lib/bcore/latest/noarch/src/python/bprocess/controller.py", line 668, in _resolve_package_alias
    package = fpackage_by_name(package.data().alias)
  File "[..]dependencies/lib/bcore/latest/noarch/src/python/bprocess/plugins/be_go.py", line 80, in <lambda>
    package = ProcessController._resolve_package_alias(package, lambda n: self._to_package(n, packages[n]))
KeyError: 'omcopy-exec'

The bcore code base is more than a year old by now, was upgraded and patched up, leaving the existing rst documentation in a miserable state. What's there might be outdated, miss the main points, or may simply not be a good read. Other pieces have no documentation at all.

This ticket is not about writing the actual documentation, but to determine the tools used to produce documentation, may it be in-code docstrings, or custom written high-level manuals.

Whatever result I come up with, it must be easily transferable to all the other repositories I have, and integrate nicely with any existing build system, like bbuild .

Wishlist

The fat ones are those that I don't want to live without.

• show up on read the docs
• easily build static documentation
• includes a tool to produce browsable documentation from source
• has a tool to build high-level documentation in an easy to use, yet powerful enough markup language
• toolchain supports multiple outputs natively, namely PDF, ePub (may also be provided by readthedocs)
• Setup a workflow that encourages sketching the documentation while the documented item takes shape.
• Help making code easily usable by providing sublime text snippets right away !! This will be the killer feature make using frameworks easy. After all, you don't want to study docs or copy-paste boilerplate.
• unit-testable examples. Without this feature, docs go out of sync way too easily
• hierarchical documentation allows to build compendiums, which are made up of documentation of various projects.
• multi-version docs - various versions of the documentation must remain accessible, each release has its own. gh-pages can't do it natively, readthedocs should be able to.

Experiment

Try out the toolchain using the existing documentation of bcore, without editing it much except for the possibly required format conversion. Be sure it works from start to end.
#19 seems like an ideal candidate for this.

Previously, when the basic delegate implementation had to get smarter, they did so by adding plenty of magic constants into the base delegate implementation.

This is fine, as anyone can define a new delegate and override or amend them. However, it's still magic, that should in fact be configurable using the process-manager key.

It will be fairly straight-forward to implement, and will make the system so much nicer. At least if the default values are the ones currently used to prevent people from having to configure the common cases every time.

This one need proper review, and either it is removed from the feature set, or it will trigger a big warning when used.

It shouldn't be needed, and people can actually fall for it not knowing it is in fact enabled.

When there are software distributions, it can be a disadvantage to always pull in configuration from the CWD. If the tool is invoked from a directory which contains conflicting configuration, it will not even startup.

In the case of pgit, it might just not be able to find bcore anymore, which can only be circumvented by specifying the location on the commandline, and thus setting an override like so

pgit submodule query ---packages.bcore.trees=/abs/path/to/dependencies/lib/bcore/latest/noarch

When it's possible to manipulate behaviour that early in the startup process, it should as well be possible to manipulate others as well.

Possible Solutions

Provide different versions of the bootstrapper, which launch the process controller with different flags.

As submitted by @mottosso

Reading through the documents you gave me and have a few questions.

From https://github.com/Byron/bcore/blob/master/README.md

1. What problems are bcore meant to solve?
2. In what scenario would I use bcore?
3. What can't I do without bcore?

I'm think some of the below questions might get answered by answering those above, but if not:

bootstrapper for dependency handling and pre-runtime configuration of programs

1. From the perspective of a new user and first-time pipeline td, what does "bootstrapper" mean?

   file based, cascading key-value store with strong schema
   

2. From the same perspective, in addition to someone who hasn't used databases before (which is where I assume this terminology is coming from, if not, do correct me), what is a "schema"?

   small utility types too numerous to mention
   

3. Like what? Solving what problem? For who?
   a simple framework for writing and running quality checks

4. Quality of what? What is quality?

It happens occasionally that you want to override a package version if just a certain main package is started.

Right now this only works from within a separate configuration directory, but it's worth thinking about overrides on package level to add some convenience.

It should also be easier to have a new package which inherits all data from another package (see related issue).

At the very core of anything in bcore stands a simple, highly abstracted diff algorithm. The current implementation is recursive, which is straightforward to implement, but costly and sometimes impractical.

The reason for it being costly is the considerable amount of stack work being done. It's impractical because some delegates have to actively maintain their own stack-like information to do their work.

To my mind it would be better to have a second implementation, which operates non-recursively and which provides its manually maintained stacks to the delegate. That way, the delegate may essentially remain similar in implementation, but use the manual stack for its own purposes.

As a final goal, the new algorithm implementation should replace the current one, which should be beneficial for the overall performance of the kvstore, and thus bcore applications in general.

Currently, a typical proxy package looks like this:

packages:
  bqt: 
    trees: '{assembly.trees.core}/src/integrations/cute'
    configuration:
      trees: etc
      files: etc/subdir/special.yaml

Even though the split in trees and files maps well to the underlying API, it's nothing that needs to be told. Instead, the system can figure this out by itself quite easily, which would allow the above example to look like this:

packages:
  bqt: 
    trees: '{assembly.trees.core}/src/integrations/cute'
    include:
      - etc
      - etc/subdir/special.yaml

Same goes for package-manager.configuration

Currently it doesn't work, but it should be easily fixable.

In the same moment, this issue will be used to try to close it using a particular commit. Lets see how that works.

In order to be more compatible and to finally move towards a better future, python 3 support should be added.

Python 2.6, 2.7 and 3.3+ should be the target python versions to support.

Thanks to the latest additions, it's totally viable for each project to have it's own package configuration to the extend possible, which will aid it's usage withhin assemblies.

vpecs also have dependency issues, as they are outdated if the delegate sets the context using commandline argument paths.

Problem

Package P has a binary b, which is supposed to be launched with an interpreter i. For this, a package L is defined, which uses an executable alias to launch i, providing a full path to b.

Currently, there is no good way to specify the full path ,as there is no way to refer to the root tree of P from L.

Using the {package.P.trees.as_PathList.first_accessible_path}/bin/b syntax doesn't work, as the root tree's first value needs substitution as well, which will be applied after calling first_accessible_path. For that reason, it won't find any accessible path.

Another way might be to use a search path in some way, but it seems hacky and/or platform dependent.

Proposed Solution I

Add children to alias, name executable to implement the current behaviour. Another child should be tree, which will make the package use the tree of the given package.

A practical example could look like this:

packages:
  ##############
  pyside-uic: #
  ############
    name: PySide UI Compiler
    description: |
      Compile .ui files into python modules
    url: https://github.com/PySide
    requires:
      - python
      - pyside
    aliases: 
      executable: python
      trees: pyside
    arguments:
      prepend:
        - 'bin/pyside-uic'

Problem here is obviously that we don't know at all which argument is supposed to be a path. Something like bin/b looks like a path, but b does not.

Note that this solution would at least work with environment variables just nicely, e.g. if variables should be defined as relative to something.

A caveat is that per package, there can only be one trees-alias, even though a package could easily refer to multiple root paths.

Proposed Solution II

PS1 looks like quite some effort, making everything more complicated, without really solving the problem.

Something that could work right now is something like this:

#...
    arguments:
      prepend:
      - '$PYSIDE_TREE/bin/pyside-uic'

For envrionment variables, it's perfectly viable to have recursive environment variables, i.e. the value of a variable is yet another variable. Code that is using this would have to explicitly support this though.

A solution could be to offer a post-resolution step, which resolves all variables as much as possible (only looking at the ones we have set, of course).

Conclusion

For now, it should be PS2.

Currently the absolute path of settings files can be substituted via the {settings-files.basename} key, like {settings-files.bcore-package}. This is long and cumbersome.

Much nicer, it would be like {ext.basename}, like {yaml.bcore-package}.

Implement it in addition to the current scheme, and remove the old one when all configurations have been updated.

Currently, there is plenty of code which iterates packages and obtains all their data each time.
That's quite expensive, and should be cached. The cache must be deleted when the context stack is known to have changed, which is simple enough.

We've spoken previously about meta-classes in use across bcore. I've postponed commenting on their use, but as I've dug deeper I have yet to find a single benefit of using them and several against; mainly that one would first have to read up on meta-classes in order to understand why they are in there to begin with and secondly because they make reading (and thus learning) someone elses code that much more difficult.

Do you have an example of where a benefit of a meta-class outweighs the reduced readability and effort in learning about them?

I ask because if there is something I'm missing out on, I want in. Otherwise, would you consider pull-requests in which meta-classes have been factored out?

The essence about daemons or other long-running processes is that they will eventually run on out-dated code, and even sooner so run on out-dated configuration. The latter affects the plugin's they have loaded, which also makes up portion of the code that may update.

Usually, daemons and their code are quite stable, as they are usually just distributing calls to plugins which carry the actual implementation. Therefore, having an on-demand framework for reloading configuration and plugins as they change, are added or removed, seems like a viable option to significantly increase the usefulness of such daemons.

As with every code, plugin-dependencies could always make it prohibitive to reload them, which is why that feature should be optional. However, it should be able to add new plugins, or remove existing one, while the daemon is running.

Possible Solutions

A backround thread could be attached to one or more context instances, and monitor changes to the configuration directories and loaded files they handle.

When a change is noticed, this information is passed to a delegate to handle the matter. That way, others get a chance to react to changes in specific ways, such as visual notifications of what happened.

To aid discovering such a thread, one might consider placing it in the application context - however, this is optional and should lie in the responsibility of the caller.

It should be considered to update Application.new() with the option to automatically set something like it up.

Also note that the context may change over time, and the background thread should have a way to learn about new contexts (Either by traversing the entire ContextStack, or by learning about it's set of Contexts by the delegate)

Whenever a framework is ready for use, there should be an accompanying snippet or a set of snippets to make using the framework easy. After all, each framework requires boilerplate which is nothing most people will know by heart.

Having a package-controlled plugin which provides these snippets would make usage of bcore as easy as cloning the source, and installing the plugin.

Distributing snippets as files might be a problem as these can run out of sync with the bcore version you are actually using. Therefore, and ideally, the plugin will not install the snippets themselves, but look for snippets in all currently loaded projects, making them available dynamically.

(taken out of #19)

If there are custom tools and widgets which require a gui framework, the be command can't be used as it is not pulling in any support for it. Of course it could, but then it would always have requirements that are more expensive (and which would not exist on travis for instance).

A graphical version of be should exist, which pulls in GUI frameworks.

the 'beg' commandline tool

beg is in all aspects the same as be, but with support for graphical interfaces. g was chosen as it nicely corresponds to the 'g' in graphical, and makes a very memorable word that could be considered a somewhat funny wordplay.

A sample invocation would look like this:

beg @/projects/a launch

Any subcommand registered with beg must by definition pop up some sort of graphical user interface. It is, however, absolutely valid to have the subcommand accept options to help configure the gui.

the 'bex' commandline tool

x stands for anything, as this is a placeholder for any suffix that differentiates the be command by the interpreter it uses.

For instance, a beb tool could be configured to launch blender's python interpreter, which allows to perform various data transformation tasks from the commandline easily.

graphical program launcher

The first beg subcommand shall be launch, which will bring up a single-window launcher widget.

In a first version, it should be valid to enforce CWD or @syntax to set the context, but in the second version, it should look out for context providers which allow the user to change the context dynamically.

The tool looks pretty much like this:

+-----------------------------+
|     [Context Selector]      |
|                             |
+-----------------------------+
+-----------------------------+
|     [Executable List]       |
|  +-----------------------+  |
|  |     Executable 1      |  |
|  |                       |  |
|  +-----------------------+  |
|  +-----------------------+  |
|  |     Executable N      |  |
|  |                       |  |
|  +-----------------------+  |
|                             |
+-----------------------------+

When pulling in packages at program load time, currently it will only use the delegate of the primary application.

Nonetheless, some packages might be sophisticated enough to require their own one, when handling environment variables for example.

For that reason, custom delegates should be used for some parts of the startup process to keep certain kinds of customizations.

It must be very easy to specify import locations for sourcing code for these custom delegates, which almost always should be coming from the package itself.

That way, the valid path can be referenced via value substitution in the yaml text.

packages:
  foo:
    environment:
      variables:
          PATH: '{packages.bar.root_paths.valid}'/lib

That way, a studio configuration maintainer can pull in configuration from anywhere.

That way, department heads can maintain their own configuration, without making it necessary to allow to push into studio repositories.

If the delegate's prepare_context() method alters the process-manager key, this change will not be picked up.

This means that currently, the only way to specify process manager configuration is through files, but not at runtime using a delegate.

To my mind this shouldn't be an issue, but at least I was aware while writing the code.

This means that the actual value will be baked into the instance of the respective type, which is a huge problems for commandline overrides (and for working with it in general), as context changes won't be reflected in the value we get.

Once fixed, the engine is ready to go.

bprocess very much depends on symbolic links, which happen to be dysfunctional on windows.

The current workaround is to emulate a symbolic link by placing a side-by-side file, one per executable.

This doesn't only clutter up the directory, but it is also redundant. there is just a minor reason to specify it per file.

There should be a way to specify it just once per directory, it's bad enough to have to copy the bootstrapper code.

Proposed Solution

Keep per-executable symlink simulations, but look for a file in the directory additionally.

When recursively resolving these files, look for the executable-specific one first, second for the one in the directory of the executable.

Currently, name generators can't be verified outside their respective, specialized unit-tests. In a production environment, names and paths are volatile, and must easily be changed without breaking the system by overlooking some details in the ruleset.

Also, name generators may implement any logic, which makes it necessary for them to reuse each other directly, without forcing anyone to try reimplement particular logic. Ideally, a generator will never have to know a particular other generator, but there may be instances where this is a requirement, as two programs are somewhat tightly coupled in the workflow.

Even though it might not exactly be semantic's purpose, it must be very easy to understand what a particular name generator expects from a ruleset, and to verify it against a particular ruleset.

It should be very easy to run all name generators available in a context against templates without needing their particular host applications to be present - they must thus be standalone enough to allow that.

Additionally, it must be easy and straightforward to reuse generators implemented by others, which allows to link up outputs of one as inputs of another, just by using contextual data.

Possible Solutions

I think it would be easiest to write plugins with a clearly documented interface, which can be addressed by name or by meta-data attached to their type. That way, the standard plugin system can be used to retrieve implementations.

This plugin type should declare in its interface what data in a ruleset it requires, and it should be possible to call it for name generation even without knowing how it works. That way, a verification step can call it and gather the results.

It should be possible to inherit all settings from a parent package, but override individual properties on demand.

Currently overrides only work if you keep package name similar, and make use of the standard yaml inheritance/override system.

It's ok, but is inconvenient for testing purposes or when variations are needed.
Treat this issue critically to check if it actually is just a minor issue.

When the wrapper can't use symbolic links, and is started with an absolute path, it fails to find its implementation even though it should work.

This works just fine on a system with symlink support.

steelbook:posix byron$ $PWD/om
Traceback (most recent call last):
  File "/Volumes/repository/bin/posix/om", line 242, in <module>
    Bootstrapper().main(__file__, sys.argv[1:])
  File "/Volumes/repository/bin/posix/om", line 221, in main
    root_module, process_controller_type = self._process_controller_class(executable)
  File "/Volumes/repository/bin/posix/om", line 121, in _process_controller_class
    raise AssertionError(msg)
AssertionError: The executable /Volumes/repository/bin/posix/om must be a symlink to the bootstrapper implementation

When looking at git-python, it would be greatly beneficial to see if a submission broke anything. If it did, travis will report it accordingly, visible for everyone.

bcore should use it right away, should be simple enough.