anybox / anybox.buildbot.odoo Goto Github PK
View Code? Open in Web Editor NEWBuildbot configurator for Odoo (formerly OpenERP)
License: Other
Buildbot configurator for Odoo (formerly OpenERP)
License: Other
During some release builds, I got this on several buildslaves:
Traceback from remote host -- Traceback (most recent call last):
(...)_
File "/srv/buildslave/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 139, in maybeDeferred
result = f(*args, **kw)
File "/srv/buildslave/local/lib/python2.7/site-packages/buildslave/commands/transfer.py", line 285, in start
os.makedirs(dirname)
File "/srv/buildslave/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
exceptions.OSError: [Errno 17] File exists: '/srv/buildslave/slavedir/myproject-release-postgresql-9_3/build/../src'
exceptions.OSError: [Errno 17] File exists: '/srv/buildslave/slavedir/myproject-release-postgresql-9_3/build/../src'
Explanation: the build/
subdir does not exist, therefore os.path.exists('build/../src')
returns False, and that trigger the makedirs
on a directory that actually exists (something at lower level probably normalizes a bit more before creating)
Since this FileDownload
has been there since the inception of the 'hgtag' token, I suppose this error can happen only after previous errors at some precise point.
Anyway, workdir=src
will be cleaner than ../src
Somehow, changes in master seem to trigger builds on other branches (seen on Odoo, but would probably be the same for other repos):
http://buildbot.anybox.fr/builders/odoo-80-postgresql-9.3/builds/560
http://buildbot.anybox.fr/builders/odoo-80-postgresql-9.3/builds/569
(that's before first tests of #1 on this buildbot)
In a swarm with heterogeneous servers, it can happen that a few are less or more powerful than the others.
Buildbot provides the max_builds
parameter to control how many parallel builds a given buildslave can take, but that's not enough. For instance, at Anybox, we have currently a slow but highly threaded server. It won't be slower for 6 builds than for one. Still, we prefer to use it only if the quicker ones are already busy.
Buildbot allows customization of the choice by means of the nextSlave
callable. We could write one using a slave_priority
property.
In some case, it's useful to deactivate some modules before launching tests or doing upgrades.
For instance, we have currently a project that depends on ldap_synchronization, and the upgrade script can't run without a properly configured LDAP server. Of course it'd be even better to setup a full integration testing harness, but we don't want to depend too hard on that, especially since in any case it'd need a modification of that module.
Failure: twisted.internet.defer.FirstError: FirstError[#12, [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.defer.FirstError'>: FirstError[#3, [Failure instance: Traceback: <type 'exceptions.ValueError'>: unsupported format character '{' (0x7b) at index 1
According to the doc, this is indeed typical of wrong format strings in Python (on which properties interpolation relies)
This is a long standing improvement wish, see the original writing on launchpad
In short, as projects evolve, their dependencies change, and nobody manages to keep the watch list up to date (it could very well be under someone else's responsibility). Consequences are the natural ones: builders keep running for addons that have long been fixed in all incarnations of the buildout, people don't think to track the bulk of odoo and get nasty surprises whereas the buildbot should have warned them.
What held this back for so long is that the polling is done from the buildmaster, hence the watch
parameters are set in the master-side MANIFEST.cfg
. Updating it would mean many things that the buildmaster is not really fit to perform:
This means that the buildmaster must reimplement and run a good deal what the slaves perform, and this almost defeats the idea of a distributed system.
So here's a new yet obvious idea: this just means that this work to be done on the buildslaves! And indeed, in the course of an ordinary build, the buildout lies open, ready for introspection of what is from a live revision/commit and what is not. The helper methods of the recipe are even there to serve, if needed.
At Anybox, we already have crons that does bin/buildbot reconfig
every hour, so we don't even need hot-reloading of the configuration upon completion of the analysis, although that'd also be fun. The following should be enough:
MANIFEST.cfg
bin/buildout
and send it back to the master with a FileUpload
step.MANIFEST.cfg
: auto-watch
, defaulting to True
.That's not a very tight design, and in particular, it means that a change in dependencies would not be immediately applied master-side, but who would care about one hour lag in that context whereas our problem today is that outdated watches stay for weeks if not months ?
That being said, the parsing of those independent watch directives should not fail ever, otherwise one bogus conf would stop the whole updating for a long while, and jeopardize the next restart if needed (this is not new). That raises the question of properly reporting errors
Cc @sbidoul @jssuzanne @pverkest @ccomb for feedback.
I'm willing to come up with a prototype real soon now, being tired of having errors that should have already been detected each time I reforce a bunch of builds to test something else than the builds outcomes themselves.
This is for database upgrade tests. Seen on an OpenERP 7.0 project, these error swallowings:
except ImportError:
_logger.error('module %(addon)s: Unable to load %(stage)s-migration file %(file)s' % mergedict({'file': pyfile}, strfmt))
raise
except AttributeError:
_logger.error('module %(addon)s: Each %(stage)s-migration file must have a "migrate(cr, installed_version)" function' % strfmt)
trivia: in that case, the AttributeError
was somewhere down the stack, i.e., not for the reason mentioned in the log
Currently, the make_pollers()
method of MultiWatcher
accepts an argument (default 600s), but
It's natural however to have different polling intervals, especially if some sources have change hooks, i.e., push notifications. Indeed in that case, a much bigger interval is in order, just to catch up if some notification got lost.
It would be much more powerful to register a callable that can return the proper value and will be called at poller instantiation time.
develop-eggs is known to be the most frequent source of erratic stateful behaviour in buildout, to the point that some people actually propose to wipe it afresh at each bootstrap, but that's not currently the case.
I have a build that runs with an outdated version of anybox.recipe.openerp on exactly one buildslave (out of half a dozen), just because of this.
Since we're reboostraping at each build anyway, we should also remove develop-eggs at each build before the bootstrap recreates a new one for whatever legitimate use it may have.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.