bundlewrap / bundlewrap Goto Github PK

Bundles can have this:

checks = {
    'my_awesome_check': {
        'command': "/usr/bin/foo -x -y -z",
        'expected_output': "Everything is OK",
        # you can also supply a callable:
        'expected_output': lambda output: "OK" in output,
    },
}

Then:

$ bw check my_node
my_node:my_awesome_check:OK
my_node:my_failing_check:FAIL

It's a poor man's monitoring.

Having to deal with a templating engine when writing scripts is a bit annoying. One could mark scripts as "binary" but then you don't get a diff.

bw apply and bw run should be able to automatically notify some services when they start and finish:

• Campfire
• HipChat
• IRC
• local Mac OS notification center
• XMPP

(should probably create separate issues for there once someone starts working on this)

Currently, blockwart is unable to deal with anything other than ASCII. Just try to use a german ümläüt in a template.

I think I fixed it. You might want to have a look at this, though, before merging it to master. Hence the new branch.

shouldn't display diff if file is above a certain size

extremely long lines should be truncated

Why not have this:

groups = {
    'group1': {
        'members': (
            'single_node',
            r"node\d+",
        ),
    },
}

instead of the current:

groups = {
    'group1': {
        'members': (
            'single_node',
        ),
    },
}

group_patterns = {
        r"node\d+": 'group1',
}

Note that we need to put an implicit ^ and $ around member names and patterns.

I use different SSH keys for different servers. Everything is configured in my ~/.ssh/config. In blockwart, I currently use this in my nodes.py:

from fabric.state import env
env.use_ssh_config = True

It's a bit obscure and hacky. How about enabling this by default? I think users would expect blockwart to respect their ~/.ssh/config.

I don't know about all the corner cases, though. Thoughts?

bw run in particular should show output from remote commands line by line as it happens, not all at once when the command has exited.

symlinks should only be considered failed when they point to the wrong target, we don't care about the target actually existing

place a lock file on node while run/apply is executing

add a timestamp-file containing the time of last apply and git ref used to each node

make it configurable (off by default) through #21

Houston, we have a problem!

(virtualenv)$ mkdir foo ; cd foo
(virtualenv)$ bw repo create
(virtualenv)$ vim nodes.py     # change "localhost" to name of my VM
(virtualenv)$ bw apply node1
node1: run started at 2013-12-08 09:47:42
Process Process-2:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/void/git/blockwart/src/blockwart/concurrency.py", line 57, in _worker_process
    message = pipe.recv()
  File "/home/void/git/blockwart/src/blockwart/repo.py", line 58, in __getattr__
    if attrname not in self.__module_cache:
  File "/home/void/git/blockwart/src/blockwart/repo.py", line 58, in __getattr__
    if attrname not in self.__module_cache:
  File "/home/void/git/blockwart/src/blockwart/repo.py", line 58, in __getattr__
    if attrname not in self.__module_cache:
  ...
RuntimeError: maximum recursion depth exceeded while calling a Python object
^C

I'm on the current HEAD:

(virtualenv)$ pwd
/home/void/git/blockwart
(virtualenv)$ git rev-parse HEAD
c4a88ff60c9443ff0109b033d4fc977062b00865
(virtualenv)$ git status
On branch master
Your branch is up-to-date with 'trehn/master'.

nothing to commit, working directory clean

How about a trivial item type that allows you to do this in a bundle.py:

metadata = {
    'something': {
        'data': 'whatever',
    },
    'foobar': {
        'data': [
            'some', 'list', 'of', 'things',
        ],
    },
}

In mako templates, you can then do this:

% for i in bundle.items:
%   if i.id == 'metadata:foobar':
%     for k in i.attributes['data']:
... do something with ${k} ...
%     endfor
%   endif
% endfor

Exemplary use case: For my private server, I've created a bundle called mailserver. It configures everything related to mail. Using the metadata item type, it also creates the mailboxes itself:

metadata = {
    'mailboxes': {
        'data': [
            ('mailboxname1', '{SSHA512}somepasswordhash'),
            ('otherbox', '{SSHA512}somepasswordhash'),
            ('blerch', '{SSHA512}somepasswordhash'),
        ],
    },
    'aliases': {
        'data': [
            ('[email protected]', 'mailboxname1@vmail'),
            ('[email protected]', 'mailboxname1@vmail'),
            ('[email protected]', '[email protected]'),
        ],
    }
}

Using a short snippet of code in bundle.py, I add additional items to files and directories based on that metadata (essentially this is a for i in metadata['mailboxes']['data'] ...). Plus, it can be used in mako templates which means that I can create my dovecot user configuration and exim mail aliases based on the bundle's metadata.

Is this a sane idea? Am I missing some concepts? In any case, it's pretty convenient because I can configure "my whole mailserver" (mailboxes and aliases) in one file.

Currently people just running "bw apply node" don't see any output whatsoever (unless there's any kind of error). That's probably not a good default.

I think we should make -v the default and add a --silent instead.

same usage as bw apply, but only collects item.get_status and prints non-correct items

I'd like to be able to:

• Build packages from the AUR
• Build completely custom packages (PKGBUILD and stuff somewhere on the local machine)
• Install prebuilt packages

The easy way out is to build the packages locally. This might pull in a lot of unwanted packages on the local host, but building stuff on nodes is quite complex because we'd have to set a build environment first (install base-devel, adjust makepkg.conf) and then run makepkg … ugh.

A useradd foo also creates a group called foo if USERGROUPS_ENAB yes is set in /etc/login.defs. This is the default for Ubuntu and Arch Linux (maybe others too).

Plus, useradd foo fails if the group foo already exists. It tells me to use -g.

So:

• Currently, there is no way to add a user to a group with the same name. (As long as this group was specified in a bundle. The implicitly created group works of course.)
• Adding a new user always leaves a dangling group with the same name.

Ideas:

• Add new users with a single useradd [a lot of options] instead of useradd+usermod.
• Same for groups.

The hard part will be to figure out a way to install them properly. setuptools does not provide a built-in mechanism for this.

Upstream bug report: fabric/fabric#700

For us, it means that download() is pretty useless. A lot of stuff is not readable for ordinary users.

I'm inclined to implement the sudo cat hackaround.

I was thinking about #20 when I realized I don't want notification code in Blockwart itself. I'd prefer for repositories to have a hooks directory (similar to git) that allows people to run custom code on certain occasions.

Wondering if there is any reason to require Python code (e.g. to pass the node object you're working on) or just any executable...

Applies to pkg_apt and pkg_pacman.

Virtually any package manager uses locking to prevent itself from running multiple times.

provide a lib subdirectory in a repo to be added to sys.path so stuff in there can be imported by bundles etc.

It appears we upload a file with the standard permissions of your user and THEN adjust the permissions. That's fatal for sudo. sudo refuses to run if /etc/sudoers is not owned by root:

pi@raspberrypi ~ $ sudo id
sudo: /etc/sudoers is owned by uid 1000, should be 0
sudo: no valid sudoers sources found, quitting

Which, of course, means that we can't fix the permissions afterwards.

we need a local config file for storing notification API tokens and the like

for running commands on every apply

it appears to be a standard feature, but doesn't fit in the whole declarative concept

need to think about this

On current master. This empty line shouldn't be there:

(virtualenv)$ bw run -f testvm -p 2 'echo hi >&2'
[overqemu20] err: hi

[testvm] ✓ completed successfully after 0.191166s
(virtualenv)$

It's interesting to note that I sometimes get this instead:

(virtualenv)$ bw run -f testvm -p 2 'echo hi >&2'

[overqemu20] err: hi
[testvm] ✓ completed successfully after 0.221963s
(virtualenv)$

I used -p 2 simply to trigger interactive = False. It doesn't matter if there's one or two workers.

So far, I narrowed it down to combine_stderr. If it's effectively False (remember, for pty = True, combine_stderr has no effect), then I can see the bug. Appears to be something deeper inside Fabric. LineBuffer doesn't appear to be the cause.

I labeled it as "easy" because it's a very minor bug. I guess, this won't be easy to fix, though.

(We stumbled upon something similar in #10. However, that empty line wasn't there.)

tell people about node.name etc.

Just like #16 but for Arch and Fedora and stuff. Only support doing a systemctl {start,stop} ..., no enable/disable as this can be easily done using symlinks. (And #16 doesn't support enabling/disabling stuff, either.)

store some configuration in the repository itself to be shared along with the repo

remove everything in a directory that is not managed by Blockwart

not sure if want

This only sets the number of node_workers:

$ bw -v apply -p 1 ubuntu

This call still runs 4 parallel workers for this single node. It would be nice if -p also applied to workers in Node.apply(). Alternative: Add -P.

Right now, the only way to get one worker per node is interactive mode.

provide an easy way of previewing generated files/configuration

bw preview mynode

Shows a summary of all items (no file contents).

bw preview mynode -f /foo/bar

Shows the expected contents of /foo/bar.

Regarding interactive prompts:

Right now, all output from Fabric is being fed through OutputStreamer and _format_output() and, in turn, through LOG. All these parts think in terms of lines: LOG.info() logs a line. (And OutputStreamer is a line buffer, actually.)

When it comes to interactive prompts, this won't work. We would need to be able to LOG.info() parts of lines.

How about distinguishing between "bw run for a lot of parallel operations" and "interactive bw run"? A "bw run -p 8 ..." works fine as it is now. With "-p 1" I would expect everything to be fully interactive.

For an interactive "bw run", I think it's best to let Fabric do its thing. Just pass sys.stdout/sys.stderr to Fabric and that's it. If we wanted to keep feeding stuff through LOG, we'd essentially be duplicating Fabric's efforts in dealing with interactive I/O.

Have a look at the run-direct branch and let me know what you think. :-) The code may not be perfect yet but it's the way to go IMHO.

Maybe it'd be worthwhile to have a separate web app that accepts statistics uploads and keeps records of applies etc.

Should it include some rudimentary node management features to act as a node database for managing IP addresses etc.?