Giter Club home page Giter Club logo

libral's Introduction

Libral

Build Status

Libral is a systems management library that makes it possible to query and modify system resources (files, packages, services, etc.) through a desired-state API. Its goals are to:

  • Provide a management API for UNIX-like systems to query and modify system resources
  • Make managing new kinds of resources very simple; in particular, there is no need to write native code to add new providers and they can be arbitrary scripts that adhere to one of a number of simple calling conventions
  • Express both current state and desired state in a simple, unified form
  • Guarantee that all changes are done idempotently by enforcing desired state only when changes are needed
  • Have a very small default footprint that enables use of libral in resource-constrained environments such as devices or containers
  • Be versatile enough to serve as the basis of more extensive configuration management systems, such as Puppet, without being directly dependent on any one of them.

Getting in touch

  • Mailing list: libral
  • IRC: irc.freenode.net:#libral

Building and installation

If you just want to quickly try out libral, you can download a precompiled tarball (GPG signature) that should work on any Linux machine that has glibc 2.12 or later. (It might actually work with glibc 2.8 or later.) If you succeed in that, please let us know.

We have reports of the precompiled binaries working on 64-bit Fedora 21-25, RHEL 5-7 (and related distributions like CentOS), Debian 6-9, Ubuntu 10.04-16.10, and SLES 11-12. The binaries will not work on 32-bit and pre-glibc 2.8 systems: RHEL 4, SLES 10, ...

In case you do need to build from source, which is not required for provider development, only if you want to work on the core libral library, this document contains instructions on building libral.

Docker

You can also try out libral in the context of a Docker container. The Dockerfile in the repository allows for building an image quickly based on the above mentioned precompiled tarball.

docker build -t puppet/libral .

Running this can then be done with Docker, for instance the following invocation will launch ralsh in the context of the container.

docker run --rm -t puppet/libral

This is intended for exploring the CLI and experimenting with providers.

Usage

After you build libral, or after you download and unpack the precompiled tarball, you can try things out by running ralsh:

    # If you downloaded the precompiled tarball
    alias ralsh=$TARBALL_LOCATION/ral/bin/ralsh

    # Only if you built libral from source
    export RALSH_DATA_DIR=$LIBRAL_CHECKOUT/data
    alias ralsh=$LIBRAL_CHECKOUT/bin/ralsh

    # list available types
    ralsh
    # list all instances of a type
    ralsh mount
    # list a specific instance
    ralsh service crond
    # make a change for the better
    ralsh service crond ensure=stopped

    # Do the same against a remote system to which you have ssh access
    ralsh -t ahost
    ralsh -t ahost package sudo

The default output from ralsh is meant for human consumption and looks a lot like Puppet. It is also possible to have ralsh produce JSON output by passing the --json flag.

Many of the providers that libral knows about are separate scripts. ralsh searches them in the following order. In each case, the providers must be executable scripts in a subdirectory providers in the mentioned directory ending in .prov:

  • if the environment variable RALSH_DATA_DIR is set, look in the directory this variable is set to
  • if the --include option to ralsh is given, look in that directory (the option can be given multiple times)
  • default to a directory determined at build time, by default /usr/share/libral/data

Running inside a container

The precompiled tarball contains a statically linked ralsh and all supporting files. After unpacking the tarball, you can copy it into a container and run it like this:

    CONTAINER=<some_container>
    docker cp $TARBALL_LOCATION/ral $CONTAINER:/tmp
    docker exec $CONTAINER /bin/sh -c /tmp/ral/bin/ralsh
    docker exec $CONTAINER /bin/sh -c '/tmp/ral/bin/ralsh user root'

Writing providers

What resources libral can manage is determined by what providers are available. Some providers are built in and implemented in C++, but doing that is of course labor intensive and should only be done for good reason. It is much simpler, and recommended, that new providers first be implemented as external providers. External providers are nothing more than scripts or other executables that follow one of libral's calling conventions. The different calling conventions trade off implementation complexity for expressive power.

The following calling conventions are available. If you are just getting started with libral, you should write your first providers using the simple or json calling convention:

  • simple
  • json: input/output via JSON
  • json_batch (planned,maybe): input/output via JSON, can operate on multiple resources at once
  • native

For all of these, you will also want to read up on the metadata that each provider needs to produce to describe itself.

To start a new provider, follow these steps:

  1. Decide on some working directory DIR, run mkdir $DIR/providers, and
  2. Create a file $DIR/providers/myprovider.prov and make it executable
  3. Make sure that running myprovider.prov ral_action=describe returns valid provider metadata, especially a valid type. For now it doesn't matter what the type is, let's call it MYTYPE
  4. Run ralsh -I $DIR $MYTYPE. This will ask the provider to list all instances of the type using the list action. Get that working
  5. Run ralsh -I $DIR $MYTYPE NAME. This will ask the provider to find a resource with name NAME using the find action. Get that working, too.
  6. Run ralsh -I $DIR $MYTYPE NAME ATTR=VALUE.... This will ask the provider to update the resource NAME using the update action.
  7. You're done; your provider is ready for a pull request here ;)

Todo list

  • finish mount provider
  • add a shell provider
  • event reporting on update
  • simple type system and checking for provider attributes
  • produce help/details about providers
  • add a json calling convention and use it in a provider
  • more core providers
    • cron
    • file
    • group (groupXXX)
    • host
    • package (besides dnf and yum)
    • service (besides systemd, upstart and sysv)
    • user (userXXX)
  • even more core providers
    • interface
    • k5login
    • mailalias
    • selboolean
    • selmodule
    • sshkey
    • ssh-authorized-key
    • vlan
    • yumrepo
  • add a remote provider (using an HTTP API)
  • adapt providers to multiple OS (maybe using mount)
  • add support for running providers over ssh (see this PR for details)
  • noop mode
  • expand the type system to cover as much of Puppet 4 as is reasonable

Some language

Naming things is hard; here's the terms libral uses:

  • Type: the abstract description of an entity we need to manage; this is the external interface through which entities are managed. I am not entirely convinced this belongs in libral at all
  • Provider: something that knows how to manage a certain kind of thing with very specific means; for example something that knows how to manage users with the user* commands, or how to manage mounts on Linux
  • Resource: an instance of something that a provider manages. This is closely tied both to what is being managed and how it is being managed. The important thing is that resources expose a desired-state interface and therefore abstract away the details of how changes are made

FIXME: we need some conventions around some special resource properties; especially, namevar should always be 'name' and the primary key for all resources from this provider, and 'ensure' should have a special meaning (or should it?)

Open questions

  • Do we need types at all at this level ?
  • Can we get away without any provider selection logic ? There are two reasons why provider selection is necessary:
    • adjust to system differences. Could we push those to compile time ?
    • manage different things that are somewhat similar, like system packages and gems, or local users and LDAP users ? This would push this modelling decision into a layer above libral.
  • Would it be better to make providers responsible for event generation rather than doing that in the framework ?

libral's People

Contributors

domcleal avatar garethr avatar igalic avatar johnmccabe avatar lutter avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libral's Issues

ral-check: add a test harness for unit testing providers

There should be a separate tool, ral-check which can be used to unit test providers. The testing will be black box testing, i.e. solely interact with the provider through its API. Writing tests should be very simple, ideally data driven.

A sketch of what such data driven tests look like is in this gist

Running tests should then just be a matter of executing ral-check test.yaml

Ralsh not picking up providers automatically

According to the readme providers should be discovered either via the ENV, a flag or from a known directory. From some basic testing this doesn't appear to be the case?

(Note this testing was done in a Docker container running ubuntu:latest)

I have the example providers available:

$ ls /usr/share/libral/data/providers/
metadata.yaml  mruby.prov  python.prov  simple.prov

If we try and load them using the ENV they are not picked up.

$ RALSH_DATA_DIR=/usr/share/libral/data/ ralsh
2017-06-23 08:10:30.505205 WARN  ral - provider[/opt/ral/data/providers/yum.prov]: ignored as it exited with status 2
2017-06-23 08:10:31.447939 WARN  ral - provider[/opt/ral/data/providers/dnf.prov]: ignored as it exited with status 2
file::posix
group::groupadd
host::aug
mount::aug
package::apt
service::systemd
ssh_authorized_key::ssh_authorized_key
user::useradd

If we assume they should be picked up because they are in the default directory from the README they are also not picked up.

$ ralsh
2017-06-23 08:10:38.349373 WARN  ral - provider[/opt/ral/data/providers/yum.prov]: ignored as it exited with status 2
2017-06-23 08:10:39.266445 WARN  ral - provider[/opt/ral/data/providers/dnf.prov]: ignored as it exited with status 2
file::posix
group::groupadd
host::aug
mount::aug
package::apt
service::systemd
ssh_authorized_key::ssh_authorized_key
user::useradd

But if we use --include they are picked up. Note the python provider warning and the mruby provider being loaded.

$ ralsh --include=/usr/share/libral/data/
2017-06-23 08:11:05.734880 WARN  ral - provider[/usr/share/libral/data//providers/python.prov]: ignored as it exited with status 2
2017-06-23 08:11:08.123525 WARN  ral - provider[/opt/ral/data/providers/yum.prov]: ignored as it exited with status 2
2017-06-23 08:11:09.120289 WARN  ral - provider[/opt/ral/data/providers/dnf.prov]: ignored as it exited with status 2
example::mruby
file::posix
group::groupadd
host::aug
mount::aug
package::apt
service::systemd
ssh_authorized_key::ssh_authorized_key
user::useradd

Based on the README I would have expected the same output for all of these commands.

Docker Travis builds are bogus for PR's

The Docker Travis builds for PR's do not check out the branch for the PR, instead they just build master.
The script contrib/docker/scripts/build.sh needs to be updated to check out the appropriate branch.

Provider metadata should let providers opt into noop mode

Provider metadata should have a field noop_safe: true|false which indicates whether the provider can work in noop mode. If noop_safe is false, the provider will only be invoked with noop=false, and noop mode for that provider will be simulated by simply diffing the is and should values for an update.

Update/create service provider to support service gathering with Upstart on Ubuntu1404

Currently running libral on an Ubuntu1404 host results in no service resources being returned (also tested via libral). Looks like the sysv provider doesn't play nicely with upstart.

$ root@dee8qjpx8rsqkk7:~/ral/bin# cat /etc/os-release
NAME="Ubuntu"
VERSION="14.04.2 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.2 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

$ root@dee8qjpx8rsqkk7:~/ral/bin# ./ralsh
file::posix
group::groupadd
host::aug
mount::aug
package::apt
ssh_authorized_key::ssh_authorized_key
user::useradd

$ root@dee8qjpx8rsqkk7:~/ral/bin# ./ralsh service
unknown provider: 'service'
run 'ralsh' to see a list of all providers

ralsh should be able to print a resource in a format that can be enforced with ralsh

When running ralsh PROV NAME there should be a way to get the output in a form that makes it easy to cut&paste it to another system to run ralsh PROV NAME A1=V1 A2=V2 ... An=Vn to create the same resource there.

This could also be done by making ralsh read JSON from stdin, so that one could write

  ssh node1 ralsh -j PROV NAME | ssh node2 ralsh -j --from-stdin

to clone a resource to another system.

Build error for f25 image in contrib

Running make f25 in the contrib/docker directory fails with the following error:

[ 70%] Build mruby
/bin/sh: rake: command not found
make[2]: *** [mruby/host/bin/mruby] Error 127
mruby/CMakeFiles/mruby.dir/build.make:345: recipe for target 'mruby/host/bin/mruby' failed
make[1]: *** [mruby/CMakeFiles/mruby.dir/all] Error 2
CMakeFiles/Makefile2:389: recipe for target 'mruby/CMakeFiles/mruby.dir/all' failed
make: *** [all] Error 2
Makefile:140: recipe for target 'all' failed
The command '/bin/sh -c git -C /usr/src clone https://github.com/puppetlabs/libral &&     mkdir -p /usr/src/libral/build &&     cd /usr/src/libral/build &&     cmake .. &&     make all install' returned a non-zero code: 2
make: *** [f25] Error 2

ralsh: allow dealing with multiple resources

Right now, ralsh only operates on one resource. It should also be able to operate on multiple resources, e.g.

ralsh package kernel glibc augeas

to list multiple packages, and

ralsh service httpd ensure=running smartd ensure=stopped docker ensure=running enable=true

to modify multiple resources in one go.

This mainly has repercussions for the output, both plain text and the JSON output.

Systemd provider fails on version 208

Running the systemd provider on version 208 fails with the following error (RHEL7 without an update):

[root@ht5qnmvyyhs09sm providers]# ./systemd.prov ral_action=find name=crond
# simple
Invalid number of arguments.
name: crond
ral_unknown: true
[root@ht5qnmvyyhs09sm providers]# systemctl --version
systemd 208
+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ

But on version 219 it works as expected:

[root@ht5qnmvyyhs09sm providers]# ./systemd.prov ral_action=find name=crond
# simple
name: crond
enable: true
ensure: running
[root@ht5qnmvyyhs09sm providers]# systemctl --version
systemd 219
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

The underlying command isn't working on 208.

systemctl list-unit-files -t service --no-pager --no-legend crond.service

Systemd provider does not find unloaded unit files

Currently the systemd provider fails to list stopped services reliably due to the following command excluding non-loaded units (resulting in the subsequent join failing):

systemctl list-units --type service --all --no-pager  --no-legend

For example, on a fresh CentOS7 or RHEL7 host perform the following:

yum -y install httpd
systemctl list-units --type service --all --no-pager  --no-legend
systemctl start httpd
systemctl list-units --type service --all --no-pager  --no-legend

Invoking the systemd.prov directly also demonstrates this problem.

ralsh: make sure we have a resource name when setting properties

ralsh gets fooled by the mistake of typing ralsh group ensure=absent and tries to find a group called ensure=absent. It should reject a name with an = in it, and instead tell the user that they need to provider a name, e.g. run ralsh group tape ensure=absent

Rename operations for hte simple calling convention

Simple still uses list/find/update which is confusing since everything else uses set/get. Rename list to get, and update to set, but keep the semantics of set working on only a single resource; for get, either never pass any names and have a separate 'find' operation, or allow optionally passing a single name to 'get'.

mruby-iijson fails to build and is pulled from HEAD

Looks like a change introduced yesterday (iij/mruby-iijson@6146eec) has resulted in mruby-iijson failing to build with this error:

CC    ../../../../../../../../../../../../../usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c -> ../../../../../../../../../../../../../usr/src/libral/build/mruby/host/mrbgems/mruby-iijson/src/json.o
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c: In function 'json_parse_number2':
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:324:39: error: expected ')' before 'MRB_PRId'
   i = snprintf(buf, sizeof(buf), "%s%"MRB_PRId"%c",
                                       ^
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: format '%s' expects a matching 'char *' argument [-Wformat=]
       num, ch);
       ^
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: spurious trailing '%' in format [-Wformat=]
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: format '%s' expects a matching 'char *' argument [-Wformat=]
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: spurious trailing '%' in format [-Wformat=]
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c: In function 'json_parse_number2':
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:324:39: error: expected ')' before 'MRB_PRId'
   i = snprintf(buf, sizeof(buf), "%s%"MRB_PRId"%c",
                                       ^
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: format '%s' expects a matching 'char *' argument [-Wformat=]
       num, ch);
       ^
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: spurious trailing '%' in format [-Wformat=]
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: format '%s' expects a matching 'char *' argument [-Wformat=]
/usr/src/libral/build/mruby/mrbgems/mruby-iijson/src/json.c:326:7: warning: spurious trailing '%' in format [-Wformat=]
rake aborted!

Should we be pulling from HEAD, or a specific sha (since there aren't any release tags)?

Pass provider configuration into get

With quite a few providers, get can only retrieve resources if it is given additional parameters; e.g., ssh_authorized_keys needs a user or target parameter.

Make it possible to mark some attributes as config which means they need to be passed to the get operation, too.

There's also an interaction with what should be considered the primary key of a resource: some config parameters (e.g. user or target) should be used as part of the pk; in other situations, e.g., when a password needs to be provided to retrieve a resource, that password should not be part of the pk.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.