Giter Club home page Giter Club logo

theblues's Introduction

theblues

Python library for using the juju charmstore API.

Installation

The easiest way to install theblues is via pip:

$ pip install theblues

Parts of theblues require the use of macaroons (e.g. parsing the return of Charmstore.fetch_macaroon or interacting with IdentityManager.discharge).

To use authenticated aspects of theblues, e.g. JIMM, you'll need to be able to manage macaroons. theblues was developed around libmacaroons. On ubuntu, you can get libmacaroons from a ppa:

$ sudo add-apt-repository ppa:yellow/ppa -y
$ apt-get install libmacaroons0 python-macaroons libsodium13

Without these, theblues cannot make any authenticated requests to the charm store or other services, but is usable to communicate with the charm store for things like looking up charm information.

Usage

Interacting with the charmstore is pretty simple. To look up an entity on the charmstore (e.g. a charm or bundle):

>>> from theblues.charmstore import CharmStore
>>> cs = CharmStore('https://api.jujucharms.com/v4')
>>> entity = cs.entity('wordpress')
>>> entity['Id']
u'cs:trusty/wordpress-2'

Data for an entity is contained in the Meta item of the response, matching the json returned from the charmstores:

>>> entity['Meta']['charm-metadata']['Name']
u'wordpress'

You can also get files for the entity:

>>> cs.files('wordpress')['hooks/install']
u'https://api.jujucharms.com/v4/wordpress/archive/hooks/install
>>> hook = cs.files('wordpress', filename='hooks/install', read_file=True)
>>> print hook
#!/bin/bash

set -xe
...
<snipped for length>
...
juju-log "So, environment is setup. We'll wait for some hooks to fire off before we get all crazy"

To see all methods available, refer to the full docs.

theblues's People

Contributors

anthonydillon avatar chrismacnaughton avatar fabricematrat avatar frankban avatar jcsackett avatar johnsca avatar jrwren avatar jujugui avatar kadams54 avatar marcoceppi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

theblues's Issues

theblues.errors.ServerError: Request timed out: https://api.jujucharms.com/v4...timeout: 3.05

We have been seeing sustained and noticeable charm store api timeouts in OpenStack Charm CI. This causes false test failures in our charm dev/test gate.

Can this be made more resilient and/or configurable, such as retry threshold and timeout backoffs?

DEBUG:runner:Traceback (most recent call last):
DEBUG:runner:  File "/tmp/bundletester-MyqTGN/ceph-osd/tests/gate-basic-xenial-mitaka", line 22, in <module>
DEBUG:runner:    deployment = CephOsdBasicDeployment(series='xenial')
DEBUG:runner:  File "/tmp/bundletester-MyqTGN/ceph-osd/tests/basic_deployment.py", line 40, in __init__
DEBUG:runner:    self._add_services()
DEBUG:runner:  File "/tmp/bundletester-MyqTGN/ceph-osd/tests/basic_deployment.py", line 72, in _add_services
DEBUG:runner:    other_services)
DEBUG:runner:  File "/tmp/bundletester-MyqTGN/ceph-osd/tests/charmhelpers/contrib/openstack/amulet/deployment.py", line 147, in _add_services
DEBUG:runner:    other_services)
DEBUG:runner:  File "/tmp/bundletester-MyqTGN/ceph-osd/tests/charmhelpers/contrib/amulet/deployment.py", line 67, in _add_services
DEBUG:runner:    constraints=svc.get('constraints'))
DEBUG:runner:  File "/var/lib/jenkins/checkout/0/ceph-osd/.tox/func27-smoke/local/lib/python2.7/site-packages/amulet/deployer.py", line 208, in add
DEBUG:runner:    service_name, charm, branch=branch, series=service['series'])
DEBUG:runner:  File "/var/lib/jenkins/checkout/0/ceph-osd/.tox/func27-smoke/local/lib/python2.7/site-packages/amulet/charm.py", line 57, in fetch
DEBUG:runner:    series=series)
DEBUG:runner:  File "/var/lib/jenkins/checkout/0/ceph-osd/.tox/func27-smoke/local/lib/python2.7/site-packages/amulet/charm.py", line 42, in get_charm
DEBUG:runner:    return Charm(with_series(charm_path, series))
DEBUG:runner:  File "/var/lib/jenkins/checkout/0/ceph-osd/.tox/func27-smoke/local/lib/python2.7/site-packages/charmstore/lib.py", line 155, in __init__
DEBUG:runner:    super(Charm, self).__init__(id, api)
DEBUG:runner:  File "/var/lib/jenkins/checkout/0/ceph-osd/.tox/func27-smoke/local/lib/python2.7/site-packages/charmstore/lib.py", line 105, in __init__
DEBUG:runner:    AVAILABLE_INCLUDES).get('Meta')
DEBUG:runner:  File "/var/lib/jenkins/checkout/0/ceph-osd/.tox/func27-smoke/local/lib/python2.7/site-packages/theblues/charmstore.py", line 107, in _meta
DEBUG:runner:    data = self._get(url)
DEBUG:runner:  File "/var/lib/jenkins/checkout/0/ceph-osd/.tox/func27-smoke/local/lib/python2.7/site-packages/theblues/charmstore.py", line 78, in _get
DEBUG:runner:    raise ServerError(message)

DEBUG:runner:theblues.errors.ServerError: Request timed out: https://api.jujucharms.com/v4/~openstack-charmers-next/xenial/ceph-mon/meta/any?include=bundle-machine-count&include=bundle-metadata&include=bundle-unit-count&include=bundles-containing&include=charm-actions&include=charm-config&include=charm-metadata&include=common-info&include=extra-info&include=revision-info&include=stats&include=supported-series&include=manifest&include=tags&include=promulgated&include=perm&include=id timeout: 3.05

CharmStore.entity() fails with "unrecognized metadata name "published""

Attempting to access any entity fails with: unrecognized metadata name "published"

>>> from theblues.charmstore import CharmStore
>>> cs = CharmStore('https://api.jujucharms.com/v4')

>>> cs.entityId('ubuntu')
'cs:xenial/ubuntu-8'

>>> cs.entity('ubuntu')
ERROR:root:Error during request: https://api.jujucharms.com/v4/ubuntu/meta/any?include=bundle-machine-count&include=bundle-metadata&include=bundle-unit-count&include=bundles-containing&include=charm-config&include=charm-metadata&include=common-info&include=extra-info&include=owner&include=revision-info&include=published&include=stats&include=resources&include=supported-series&include=terms status code:(500) message: {"Message":"unrecognized metadata name \"published\"","Code":""}
Traceback (most recent call last):
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 57, in _get
    response.raise_for_status()
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.jujucharms.com/v4/ubuntu/meta/any?include=bundle-machine-count&include=bundle-metadata&include=bundle-unit-count&include=bundles-containing&include=charm-config&include=charm-metadata&include=common-info&include=extra-info&include=owner&include=revision-info&include=published&include=stats&include=resources&include=supported-series&include=terms

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 135, in entity
    return self._meta(entity_id, includes, channel=channel)
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 106, in _meta
    data = self._get(url)
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 72, in _get
    message)
theblues.errors.ServerError: (500, '{"Message":"unrecognized metadata name \\"published\\"","Code":""}', 'Error during request: https://api.jujucharms.com/v4/ubuntu/meta/any?include=bundle-machine-count&include=bundle-metadata&include=bundle-unit-count&include=bundles-containing&include=charm-config&include=charm-metadata&include=common-info&include=extra-info&include=owner&include=revision-info&include=published&include=stats&include=resources&include=supported-series&include=terms status code:(500) message: {"Message":"unrecognized metadata name \\"published\\"","Code":""}')

Charm store / methods reject full entity IDs

The output from CharmStore.entityId() cannot be used to reference a charm:

>>> from theblues.charmstore import CharmStore
>>> cs = CharmStore('https://api.jujucharms.com/v4')

>>> cs.entityId('wiki-simple')
'cs:bundle/wiki-simple-4'

>>> cs.entityId(cs.entityId('wiki-simple'))
Traceback (most recent call last):
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 57, in _get
    response.raise_for_status()
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.jujucharms.com/v4/cs:bundle/wiki-simple-4/meta/any

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 325, in entityId
    data = self._get(_add_channel(url, channel))
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 61, in _get
    raise EntityNotFound(url)
theblues.errors.EntityNotFound: https://api.jujucharms.com/v4/cs:bundle/wiki-simple-4/meta/any

>>> cs.files('wiki-simple', filename='bundle.yaml')
'https://api.jujucharms.com/v4/wiki-simple/archive/bundle.yaml'

>>> cs.files(cs.entityId('wiki-simple'), filename='bundle.yaml')
Traceback (most recent call last):
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 57, in _get
    response.raise_for_status()
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://api.jujucharms.com/v4/cs:bundle/wiki-simple-4/meta/manifest

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 273, in files
    manifest = self._get(manifest_url)
  File "/home/johnsca/juju/python-libjuju/.tox/py35/lib/python3.5/site-packages/theblues/charmstore.py", line 61, in _get
    raise EntityNotFound(url)
theblues.errors.EntityNotFound: https://api.jujucharms.com/v4/cs:bundle/wiki-simple-4/meta/manifest

AttributeError: 'ConnectionError' object has no attribute 'message'

The original failure that caused this was a DNS resolution failure, but in attempting to turn that into a ServerError, theblues failed with this. Using something like str(exc) would be safest, but you could also do something with getattr(). I think some exception types use msg rather than message, as well.

AttributeError: 'ConnectionError' object has no attribute 'message'
  File "conjureup/controllers/deploy/tui.py", line 34, in do_deploy
    await asyncio.gather(*tasks)
  File "conjureup/juju.py", line 622, in deploy_service
    app_inst = await app.juju.client.deploy(**deploy_args)
  File "juju/model.py", line 1081, in deploy
    entity = await self.charmstore.entity(entity_url, channel=channel)
  File "juju/model.py", line 1926, in coro
    return await self.loop.run_in_executor(None, method)
  File "concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "theblues/charmstore.py", line 136, in entity
    return self._meta(entity_id, includes, channel=channel)
  File "theblues/charmstore.py", line 107, in _meta
    data = self._get(url)
  File "theblues/charmstore.py", line 83, in _get
    message=exc.message)

API version should be set by the library instead of supplied by user

The comments and repo summary say to use v4 of the API, but, as reported in #43, that now fails with the current version of this library.

The library should have the supported API version in the code, instead of relying on it being provided by the user, so that a given release will always reference the API version that it works with.

Set minimum requirement for dependencies

Fixing requirements to specific versions of requests and libcharmstore creates some challenges when trying to interop theblues with other python modules that require newer versions of requests.

Please could a minimum version requirement be set rather than a fixed version.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.