bbc / nmos-common Goto Github PK

View Code? Open in Web Editor NEW

6.0 22.0 5.0 577 KB

Common components and SDK for interacting with BBC implementations of AMWA NMOS APIs

License: Apache License 2.0

Makefile 0.68% Python 99.32%

rd-project rd-section-apmm rd-stability-green nmos sdk

nmos-common's Introduction

NMOS Common Library Components

Common Python library for BBC reference implementations of NMOS APIs.

Introduction

This library provides common components required by NMOS API implementations. Additionally, it provides SDK components for interacting with the nmos-node, nmos-registration and nmos-query implementations.

Installation

Requirements

Linux (untested on Windows and Mac)
Python 2.7 or 3.3+
Python Pip

Steps

# Install Python setuptools
$ pip install setuptools

# Install the library
$ sudo python setup.py install

Configuration

This library provides a number of configuration options which are used by its own components and by services which depend upon the library. Configuration is provided by a JSON object within the file /etc/nmoscommon/config.json. The following attributes may be set in the object:

priority: [integer] Overrides the default behaviour for interaction with a Registration API, causing it to prefer a particular instance if set to a value greater than or equal to 100. Default: 0.
https_mode: [string] Switches the library between HTTP and HTTPS operation. "disabled" indicates HTTP mode is in use, "enabled" indicates HTTPS mode is in use. Default: "disabled".
prefer_ipv6: [boolean] Switches the libary into IPv6 mode. Default: false.
prefer_hostnames: [boolean] Causes hostnames to be surfaced via APIs rather than IP addresses, and to cause any HTTP requests to use hostnames rather than IP addresses where possible. This is important when operating in HTTPS mode. Default: false.
node_hostname: [string] Overrides the default fully qualified DNS name for the system. Default: null.
fix_proxy: [string] The number (e.g. "1", "2") of reverse proxies the API is running behind, enabling URLs and headers to show the correct address. Also accepts "enabled" (same as "1") or "disabled". Default: "disabled".
logging.level: [string] Sets the log level to one of "DEBUG", "INFO", "WARN", "ERROR" or "FATAL". Default: "DEBUG".
logging.fileLocation: [string] The path to write the log file to. Default: "/var/log/nmos.log".
logging.output: [array] An array of locations to write log messages to, which may include "file" and "stdout". Default: ["file", "stdout"].
nodefacade.NODE_REGVERSION: [string] Which Registration API version to look for when performing registrations from a Node API. Default: "v1.2".
oauth_mode: [boolean] Indicates whether use of JSON Web Tokens and OAuth authorisation is expected. Default: false.
dns_discover: [boolean] Indicates whether browsing of DNS-SD via unicast DNS is enabled. Default: true.
mdns_discover: [boolean] Indicates whether browsing of DNS-SD via multicast DNS is enabled. Default: true.
interfaces: [mixed] An array containing network interfaces, via which mDNS announcements and browsing should be performed. Default: [] (uses the default gateway interface).

An example configuration file is shown below:

{
  "priority": 0,
  "https_mode": "disabled",
  "prefer_ipv6": false,
  "prefer_hostnames": false,
  "node_hostname": null,
  "fix_proxy": "disabled",
  "logging": {
    "level": "DEBUG",
    "fileLocation": "/var/log/nmos.log",
    "output": [
      "file",
      "stdout"
    ]
  },
  "nodefacade": {
    "NODE_REGVERSION": "v1.2"
  },
  "oauth_mode": false,
  "dns_discover": true,
  "mdns_discover": false,
  "interfaces": ["ens1f0", "eno1"]
}

Documentation

Deprecated Components

The following components are deprecated and are maintained here as a compatibility interface to alternative underlying libraries. New code should make direct use of the underlying libraries as opposed to the classes and functions provided here.

aggregator.py: Now provided by 'nmosnode'
facade.py: Now provided by 'nmosnode'
json.py: Now provided by 'mediajson'
mdnsbridge.py: Now provided by 'mdnsbridge'
ptptime.py: Now provided by 'mediatimestamp'
timestamp.py: Now provided by 'mediatimestamp'

Time

This library makes use of the underlying 'mediatimestamp' library, which may be used to get the current TAI time. The system will provide the time in UTC. As TAI does not account for leap seconds it maintains an offset from UTC that changes every time a leap second occurs1. The library contains a table of leap seconds, which is up to date as of the end of 2017. Users of this library should ensure this table is up to date by checking with the IERS who are responsible for the scheduling of leap seconds and publish decisions in their Bulletin C.

Interaction with pyipputils

When this library is used on a system where the BBC R&D internal pyipputils is installed the following libraries will automatically revert to using their pyipputils equivalents to ensure correct behaviour in house. These are:

logger.py Uses pyipputils IppLogger class instead so that underlying C logging libraries are still used in order to maintain a single log file for all of IP Studio.

Development

Testing

# Run the tests
$ make test

Packaging

# Debian packaging
$ make deb

# RPM packaging
$ make rpm

Versioning

We use Semantic Versioning for this repository

Changelog

See CHANGELOG.md

Contributing

See CONTRIBUTING.md

Please ensure you have run the test suite before submitting a Pull Request, and include a version bump in line with our Versioning policy.

License

See LICENSE.md

nmos-common's People

Contributors

Stargazers

Watchers

Forkers

jamesba jasperpeeters alexrawcliffe nigelmegitt pkeroulas

nmos-common's Issues

Jenkins tests fail because of missing systemd dependency

Jenkins tests currently fail, by the looks of it because a dependency on the systemd module is missing.

https://jenkins.rd.bbc.co.uk/job/ap.python-service-nmos-mdns-bridge/20/console

License does not appear correctly via GitHub UI

Suspect we should be using a more standard LICENSE.md matching the Apache license.

https://choosealicense.com/licenses/apache-2.0/#

.local hostnames don't always resolve

When browsing for mDNS service types there is both a 'name' and a 'hostname' advertised for each record. These may match or may not. If they don't it may be that only the hostname resolves. Changes in 8614e3d#diff-ab70800f5629db7f7853e41f1814ce58L74 should address use of the entity which is most likely to resolve, but I note that in our mDNS advertisements we typically set the 'name' to be the same as the 'hostname'. We should check that this works as expected, and if not we should instead set the hostname to something which avahi-daemon is responsible for resolving for us.

Dependencies missing unless version is incremented

Over multiple trials, the deb package dependencies 'python-flask-oauthlib' and 'python-flask-sockets' fail to be installed when installing the package. This is despite them correctly appearing in the Depends: line of the built .deb

I've noted that by incrementing the version in the Debian changelog (for example to 0.1.1) this starts working correctly. If you then decrement again and install the package it again omits these dependencies. I'm yet to fully understand why these two specific packages seem to strongly linked to this package's version.

Pure python logger outputs to stderr twice

When using pure python logging, everything seems to get outputted to stderr twice. This has been observed under python3. Untested under python2.

Python 3 test dependencies not installed by default

I think I have completed all set-up of Vagrant and Virtualbox and VMs staretd successfully.
Host machine can access to node VM and regquery VM by SSH.

How ever, when test on nmos-common was run, such error message occurred.

vagrant@node:~/nmos-common$ sudo make test
tox -c /home/vagrant/nmos-common/tox.ini
GLOB sdist-make: /home/vagrant/nmos-common/setup.py
py27 create: /home/vagrant/nmos-common/.tox/py27
py27 installdeps: coverage, mock
py27 inst: /home/vagrant/nmos-common/.tox/dist/nmoscommon-0.6.6.zip
py27 installed: The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,certifi==2018.8.13,chardet==3.0.4,click==6.7,coverage==4.5.1,Flask==1.0.2,Flask-OAuthlib==0.9.5,Flask-Sockets==0.2.1,funcsigs==1.0.2,functools32==3.2.3.post2,gevent==1.3.6,gevent-websocket==0.10.1,greenlet==0.4.14,idna==2.7,itsdangerous==0.24,Jinja2==2.10,jsonschema==2.6.0,MarkupSafe==1.0,mediajson==1.0.0,mediatimestamp==1.0.1,mock==2.0.0,netifaces==0.10.7,nmoscommon==0.6.6,oauthlib==2.1.0,pbr==4.2.0,pybonjour==1.1.1,Pygments==2.2.0,python-dateutil==2.7.3,pyzmq==17.1.2,requests==2.19.1,requests-oauthlib==1.0.0,six==1.11.0,socketIO-client==0.7.2,ujson==1.35,urllib3==1.23,websocket-client==0.51.0,Werkzeug==0.14.1,ws4py==0.5.1,wsaccel==0.6.2
py27 runtests: PYTHONHASHSEED='1993019345'
py27 runtests: commands[0] | coverage run --source=./nmoscommon -m unittest discover -s tests
.......................................................................................authenticating: None
................Opening Websocket 05f1c8da-dc36-11e7-af05-7fe527dcf7ab at path /, Receiving ...
Websocket 05f1c8da-dc36-11e7-af05-7fe527dcf7ab closed
.Opening Websocket 05f1c8da-dc36-11e7-af05-7fe527dcf7ab at path /, Receiving ...
Websocket 05f1c8da-dc36-11e7-af05-7fe527dcf7ab closed
[]
........................authenticating: jkhndgkjsj.jkuhjhn
token result code: 404
result payload: <MagicMock name='fp.read()' id='140041619044688'>
Authentication Failed
....authenticating: jkhndgkjsj.jkuhjhn
token result code: 200
result payload: {"userid": "FAKE_USER_ID", "token": "jkhndgkjsj.jkuhjhn"}
authorizing: jkhndgkjsj.jkuhjhn
Authorization Failed.
..authenticating: jkhndgkjsj.jkuhjhn
token result code: 200
result payload: {"userid": "FAKE_USER_ID", "token": null}
Authentication Failed
.authenticating: jkhndgkjsj.jkuhjhn
token result code: 200
result payload: {"userid": "FAKE_USER_ID", "token": "jkhndgkjsj.jkuhjhn"}
authorizing: jkhndgkjsj.jkuhjhn
.Could not find token
..........Exception loading config: 
......2018-08-20 08:34:38,930 : test-logger : DEBUG : jh,dnv [
.2018-08-20 08:34:38,931 : test-logger : ERROR : skjhd;khj
2018-08-20 08:34:38,931 : test-logger : ERROR : skjhd;khj
.2018-08-20 08:34:38,934 : test-logger : CRITICAL : aslkdjhnjksh
2018-08-20 08:34:38,934 : test-logger : CRITICAL : aslkdjhnjksh
2018-08-20 08:34:38,934 : test-logger : CRITICAL : aslkdjhnjksh
.2018-08-20 08:34:38,936 : test-logger : INFO : fh io.ue8yr
2018-08-20 08:34:38,936 : test-logger : INFO : fh io.ue8yr
2018-08-20 08:34:38,936 : test-logger : INFO : fh io.ue8yr
2018-08-20 08:34:38,936 : test-logger : INFO : fh io.ue8yr
...2018-08-20 08:34:38,939 : test-logger : WARNING : e;l;dgfl;k
2018-08-20 08:34:38,939 : test-logger : WARNING : e;l;dgfl;k
2018-08-20 08:34:38,939 : test-logger : WARNING : e;l;dgfl;k
2018-08-20 08:34:38,939 : test-logger : WARNING : e;l;dgfl;k
2018-08-20 08:34:38,939 : test-logger : WARNING : e;l;dgfl;k
2018-08-20 08:34:38,939 : test-logger : WARNING : e;l;dgfl;k
.Got Data
.Got Data
.Got Data
............................Listening on ipc://tmp.test
...............................................................................................................ERROR! Port still 0!
.HttpServer failed to start
..HttpServer running
.HttpServer running
.HttpServer running
.HttpServer running
.
----------------------------------------------------------------------
Ran 307 tests in 4.683s

OK
py27 runtests: commands[1] | coverage annotate
py27 runtests: commands[2] | coverage report
Name                                  Stmts   Miss  Cover
---------------------------------------------------------
nmoscommon/InteractiveServer.py          27      0   100%
nmoscommon/__init__.py                    2      0   100%
nmoscommon/aggregator.py                275      0   100%
nmoscommon/facade.py                    168      0   100%
nmoscommon/flask_cors.py                 35      0   100%
nmoscommon/httpserver.py                 58      0   100%
nmoscommon/ipc.py                       126      2    98%
nmoscommon/json.py                        7      0   100%
nmoscommon/logger.py                     47      4    91%
nmoscommon/mdns/__init__.py               5      0   100%
nmoscommon/mdns/avahidbus.py            115    115     0%
nmoscommon/mdns/bonjour.py              141      0   100%
nmoscommon/mdnsbridge.py                 76      0   100%
nmoscommon/nmoscommonconfig.py           12      0   100%
nmoscommon/ptptime.py                    29      0   100%
nmoscommon/query.py                      87      0   100%
nmoscommon/rql/__init__.py                7      7     0%
nmoscommon/rql/converters.py             69     69     0%
nmoscommon/rql/dateutil/__init__.py       2      2     0%
nmoscommon/rql/dateutil/util.py          40     40     0%
nmoscommon/rql/parser.py                 97     97     0%
nmoscommon/rql/query/__init__.py          0      0   100%
nmoscommon/rql/query/mongodb.py          51     51     0%
nmoscommon/timestamp.py                   5      0   100%
nmoscommon/utils.py                      34      0   100%
nmoscommon/webSocketClient.py            44      0   100%
nmoscommon/webapi.py                    452     22    95%
nmoscommon/zmqserver.py                  60      0   100%
---------------------------------------------------------
TOTAL                                  2071    409    80%
py3 create: /home/vagrant/nmos-common/.tox/py3
py3 installdeps: coverage, mock
py3 inst: /home/vagrant/nmos-common/.tox/dist/nmoscommon-0.6.6.zip
ERROR: invocation failed (exit code 1), logfile: /home/vagrant/nmos-common/.tox/py3/log/py3-2.log
ERROR: actionid: py3
msg: installpkg
cmdargs: ['/home/vagrant/nmos-common/.tox/py3/bin/pip', 'install', '--process-dependency-links', '/home/vagrant/nmos-common/.tox/dist/nmoscommon-0.6.6.zip']

The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/vagrant/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Processing ./.tox/dist/nmoscommon-0.6.6.zip
DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release. You can find discussion regarding this at https://github.com/pypa/pip/issues/4187.
Collecting gevent>=1.2.2 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/a9/33/1fe2c8b3f9b6d0b667e42c8902beb6d944149e0330f564901d50fec76e41/gevent-1.3.6-cp35-cp35m-manylinux1_x86_64.whl (4.4MB)
Collecting greenlet>=0.4.13 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/e0/69/3cd1d75ccf1e38f9cd701833daecd3c5ac5de949653b559cee735e4ad4cd/greenlet-0.4.14-cp35-cp35m-manylinux1_x86_64.whl (41kB)
Collecting gevent-websocket>=0.9.3 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/7b/84/2dc373eb6493e00c884cc11e6c059ec97abae2678d42f06bf780570b0193/gevent_websocket-0.10.1-py3-none-any.whl
Requirement already satisfied: six>=1.10.0 in ./.tox/py3/lib/python3.5/site-packages (from nmoscommon==0.6.6) (1.11.0)
Collecting flask>=0.10.1 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/7f/e7/08578774ed4536d3242b14dacb4696386634607af824ea997202cd0edb4b/Flask-1.0.2-py2.py3-none-any.whl (91kB)
Collecting jinja2>=2.7.2 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/7f/ff/ae64bacdfc95f27a016a7bed8e8686763ba4d277a78ca76f32659220a731/Jinja2-2.10-py2.py3-none-any.whl (126kB)
Collecting werkzeug>=0.9.4 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
Collecting itsdangerous>=0.24 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/dc/b4/a60bcdba945c00f6d608d8975131ab3f25b22f2bcfe1dab221165194b2d4/itsdangerous-0.24.tar.gz (46kB)
Collecting socketio-client>=0.5.3 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/12/d4/abeb2596c2f16276c66910362b27d04b8d2cf12a746dcccf1d00de3f691b/socketIO-client-0.7.2.tar.gz
Collecting flask-sockets>=0.1 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/62/a1/2295e41c32f591093cce519282ceeb6a968945bf497faa83a124e206aa26/Flask_Sockets-0.2.1-py2.py3-none-any.whl
Collecting pyzmq>=15.2.0 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/9a/f9/6d7d3d1c83159997e2e7bc74d11e84a83aa1e7f85e6f028341414e7c2141/pyzmq-17.1.2-cp35-cp35m-manylinux1_x86_64.whl (995kB)
Collecting pygments>=1.6 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/02/ee/b6e02dc6529e82b75bb06823ff7d005b141037cb1416b10c6f00fc419dca/Pygments-2.2.0-py2.py3-none-any.whl (841kB)
Collecting python-dateutil>=2.4.2 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl (211kB)
Collecting flask-oauthlib>=0.9.1 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/da/ec/45bf1c4bb0003c80abed5502c679ac849d95e1899aaebe0891f20267bda9/Flask-OAuthlib-0.9.5.tar.gz
Collecting ws4py>=0.3.2 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/53/20/4019a739b2eefe9282d3822ef6a225250af964b117356971bd55e274193c/ws4py-0.5.1.tar.gz (51kB)
Collecting requests>=2.6 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl (91kB)
Collecting jsonschema>=2.3.0 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl
Collecting netifaces>=0.10.6 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/e9/25/6df71623bc2867ead3323265d548aa9bd27bfa5fd22b2486185d68b81c2b/netifaces-0.10.7-cp35-cp35m-manylinux1_x86_64.whl
Collecting websocket-client>=0.18.0 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/6a/0b/fe3a2d47066005c928056f60ed308036485a7d8bcfa6595d4f7a45f1b938/websocket_client-0.51.0-py2.py3-none-any.whl (198kB)
Collecting ujson>=1.33 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/16/c4/79f3409bc710559015464e5f49b9879430d8f87498ecdc335899732e5377/ujson-1.35.tar.gz (192kB)
Collecting wsaccel>=0.6.2 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/52/46/9ef0744b434ac723ae3aeddcb92ab29af14f8912f53f47590b5a0db0b9d6/wsaccel-0.6.2.tar.gz
Collecting mediatimestamp>=1.0.1 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/66/6f/8a6fbd313df6498d69c0101ea891a6461dede53b0e62457164617acf0567/mediatimestamp-1.0.1-py3-none-any.whl
Collecting mediajson>=1.0.0 (from nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/c9/83/2409b6ea72474ed49ddf016d03ca380dca8ece8a43b4d96409e82eaecc60/mediajson-1.0.0-py3-none-any.whl
Collecting click>=5.1 (from flask>=0.10.1->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/34/c1/8806f99713ddb993c5366c362b2f908f18269f8d792aff1abfd700775a77/click-6.7-py2.py3-none-any.whl (71kB)
Collecting MarkupSafe>=0.23 (from jinja2>=2.7.2->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/4d/de/32d741db316d8fdb7680822dd37001ef7a448255de9699ab4bfcbdf4172b/MarkupSafe-1.0.tar.gz
Collecting oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2 (from flask-oauthlib>=0.9.1->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/e6/d1/ddd9cfea3e736399b97ded5c2dd62d1322adef4a72d816f1ed1049d6a179/oauthlib-2.1.0-py2.py3-none-any.whl (121kB)
Collecting requests-oauthlib>=0.6.2 (from flask-oauthlib>=0.9.1->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/94/e7/c250d122992e1561690d9c0f7856dadb79d61fd4bdd0e598087dce607f6c/requests_oauthlib-1.0.0-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests>=2.6->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
Collecting certifi>=2017.4.17 (from requests>=2.6->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/16/1f/50d729c104b21c1042aa51560da6141d1cab476ba7015d92b2111c8db841/certifi-2018.8.13-py2.py3-none-any.whl (146kB)
Collecting idna<2.8,>=2.5 (from requests>=2.6->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl (58kB)
Collecting urllib3<1.24,>=1.21.1 (from requests>=2.6->nmoscommon==0.6.6)
  Downloading https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl (133kB)
Installing collected packages: greenlet, gevent, gevent-websocket, itsdangerous, werkzeug, MarkupSafe, jinja2, click, flask, chardet, certifi, idna, urllib3, requests, websocket-client, socketio-client, flask-sockets, pyzmq, pygments, python-dateutil, oauthlib, requests-oauthlib, flask-oauthlib, ws4py, jsonschema, netifaces, ujson, wsaccel, mediatimestamp, mediajson, nmoscommon
  Running setup.py install for itsdangerous: started
    Running setup.py install for itsdangerous: finished with status 'done'
  Running setup.py install for MarkupSafe: started
    Running setup.py install for MarkupSafe: finished with status 'done'
  Running setup.py install for socketio-client: started
    Running setup.py install for socketio-client: finished with status 'done'
  Running setup.py install for flask-oauthlib: started
    Running setup.py install for flask-oauthlib: finished with status 'done'
  Running setup.py install for ws4py: started
    Running setup.py install for ws4py: finished with status 'done'
  Running setup.py install for ujson: started
    Running setup.py install for ujson: finished with status 'error'
    Complete output from command /home/vagrant/nmos-common/.tox/py3/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-s9opp971/ujson/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-824pw6vm/install-record.txt --single-version-externally-managed --compile --install-headers /home/vagrant/nmos-common/.tox/py3/include/site/python3.5/ujson:
    running install
    running build
    running build_ext
    building 'ujson' extension
    creating build
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/python
    creating build/temp.linux-x86_64-3.5/lib
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I./python -I./lib -I/usr/include/python3.5m -I/home/vagrant/nmos-common/.tox/py3/include/python3.5m -c ./python/ujson.c -o build/temp.linux-x86_64-3.5/./python/ujson.o -D_GNU_SOURCE
    In file included from ./python/ujson.c:39:0:
    ./python/py_defines.h:39:20: fatal error: Python.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/home/vagrant/nmos-common/.tox/py3/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-s9opp971/ujson/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-824pw6vm/install-record.txt --single-version-externally-managed --compile --install-headers /home/vagrant/nmos-common/.tox/py3/include/site/python3.5/ujson" failed with error code 1 in /tmp/pip-install-s9opp971/ujson/

py3 installed: The directory '/home/vagrant/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.,certifi==2018.8.13,chardet==3.0.4,click==6.7,coverage==4.5.1,Flask==1.0.2,Flask-OAuthlib==0.9.5,Flask-Sockets==0.2.1,gevent==1.3.6,gevent-websocket==0.10.1,greenlet==0.4.14,idna==2.7,itsdangerous==0.24,Jinja2==2.10,jsonschema==2.6.0,MarkupSafe==1.0,mock==2.0.0,netifaces==0.10.7,nmoscommon==0.6.6,oauthlib==2.1.0,pbr==4.2.0,Pygments==2.2.0,python-dateutil==2.7.3,pyzmq==17.1.2,requests==2.19.1,requests-oauthlib==1.0.0,six==1.11.0,socketIO-client==0.7.2,urllib3==1.23,websocket-client==0.51.0,Werkzeug==0.14.1,ws4py==0.5.1
___________________________________ summary ____________________________________
  py27: commands succeeded
ERROR:   py3: InvocationError for command /home/vagrant/nmos-common/.tox/py3/bin/pip install --process-dependency-links /home/vagrant/nmos-common/.tox/dist/nmoscommon-0.6.6.zip (see /home/vagrant/nmos-common/.tox/py3/log/py3-2.log) (exited with code 1)
Makefile:53: recipe for target 'test' failed
make: *** [test] Error 1

If you know cause of it, please give me information.

test_json throwing errors

Seen when running make test

ERROR: test_dump_nmos_json (test_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jamessa/BBC/IPStudio/dev/nmos-common/tests/test_json.py", line 89, in test_dump_nmos_json
    decoded = nmos_json.loads(fp.getvalue())
  File "nmoscommon/json.py", line 73, in loads
    return json.loads(*args, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "nmoscommon/json.py", line 96, in raw_decode
    value = self._reinterpret_object(value)
  File "nmoscommon/json.py", line 105, in _reinterpret_object
    o[key] = self._reinterpret_object(o[key])
  File "nmoscommon/json.py", line 113, in _reinterpret_object
    return Timestamp.from_tai_sec_nsec(o)
  File "nmoscommon/timestamp.py", line 498, in from_tai_sec_nsec
    return cls.from_sec_nsec(ts_str)
  File "nmoscommon/timestamp.py", line 171, in from_sec_nsec
    return cls(sec=sec, ns=ns, sign=sign)
  File "nmoscommon/timestamp.py", line 478, in __init__
    super(Timestamp, self).__init__(sec, ns, sign)
TypeError: super(type, obj): obj must be an instance or subtype of type

======================================================================
ERROR: test_dumps_nmos_json (test_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jamessa/BBC/IPStudio/dev/nmos-common/tests/test_json.py", line 94, in test_dumps_nmos_json
    decoded = nmos_json.loads(encoded)
  File "nmoscommon/json.py", line 73, in loads
    return json.loads(*args, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "nmoscommon/json.py", line 96, in raw_decode
    value = self._reinterpret_object(value)
  File "nmoscommon/json.py", line 105, in _reinterpret_object
    o[key] = self._reinterpret_object(o[key])
  File "nmoscommon/json.py", line 113, in _reinterpret_object
    return Timestamp.from_tai_sec_nsec(o)
  File "nmoscommon/timestamp.py", line 498, in from_tai_sec_nsec
    return cls.from_sec_nsec(ts_str)
  File "nmoscommon/timestamp.py", line 171, in from_sec_nsec
    return cls(sec=sec, ns=ns, sign=sign)
  File "nmoscommon/timestamp.py", line 478, in __init__
    super(Timestamp, self).__init__(sec, ns, sign)
TypeError: super(type, obj): obj must be an instance or subtype of type

======================================================================
ERROR: test_load_nmos_json (test_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jamessa/BBC/IPStudio/dev/nmos-common/tests/test_json.py", line 101, in test_load_nmos_json
    decoded = nmos_json.load(fp)
  File "nmoscommon/json.py", line 65, in load
    return json.load(*args, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/usr/lib/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "nmoscommon/json.py", line 96, in raw_decode
    value = self._reinterpret_object(value)
  File "nmoscommon/json.py", line 105, in _reinterpret_object
    o[key] = self._reinterpret_object(o[key])
  File "nmoscommon/json.py", line 113, in _reinterpret_object
    return Timestamp.from_tai_sec_nsec(o)
  File "nmoscommon/timestamp.py", line 498, in from_tai_sec_nsec
    return cls.from_sec_nsec(ts_str)
  File "nmoscommon/timestamp.py", line 171, in from_sec_nsec
    return cls(sec=sec, ns=ns, sign=sign)
  File "nmoscommon/timestamp.py", line 478, in __init__
    super(Timestamp, self).__init__(sec, ns, sign)
TypeError: super(type, obj): obj must be an instance or subtype of type

======================================================================
ERROR: test_loads_nmos_json (test_json.TestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jamessa/BBC/IPStudio/dev/nmos-common/tests/test_json.py", line 106, in test_loads_nmos_json
    decoded = nmos_json.loads(NMOS_JSON_STRING)
  File "nmoscommon/json.py", line 73, in loads
    return json.loads(*args, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "nmoscommon/json.py", line 96, in raw_decode
    value = self._reinterpret_object(value)
  File "nmoscommon/json.py", line 105, in _reinterpret_object
    o[key] = self._reinterpret_object(o[key])
  File "nmoscommon/json.py", line 113, in _reinterpret_object
    return Timestamp.from_tai_sec_nsec(o)
  File "nmoscommon/timestamp.py", line 498, in from_tai_sec_nsec
    return cls.from_sec_nsec(ts_str)
  File "nmoscommon/timestamp.py", line 171, in from_sec_nsec
    return cls(sec=sec, ns=ns, sign=sign)
  File "nmoscommon/timestamp.py", line 478, in __init__
    super(Timestamp, self).__init__(sec, ns, sign)
TypeError: super(type, obj): obj must be an instance or subtype of type

Dateutil version capped at <2.8.1

Tagging @bbc/rd-apmm-cloudfit

Capped here: #53b4d43
In response to boto/boto3@c84bb40
Which pulls in this change from botocore:
boto/botocore@e87e7a7

Should uncap once boto project is sorted.

API constructor failure not signalled in HTTPServer start

If the API constructor raises an exception at https://github.com/bbc/nmos-common/blob/master/nmoscommon/httpserver.py#L43 then the HTTPServer thread exits but doesn't set started (and failed). A client that is waiting for started to be set will wait forever. A similar issue could arise if the socket calls raise an exception.

Aggregator.py fails to set Content-Type header in interactions with Registration API

Noted by Tektronix. This is a breach of the spec, but our registry isn't strictly checking this as it doesn't expect to receive anything other than JSON.

Update ProxyFix to work with Werkzeug 0.15

Werkzeug 0.15 makes some changes to ProxyFix to move where it is imported from and use a different signature, which is not backwards compatible with previous versions. For now Werkzeug's version has been pinned before 0.15.0, but in future we should make it compatible instead. The main blocker is having Werkzeug 0.15 in the Debian/Ubuntu apt repos.

Arose because of #91 and tags/archive/sammg-update-proxyfix illustrates how to do it.

Negative timestamps/offsets sometimes returned as positive in strings

The various to string routines (inc to_sec_nsec and to_sec_frac) use self.sign * self.sec for returning the second component when self.sec is 0, this will always return 0. Any timestamp/timeoffset between -0:MAX_NANOSEC and -0:1 will return their absolute equivalent.

Default IPC proxy timeout is too small in facade.py

This is likely to be a compound issue. When using facade.py to register many thousands of resources with the Node API, the process slows down. When this happens the default IPC Proxy timeout of 100ms is regularly breached. Upon timeout a new connection is made and the old one never appears to be closed (causing a file descriptor leak). There are likely to be a number of small changes required to account for this:

Increasing the default Proxy timeout (to a second for example) to work around the slowdown in the Node API
Ideally improve performance within the Node API, as it becomes unworkably slow beyond around 2000 resources, and breaches the timeout of 1 second at around 4000 resources
Ensure that when timeouts do occur, the Proxy connections are properly closed, and the caller handles the error condition correctly (it will likely be out of sync with the Node API as a result)

At present the interface is limited to around 1200 resources before it starts to cause errors on my test host.

oauth dependency version conflict

A change has been made to recently to requests-oauthlib (see requests/requests-oauthlib@41706b5 and requests/requests-oauthlib@53481e3) resulting in a requirement for oauthlib>=3.0.0.

However, flask-oauthlib requires oauthlib>=1.1.2,!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0 (https://github.com/lepture/flask-oauthlib/blob/master/setup.py#L46).

oauthlib<3.0.0 and oauthlib>=3.0.0 can't be satisfied, resulting in an error when creating a package.

Missing Debian dependency

Currently debugging the node facade not starting on a new machine, the error I get is:

Traceback (most recent call last):
  File "/usr/bin/nmosnode", line 6, in <module>
    from nmosnode.nodefacadeservice import NodeFacadeService
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nmosnode/nodefacadeservice.py", line 29, in <module>
    from api import FacadeAPI
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nmosnode/api.py", line 15, in <module>
    from nmoscommon.webapi import *
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nmoscommon/webapi.py", line 21, in <module>
    from flask_sockets import Sockets
  File "/usr/lib/python2.7/dist-packages/gevent/builtins.py", line 93, in __import__
    result = _import(*args, **kwargs)
ImportError: No module named flask_sockets

I believe this repo is where flask_sockets is imported, so python-flask-sockets should be a dependency of the Debian repo.

Calling stop() on MDNSEngine causes AttributeError

>>> from nmoscommon.mdns import MDNSEngine
>>> mdns = MDNSEngine()
>>> mdns.start()
>>> mdns.stop()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/dist-packages/nmoscommon/mdns/avahidbus.py", line 52, in stop
    self._mainloop.quit()
AttributeError: 'MDNSEngine' object has no attribute '_mainloop'

Potential unhandled exception in dnsUtils

https://github.com/bbc/nmos-common/blob/master/nmoscommon/mdns/dnsUtils.py#L23

If no search domain at all is configured for a system then this will raise an exception which isn't caught.