canonical-ols / talisker Goto Github PK

View Code? Open in Web Editor NEW

30.0 30.0 26.0 1.48 MB

An enhanced WSGI runtime for your apps

License: Other

Makefile 2.27% Python 97.42% Shell 0.32%

talisker's People

Contributors

Stargazers

Watchers

talisker's Issues

Talisker uses proxy address instead of client address for private endpoint filtering

Clients can access any private endpoint by spoofing the proxy header:

$ curl https://myservice/_status/metrics -H 'X-Forwarded-For: 127.0.0.1'
# HELP some_metric
# TYPE some_metric counter
some_metric 1.0

I'm hitting taliskers's private endpoints behind a reverse proxy (which correctly sets in this case X-Forwarded-For: 127.0.0.1 <client.ip>). TALISKER_NETWORKS is not set; though I believe that by setting TALISKER_NETWORKS=<internal.network> all private endpoints will be accessible by remote clients for this reason.

talisker should be explicit about it's dependency on celery >= 3.1.0

The minimum supported version of celery (3.1.0) is only listed in the requirements.devel file. This should really be specified in setup.py as it's a run-time minimum version requirement as well.

Remove werkzeug as a dependency

We used werkzeug for WSGI responses and for local thread/greenlet context, but it would be good to remove it. This would mean some simple WSGI boilerplate for endpoints, but also need a replacement for the local context.

The obvious choice for this is the new stdlib contextvars, (with pre 3.7 backport), so we should see if that would enable dropping werkzeug

Type error emiting request status metric

When running on Python 2, I'm getting a TypeError on each request when Gunicorn emits stats.

2017-04-10 14:40:07.928Z ERROR gunicorn.error "Error handling request"
Traceback (most recent call last):
  File "env/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 185, in handle_request
    self.log.access(resp, req, environ, request_time)
  File "env/local/lib/python2.7/site-packages/talisker/gunicorn.py", line 126, in access
    self.increment("gunicorn.request.status.%d" % status, 1)
TypeError: %d format: a number is required, not str

Python 2.7.6
talisker==0.9.1
gunicorn==19.6.0

Enable logging for Django management commands

Projects usually use management commands for cronjobs and other offline tasks. It'd be great if these scripts could take advantage of talisker's logging setup.

add introspection

Could it be possible to add introspection of gunicorn via talisker ala benoitc/gunicorn#793 ? Anything that helps us check what talisker and/or gunicorn are doing when they suddenly stop replying to HTTP requests.

Sentry context not cleared in between celery tasks

Just talisker context, not the sentry one also

Alternative bind address for status endpoints

Would it be possible with gunicorn to have the /_status endpoints exposed over a different interface:port than the primary one?

Our use case is that we run an internal network where metrics are exposed separate from the primary one that user traffic is served off of.

Talisker 0.9.3 inserts \x1b into log files

As it says on the tin. We're running:

talisker 0.9.3
python 3.4.3

deployed on trusty.

We think this was introduced when talisker added automatic colouration when stderr is a tty. Perhaps for some reason talisker thinks an upstart-owned stderr is a tty, and so applies coloration to it's log output?

Better grok filter

Current grok filter uses a combination of filters to get what we want, with some limitations and performance issues.

It's probably better to write a custom ruby filter that parses specifically the talisker format:

we can avoid regex altogether,
we can support escapes in logfmt values
we can unit test the filter function w/o logstash
probably faster (would need verifying)
no need for patterns, so just simple drop in.

Talisker 0.9.2 breaks /_status/error endpoint

Talisker 0.9.2, deployed on trusty and python 3.4.3 has a broken /_status/error endpoint.l Hitting that endpoint with curl produces the following output on the terminal where talisker is running:

2017-04-12 02:03:08.718Z INFO gunicorn.access "GET /_status/error" method=GET path=None qs= status=500 ip=127.0.0.1 proto=HTTP/1.1 length=0 referrer=None ua=None duration=2.164 request_id=371502e1-c5bc-4fed-ac94-44983
63065de
--- Logging error ---
Traceback (most recent call last):
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/raven/middleware.py", line 98, in __call__
    iterable = self.application(environ, start_response)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/context.py", line 46, in middleware
    return ClosingIterator(app(environ, start_response), clear)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/request_id.py", line 106, in __call__
    return self.app(environ, add_id_header)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/wsgi.py", line 44, in middleware
    return app(environ, start_response)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/endpoints.py", line 139, in __call__
    response = func(request)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/endpoints.py", line 81, in wrapper
    return f(self, request)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/endpoints.py", line 192, in error
    raise TestException('this is a test, ignore')
talisker.endpoints.TestException: this is a test, ignore

During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
    worker.init_process()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/base.py", line 131, in init_process
    self.run()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 124, in run
    self.run_for_one(timeout)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 68, in run_for_one
    self.accept(listener)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 30, in accept
    self.handle(listener, client, addr)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 156, in handle
    self.handle_error(req, client, addr, e)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/base.py", line 248, in handle_error
    self.log.access(resp, req, environ, request_time)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/gunicorn.py", line 133, in access
    "gunicorn.request.status.{}".format(resp.status_code), 1)
AttributeError: 'Response' object has no attribute 'status_code'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/logging/__init__.py", line 978, in emit
    msg = self.format(record)
  File "/usr/lib/python3.4/logging/__init__.py", line 828, in format
    return fmt.format(record)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/logs.py", line 378, in format
    return super(ColoredFormatter, self).format(record)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/logs.py", line 280, in format
    record.message = self.escape_quotes(record.getMessage())
  File "/usr/lib/python3.4/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/home/ubuntu/code/canonical/snap-delta-service/env/bin/talisker", line 11, in <module>
    sys.exit(run())
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/gunicorn.py", line 268, in run
    return app.run()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/app/base.py", line 203, in run
    super(Application, self).run()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/arbiter.py", line 211, in run
    self.manage_workers()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/arbiter.py", line 544, in manage_workers
    self.spawn_workers()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/arbiter.py", line 611, in spawn_workers
    self.spawn_worker()
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    self.log.exception("Exception in worker process"),
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/gunicorn.py", line 110, in exception
    super().exception(self, msg, *args, **kwargs)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/glogging.py", line 253, in exception
    self.error_log.exception(msg, *args, **kwargs)
  File "/usr/lib/python3.4/logging/__init__.py", line 1310, in exception
    self.error(msg, *args, **kwargs)
  File "/usr/lib/python3.4/logging/__init__.py", line 1303, in error
    self._log(ERROR, msg, args, **kwargs)
  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/logs.py", line 250, in _log
    super()._log(level, msg, *args, **kwargs)
Message: <talisker.gunicorn.GunicornLogger object at 0x2acf62b99e48>
Arguments: ('Exception in worker process',)
Top level Sentry exception caught - failed creating log record
b'<talisker.gunicorn.GunicornLogger object at 0x2acf62b99e48>'
b'Traceback (most recent call last):\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 135, in handle\n    self.handle_request(listener, req, clien
t, addr)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 176, in handle_request\n    respiter = self.wsgi(environ, resp.start_response)\n  File "
/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/raven/middleware.py", line 98, in __call__\n    iterable = self.application(environ, start_response)\n  File "/home/ubuntu/code/canonical/
snap-delta-service/env/lib/python3.4/site-packages/talisker/context.py", line 46, in middleware\n    return ClosingIterator(app(environ, start_response), clear)\n  File "/home/ubuntu/code/canonical/snap-delta-service/
env/lib/python3.4/site-packages/talisker/request_id.py", line 106, in __call__\n    return self.app(environ, add_id_header)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talis
ker/wsgi.py", line 44, in middleware\n    return app(environ, start_response)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/endpoints.py", line 139, in __call__\n    
response = func(request)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/endpoints.py", line 81, in wrapper\n    return f(self, request)\n  File "/home/ubuntu/code/cano
nical/snap-delta-service/env/lib/python3.4/site-packages/talisker/endpoints.py", line 192, in error\n    raise TestException(\'this is a test, ignore\')\ntalisker.endpoints.TestException: this is a test, ignore\n\nDur
ing handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/arbiter.py", line
 578, in spawn_worker\n    worker.init_process()\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/base.py", line 131, in init_process\n    self.run()\n  File "/h
ome/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 124, in run\n    self.run_for_one(timeout)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/p
ython3.4/site-packages/gunicorn/workers/sync.py", line 68, in run_for_one\n    self.accept(listener)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", l
ine 30, in accept\n    self.handle(listener, client, addr)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/sync.py", line 156, in handle\n    self.handle_error(
req, client, addr, e)\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/gunicorn/workers/base.py", line 248, in handle_error\n    self.log.access(resp, req, environ, request_time)
\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/gunicorn.py", line 133, in access\n    "gunicorn.request.status.{}".format(resp.status_code), 1)\nAttributeError: \'Res
ponse\' object has no attribute \'status_code\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/home/ubuntu/code/canonical/snap-delta-service/env/
lib/python3.4/site-packages/raven/handlers/logging.py", line 83, in emit\n    self.format(record)\n  File "/usr/lib/python3.4/logging/__init__.py", line 828, in format\n    return fmt.format(record)\n  File "/home/ubu
ntu/code/canonical/snap-delta-service/env/lib/python3.4/site-packages/talisker/logs.py", line 280, in format\n    record.message = self.escape_quotes(record.getMessage())\n  File "/usr/lib/python3.4/logging/__init__.p
y", line 328, in getMessage\n    msg = msg % self.args\nTypeError: not all arguments converted during string formatting\n'

I'm not sure exactly what causes this issue, but can confirm that the same thing happens with python 3.5.2 on yakkety.

Upgrade raven to sentry-sdk

raven is getting deprecated and being replaced by sentry-sdk

It could be nice to make the upgrade to get the right DSN configured now from sentry's settings (<key>:<secret>@sentry.io/<project> to https://<key>@sentry.io/<project>) in case sentry stops supporting it, and all the updates that the new SDK can bring us.

Add live status checks on talisker.run

It'd be great if there was something similar to taliskers's /_status/, /_status/info/, /_status/test, etc API when running a service through talisker.run. Maybe a command line flag, or envvars?

Include ANALYZE results in postgres sentry reports

When collecting sql SQLqueries to include in sentry reports, it would be need to include the results of ANALYZE on those queries.

This should be ok, performance wise, as usually the sentry report is generated at sent after the response has been sent to the user, but maybe there should be a limit on the number of ANALYZES, and we do them against slowest queries first.

We should only collect these in case of error, using raven's callback api, as while not expensive

No module named "celery"

If I install Talisker, and try to run with gevent, I get a celery error:

$ talisker.gunicorn.gevent webapp.app:app --bind 0.0.0.0:8030 --worker-class gevent --name talisker-6fd2e47c12b8 --reload --log-level debug --timeout 9999
2018-12-14 17:10:07.935Z INFO talisker.sentry.TaliskerSentryClient "Raven is not configured (logging is disabled). Please see the documentation for more information."
2018-12-14 17:10:07.939Z INFO talisker.statsd "configuring statsd DummyClient"
Traceback (most recent call last):
  File "/usr/local/bin/talisker.gunicorn.gevent", line 11, in <module>
    sys.exit(run_gunicorn_gevent())
  File "/usr/local/lib/python3.6/dist-packages/talisker/__init__.py", line 266, in run_gunicorn_gevent
    run_gunicorn()
  File "/usr/local/lib/python3.6/dist-packages/talisker/__init__.py", line 239, in run_gunicorn
    import talisker.celery
  File "/usr/local/lib/python3.6/dist-packages/talisker/celery.py", line 35, in <module>
    from raven.contrib.celery import SentryCeleryHandler
  File "/usr/local/lib/python3.6/dist-packages/raven/contrib/celery/__init__.py", line 12, in <module>
    from celery.exceptions import SoftTimeLimitExceeded
ModuleNotFoundError: No module named 'celery'

Explicitly installing celery 4.2.1 fixes this issue.

Warn user when they try to use gunicorns --log-level

Should explain DEBUGLOG mechanism

Perhaps consider changing the argparse parser message for --log-level and other things in talisker case?

add support for raven

This should include general wsgi middleware, and logging.

Also, integrations for django and flask would be good.

Requests error during ssl with eventlet

Hey, thanks for your help with my previous noob question with PyCharm and debugging. Your link totally worked. Funny i didn't know about that before because i have never been able to dev in the same env as prod. Which is why i have been so excited about talisker. Hopefully will finally catch some prod env bugs before deployment.

I pretty much got my entire setup refactored to using talisker which was awesome and started testing some things. There seems to be an error when making any external https requests. This time i have a small test case for you, i just hope it isn't macOS related. The error isn't surfaced so i don't have that.

Command is talisker -k eventlet taltest:app.
Changing to gunicorn -k eventlet taltest:app does not error so i hoped you would know how to surface the error and then it would be an easy fix.

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""Talisker repro with ssl and requests on py 3.6

eventlet==0.21.0
Flask==0.12.2
talisker==0.9.5
"""

from flask import Flask
app = Flask(__name__)

import requests
print(requests.get("https://google.com"))

One weird thing is i changed talisker.gunicorn.run to below just to play with things. And the error didn't go away. Which stumped me since i copied the lines from the gunicorn.wsgiapp.run method.

def run():
  # config = talisker.initialise()
  # talisker.celery.enable_signals()
  # return TaliskerApplication(
  #     "%(prog)s [OPTIONS] [APP_MODULE]", config['devel'], config['debuglog'])
  from gunicorn.app.wsgiapp import WSGIApplication
  return WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()

Missing metrics with multiple workers

On Ubuntu 16.04 LTS, when running talisker 0.9.10 (reproduced in 0.9.9 too) with multiple workers talisker -w 2 the /status/_metrics endpoint is empty

Use simpler fs based locking for multiproc prometheus

Talisker currently uses a multiprocessing.Lock to synchronize metrics clean up. This is non-trivial and probably unnecessary. We could instead use a simpler filesystem based locking, along the lines of

master - writes a monotonic counter to file whenever it finishes aggregating metrics
worker - reads the value of the file before collecting metrics, collects, that before rendering, checks it hasn't changed. If it has, either abort, or maybe retry once.

This should protect against the worker reading an inconsistent state on disk. We can only get away with it because metrics are non-transactional data that we can fail on if needed, of course.

talisker with latest gunicorn doesn't support umask '0'

When running talisker 0.9.5 with gunicorn 19.7.1 and passing

--umask=0

I get the following error

talisker.gunicorn: error: argument -m/--umask: invalid auto_int value: '0'

talisker > 0.6.7 does not work with celery 3.1.23

We had deployed delta-service with talisker 0.6.7 and celery 3.1.23. This combination was working fine. Upon upgrading to talisker 0.9.0, we can now no longer queue tasks:

https://sentry.ols.staging.canonical.com/canonical/delta-service/issues/43/

The version of celery has not changed. Downgrading talisker to 0.6.7 fixes the problem. I realise this is super odd - the stack trace is entirely inside celery itself... but perhaps talisker patches some aspect of celery?

I downgraded talisker on stg, and verified that everything started working again. Then upgraded again, and verified that it broke again. The only package that changed was talisker.

celery 4.0 is out now, but the talisker docs do not mention whether 4.x is supported - I'm n ot sure if upgrading celery and talisker will help (and I wanted to avoid too many simultaneously moving parts).

Prometheus opens file with master pid

Due to prometheus_client implementation, any metrics created early will use the master pid in it's mmaped filename. The first time a metric is written, this will be fixed, as it checks for pid changes on writing.

This means there might be an empty {histogram,counter}_$MASTERPID.db file left around for the duration of the gunicorn master.

Not a huge issue, but would be nice to fix at some point, but is not easy, given prometheus_client implementation choices.

Could /_status/{missing-page} pass through to the application instead of showing a body-less 404 page?

While investigating #194, I noticed that https://cn.ubuntu.com/_status/check/ leads to a page that looks (in Chromium/Chrome) very much like a "connection failed" type error:

Although on closer inspection it is in fact a "404" response with an empty body.

The same is true of any unrecognised route under /_status.

Would it be possible to simply pass these unrecognised routes through to the application so that they show the normal application 404 page?

Multiline messages overflow into traceback (grok)

Multiline log messages appear to overflow into 'traceback' resulting in status, ip etc. not being logged. This is quite misleading as we often filter out 200 status to look for issues.

See: https://logging.ols.canonical.com/app/kibana#/doc/logstash-*/logstash-2017.05.18/talisker?id=AVwdUQf2ZwZISI6xIgcV&_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15m,mode:quick,to:now))

Hopefully addressed by #87 but worth confirming.

Feature request: A testable sentry client

It would be nice if we could have some way to assert that a sentry report was sent within a test. Perhaps an interface similar to that provided by the talisker.statsd client could work? Ideally we'd be able to assert on the contents of the report as well.

How prometheus is supposed to collect metrics from celery?

Do not swallow Sentry capture's result

As per the sentry doc https://docs.sentry.io/clients/python/api/, "capture" returns a "a tuple with a 32-length string identifying this event" which Talisker is not returning back in its SentryClient implementation.

AttributeErrortalisker.prometheus in write_metrics

We're seeing a lot of the following error in our Sentry for ubuntu.com:

AttributeError

module 'prometheus_client.core' has no attribute '_MmapedDict'

    talisker/prometheus.py in write_metrics at line 349

            def key_func(metric_name, name, labelnames, labelvalues):
                return json.dumps(
                    (metric_name, name, tuple(labels), tuple(labels.values()))
                )
        histograms = core._MmapedDict(histogram_file) <---
        counters = core._MmapedDict(counter_file)
        try:
            for metric in metrics:
                if metric.type == 'histogram':

talisker/prometheus.py in prometheus_cleanup_worker at line 210
talisker/gunicorn.py in handle_custom at line 81

Any ideas?

API changes for the logger in Flask 1.0.0+

When upgrading to FLASK v1.0.0+ I get this error:

Traceback (most recent call last):
  File "PATH/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "PATH/app.py", line 40, in <module>
    talisker.flask.register(app)
  File "PATH/lib/python3.6/site-packages/talisker/flask.py", line 62, in register
    app._logger = logging.getLogger(app.logger_name)
AttributeError: 'Flask' object has no attribute 'logger_name'

From changelog:

Flask.logger has been simplified. LOGGER_NAME and LOGGER_HANDLER_POLICY config was removed. The logger is always named flask.app. The level is only set on first access, it doesn’t check Flask.debug each time. Only one format is used, not different ones depending on Flask.debug. No handlers are removed, and a handler is only added if no handlers are already configured.

/_status/check requires version-info.txt

I expected the /_status_check endpoint to work regardless of whether the version-info.txt file existed or not (same as /_status/ping). But it doesn't seem to - e.g. see https://cn.ubuntu.com/_status/check vs https://snapcraft.io/_status/check.

Feature request: requests per-request timing metric customisation

Talisker will emit a statsd metric for every inter-service HTTP request made with talisker.requests. However, the metric name is calculated using the URL used in the request. The issue I'm facing is that we use an internal IP for the remote service rather than a domain name (in this case the target service has no domain name, and we don't want to add one).

Initially I thought that this meant I probably couldn't use talisker's per-request metrics feature. However, I figured I'd open this bug to see if there's a viable workaround.

Perhaps we could have an alternative interface in talisker.requests that allows a caller to pass a service name, which talisker could then use instead of the IP address when emitting the metric?

Support PyPy

We used to support pypy, but the introduction of psycopg2 support causes issues, so we have removed it.

We should be able to leverage psycopg2cffi as a drop in, but managing the intstall_requires needs some care. Also, the pytest-postgresql test plugin we use doesn't support using psycopg2cffi, so we'd need to fork/upstream a change that allows it to depend on/use pyscopg2cffi on pypy.

Sentry integration in talisker 0.9.3 does not catch exceptions from views

Uncaught exceptions from the /_status/test/sentry endpoint, and uncaught exceptions when loading the application are correctly reported. However, uncaught exceptions from flask views are not reported, making the sentry integration not terribly useful.

My test view is this:

def breaks():
    raise Exception("This should be reported")

Hitting this view with curl results in a 500 response, but no report in sentry.

Wrong errors raised in tests?

Please tell if it's intended or just not fixed yet.

_____________________________________________________________________________ test_formatter_with_exception _____________________________________________________________________________

    def test_formatter_with_exception():
        fmt = logs.StructuredFormatter()

        try:
            e = Exception()
            e.errno = 101
            e.strerror = 'some error'
            raise e
        except Exception:
            record = make_record({})
            record.exc_info = sys.exc_info()
            log = fmt.format(record)
        assert '\n' in log
        output = log.splitlines()
        timestamp, level, name, msg, structured = parse_logfmt(output[0])
        assert timestamp == TIMESTAMP
        assert level == 'INFO'
        assert name == 'name'
        assert msg == "msg here"
>       assert structured == {
            'errno': 'ENETUNREACH',
            'strerror': 'some error',
        }
E       AssertionError: assert {u'errno': u'...u'some error'} == {'errno': 'ENE... 'some error'}
E         Common items:
E         {u'strerror': u'some error'}
E         Differing items:
E         {'errno': 'ETIME'} != {'errno': 'ENETUNREACH'}
E         Full diff:
E         - {u'errno': u'ETIME', u'strerror': u'some error'}
E         ?                ^^
E         + {u'errno': u'ENETUNREACH', u'strerror': u'some error'}
E         ?               ++ ^^^ +++

/Users/.../work/.../talisker/tests/test_logging.py:275: AssertionError
_______________________________________________________________________________ test_get_errno_fields_dns _______________________________________________________________________________

    def test_get_errno_fields_dns():
        exc = None
        try:
            import socket
            s = socket.socket()
            s.connect(('some-host-name-that-will-not-resolve.com', 54321))
        except Exception as e:
            exc = e

>       assert util.get_errno_fields(exc) == {
            'errno': 'EAI_NONAME',
            'strerror': 'Name or service not known',
        }
E       AssertionError: assert {'errno': 'EN...or not known'} == {'errno': 'EAI...ce not known'}
E         Differing items:
E         {'errno': 'ENOEXEC'} != {'errno': 'EAI_NONAME'}
E         {'strerror': 'nodename nor servname provided, or not known'} != {'strerror': 'Name or service not known'}
E         Full diff:
E         + {u'errno': u'EAI_NONAME', u'strerror': u'Name or service not known'}
E         - {u'errno': 'ENOEXEC',
E         -  u'strerror': 'nodename nor servname provided, or not known'}

/Users/.../work/.../talisker/tests/test_util.py:110: AssertionError

OS name and version:
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian

Any details about your local setup that might be helpful in troubleshooting:
Docker container with gunicorn, django, python_prometheus, etc...
python2.7.16

Detailed steps to reproduce the bug:

Install deps and run pytest -vv

talisker.logs.extra_logging should restore context to original state on exit, rather than delete

Colours in default ubuntu / tmux too obnoxious

talisker 0.9.1 breaks with older setuptools

Talisker 0.9.1's celery entry point fails - I think when run under python3:

�[2;34m2017-03-24 02:31:33.190Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:33.192Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:33.559Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:33.561Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:33.922Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:33.924Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:34.283Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:34.284Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:34.624Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:34.625Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:34.962Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:34.963Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:35.297Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:35.298Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:35.657Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:35.659Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:35.999Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:36.000Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:36.331Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:36.332Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:36.680Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:36.681Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:37.038Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:37.040Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:37.385Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:37.387Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:37.739Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:37.740Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:38.061Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:38.062Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:38.409Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:38.410Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:38.759Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:38.760Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:39.117Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:39.118Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:39.466Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:39.467Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'
�[2;34m2017-03-24 02:31:39.808Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.sentry.TaliskerSentryClient �[0m"�[1;37mRaven is not configured (logging is disabled). Please see the documentation for more information.�[0m"
�[2;34m2017-03-24 02:31:39.810Z �[0m�[0;32mINFO�[0m �[0;33mtalisker.statsd �[0m"�[1;37mconfiguring statsd via environment�[0m" �[3;36mSTATSD_DSN=udp://localhost:8125/�[0m
Traceback (most recent call last):
  File "./env/bin/talisker.celery", line 11, in <module>
    sys.exit(main())
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/celery.py", line 177, in main
    ensure_extra_versions_supported('celery')
  File "/srv/snap-delta-service/production/snap-delta-service/env/lib/python3.4/site-packages/talisker/util.py", line 55, in ensure_extra_versions_supported
    if pkg.version not in requirement.specifier:
AttributeError: 'Requirement' object has no attribute 'specifier'

We've reverted to 0.9.0 for the time being.

Ability to provide version-info.txt as environment variable?

It's useful to have the COMMIT_ID available to our application, and the best way for us to provide it is as an environment variable. But talisker reads it from the version-info.txt file instead. Would it be possible for talisker to pick it up from either the file or an environment variable? Just to save us having to export it in 2 different ways.

Explore using pg_virtualenv for test pg, rather than depend on system postgresql

http://manpages.ubuntu.com/manpages/xenial/man1/pg_virtualenv.1.html

Add per request deadlines/budgets

Each request has start time and a budget, set by in talisker config, but overridable per view.

Every network call that talisker has support for (requests and postgres, mainly) should set timeouts for those network calls to the remaining budget by default. It should also be easy to get the remaining budget for the current HTTP request, so it can easily be use with other libraries (e.g. cassandra).

HTTP calls should also include the deadline time in a header, so internal services can respect the remaining for a request in the future.

Document RootLogger limitations

The logging.root is a module global RootLogger instance created at import time.

As such it's type is Logger, not talisker.Structured logger. This means that it doesnot support logfmt tags.

But replacing it is hack, and fraught with corner cases, so we will leave it alone. An application should not be using the root logger anyway.

But we should document this limitation.

Data race with new prometheus aggregation

Talisker 0.9.14 introduced a workaround for a memory leak in prometheus_client's multiprocess support (see prometheus/client_python#275)

This workaround relied on the child_exit server hook, and the assumption that signal handling was serialized by the gunicorn main process. This turned out to to be true for every signal except SIGCHLD, which is the one that fires child_exit.

When you have large numbers of workers, this results in multiple SIGCHLDs being sent to the master process, and later signals can interrupt a running child_exit. Talisker did not protect against this, which could result in some data loss. Additionally, because child_exit is run in a signal handler context, it cannot safely do any IO, which was causing some additional issues.

#296 attempts to fix this issue, by doing as little as possible in child_exit, and moving the handling into the main gunicorn master loop, which seems to fix the problem.

Include query parameters in sentry reports

Currently, SQL queries are included as breadcrumbs, but with placeholders rather than with actual parameters, as this may leak senstive info (e.g. sessions or hashes). This limits their usefulness when debugging issues.

Instead, we should include the parameters in the sentry breadcrumb, but also have a blacklist of sensitive table names, that if present in the query, mean we fall back to the placeholder version of the query to avoid leaking.

We should probably have a default blacklist, e.g. 'user', 'session', as well as allow user to add their own blacklisted tables for their app.

Use statsd pipelines

We should start a pipeline on each request and have get_client() return that, and flush it when request is finished

Version 0.9.14 from pip doesn't seem to be latest

Installing latest version from pip (0.9.14) doesn't seem to solve stats import error: ImportError: cannot import name 'StatsClientBase'.

It seems that 0.9.14 released to pip is different from latest 0.9.14 tag in GitHub.

This was found while debugging canonical/snapcraft.io#1042

Invalid .pyup.yml detected

The bot encountered an error in your .pyup.yml config file:

  in "<unicode string>", line 4, column 1:
    update: all
    ^
expected <block end>, but found '<block sequence start>'
  in "<unicode string>", line 29, column 3:
      - setup.cfg:
      ^

You can validate it with this online YAML parser or by taking a look at the Documentation.

Crash on startup with `--statsd-*` command line

Hi! Talisker crashes if Gunicorn's --statsd-host or --statsd-prefix are set (regardless of STATSD_DSN env var being set or not).

$ talisker django_project.wsgi:application --bind 0.0.0.0:8000 --statsd-host=test:123 --statsd-prefix=test
2017-04-10 14:10:47.196Z INFO talisker.sentry.TaliskerSentryClient "Raven is not configured (logging is disabled). Please see the documentation for more information."
2017-04-10 14:10:47.196Z INFO talisker.statsd "configuring statsd DummyClient"
2017-04-10 14:10:47.244Z INFO talisker.celery "enabled talisker celery signals"
2017-04-10 14:10:47.250Z WARNING talisker.gunicorn "ignoring gunicorn statsd config, as has no effect when using talisker, as it uses STATS_DSN env var" statsd_prefix= statsd_host="('test', 123)"
Traceback (most recent call last):
  File "env/bin/talisker", line 11, in <module>
    sys.exit(run())
  File "env/local/lib/python2.7/site-packages/talisker/gunicorn.py", line 261, in run
    return app.run()
  File "env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 192, in run
    super(Application, self).run()
  File "env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 72, in run
    Arbiter(self).run()
  File "env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 61, in __init__
    self.setup(app)
  File "env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 95, in setup
    self.log = self.cfg.logger_class(app.cfg)
  File "env/local/lib/python2.7/site-packages/talisker/gunicorn.py", line 68, in __init__
    super(GunicornLogger, self).__init__(cfg)
  File "env/local/lib/python2.7/site-packages/gunicorn/instrument/statsd.py", line 31, in __init__
    self.prefix = sub(r"^(.+[^.]+)\.*$", "\g<1>.", cfg.statsd_prefix)
  File "env/lib/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or buffer

Python 2.7.6
talisker==0.9.1
gunicorn==19.6.0

Add support for graylog

Talisker has support for logstash, with a custom logstash filter for parsing its log output format.

We should look at adding support for graylog parsing the format also.

RecursionError: maximum recursion using flask-socketio

First, glad i found this. I have been trying to roll my own setup for a while and really happy to find something well thought out and tested.

Due to using flask socketio my entry point has looked like
gunicorn --worker-class eventlet -w 1 --bind 127.0.0.1:8070 --timeout 1800 main:app
or with gevent
gunicorn -k gevent -w 1 --bind 127.0.0.1:8070 --timeout 1800 main:app

Changing either of those to talisker logs the below then dies from a stack overflow.
Is there anything envars required or docs i might have missed that would cause this? Thanks for any help.

2017-05-27 04:48:20.887Z INFO talisker.sentry.TaliskerSentryClient "Raven is not configured (logging is disabled). Please see the documentation for more information."
2017-05-27 04:48:20.888Z INFO talisker.statsd "configuring statsd DummyClient"
2017-05-27 04:48:21.034Z INFO talisker.celery "enabled talisker celery signals"
2017-05-27 04:48:21.173Z INFO gunicorn.error "Starting gunicorn 19.7.1"
2017-05-27 04:48:21.175Z INFO gunicorn.error "Listening at: http://127.0.0.1:8070 (18517)"
2017-05-27 04:48:21.175Z INFO gunicorn.error "Using worker: gevent"
2017-05-27 04:48:21.294Z INFO gunicorn.error "Booting worker with pid: 18523"
Fatal Python error: Cannot recover from stack overflow.

Thread 0x000070000d527000 (most recent call first):
  File "/Users/john/.virtualenvs/soc/lib/python3.6/site-packages/eventlet/hubs/kqueue.py", line 98 in wait
  File "/Users/john/.virtualenvs/soc/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 347 in run

Thread 0x000070000d024000 (most recent call first):
  File "/Users/john/.virtualenvs/soc/lib/python3.6/site-packages/eventlet/hubs/kqueue.py", line 98 in wait
  File "/Users/john/.virtualenvs/soc/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 347 in run

Thread 0x000070000cb21000 (most recent call first):
  File "/Applications/PyCharm 2017.2 EAP.app/Contents/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 353 in _on_run
  File "/Applications/PyCharm 2017.2 EAP.app/Contents/helpers/pydev/_pydevd_bundle/pydevd_comm.py", line 291 in run
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 916 in _bootstrap_inner
  File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/threading.py", line 884 in _bootstrap

Thread 0x000070000c61e000 (most recent call first):
  File "/Users/john/.virtualenvs/soc/lib/python3.6/site-packages/eventlet/hubs/kqueue.py", line 98 in wait
  File "/Users/john/.virtualenvs/soc/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 347 in run

Current thread 0x00007fff946da3c0 (most recent call first):
  File "/Users/john/.virtualenvs/soc/lib/python3.6/site-packages/eventlet/green/threading.py", line 109 in current_thread

canonical-ols / talisker Goto Github PK

talisker's People

Contributors

Stargazers

Watchers

Forkers

talisker's Issues

Recommend Projects

Recommend Topics

Recommend Org