tgres / tgres Goto Github PK

Time Series in Go and PostgreSQL

License: Apache License 2.0

Makefile 0.17% Go 99.83%

go golang grafana graphite postgresql-database statsd time-series

tgres's Introduction

Tgres is a program comprised of several packages which together can be used to receive, store and present time-series data using a relational database as persistent storage (currently only PostgreSQL).

You can currently use the standalone Tgres daemon as Graphite-like API and Statsd replacement all-in-one, or you can use the Tgres packages to incorporate time series collection and reporting functionality into your application.

See GoDoc for package details.

Whether you use standalone Tgres or as a package, the time series data will appear in your database in a compact and efficient format (by default as a view called tv), while at the same time simple to process using any other tool, language, or framework because it is just a table (or a view, rather). For a more detailed description of how Tgres stores data see this article

Current Status

Feb 7 2018: This project is not actively maintained. You may find quite a bit of time-series wisdom here, but there are probably still a lot of bugs.

Jul 5 2017: See this status update

Jun 15 2017: Many big changes since March, most notably data point versioning and instoduction of ds_state and rra_state tables which contain frequently changed attributes as arrays, similar to the way data points are stored eliminating the need to update ds and rra tables, these are now essentially immutable. Ability to delete series with NOTIFY to Tgres to purge it from the cache.

Mar 22 2017: Version 0.10.0b was tagged. This is our first beta (which is more stable than alpha). Please try it out, and take a minute to open an issue or even a PR if you see/fix any problems. Your feedback is most appreciated!

Feb 2017 Note: A major change in the database structure has been made, Tgres now uses the "write optimized" / "vertical" storage. This change affected most of the internal code, and as far overall status, it set us back a bit, all tests are currently broken, but on the bright side, write performance is amazing now.

Phase 1 or proof-of-concept for the project is the ability to (mostly) act as a drop-in replacement for Graphite (except for chart generation) and Statsd. Currently Tgres supports nearly all of Graphite functions.

As of Aug 2016 Tgres is feature-complete for phase 1, which means that the development will focus on tests, documentation and stability for a while.

Tgres is not ready for production use, but is definitely stable enough for tinkering for those interested.

Getting Started

You need a newer Go (1.7+) and PostgreSQL 9.5 or later. To get the daemon compiled all you need is:

$ go get github.com/tgres/tgres

Now you should have a tgres binary in $GOPATH/bin.

There is also a Makefile which lets you build Tgres with make which will use a slightly more elaborate command and the resulting tgres binary will be able to report its build time and git revision, but otherwise it's the same.

Look in $GOPATH/src/github.com/tgres/tgres/etc for a sample config file. Make a copy of this file and edit it, at the very least check the db-connect-string setting. Also check log-file directory, it must be writable.

The user of the PostgreSQL database needs CREATE TABLE permissions. On first run tgres will create three tables (ds, rra and ts) and two views (tv and tvd).

Tgres is invoked like this:

$ $GOPATH/bin/tgres -c /path/to/config

For Developers

There is nothing specific you need to know. If you'd like to submit a bug fix, or for anything else - use Github.

Migrating Graphite Data

Included in cmd/whisper_import is a program that can copy whisper data into Tgres, its command-line arguments are self-explanatory. You should be able to start sending data to Tgres and then migrate your Graphite data retroactively by running whisper_import to avoid gaps in data. It's probably a good idea to test a small subset of series first, migrations can be time consuming and resource-intensive.

tgres's People

Contributors

Stargazers

Watchers

tgres's Issues

HTTP /render endpoint not returning data

What is the correct format for querying data from the render endpoint?
I am running tgres in Docker with a Postgres instance and I am populating the ts table with
while true; do echo -n "carbon.823:$((RANDOM % 100))|c" | nc -w 1 -u 127.0.0.1 8125; done in a separate terminal. I see data in the ts table, but all of my queries at the HTTP /render endpoint returns "SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data". I am (for the most part) formatting queries such that they work with Graphite.

Example queries:
localhost:8888/render?target=carbon.823
localhost:8888/render?target=stats.cabon.823
http://localhost:8888/render/?target=stats.carbon.823&from=-1d&maxDataPoints=100

Tgres is not exiting properly

[36194] 2016/10/18 16:34:31 Read config file: './etc/tgres.conf'.
[36194] 2016/10/18 16:34:31 Logs will remain on stdout
[36194] 2016/10/18 16:34:31 Data Sources will be flushed when cached points exceeds 4096 (max-cached-points).
[36194] 2016/10/18 16:34:31 Data Sources will be flushed after (approximately) 5s (max-cache-duration).
[36194] 2016/10/18 16:34:31 A Data Source will be flushed at most once per 1s (min-cache-duration).
[36194] 2016/10/18 16:34:31 Stats (a la statsd) will be flushed every 10s (stat-flush-interval).
[36194] 2016/10/18 16:34:31 Number of workers (and flushers) will be 4.
[36194] 2016/10/18 16:34:31 Pid saved in /Users/jcreasy/code/tgres/tgres.pid.
[36194] 2016/10/18 16:34:31 Initialized DB connection.
Graphite text protocol Listening on 0.0.0.0:2003
Graphite UDP protocol Listening on 0.0.0.0:2003
Graphite Pickle protocol Listening on 0.0.0.0:2004
Statsd UDP protocol Listening on 0.0.0.0:8125
HTTP protocol Listening on 0.0.0.0:8888
[36194] 2016/10/18 16:34:31 Receiver: starting...
[36194] 2016/10/18 16:34:31 Starting 4 workers...
[36194] 2016/10/18 16:34:31 Starting 4 flushers...
[36194] 2016/10/18 16:34:31 Starting aggWorker...
[36194] 2016/10/18 16:34:31 Starting pacedMetricWorker...
[36194] 2016/10/18 16:34:31   - worker(3) started.
[36194] 2016/10/18 16:34:31 pacedMetricWorker: started.
[36194] 2016/10/18 16:34:31   - flusher(3) started.
[36194] 2016/10/18 16:34:31   - worker(0) started.
[36194] 2016/10/18 16:34:31   - worker(1) started.
[36194] 2016/10/18 16:34:31   - worker(2) started.
[36194] 2016/10/18 16:34:31 aggWorker: started.
[36194] 2016/10/18 16:34:31   - flusher(0) started.
[36194] 2016/10/18 16:34:31   - flusher(1) started.
[36194] 2016/10/18 16:34:31 aggWorker: adding the aggregator.Aggregator DistDatum to the cluster
[36194] 2016/10/18 16:34:31   - flusher(2) started.
[36194] 2016/10/18 16:34:31 Receiver: All workers running, starting dispatcher.
[36194] 2016/10/18 16:34:31 dispatcher(): marking cluster node as Ready.
[36194] 2016/10/18 16:34:31 Transition(): Starting...
[36194] 2016/10/18 16:34:31 Transition(): Id aggregator.Aggregator:1 (TheAggregator) is moving to this node from node <nil>
[36194] 2016/10/18 16:34:31 Receiver: Ready.
[36194] 2016/10/18 16:34:31 Transition(): Waiting on 0 relinquish messages... (timeout 45s) map[]
[36194] 2016/10/18 16:34:31 Transition(): Complete!

^C[36194] 2016/10/18 16:34:47 Got signal: interrupt
[36194] 2016/10/18 16:34:47 Gracefully exiting...
[36194] 2016/10/18 16:34:47 Waiting for all TCP connections to finish...
[36194] 2016/10/18 16:34:47 TCP connections finished.
[36194] 2016/10/18 16:34:47 Transition(): Starting...
[36194] 2016/10/18 16:34:47 handleStatsdTextProtocol(): Error reading: %v read udp [::]:8125: use of closed network connection
[36194] 2016/10/18 16:34:47 Closing dispatcher channel...
[36194] 2016/10/18 16:34:47 handleGraphiteTextProtocol(): Error reading: %v read udp [::]:2003: use of closed network connection
[36194] 2016/10/18 16:34:47 Transition(): Calling Relinquish for aggregator.Aggregator:1 (TheAggregator).
[36194] 2016/10/18 16:34:47 Transition(): Waiting on 0 relinquish messages... (timeout 45s) map[]
[36194] 2016/10/18 16:34:47 Transition(): Complete!
[36194] 2016/10/18 16:34:47 dispatcher(): channel closed, shutting down
[36194] 2016/10/18 16:34:47 Dispatcher finished.
[36194] 2016/10/18 16:34:47 stopPacedMetricWorker(): closing paced metric channel...
[36194] 2016/10/18 16:34:47 stopPacedMetricWorker(): waiting for paced metric worker to finish...
[36194] 2016/10/18 16:34:47 stopPacedMetricWorker(): paced metric worker finished.
[36194] 2016/10/18 16:34:47 stopAggWorker(): closing agg channel...
[36194] 2016/10/18 16:34:47 stopAggWorker(): waiting for agg worker to finish...
[36194] 2016/10/18 16:34:47 aggWorker: channel closed, performing last flush
[36194] 2016/10/18 16:34:47 stopAggWorker(): agg worker finished.
[36194] 2016/10/18 16:34:47 stopWorkers(): closing all worker channels...
[36194] 2016/10/18 16:34:47 stopWorkers(): waiting for workers to finish...
[36194] 2016/10/18 16:34:47 stopWorkers(): all workers finished.
[36194] 2016/10/18 16:34:47 stopFlushers(): closing all flusher channels...
[36194] 2016/10/18 16:34:47 stopFlushers(): waiting for flushers to finish...
[36194] 2016/10/18 16:34:47 flusher(3): channel closed, exiting
[36194] 2016/10/18 16:34:47 flusher(0): channel closed, exiting
[36194] 2016/10/18 16:34:47 flusher(1): channel closed, exiting
[36194] 2016/10/18 16:34:47 flusher(2): channel closed, exiting

[36194] 2016/10/18 16:34:49 pacedMetricWorker: dropping flush timer on the floor - busy system?
^C[36194] 2016/10/18 16:34:50 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:34:51 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:34:52 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:34:53 pacedMetricWorker: dropping flush timer on the floor - busy system?
^C[36194] 2016/10/18 16:34:54 pacedMetricWorker: dropping flush timer on the floor - busy system?
^C^C^C^C^C^C^C^C^C^C[36194] 2016/10/18 16:34:55 pacedMetricWorker: dropping flush timer on the floor - busy system?
^C[36194] 2016/10/18 16:34:56 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:34:57 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:34:58 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:34:59 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:00 aggWorker: dropping aggreagator flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:00 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:01 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:02 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:03 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:04 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:05 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:06 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:07 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:08 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:09 pacedMetricWorker: dropping flush timer on the floor - busy system?
[36194] 2016/10/18 16:35:10 aggWorker: dropping aggreagator flush timer on the floor - busy system?
^C[36194] 2016/10/18 16:35:10 pacedMetricWorker: dropping flush timer on the floor - busy system?```

Can't build on Windows?

I'm on Windows 10.

"go get github.com/tgres/tgres"

Gives me the following output

daemon.go:232: undefined: syscall.Kill
daemon.go:235: undefined: syscall.SIGUSR1
daemon.go:394: undefined: syscall.Kill
daemon.go:394: undefined: syscall.SIGUSR1

A quick Google search leads me to believe that it's using some very unix-specific things that won't work on Windows.

undefined: pq.Array golang go version go1.8 darwin/amd64

# github.com/tgres/tgres/serde
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:171: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:179: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:221: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:228: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:293: undefined: pq.Array
../../.go/src/github.com/tgres/tgres/serde/postgres_common.go:302: undefined: pq.Array```

averageSeries with wildcard causes 'invalid memory address or nil pointer dereference'

We have two metrics:

local.test.a
local.test.b

We are producing some random data points for these metrics. When we are trying to render one of them, everything looks fine:

$ curl -s "http://localhost:8888/render?target=averageSeries(local.test.a)&from=$( date +%s --date=-5min )&maxDataPoints=10&format=json"
[
{"target": "averageSeries(local.test.a)", "datapoints": [
[null, 1482321120],[4.733333333333334, 1482321180],[5.383333333333332, 1482321240],[4.783333333333332, 1482321300],[4.1833333333333345, 1482321360],[5.216666666666667, 1482321420]]}]

But if we try to have average across both metrics:

$ curl -s "http://localhost:8888/render?target=averageSeries(local.test.*)&from=$( date +%s --date=-5min )&maxDataPoints=10&format=json"

then we are getting an error and the following stack trace in the logs:

[9] 2016/12/21 11:51:26 http: panic serving 172.17.0.1:54804: runtime error: invalid memory address or nil pointer dereference
goroutine 14 [running]:
net/http.(*conn).serve.func1(0xc820092a80)
	/usr/lib/go-1.6/src/net/http/server.go:1389 +0xc1
panic(0xaedd80, 0xc820010080)
	/usr/lib/go-1.6/src/runtime/panic.go:443 +0x4e9
database/sql.(*Rows).Close(0x0, 0x0, 0x0)
	/usr/lib/go-1.6/src/database/sql/sql.go:1863 +0x28
github.com/tgres/tgres/serde.(*dbSeries).Close(0xc820109ba0, 0x0, 0x0)
	/usr/local/src/github.com/tgres/tgres/serde/dbseries.go:190 +0x34
github.com/tgres/tgres/dsl.(*SeriesList).Close(0xc820388600, 0x0, 0x0)
	/usr/local/src/github.com/tgres/tgres/dsl/funcs.go:560 +0xa6
github.com/tgres/tgres/http.GraphiteRenderHandler.func1(0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/local/src/github.com/tgres/tgres/http/graphite.go:122 +0x1243
net/http.HandlerFunc.ServeHTTP(0xc820232010, 0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/lib/go-1.6/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82000b710, 0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/lib/go-1.6/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc820236000, 0x7f849dfd1f70, 0xc820109520, 0xc8201180e0)
	/usr/lib/go-1.6/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc820092a80)
	/usr/lib/go-1.6/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
	/usr/lib/go-1.6/src/net/http/server.go:2137 +0x44e

Could it be that in dbseries.go:190 we are trying to call Close() on a nil pointer?

Metric listing is incorrect sometimes

foo.*.blah.*.blah doesn't always work (see tgres.*.receiver.workers.*)

also drop downs should distinctify the list, e.g. tgres.*.receiver.workers.*.channel.* should not list fill_percent 4 times.

Viewing newly added metrics requires restart

I'm sending graphite formatted metrics to tgres from a script to test it out, and the /render endpoint returns an empty set for any metrics added after tgres was started.

To resolve I can force tgres to restart, and then the data is available. Note that all the points that were sent are visible, not just those that were created after the restart. You can see what I mean in the following grafana screenshots:

Before restart

After tgres restart at ~07:12

The gap / backfill problem.

If we have a series that has not received a data point for a while (i.e. a number of steps), the next data point will cause RRD to fill the gap. This will in turn cause a lot of rows to get updated in the database. If there are many such "stale" series suddenly brought back to life, it causes the vcache to temporarily grow very large followed by a lot of db activity. On a busy set up it is sufficient to leave Tgres stopped for an hour or two to experience this, and there is no easy work-around, it's possible to end up in a situation when Tgres just cannot be started back up. As a band-aid we could impose a limit on the vcache size, but that doesn't really solve the problem.

One potential way to solve this problem is to not back-fill if the gap exceeds a certain size. The problem is that due to the round-robin nature of our storage, we cannot simply leave slots untouched because they may have previous data in them ("shadow data"). The way to address the sub-problem of shadow data is to add some kind of a version to each data point. Whisper, for example, stores the timestamp, and so if a data point timestamp is outside the range expected, it can be ignored as "expired".

Adding a timestamp to seems like a lot of extra data for not much gain, but alternatively it could be a 1 byte version. 1 byte would prevent shadow data across 256 iterations of the db, which seems sufficient. If an RRA has been stale for more than 256 times its span, we could have a special rule that prevents its update (i.e. you have to delete and re-create it, for example).

This would be another backwards-incompatible db schema change, alas.

Edit: Smallest PG type appears to be SMALLINT which is 2 bytes.

isNonNull is incorrect

isNonNull() should return the same number of series as it was given, with just 0 or 1s. Our current implementation ads them up and returns a single series, but it shouldn't do this.

(As such, it probably shouldn't be in the "combine" list in Grafana either as it does not combine.)

Failed to get final advertise address

Using the default config (except a different db connection string), I get the following error:

Error initializing cluster, exiting: Failed to get final advertise address: %!v(MISSING)

To fix this, I set the TGRES_BIND environment variable (to "0.0.0.0" to match the config), but I had to read the source to find it and I'm not sure it's the correct fix.

Graceful restart broken

This relates to #4 - since implementing a background/daemon option requires a gracefulRestart immediately on start up.

[1962] 2016/11/02 13:29:16 Got signal: hangup
[1962] 2016/11/02 13:29:16 gracefulRestart(): Beginning graceful restart with sockets: [0xc42015e128 0xc42015e138 0xc42015e140 0xc42015e148 0xc42015e150] and protos: "gt,gu,gp,su,www"
[1962] 2016/11/02 13:29:16 gracefulRestart(): Forked child, waiting to be killed...
[1992] 2016/11/02 13:29:16 Server starting.
[1992] 2016/11/02 13:29:16 Data Sources will be flushed when cached points exceeds 4096 (max-cached-points).
[1992] 2016/11/02 13:29:16 Data Sources will be flushed after (approximately) 5s (max-cache-duration).
[1992] 2016/11/02 13:29:16 A Data Source will be flushed at most once per 1s (min-cache-duration).
[1992] 2016/11/02 13:29:16 Stats (a la statsd) will be flushed every 10s (stat-flush-interval).
[1992] 2016/11/02 13:29:16 Number of workers (and flushers) will be 4.
[1992] 2016/11/02 13:29:16 Pid saved in /home/grisha/golang/src/github.com/tgres/tgres/tgres.pid.
[1992] 2016/11/02 13:29:16 Initialized DB connection.
[1992] 2016/11/02 13:29:16 Error initializing cluster, exiting: Failed to start TCP listener. Err: listen tcp 0.0.0.0:7946: bind: address already in use
[1992] 2016/11/02 13:29:16 main: Waiting for all other goroutines to finish...
[1992] 2016/11/02 13:29:16 main: All goroutines finished, exiting.

Value Lists not supported

It seems that "Value Lists" syntax {foo,bar,...} from http://graphite.readthedocs.io/en/latest/render_api.html#paths-and-wildcards is not supported by tgres.

When testing with curl (note that curl has its own idea about interpreting curly braces, so we need to escape them):

$ curl -v 'http://localhost:8888/render?target=local.random.\{a,b\}&maxDataPoints=10&from=-1h'

I'm getting the following error in the log:

[8] 2016/12/22 15:12:32 RenderHandler(): Error parsing "group(\"local.random.\"{a,b})": 1:22: missing ',' in argument list

This seems to be caused by the fact that we use go parser here: https://github.com/tgres/tgres/blob/master/dsl/dsl.go#L45

Collectd support

Docs on integrating collectd with tgres would be nice.

If this is possible of course :)

Connecting to Grafana

Hi. This is more a question about documentation.

I see you've had success using tgres as a data source for Grafana. Did you have to do anything special to get this working?

I really agree with your premise that time series should be stored in a relational database, so I'm hoping I can get this working.

Add foreground option for Docker use

I'd like to use this tool, and I'm happy to setup a Dockerfile for it.

If you want to add a branch where you are doing the WIP receiver refactor, I can contribute on that also. Or I can wait until you're at a milestone.

provide a way to delete data sources

I'm not sure if this is desirable with the way tgres stores data, but it would be helpful to have a command (or postgresql function) to delete a data source (or documentation if this exists already). #

fatal error: concurrent map iteration and map write

Hi,
I installed the latest tgres, and postgres.
here's my config:


# This is a TOML file: https://github.com/toml-lang/toml

min-step                = "10s"

# 0 - unlilimited (default). points in excess are discarded
max-receiver-queue-size  = 1000000000

# number of flushers == number of workers
workers                 = 4

pid-file =                 "tgres.pid"
log-file =                 "log/tgres.log"
log-cycle-interval =       "24h"

http-listen-spec            = "0.0.0.0:8888"
graphite-line-listen-spec   = "0.0.0.0:2003"
graphite-text-listen-spec   = "0.0.0.0:2003"
graphite-udp-listen-spec    = "0.0.0.0:2003"
graphite-pickle-listen-spec = "0.0.0.0:2004"

statsd-text-listen-spec     = "0.0.0.0:8125"
statsd-udp-listen-spec      = "0.0.0.0:8125"
stat-flush-interval         = "10s"
stats-name-prefix           = "stats"

# RedHat and some others:
#db-connect-string = "host=/tmp dbname=tgres sslmode=disable"
# Debian and some others:
db-connect-string = "host=/var/run/postgresql dbname=tgres sslmode=disable"

[[ds]]
regexp = ".*"
step = "10s"
heartbeat = "2h"
# rra is "[wmean|min|max|last:]ts:ts[:xff]"
# function is not case-sensitive, default is "wmean".
rras = ["10s:6h", "1m:24h", "10m:93d", "1d:5y:1"]

I created a postgres role and ran createdb tgres, then was able to launch tgres.
i started to write some fake data into port 2003 and loaded up http://localhost:8888/ in my browser. but then tgres aborted, as you can see in the output below.

tgres -c tgres.conf
[16639] 2017/03/12 22:26:18 Tgres starting.
[16639] 2017/03/12 22:26:18 Using config file: 'tgres.conf'.
[16639] 2017/03/12 22:26:18 Logs will be written to '/home/dieter/go/src/github.com/tgres/tgres/etc/log/tgres.log'.
[16639] 2017/03/12 22:26:18 Will cycle logs every 24h0m0s (log-cycle-interval).
[16639] 2017/03/12 22:26:18 All further status messages will be written to log file(s) in '/home/dieter/go/src/github.com/tgres/tgres/etc/log/'.
Graphite text protocol Listening on 0.0.0.0:2003
Graphite UDP protocol Listening on 0.0.0.0:2003
Graphite Pickle protocol Listening on 0.0.0.0:2004
Statsd UDP protocol Listening on 0.0.0.0:8125
HTTP protocol Listening on 0.0.0.0:8888
fatal error: concurrent map iteration and map write

goroutine 65 [running]:
runtime.throw(0x99daef, 0x26)
	/usr/lib/go/src/runtime/panic.go:596 +0x95 fp=0xc420384bf8 sp=0xc420384bd8
runtime.mapiternext(0xc420384d98)
	/usr/lib/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc420384ca8 sp=0xc420384bf8
github.com/tgres/tgres/receiver.(*verticalCache).update(0xc4201f6380, 0xbf0dc0, 0xc42013c040)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/vcache.go:93 +0x2af fp=0xc420384e08 sp=0xc420384ca8
github.com/tgres/tgres/receiver.(*dsFlusher).verticalFlush(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:90 +0x9a fp=0xc420384e58 sp=0xc420384e08
github.com/tgres/tgres/receiver.(*dsFlusher).flushToVCache(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:108 +0x5b fp=0xc420384e80 sp=0xc420384e58
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:76 +0x197 fp=0xc420384f18 sp=0xc420384e80
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167 fp=0xc420384fa8 sp=0xc420384f18
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc420384fb0 sp=0xc420384fa8
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 1 [chan receive]:
github.com/tgres/tgres/daemon.glob..func11(0xc42000b100, 0xc42015fe40, 0x7ffe3bb06262, 0xa, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/daemon.go:131 +0x133
github.com/tgres/tgres/daemon.Init(0x7ffe3bb06262, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400000)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/daemon.go:249 +0x1156
main.main()
	/home/dieter/go/src/github.com/tgres/tgres/main.go:70 +0x3d1

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 5 [syscall]:
os/signal.signal_recv(0x0)
	/usr/lib/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
	/usr/lib/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
	/usr/lib/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 19 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func11.1(0xbe7780, 0xc42000b100, 0xc4200676e0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:174 +0x2a
created by github.com/tgres/tgres/receiver.glob..func11
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:177 +0x5d

goroutine 10 [chan receive]:
github.com/tgres/tgres/daemon.glob..func17.1(0xc420168180, 0x3c)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:71 +0x4c
created by github.com/tgres/tgres/daemon.glob..func17
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:79 +0x6c

goroutine 11 [sleep]:
time.Sleep(0x4e94914f0000)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/daemon.glob..func17.2(0x4e94914f0000)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:83 +0x2b
created by github.com/tgres/tgres/daemon.glob..func17
	/home/dieter/go/src/github.com/tgres/tgres/daemon/log.go:89 +0x8e

goroutine 12 [chan receive]:
database/sql.(*DB).connectionOpener(0xc420085180)
	/usr/lib/go/src/database/sql/sql.go:837 +0x4a
created by database/sql.Open
	/usr/lib/go/src/database/sql/sql.go:582 +0x212

goroutine 14 [chan receive]:
github.com/tgres/tgres/graceful.NewListener.func1(0xc42000e160)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:52 +0x5e
created by github.com/tgres/tgres/graceful.NewListener
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:55 +0x108

goroutine 15 [IO wait]:
net.runtime_pollWait(0x7f82c5716118, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c848, 0x72, 0x0, 0xc4203457c0)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c848, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005c7e0, 0x0, 0xbe5180, 0xc4203457c0)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e158, 0xc42002b6d0, 0x40e4e8, 0x94a3a0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e158, 0x1, 0xc42002b738, 0x43372b, 0xc42002b710)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
github.com/tgres/tgres/graceful.(*Listener).Accept(0xc420167940, 0x9adb00, 0xc42000b100, 0xbee180, 0xc42035ca30)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:68 +0x37
github.com/tgres/tgres/daemon.(*graphiteTextServiceManager).graphiteTextServer(0xc4201678e0, 0x31, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:417 +0x3d
created by github.com/tgres/tgres/daemon.(*graphiteTextServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:408 +0x207

goroutine 16 [IO wait]:
net.runtime_pollWait(0x7f82c5716058, 0x72, 0x7)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c8b8, 0x72, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c8b8, 0xc4201aa000, 0x1000)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42005c850, 0xc4201aa000, 0x1000, 0x1000, 0x0, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e168, 0xc4201aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/net.go:181 +0x70
bufio.(*Scanner).Scan(0xc42002bf30, 0xbee0c0)
	/usr/lib/go/src/bufio/scan.go:207 +0x294
github.com/tgres/tgres/daemon.handleGraphiteTextProtocol(0xc42000b100, 0xbee0c0, 0xc42000e168, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:455 +0x105
created by github.com/tgres/tgres/daemon.(*graphiteUdpTextServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:357 +0x213

goroutine 34 [chan receive]:
github.com/tgres/tgres/graceful.NewListener.func1(0xc42000e178)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:52 +0x5e
created by github.com/tgres/tgres/graceful.NewListener
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:55 +0x108

goroutine 35 [IO wait]:
net.runtime_pollWait(0x7f82c5715f98, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c928, 0x72, 0x0, 0xc420120020)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c928, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005c8c0, 0x0, 0xbe5180, 0xc420120020)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e170, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e170, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
github.com/tgres/tgres/graceful.(*Listener).Accept(0xc4201679a0, 0x0, 0x0, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:68 +0x37
github.com/tgres/tgres/daemon.(*graphitePickleServiceManager).graphitePickleServer(0xc420167900, 0x1, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:223 +0x3d
created by github.com/tgres/tgres/daemon.(*graphitePickleServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:214 +0x1e6

goroutine 36 [IO wait]:
net.runtime_pollWait(0x7f82c5715ed8, 0x72, 0x9)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005c998, 0x72, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005c998, 0xc4201a8000, 0x1000)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc42005c930, 0xc4201a8000, 0x1000, 0x1000, 0x0, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e180, 0xc4201a8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/net.go:181 +0x70
bufio.(*Scanner).Scan(0xc42002d730, 0xbee0c0)
	/usr/lib/go/src/bufio/scan.go:207 +0x294
github.com/tgres/tgres/daemon.handleStatsdTextProtocol(0xc42000b100, 0xbee0c0, 0xc42000e180, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:509 +0x105
created by github.com/tgres/tgres/daemon.(*statsdUdpTextServiceManager).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:576 +0x213

goroutine 37 [chan receive]:
github.com/tgres/tgres/graceful.NewListener.func1(0xc42000e190)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:52 +0x5e
created by github.com/tgres/tgres/graceful.NewListener
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:55 +0x108

goroutine 38 [IO wait]:
net.runtime_pollWait(0x7f82c5715e18, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005ca08, 0x72, 0x0, 0xc420120080)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005ca08, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005c9a0, 0x0, 0xbe5180, 0xc420120080)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e188, 0x429f7f, 0xc42003ce48, 0x410d18)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e188, 0xc420110240, 0xc42003ceb0, 0x4295de, 0xc420110240)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
github.com/tgres/tgres/graceful.(*Listener).Accept(0xc420167a00, 0xc420110210, 0x8eabe0, 0xc10b50, 0x934320)
	/home/dieter/go/src/github.com/tgres/tgres/graceful/graceful.go:68 +0x37
net/http.(*Server).Serve(0xc4201c2000, 0xbe9d80, 0xc420167a00, 0x0, 0x0)
	/usr/lib/go/src/net/http/server.go:2643 +0x228
github.com/tgres/tgres/daemon.httpServer(0xc42015e370, 0xc, 0xbe9d80, 0xc420167a00, 0xc42000b100, 0xbec860, 0xc4201678c0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/http.go:51 +0x22f
created by github.com/tgres/tgres/daemon.(*wwwServer).Start
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:163 +0x2e1

goroutine 39 [IO wait]:
net.runtime_pollWait(0x7f82c5715d58, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005ca78, 0x72, 0x0, 0xc4201c0000)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005ca78, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005ca10, 0x0, 0xbe5180, 0xc4201c0000)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e1a0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e1a0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock.go:215 +0x49
github.com/hashicorp/memberlist.(*Memberlist).tcpListen(0xc42015a2a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/net.go:180 +0x33
created by github.com/hashicorp/memberlist.newMemberlist
	/home/dieter/go/src/github.com/hashicorp/memberlist/memberlist.go:123 +0x79d

goroutine 40 [IO wait]:
net.runtime_pollWait(0x7f82c5715c98, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005cae8, 0x72, 0x0, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005cae8, 0xc4201ac000, 0x10000)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).readFrom(0xc42005ca80, 0xc4201ac000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0xbe6780, 0xbe2648)
	/usr/lib/go/src/net/fd_unix.go:277 +0x1cc
net.(*UDPConn).readFrom(0xc42000e1a8, 0xc4201ac000, 0x10000, 0x10000, 0xc42018b6b8, 0x4407cb, 0x10000, 0x8cc520)
	/usr/lib/go/src/net/udpsock_posix.go:47 +0x6a
net.(*UDPConn).ReadFrom(0xc42000e1a8, 0xc4201ac000, 0x10000, 0x10000, 0x10000, 0x10000, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/udpsock.go:109 +0x75
github.com/hashicorp/memberlist.(*Memberlist).udpListen(0xc42015a2a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/net.go:275 +0xc6
created by github.com/hashicorp/memberlist.newMemberlist
	/home/dieter/go/src/github.com/hashicorp/memberlist/memberlist.go:124 +0x7c2

goroutine 41 [select]:
github.com/hashicorp/memberlist.(*Memberlist).udpHandler(0xc42015a2a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/net.go:361 +0x333
created by github.com/hashicorp/memberlist.newMemberlist
	/home/dieter/go/src/github.com/hashicorp/memberlist/memberlist.go:125 +0x7e7

goroutine 42 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc42015a2a0, 0x3b9aca00, 0xc420067080, 0xc420067020, 0xc4201f01a0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:105 +0x196
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:71 +0x3f0

goroutine 43 [select]:
github.com/hashicorp/memberlist.(*Memberlist).pushPullTrigger(0xc42015a2a0, 0xc420067020)
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:132 +0x1ed
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:77 +0x341

goroutine 44 [select]:
github.com/hashicorp/memberlist.(*Memberlist).triggerFunc(0xc42015a2a0, 0xbebc200, 0xc4200670e0, 0xc420067020, 0xc4201f01b0)
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:105 +0x196
created by github.com/hashicorp/memberlist.(*Memberlist).schedule
	/home/dieter/go/src/github.com/hashicorp/memberlist/state.go:83 +0x1ed

goroutine 45 [chan receive]:
github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType.func1(0xc42000e1c8, 0xc4200acfd0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:333 +0x65
created by github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:360 +0x113

goroutine 46 [IO wait]:
net.runtime_pollWait(0x7f82c5715bd8, 0x72, 0x0)
	/usr/lib/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc42005d028, 0x72, 0x0, 0xc4201fa020)
	/usr/lib/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc42005d028, 0xffffffffffffffff, 0x0)
	/usr/lib/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).accept(0xc42005cfc0, 0x0, 0xbe5180, 0xc4201fa020)
	/usr/lib/go/src/net/fd_unix.go:430 +0x1e5
net.(*TCPListener).accept(0xc42000e1f8, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).Accept(0xc42000e1f8, 0x0, 0x0, 0x0, 0x0)
	/usr/lib/go/src/net/tcpsock.go:228 +0x49
net/rpc.(*Server).Accept(0xc4200157c0, 0xbeaa40, 0xc42000e1f8)
	/usr/lib/go/src/net/rpc/server.go:623 +0x3b
net/rpc.Accept(0xbeaa40, 0xc42000e1f8)
	/usr/lib/go/src/net/rpc/server.go:681 +0x41
github.com/tgres/tgres/cluster.NewClusterBind.func1(0xc4200acfd0)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:156 +0x3e
created by github.com/tgres/tgres/cluster.NewClusterBind
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:158 +0x781

goroutine 66 [chan receive]:
github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType.func1(0xc42011e008, 0xc4200acfd0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:333 +0x65
created by github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:360 +0x113

goroutine 48 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f63c0, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 49 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f63e0, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 50 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f6400, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 51 [chan receive]:
github.com/tgres/tgres/receiver.glob..func15(0xbeb940, 0xc4201f6420, 0xbe7800, 0xc420085220, 0xc4200673e0, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:243 +0x1f2
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:60 +0x31e

goroutine 52 [sleep]:
time.Sleep(0x2540be400)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func16(0xc4201f6380, 0xc4200673e0, 0xbe7800, 0xc420085220, 0x2540be400, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:304 +0x2f
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:62 +0x39f

goroutine 53 [chan receive]:
github.com/tgres/tgres/receiver.glob..func14(0xbeb940, 0xc4201f6460, 0xbeddc0, 0xc4200191d0, 0xc420067440, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:173 +0x23d
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:67 +0x4f8

goroutine 54 [sleep]:
time.Sleep(0x37e11d600)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.reportTsTableSize(0x7f82c571a458, 0xc420085220, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:325 +0x34
created by github.com/tgres/tgres/receiver.(*dsFlusher).start
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:71 +0x5c9

goroutine 55 [select]:
github.com/tgres/tgres/receiver.glob..func5(0xbeb940, 0xc4201f64c0, 0xc420066cc0, 0xbee4e0, 0xc4200acfd0, 0x2540be400, 0xc42015ead0, 0x5, 0xbe7780, 0xc42000b100, ...)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:142 +0x603
created by github.com/tgres/tgres/receiver.glob..func29
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:142 +0x168

goroutine 67 [chan receive]:
github.com/tgres/tgres/receiver.glob..func1(0x98bcc5, 0x9, 0xc420252060, 0xc420066cc0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:32 +0x7f
created by github.com/tgres/tgres/receiver.glob..func5
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:111 +0x828

goroutine 68 [sleep]:
time.Sleep(0x2540ae60b)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func2(0x98bcc5, 0x9, 0xc4202520c0, 0x2540be400)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:62 +0x126
created by github.com/tgres/tgres/receiver.glob..func5
	/home/dieter/go/src/github.com/tgres/tgres/receiver/aggworker.go:115 +0x118

goroutine 56 [select]:
github.com/tgres/tgres/receiver.glob..func19(0xbeb940, 0xc4201f6500, 0xc420066d20, 0xbe4540, 0xc42000b100, 0xbe4580, 0xc42000b100, 0x3b9aca00, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/pacedmetric.go:87 +0x73b
created by github.com/tgres/tgres/receiver.glob..func30
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:148 +0x15e

goroutine 57 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.glob..func18(0xc42005d030, 0x3b9aca00, 0x991065, 0x11)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/pacedmetric.go:64 +0x4f
created by github.com/tgres/tgres/receiver.glob..func19
	/home/dieter/go/src/github.com/tgres/tgres/receiver/pacedmetric.go:81 +0x16a

goroutine 58 [runnable]:
github.com/tgres/tgres/receiver.glob..func12(0xbeb940, 0xc4201f65a0, 0xc420066c60, 0x4, 0xbee4e0, 0xc4200acfd0, 0xbe7780, 0xc42000b100, 0xc42005c770, 0xbeddc0, ...)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:259 +0xf58
created by github.com/tgres/tgres/receiver.glob..func21
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:68 +0x431

goroutine 59 [chan receive]:
github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType.func1(0xc42000e228, 0xc4200acfd0, 0x2)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:333 +0x65
created by github.com/tgres/tgres/cluster.(*Cluster).RegisterMsgType
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:360 +0x113

goroutine 60 [chan receive]:
github.com/tgres/tgres/receiver.glob..func6(0xc420067560, 0xc420066c60)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:32 +0x6e
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:224 +0xffe

goroutine 61 [sleep]:
time.Sleep(0x3b9aca00)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.reportOverrunQueueSize(0xc4201f65c0, 0xbe7780, 0xc42000b100, 0x3b9aca00)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:156 +0x2b
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:229 +0x140

goroutine 62 [select]:
github.com/tgres/tgres/receiver.elasticCh(0xc420066c60, 0xc420067680, 0xc4201f65c0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/elastic_ch.go:56 +0x5ba
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:232 +0x19d

goroutine 63 [chan receive]:
github.com/tgres/tgres/receiver.glob..func11(0xc4200676e0, 0xc420066c60, 0xc42005c770, 0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:180 +0x93
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:238 +0x214

goroutine 64 [semacquire]:
sync.runtime_SemacquireMutex(0xc420370884)
	/usr/lib/go/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc420370880)
	/usr/lib/go/src/sync/mutex.go:87 +0x9d
github.com/tgres/tgres/receiver.(*cachedDs).processIncoming(0xc42026c360, 0x0, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/dscache.go:175 +0x6b
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:69 +0x40
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 82 [runnable]:
sync.runtime_Semrelease(0xc420370884)
	/usr/lib/go/src/runtime/sema.go:56 +0x3a
sync.(*Mutex).Unlock(0xc420370880)
	/usr/lib/go/src/sync/mutex.go:126 +0x81
github.com/tgres/tgres/receiver.(*cachedDs).processIncoming(0xc42026c360, 0x0, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/dscache.go:189 +0x37a
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:69 +0x40
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x2)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 83 [semacquire]:
sync.runtime_SemacquireMutex(0xc4203702e4)
	/usr/lib/go/src/runtime/sema.go:62 +0x34
sync.(*Mutex).Lock(0xc4203702e0)
	/usr/lib/go/src/sync/mutex.go:87 +0x9d
github.com/tgres/tgres/receiver.(*verticalCache).update(0xc4201f6380, 0xbf0dc0, 0xc42013c040)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/vcache.go:90 +0x147
github.com/tgres/tgres/receiver.(*dsFlusher).verticalFlush(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:90 +0x9a
github.com/tgres/tgres/receiver.(*dsFlusher).flushToVCache(0xc4200191d0, 0xbf02c0, 0xc420378ed0)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/flusher.go:108 +0x5b
github.com/tgres/tgres/receiver.glob..func8(0xc42026c360, 0xbeddc0, 0xc4200191d0, 0x1)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:76 +0x197
github.com/tgres/tgres/receiver.glob..func13(0xc4201f0500, 0xc420067740, 0xbeddc0, 0xc4200191d0, 0xbe7780, 0xc42000b100, 0x3)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:351 +0x167
created by github.com/tgres/tgres/receiver.glob..func12
	/home/dieter/go/src/github.com/tgres/tgres/receiver/director.go:245 +0x384

goroutine 20 [sleep]:
time.Sleep(0x12a05f200)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/receiver.reportRuntime(0xbe7780, 0xc42000b100)
	/home/dieter/go/src/github.com/tgres/tgres/receiver/runtime.go:50 +0x34
created by github.com/tgres/tgres/receiver.glob..func21
	/home/dieter/go/src/github.com/tgres/tgres/receiver/startstop.go:72 +0x4a7

goroutine 21 [select, locked to thread]:
runtime.gopark(0x9ae7a0, 0x0, 0x9896b0, 0x6, 0x18, 0x2)
	/usr/lib/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc4201caf50, 0x0, 0x18)
	/usr/lib/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc4201caf50)
	/usr/lib/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
	/usr/lib/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
	/usr/lib/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 69 [sleep]:
time.Sleep(0xa7a358200)
	/usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/tgres/tgres/cluster.(*Cluster).Transition.func3.1(0xa7a358200, 0xc42026e000)
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:748 +0x2b
created by github.com/tgres/tgres/cluster.(*Cluster).Transition.func3
	/home/dieter/go/src/github.com/tgres/tgres/cluster/cluster.go:750 +0x1b3

goroutine 391 [runnable]:
fmt.(*ss).SkipSpace(0xc4202145a0)
	/usr/lib/go/src/fmt/scan.go:304 +0x3f
fmt.(*ss).doScanf(0xc4202145a0, 0x98ac11, 0x8, 0xc420389e10, 0x3, 0x3, 0x0, 0x0, 0x0)
	/usr/lib/go/src/fmt/scan.go:1188 +0x354
fmt.Fscanf(0xbe4300, 0xc420243c80, 0x98ac11, 0x8, 0xc420389e10, 0x3, 0x3, 0x8, 0x8c8520, 0xc42018ec01)
	/usr/lib/go/src/fmt/scan.go:143 +0xc5
fmt.Sscanf(0xc420415290, 0x25, 0x98ac11, 0x8, 0xc420389e10, 0x3, 0x3, 0x500, 0xc420415290, 0xc420389de8)
	/usr/lib/go/src/fmt/scan.go:114 +0xad
github.com/tgres/tgres/daemon.parseGraphitePacket(0xc420415290, 0x25, 0x25, 0x52a, 0xc420415290, 0x25, 0xc1f780, 0x4102894000000000, 0x0, 0x0)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:484 +0x15d
github.com/tgres/tgres/daemon.handleGraphiteTextProtocol(0xc42000b100, 0xbee180, 0xc42035ca30, 0xa)
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:458 +0x15e
created by github.com/tgres/tgres/daemon.(*graphiteTextServiceManager).graphiteTextServer
	/home/dieter/go/src/github.com/tgres/tgres/daemon/services.go:438 +0x214

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.