grafana / carbon-relay-ng Goto Github PK
View Code? Open in Web Editor NEWFast carbon relay+aggregator with admin interfaces for making changes online - production ready
License: Other
Fast carbon relay+aggregator with admin interfaces for making changes online - production ready
License: Other
@pauloconnor currently it checks with a regex for ip:port
, and doesn't let me submit anything else
it should also support hostnames and dns names instead of just an ip.
Following the readme to install :
go get github.com/graphite-ng/carbon-relay-ng
/opt/data/apps/go/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:7:2: local import "./routing" in non-local package
Here is my Go version :
go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/data/apps/go"
GORACE=""
GOROOT="/usr/local/go/bin"
GOTOOLDIR="/usr/local/go/bin/pkg/tool/linux_amd64"
TERM="dumb"
CC="gcc"
GOGCCFLAGS="-g -O2 -fPIC -m64 -pthread"
CXX="g++"
CGO_ENABLED="1"
go version
go version go1.2.2 linux/amd64
interesting situation happened yesterday:
we have 1 main statsdaemon, a whole bunch of collectd's (around 400), some diamonds, and some ssh tunnels writing metrics into the relay.
yesterday we wanted to run some local statsdaemons on some servers, instead of the central one, so they just took some existing metric traffic , didn't really add much new. just 7 more clients. somehow this caused our pipeline to block. not sure that carbon-relay-ng is at fault, but it feels that way :?
we do about 4k metrics/s with peaks at 20k/s when statsd flushes. presumably most clients send spread out over time, statsd sends on the minute "on the dot", maybe that's related
after updating js code, recompiling and running, browser keeps using the old js.
this bit me for example in #26
I can only change the config file and restart carbon-relay
I'm having problems installing on Ubuntu 14.04 with Go v1.2. Any help is greatly appreciated.
# export GOPATH=/tmp/foo
# export PATH=$PATH:$GOPATH/bin
# go get -d github.com/graphite-ng/carbon-relay-ng
# go get github.com/jteeuwen/go-bindata/...
# cd "$GOPATH/src/github.com/graphite-ng/carbon-relay-ng"
# git checkout v0.5
Note: checking out 'v0.5'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
[0/4613]
git checkout -b new_branch_name
HEAD is now at fa7022b... document new releases / versioning approach
# make
make: Nothing to be done for `all'.
Hello,
with some regret I read you had to close the consistent hashing bug. I would like to open this one as an enhancement/feature request for this feature, as it is the one last thing that prevents me from using carbon-relay-ng instead of the original relay-agent.
Sadly Im not a good enough developer to fix this issue, instead I tried some of the existing packages for consistent-hashing and fed them the metrics from my graphite cluster (1M+ metrics), but sadly none of them distributed the metrics in such a way as the pyhton/graphite implementation does.
I had a look at:
"github.com/golang/groupcache/consistenthash"
"github.com/ngerakines/ketama"
"github.com/stathat/consistent"
I failed to make it run with this package, due to a missing understanding how to initialise it with my two backends: "github.com/dgryski/go-ketama".
If you know of more packages, Ill happily feed my metrics into them to check the distribution ;)
Kind regards,
Marcus
the last few commits (less allocations, faster path for disabled logging, RCU-backed table config instead of mutex based, amortize cost of field splitting for aggregators) have given me about 30% decrease in cpu usage. I wonder what your results are.
Here are some more ideas to shave off more cpu time:
I tried to build carbon-relay-ng as described in [1], but it just fails.
I tried it with ubuntu-precise and ubuntu-trusty in a vagrant box, both
did not work.
Here the command which fails:
root@vagrant-ubuntu-trusty-64:~/go/src/github.com/graphite-ng/carbon-relay-ng#
./make.sh
github.com/graphite-ng/carbon-relay-ng/admin
admin/admin-http.go:141: undefined: AssetDir
dependencies were installed without problems on trusty
root@vagrant-ubuntu-trusty-64:/go/src/github.com/graphite-ng/carbon-relay-ng#/go/src/github.com/graphite-ng/carbon-relay-ng#
ls $GOPATH
bin lib pkg src
root@vagrant-ubuntu-trusty-64:
ls /root/go/src/
github.com
root@vagrant-ubuntu-trusty-64:~/go/src/github.com/graphite-ng/carbon-relay-ng#
ls /root/go/src/github.com/
BurntSushi Dieterbe elazarl gorilla graphite-ng jteeuwen kisielk
rcrowley
I verified that 5a1a5f3 is the last commit that builds for me.
[1] https://github.com/graphite-ng/carbon-relay-ng#
I hope you can help me.
just a message for all watchers that the master branch, which contains the new version, is ready for more testing! try it out and let us know what you think.
in everything-looks-good mode, still some metrics don't arrive at the destination, and sniffing traffic reveals that some seem to be duplicated. possibly related to the bufwriter, should try without that and see.
@pauloconnor that makes it easier to deploy
we should probably use https://github.com/jteeuwen/go-bindata for this
see https://www.youtube.com/watch?v=stFw0f3WEsk from 6:00 onwards, where this is explained.
There is map used for storing routes, but as it is stated here http://blog.golang.org/go-maps-in-action
"Iteration order" is not guaranteed to be the same from one iteration to the next. I wonder if it makes sense to make it more under control, especially when I want to forward only to first matching.
I understand from the README that "[carbon-relay-ng] can be restarted without dropping packets (needs testing)" but I am unclear on some things. Could you perhaps assist?
Thanks!
in relay()
when c is nil, we print the error and sleep, and then proceed with a write on the c which is still nil, causing a panic.
full output here:
(note to self: influx:2003 was down at first but than came up, but that shouldn't be related to this)
[dieter.plaetinck@dfvimeographite3 ~]$ ./carbon-relay-ng -l="0.0.0.0:2005" "=:2015" "=influx:2003"
17:06:05.408269 carbon-relay-ng.go:128: listening on 0.0.0.0:2005
17:06:05.409048 carbon-relay-ng.go:156: dial tcp 10.90.128.10:2003: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x402583]
goroutine 6 [running]:
runtime.panic(0x53c620, 0x6d7928)
/usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.write(0xc2100504e0, 0x51, 0x51, 0x0, 0x51)
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:181 +0x233
main.relay(0xc210050180, 0xc21001e5d0)
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:165 +0x1ef
created by main.main
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:115 +0x559
goroutine 1 [chan receive]:
github.com/rcrowley/goagain.Wait(0x7f95609f6440, 0xc210000130, 0x7f9560855da8, 0x401c65, 0x20)
/home/dieter/go/src/github.com/rcrowley/goagain/goagain.go:204 +0x222
github.com/rcrowley/goagain.AwaitSignals(0x7f95609f6440, 0xc210000130, 0x0, 0x0)
/home/dieter/go/src/github.com/rcrowley/goagain/legacy.go:13 +0x43
main.main()
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:143 +0x8e5
goroutine 3 [syscall]:
os/signal.loop()
/usr/lib/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
/usr/lib/go/src/pkg/os/signal/signal_unix.go:27 +0x31
goroutine 4 [chan receive]:
main.relay(0xc2100500c0, 0xc21001e480)
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:164 +0x1c3
created by main.main
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:115 +0x559
goroutine 5 [syscall]:
runtime.goexit()
/usr/lib/go/src/pkg/runtime/proc.c:1394
goroutine 7 [IO wait]:
net.runtime_pollWait(0x7f95609f61d8, 0x72, 0x0)
/usr/lib/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc210051300, 0x72, 0x7f95609f4f88, 0xb)
/usr/lib/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc210051300, 0xb, 0x7f95609f4f88)
/usr/lib/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc2100512a0, 0x5af3e8, 0x0, 0x7f95609f4f88, 0xb)
/usr/lib/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc210000130, 0xc21004a040, 0x2, 0x2)
/usr/lib/go/src/pkg/net/tcpsock_posix.go:233 +0x47
main.accept(0xc210000130, 0xc21004a040, 0x2, 0x2)
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:34 +0x27
created by main.main
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:134 +0x8c5
goroutine 9 [runnable]:
main.dispatch(0xc2100504e0, 0x51, 0x51, 0xc21004a040, 0x2, ...)
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:48 +0xfd
main.handle(0xc210000188, 0xc21004a040, 0x2, 0x2)
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:78 +0x409
created by main.accept
/home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:39 +0xf7
The code currently sends a newline after every message, even when the messages are pickled. The receiver doesn't expect this.
i had this when influxdb had an issue, the relay starts writing to spool, but it also didn't send to carbon anymore.
Hi everybody.
I'ld like to hear your thoughts on this. 1 and 2 are fairly obvious, but 3 could really use input on how you are using, or want to use the relay.
regex has much overhead. I don't think much can be done about this, so we should try to avoid regex matching when we can. I'm thinking of introducing "contains" and "starts_with" options for string checks in addition to regex, so that in many cases we could use those instead of regex and it will perform much better.
specifically, if the regex pattern is "", we shouldn't even do the regex Match() like we currently do.
I've noticed that first_only, while useful for some setups (see E below), prevents us from "also sending traffic elsewhere". so i started thinking about getting rid of first_only and making the routing/dispatching more powerful. that said, it's a neat mechanism because you can make routing decisions with just a bool check, no pattern checking needed.
so, taking a step back, i'ld like to collect use cases of how people want to use carbon-relay-ng, especially since i know some people are interested (and working on) sharding/consistent hashing, and round robin, and i've been pondering how to keep routing/dispatching performant, yet viable for various/all use cases. (ideally, without running multiple carbon-relay-ng instances if we can)
I see the following use cases
A consistent hashing / round robin to a pool of servers, for loadbalancing and HA (each pool is just one route)
B mirroring the same load to multiple machines, possibly a subset of traffic using regex pattern (for example to test influxdb next to graphite)
C sending a subset of metrics to anomaly detection
D sending all or some - of the metrics to carbon-tagger
E poor man's load balancing: sending certain metrics to server A, some others to B and maybe some others to C, using regexes to control what goes where (maybe some storage is faster then others), and using first_only to make sure we don't store metrics on more than one system.
am i missing a case? first_only is really only used for E, but it doesn't really work anymore as soon as you want to send some metrics also elsewhere like in B, C or D.
What I'm thinking is, to keep the system powerful, and still efficient, without overcomplicating things too much, we should implement E as a multi-endpoint route just like we are approaching round robin and hashing/sharding. the key is that these are multiple endpoints within one route, instead of multiple routes in the "global level"
this way, on the global routing/dispatching level, we check the pattern (and string check) of every route, and if it matches, the route gets it. but a route can be a special one with multiple endpoints,
and send traffic to those endpoints using a dedicated mechanism:
thoughts?
cc @robinbowes @rcrowley @pauloconnor @willowpet @pwielgolaski @shiaho @vladimir-smirnov-sociomantic @prune998 @curtisgithub
I'm using carbon-relay-ng with the following conf
init = [
'addBlack interface-lo',
'addBlack cpu-',
'addAgg avg ^(.*)$ 5m.$1 300 120',
'addRoute sendAllMatch carbon-default 127.0.0.1:2004 spool=true pickle=false',
]
After a few minutes I see the 5 minute metrics sent, and after that the UI breaks because the /tables endpoint hangs. I grabbed a goroutine dump and got this:
goroutine 73 [running]:
runtime/pprof.writeGoroutineStacks(0x7f147d234d90, 0xc2080ba6e0, 0x0, 0x0)
/usr/local/go/src/runtime/pprof/pprof.go:511 +0x8d
runtime/pprof.writeGoroutine(0x7f147d234d90, 0xc2080ba6e0, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/pprof/pprof.go:500 +0x4f
runtime/pprof.(*Profile).WriteTo(0xa73d80, 0x7f147d234d90, 0xc2080ba6e0, 0x2, 0x0, 0x0)
/usr/local/go/src/runtime/pprof/pprof.go:229 +0xd5
net/http/pprof.handler.ServeHTTP(0xc208ec3571, 0x9, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
/usr/local/go/src/net/http/pprof/pprof.go:169 +0x35f
net/http/pprof.Index(0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
/usr/local/go/src/net/http/pprof/pprof.go:181 +0x15e
net/http.HandlerFunc.ServeHTTP(0x921e48, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
/usr/local/go/src/net/http/server.go:1265 +0x41
net/http.(*ServeMux).ServeHTTP(0xc20803c7e0, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
/usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc2087b14a0, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
/usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2080ba140)
/usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1751 +0x35e
goroutine 1 [chan receive, 11 minutes]:
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain.Wait(0x7f147d20bf90, 0xc2080404a8, 0xc208031ce0, 0x0, 0x0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain/goagain.go:204 +0x3d4
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain.AwaitSignals(0x7f147d20bf90, 0xc2080404a8, 0x0, 0x0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain/legacy.go:13 +0x44
main.main()
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:264 +0x1593
goroutine 25 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209b92af0, 0x47, 0x47)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208040048, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 6 [syscall, 11 minutes]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35
goroutine 8 [select, 11 minutes]:
github.com/graphite-ng/carbon-relay-ng/badmetrics.(*BadMetrics).manage(0xc20803cea0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/badmetrics/badMetrics.go:65 +0x715
created by github.com/graphite-ng/carbon-relay-ng/badmetrics.New
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/badmetrics/badMetrics.go:42 +0x178
goroutine 9 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cf800, 0x3e, 0x40)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.func·010()
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:52 +0xb8
created by main.NewTable
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:54 +0x3a0
goroutine 10 [chan send, 2 minutes]:
github.com/graphite-ng/carbon-relay-ng/aggregator.(*Aggregator).Flush(0xc2080c0480, 0x5536dbd4)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/aggregator/aggregator.go:137 +0x4c3
github.com/graphite-ng/carbon-relay-ng/aggregator.(*Aggregator).run(0xc2080c0480)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/aggregator/aggregator.go:180 +0x52e
created by github.com/graphite-ng/carbon-relay-ng/aggregator.New
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/aggregator/aggregator.go:107 +0x64a
goroutine 11 [select]:
github.com/graphite-ng/carbon-relay-ng/nsqd.(*DiskQueue).ioLoop(0xc208074160)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/nsqd/diskqueue.go:566 +0x6df
created by github.com/graphite-ng/carbon-relay-ng/nsqd.NewDiskQueue
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/nsqd/diskqueue.go:87 +0x409
goroutine 12 [chan receive, 11 minutes]:
main.func·006(0xc2087b01e0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/slowchan.go:12 +0x87
created by main.NewSlowChan
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/slowchan.go:17 +0xfe
goroutine 13 [chan receive]:
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics.(*meterArbiter).tick(0xa7da80)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics/meter.go:221 +0x52
created by github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics.NewMeter
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics/meter.go:40 +0x1f7
goroutine 14 [select, 11 minutes]:
main.(*Spool).Writer(0xc208068c60)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:84 +0x9d0
created by main.NewSpool
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:68 +0x5f2
goroutine 15 [select, 11 minutes]:
main.(*Spool).Buffer(0xc208068c60)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:117 +0x271
created by main.NewSpool
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:69 +0x60f
goroutine 16 [select]:
main.(*Destination).relay(0xc208091e00)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/destination.go:258 +0x13ed
created by main.(*Destination).Run
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/destination.go:158 +0x3b5
goroutine 18 [IO wait]:
net.(*pollDesc).Wait(0xc208816300, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208816300, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2088162a0, 0x0, 0x7f147d20ad98, 0xc208dee740)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080404a8, 0xc208032df0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
main.accept(0xc2080404a8, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:75 +0x33
created by main.main
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:240 +0x14e6
goroutine 19 [IO wait, 11 minutes]:
net.(*pollDesc).Wait(0xc2088164c0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2088164c0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208816460, 0x0, 0x7f147d20ad98, 0xc208099d88)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080404b0, 0xc208020380, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc2080404b0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
github.com/graphite-ng/carbon-relay-ng/telnet.ListenAndServe(0xc208032ea0, 0xc, 0x0, 0x0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/telnet/telnet.go:49 +0xea
main.adminListener(0xc208032ea0, 0xc, 0x0, 0x0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_telnet.go:108 +0x9bf
main.func·002()
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:252 +0x3c
created by main.main
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:257 +0x1534
goroutine 20 [IO wait]:
net.(*pollDesc).Wait(0xc20882ffe0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20882ffe0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20882ff80, 0x0, 0x7f147d20ad98, 0xc208dee7b8)
/usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080405e8, 0x4ae08e, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc2080405e8, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2087b14a0, 0x7f147d20d1e8, 0xc2080405e8, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2087b14a0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc208032f40, 0xc, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:1808 +0xba
main.HttpListener(0xc208032f40, 0xc, 0xc208034410)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:211 +0xaf3
created by main.main
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:261 +0x1573
goroutine 21 [chan receive]:
main.(*keepSafe).keepClean(0xc208828690)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/keepsafe.go:29 +0x77
created by main.NewKeepSafe
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/keepsafe.go:23 +0x162
goroutine 22 [IO wait, 11 minutes]:
net.(*pollDesc).Wait(0xc2088163e0, 0x72, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2088163e0, 0x0, 0x0)
/usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208816380, 0xc208834800, 0x400, 0x400, 0x0, 0x7f147d20ad98, 0xc208826298)
/usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2080405f0, 0xc208834800, 0x400, 0x400, 0x400, 0x0, 0x0)
/usr/local/go/src/net/net.go:121 +0xdc
main.(*Conn).checkEOF(0xc2087fa000)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:110 +0x9f
created by main.NewConn
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:92 +0xcba
goroutine 23 [select]:
main.(*Conn).HandleData(0xc2087fa000)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:172 +0x1901
created by main.NewConn
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:94 +0xcd4
goroutine 24 [select]:
main.(*Conn).HandleStatus(0xc2087fa000)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:149 +0x3bd
created by main.NewConn
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:95 +0xcee
goroutine 26 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209b93680, 0x41, 0x41)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2080401d0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 27 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20dc0, 0x40, 0x40)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208040810, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 28 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20fc0, 0x40, 0x40)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208040e98, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 29 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e21040, 0x3e, 0x3e)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208fecb98, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 30 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094555e0, 0x41, 0x41)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208fed248, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 31 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209455900, 0x44, 0x44)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208fed938, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 32 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094559f0, 0x42, 0x42)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6030, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 33 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20740, 0x3e, 0x3e)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6168, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 34 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e203c0, 0x37, 0x37)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6170, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 35 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e204c0, 0x3c, 0x3c)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6468, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 36 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209455d60, 0x47, 0x47)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a65d0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 37 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209455bd0, 0x4c, 0x4c)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a65d8, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 38 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20a00, 0x38, 0x38)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6990, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 40 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfa40, 0x38, 0x38)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc20987b338, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 41 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfa80, 0x3b, 0x3b)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209918a30, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 42 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfac0, 0x37, 0x37)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2099195e8, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 43 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfd00, 0x39, 0x39)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2099bd930, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 44 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfe40, 0x3d, 0x3d)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209a57660, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 45 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094556d0, 0x42, 0x42)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209b04770, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 46 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20140, 0x37, 0x37)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209b9ab80, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 47 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e201c0, 0x40, 0x40)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c36f38, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 48 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20a80, 0x37, 0x37)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c36f20, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 49 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20c40, 0x3f, 0x3f)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c374b0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 50 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20f40, 0x38, 0x38)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209b04bd0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
goroutine 51 [chan send]:
main.(*Table).Snapshot(0xc208034410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:115 +0x3f8
main.listTable(0x7f147d234860, 0xc2080ba5a0, 0xc20801f110, 0x0, 0x0, 0xc20883dae8)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:57 +0x4d
main.handler.ServeHTTP(0x921c68, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:30 +0x55
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux.(*Router).ServeHTTP(0xc208035ef0, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux/mux.go:98 +0x297
net/http.(*ServeMux).ServeHTTP(0xc20803c7e0, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
/usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc2087b14a0, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
/usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2080ba000)
/usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1751 +0x35e
goroutine 52 [chan send, 2 minutes]:
main.(*Table).Snapshot(0xc208034410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:115 +0x3f8
main.listTable(0x7f147d234860, 0xc2080ba780, 0xc20801f2b0, 0x0, 0x0, 0xc20883fae8)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:57 +0x4d
main.handler.ServeHTTP(0x921c68, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:30 +0x55
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux.(*Router).ServeHTTP(0xc208035ef0, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux/mux.go:98 +0x297
net/http.(*ServeMux).ServeHTTP(0xc20803c7e0, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
/usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc2087b14a0, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
/usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2080ba0a0)
/usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1751 +0x35e
goroutine 71 [chan send]:
main.(*Table).Dispatch(0xc208034410, 0xc20a2865c0, 0x3c, 0x3c)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c36018, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
/Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8
routing table may have been altered via admin interface.
with a config like
addRoute sendAllMatch carbon-default host1:2005 spool=true host2:2004 spool=true
when host1 becomes slow and starts dropping packets (seen by dest ... nonBlockingSend -> dropping due to slow conn
messages and in grafana), sometimes host2 exhibits the same behavior, a drop in traffic at the same time and following the same pattern.
we must always make sure the pipeline can't hang, so i traced through the code (starting at table.Dispatch
) and made sure none of the route/destpoint/conn code can block the processing. especially of importance is the Destination.relay
loop. in there we actually have the explicit flush and shutdown handling which might take a while, but the first is seemingly never called, and the latter is definitely never called during runtime.
so i installed a profile hook by importing _ "net/http/pprof"
, loading up a relay and blasting it with input traffic to cause it to start dropping traffic, and then collected a profile:
clearly we spend too much time logging. for comparison, the other functions invoked by relay
are around 0.01s ~ 0.04s.
my current thinking is:
i've tried replacing these high-volume warning messages to info (which is appropriate, since we use it to trace what happens to individual metric messages), if we tell a loglevel below warning might cause a lot of traffic, we're good. so i ran again with warning level and tested again (this time using the /debug/vars2 because no more warn messages on screen). There's definitely a lot less time spent in logger.Info, but still some, and also significant time spent in logger.Debug. Not sure why the latter, as I changed to Info, or maybe because we now process so much traffic that the Debug stuff also starts adding up.
if you have the same address multiple times as endpoint (in the same or different routes), those are each an endpoint and conn but will use a spool file with the same name, which will conflict and cause all kinds of issues. like
2015/04/17 15:08:44 ERROR: diskqueue(spool_localhost_2005) failed to sync - rename spool/spool_localhost_2005.diskqueue.meta.dat.tmp spool/spool_localhost_2005.diskqueue.meta.dat: no such file or directory
long term, might want to use the same conn and spool for these multiple endpoints, but this would require more coordination logic. short term, we should just name them better, somehow.
When I run official code from master branch(commit dc9ebd0) with official configuration ./carbon-relay-ng carbon-relay-ng.ini
My CPU is getting crazy and consuming 100%.
I have no port open for endpoints defined in configuration, but still it should not consume 100% (it sounds like active loop).
I run it on mac.
in the loop where we read from to_unspool and route.ch we call process_packet which can take a while to return (because we write to unbuffered conn, because of udp sends, and i think because of writing to diskqueue.
this is probably a cause of #32 and #33
we should decouple this and not block any of the routing.
https://github.com/tv42/topic/blob/master/topic.go#L42-53 is an interesting idea i'm taking inspiration from.
i'm working on a fix in next
.
#21 brought a revamped admin UI as a single page app, based on a rest api exposed by the go program. neat stuff, and it got merged.
however there's a few small remarks I'ld like to addressed:
map[string]string
when I closed a route, but do not remove it from carbon-relay-ng, my other routes will not receive any data.
Hi,
I would just let you know that I have created a small (15.46 MB) Docker image for carbon-relay-ng (since I couldn't find one in the registry).
You can find it here:
https://registry.hub.docker.com/u/yasn77/docker-carbon-relay-ng/
Thanks for the great work!
Yasser
i had this happen, port was already in use by other app, carbon-relay-ng still said the http admin ui was listening. this is because we ignore the error from http.ListenAndServe
which we should of course, not do.
@pauloconnor
The README specifies that there is a version v0.5 and that the master is a work-in-progress.
However, there are no git tags/releases in the repo or changelog.
How do we use the stable version?
In golang the iteration order for map
is uncertain, even if it is certain, it will not the same as the order in configure file.
had carbon on port 2003 with route pointing to it,
restarted carbon to listen on port 2005
try to use web UI to update the route with port 2005, but it gave me the message "something something (probably the route): connection refused"
this should have worked. maybe because 2003 was now connection refused?
Hi,
I am using carbon-relay-ng and every once and a while I get a concatenated entry in my graphite database. The entry comes into carbon-relay-ng ok, but when it is forwarded to the carbon-cache server occasionally an entry becomes concatenated.
For example I get an entry like:
c01-01/:
cpuload
cputotals
ctxint
disktotals
inodeinc01-01
inodeinfo
meminfo
nettotals
pageinfo
swapinfo
tcpinfo
where the "indoeinc01-01" is wrong.
Below, 192.168.1.102 is the client. 192.168.1.101/10.0.0.1 is the carbon-relay-ng server (has multiple interfaces), and 10.0.0.6 is the carbon-cache server.
Watching with tcpflow, I see the packet come into carbon-relay-ng OK:
$ grep -a "c01-01.cpuload.avg5 0 1359480120" 192.168.001.102.39231-192.168.001.101.02003
c01-01.cpuload.avg5 0 1359480120
but when it goes out to the carbon-cache server it's not the same as when it came in:
$ grep -a inodeinc01-01 ./*
./010.000.000.001.60749-010.000.000.006.02003:c01-01.inodeinc01-01.cpuload.avg5 0 1359480120
Any thoughts on that? I'm not sure where to look next.
Thanks.
ideally we're able to expose them via expvars, and also as a carbon stream.
rcrowley/go-metrics#68 would be nice for that.
once we have the expvars, we can also show them in the web ui.
./admin_http.go:151: undefined: Asset
./admin_http.go:151: undefined: AssetDir
I've been reading configuration rules and I can understand how can't I define a group of 2 graphite-carbon servers working together in mode active/passive, something like.
clusterA = serverA_1:2003 (active) , serverA_2:2003 (passive)
clusterB = serverB_2:2003 (active), serverB_2:2003 (passive)
where I should send all metrics matching a regex "^metricsA.*" only to the first server (active) or the second one if primary fails ( passive) .
something like:
addRoute sendAllMatch regex="^metricsA.*" serverA_1:2003 serverA_2:2003
addRoute sendAllMatch regex="^metricsB.*" serverB_1:2003 serverB_2:2003
I think there is no way to indicate a active/passive balancing mode , isn't it?
Is there support for listners for plain text and pickle on relay-ng? I currently have things that send in both to an ordinary Carbon relay daemon which listens on both 2003/2004. I can see that I can send to pickle port as well so I presume that there is support for plain/pickle on the single listen address?
Fails to build (go install github.com/graphite-ng/carbon-relay-ng) with messages:
src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:205: undefined: Asset
src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:205: undefined: AssetDir
Hello,
i have this message when i try to install carbon-relay-ng
go get github.com/graphite-ng/carbon-relay-ng
/opt/golang/bin/src/github.com/graphite-ng/carbon-relay-ng/templates.go:17: undefined: Asset
Regards,
Wilfrid
Since the last merge, when I delete a route in the web-interface a press cancel, it gets deleted anyway.
Implementing a change like this is useful for protecting (non-admins) from changing configurations
I was thinking of standing up 2 carbon-relay-ng instances in a Citrix Netscaler VIP, pointing to 6 carbon daemons.
Would there be any problem with this configuration?
Currently I'm using the older, regular, carbon-relay.
When I follow the readme instructions and run make I get:
go-bindata admin_http_assets
go build
# github.com/graphite-ng/carbon-relay-ng
./carbon-relay-ng.go:110: undefined: metrics20.ValidatePacket
./conn.go:140: c.numBuffered.Dec undefined (type metrics.Gauge has no field or method Dec)
./conn.go:179: c.numBuffered.Dec undefined (type metrics.Gauge has no field or method Dec)
./destination.go:215: conn.numBuffered.Inc undefined (type metrics.Gauge has no field or method Inc)
./spool.go:93: s.numBuffered.Inc undefined (type metrics.Gauge has no field or method Inc)
./spool.go:102: s.numBuffered.Inc undefined (type metrics.Gauge has no field or method Inc)
./spool.go:121: s.numBuffered.Dec undefined (type metrics.Gauge has no field or method Dec)
make: *** [build] Error 2
Some kind of dependency issue.
Relay seems to be leaking memory. It consumes all 2GB of RAM we gave him until OOMK kicks in.
Routing rules are pretty simple:
init = [
"addBlack .net",
"addRoute sendAllMatch core prefix=org.company 12.56.176.41:2014 spool=false pickle=true 12.56.176.42:2014 spool=false pickle=true",
"addRoute sendAllMatch other prefix=stats 12.56.176.51:2014 spool=false pickle=true 12.56.176.52:2014 spool=false pickle=true"
]
18:29:49.871118 carbon-relay-ng.go:349: initializing routes...
18:29:49.878296 carbon-relay-ng.go:378: listening on 0.0.0.0:2003
18:29:49.878684 routing.go:98: influxdb (re)connecting to 127.0.0.1:2004
18:29:49.878777 routing.go:98: carbon-default (re)connecting to out:2005
18:29:49.879632 routing.go:113: influxdb connected
18:29:49.879859 routing.go:113: carbon-default connected
if we have output like the above, we might as well mention http and tcp admin endpoints
When building latest version of carbon-relay-ng like mentioned in the readme I get this error when running ./make.sh
:
routing/routing.go:153: undefined: parseDataPoint
I tried building on CentOS 6.5 and MacOSX 10.9.5
following your instructions there is an error when compiling , it seems like path is not correct or can not find go-bindata
root@nono:~/go/src/github.com/graphite-ng/carbon-relay-ng# make
go-bindata admin_http_assets
make: go-bindata: No se encontró el programa
make: *** [build] Error 127
How can I fix this problem?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.