fvbock / endless Goto Github PK
View Code? Open in Web Editor NEWZero downtime restarts for go servers (Drop in replacement for http.ListenAndServe)
License: MIT License
Zero downtime restarts for go servers (Drop in replacement for http.ListenAndServe)
License: MIT License
I tried to get the app to send a SIGHUP signal to reload itself and got a "text file busy" error
2021/11/17 17:14:30 8597 Received SIGHUP. forking.
2021/11/17 17:14:30 Restart: Failed to launch, error: fork/exec ./project: text file busy
but I manually executed "kill -1" to send a SIGHUP signal and it worked fine, reload successful
2021/11/17 17:53:28 11425 Received SIGHUP. forking.
2021/11/17 17:53:28 11475 0.0.0.0:8001
2021/11/17 17:53:28 11425 Received SIGTERM.
2021/11/17 17:53:28 11425 Waiting for connections to finish...
2021/11/17 17:53:28 11425 Serve() returning...
Hi,
When I use endless, I found the following code cause the memory leak after restart.
func main() {
fd, _ := os.Create("access.log")
defer fd.Close()
// start http server
...
}
After restarting, use lsof to list the open FDs, there are 2 FDs for access.log
Am I using it incorrectly?
Thanks!
Kevin
https://github.com/fvbock/endless/blob/master/examples/multi_port.go
it sleeps 1 second before endless.ListenAndServe
in the first goroutine. why ?
I use systemd to manage go app.
config:
ExecReload=/bin/kill -s HUP $MAINPID
Because of pid change, systemd think process failed, and use ExecStart to start process.But process was started by endless, systemd think process start failed.
How can I use endless with systemd?
Hey,
First of all thank you for this really helpful library! :-)
After having spent multiple hours testing this lib and reasoning about your code whether it is race conditions free and whether it is robust enough to survive in a high profile scenario I decided to give it a try. Since then I use endless in a mission critical production app.
All in all I'm very happy with the result. But there are situations where my Go server sporadically crashes due to a panic (sync: negative WaitGroup counter) in endless with the following trace:
goroutine 3849 [running]:
runtime.panic(0x764000, 0xc208404de0)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/panic.c:279 +0xf5
sync.(*WaitGroup).Add(0xc208042080, 0xffffffffffffffff)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/sync/waitgroup.go:64 +0x93
sync.(*WaitGroup).Done(0xc208042080)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/sync/waitgroup.go:82 +0x30
bitbucket.org/justphil/glutenplan.de/vendor/endless.endlessConn.Close(0x7f0997bd9ce0, 0xc20803e018, 0xc208042000, 0x0, 0x0)
/Users/pt/Dev/Workspaces/go/src/bitbucket.org/justphil/glutenplan.de/vendor/endless/endless.go:508 +0x48
bitbucket.org/justphil/glutenplan.de/vendor/endless.(*endlessConn).Close(0xc2083d8360, 0x0, 0x0)
<autogenerated>:16 +0xa4
net/http.(*conn).close(0xc20836e000)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/net/http/server.go:1047 +0x4f
net/http.func·011()
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/net/http/server.go:1104 +0x22a
net/http.(*conn).serve(0xc20836e000)
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/net/http/server.go:1187 +0x78b
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.3.3/libexec/src/pkg/net/http/server.go:1721 +0x313
(Note: I forked your lib and extended it with the ability to write pid files. But the same problem appears in the original lib as well.)
So, I again started to reason about the endless code in order to get rid of this problem. There are basically two possibilities how the WaitGroup counter can become negative. The first one is when hammerTime() tries to forcefully shut down the parent process. And the second one is the one that is panicking when a connection is closed. To me this is really weird because according to your code this panicking implies that endless wants to close connections that have neven been established.
Any hints on this one? :-)
Bye,
Phil
i found endless
uses syscall.SIGUSR1
,syscall.SIGUSR2
,syscall.SIGTSTP
,syscall.Kill
, these things are not defined on Go 1.6.2 on windows. so codes using endless
can not even be compiled.
endless: 2cdc20a
would you please fix this?
Hi, how to create a systemd init script for application using endless package ?
Thank you
I want to know how to use this with websocket long connection,no connection will finish~
This error is printed whenever a SIGHUP is sent to an endless server. The server continues to run but this error is causing my program which calls endless to panic. Where is the error coming from?
server.SignalHooks[endless.PRE_SIGNAL][syscall.SIGUSR1],
a.Logger.Rotate)
func Rotate(conf *toml.TomlTree) {
.....
}
How i do?
how to use with r.Run(":8080") and r.RunTLS(":8443", "server.pem", "server.key")
I was wondering about the comment in line 529:
"// returns a dup(2) - FD_CLOEXEC flag not set"
Is that strictly correct? If I call syscall.Syscall to GETFD , the value is 1
Isn't what happens that .File() actually returns a os.File with an FD with FD_CLOEXEC set (using dupCloseOnExec() in net/fd_unix.go )
And then... when you call
cmd := exec.Command(path, args...)
err = cmd.Start()
... the Go stdlib clears CLOEXEC on the files you put in cmd.ExtraFiles as a part of forkAndExecInChild() when it shuffles file descriptors around ?
2020/11/24 15:28:49 net.FileListener error: file file+net : getsockopt: socket operation on non-socket
eg ./simple_server -test
after forking would be
./simple_server -continue
- dropping the original args
This library only provides public API for http.Handler
s and not net.Listener
s :-(
endlessListener
is private for some reason.
Please make the TCP-level API available!
I found this error in my main code , relevant log is :
2015/09/07 12:37:01 [STOP - Hammer Time] Forcefully shutting down parent
2015/09/07 12:37:01 WaitGroup at 0 sync: negative WaitGroup counter
2015/09/07 12:37:01 22203 Serve() returning...
panic: sync: WaitGroup is reused before previous Wait has returned
goroutine 1 [running]:
sync.(*WaitGroup).Wait(0xc82020aee0)
/usr/local/go/src/sync/waitgroup.go:128 +0x114
github.com/fvbock/endless.(*endlessServer).Serve(0xc82020ae60, 0x7f9fbe8b1500, 0xc82059f130)
/home/azmoona/app_build/vendor/src/github.com/fvbock/endless/endless.go:166 +0x32c
github.com/fvbock/endless.(*endlessServer).ListenAndServe(0xc82020ae60, 0x0, 0x0)
/home/azmoona/app_build/vendor/src/github.com/fvbock/endless/endless.go:197 +0x42a
github.com/fvbock/endless.ListenAndServe(0xc82022a900, 0xe, 0x7f9fbe8b3740, 0xc820242300, 0x0, 0x0)
/home/azmoona/app_build/vendor/src/github.com/fvbock/endless/endless.go:136 +0x59
main.main()
/home/azmoona/app_build/src/webserver/azmoona.go:58 +0x605
build the simple example with -race
flag, and there is always a race.
./simple
2015/09/07 13:10:03 10751 localhost:4242
2015/09/07 13:10:15 10751 Received SIGHUP. forking.
2015/09/07 13:10:15 10751 Received SIGTERM.
==================
WARNING: DATA RACE
Read by goroutine 7:
github.com/fvbock/endless.(*endlessServer).shutdown()
/home/f0rud/gospace/src/github.com/fvbock/endless/endless.go:342 +0x57
github.com/fvbock/endless.(*endlessServer).handleSignals()
/home/f0rud/gospace/src/github.com/fvbock/endless/endless.go:316 +0x1272
Previous write by main goroutine:
github.com/fvbock/endless.(*endlessServer).Serve()
/home/f0rud/gospace/src/github.com/fvbock/endless/endless.go:163 +0x1f1
github.com/fvbock/endless.(*endlessServer).ListenAndServe()
/home/f0rud/gospace/src/github.com/fvbock/endless/endless.go:197 +0x526
github.com/fvbock/endless.ListenAndServe()
/home/f0rud/gospace/src/github.com/fvbock/endless/endless.go:136 +0x6a
main.main()
/home/f0rud/gospace/src/github.com/fvbock/endless/examples/simple.go:21 +0x220
Goroutine 7 (running) created at:
github.com/fvbock/endless.(*endlessServer).ListenAndServe()
/home/f0rud/gospace/src/github.com/fvbock/endless/endless.go:182 +0xbb
github.com/fvbock/endless.ListenAndServe()
/home/f0rud/gospace/src/github.com/fvbock/endless/endless.go:136 +0x6a
main.main()
/home/f0rud/gospace/src/github.com/fvbock/endless/examples/simple.go:21 +0x220
==================
2015/09/07 13:10:15 10751 127.0.0.1:4242 Listener closed.
2015/09/07 13:10:15 10751 Waiting for connections to finish...
2015/09/07 13:10:15 10751 Serve() returning...
2015/09/07 13:10:15 accept tcp 127.0.0.1:4242: use of closed network connection
2015/09/07 13:10:15 Server on 4242 stopped
2015/09/07 13:10:15 10930 localhost:4242
Found 1 data race(s)
env:
Linux l-qfy0.dba.dev.cn0 3.10.0-327.28.3.el7.x86_64 #1 SMP Thu Aug 18 19:05:49 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
go version go1.17 linux/amd64
test.go
import (
"context"
"flag"
"fmt"
"os/exec"
"strconv"
"syscall"
"github.com/fvbock/endless"
"github.com/gin-gonic/gin"
)
var ctx context.Context
var cancel context.CancelFunc
var reload bool
var port = flag.Int("port", 8080, "")
func main() {
flag.Parse()
ListenHTTP(setReload)
}
func setReload() {
reload = true
}
func ListenHTTP(reloadFunc func()) error {
var handler = gin.Default()
handler.GET("/", startDaemon())
handler.GET("/hello", hello())
ctx, cancel = context.WithCancel(context.Background())
defer cancel()
s := endless.NewServer(":"+strconv.Itoa(*port), handler)
go func() {
select {
case <-ctx.Done():
}
if s != nil {
if err := s.Shutdown(ctx); err != nil {
fmt.Println(err)
}
}
}()
s.SignalHooks[endless.PRE_SIGNAL][syscall.SIGHUP] = append(
s.SignalHooks[endless.PRE_SIGNAL][syscall.SIGHUP],
reloadFunc)
if err := s.ListenAndServe(); err != nil {
return err
}
return nil
}
func hello() gin.HandlerFunc {
return func(c *gin.Context) {
fmt.Println("hello ", *port)
}
}
func startDaemon() gin.HandlerFunc {
return func(c *gin.Context) {
start(c)
}
}
func start(c *gin.Context) (err error) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
cmd := exec.CommandContext(ctx, "sh", []string{"start.sh"}...)
err = cmd.Run()
if err != nil {
fmt.Println(err)
return
}
return
}
start.sh
nohup /home/qfy/tmp/test --port=8081 > /dev/null 2>&1 &
basedir:/root/tmp
steps:
4.then send http requst,but 8081 is not listening
Is that a bug need to fix ?
And how can I resolve it.
Expected behavior: endless to append whatever runtime args it needs, but preserve the application command line params.
Actual behavior: endless takes over the command line params, and treats all application params as illegal
What's most curious is that args required by the main app are still considered required, even though providing them is disallowed by endless.
1074)internationalsos/riskratings % ./rr -h
Usage of ./rr:
-continue=false: Dummy -- needed by endless
-debug=false: Enable debugging
-pass="": Password
-port=":8080": Listen port
-refresh=10: Cache refresh time, in minutes
-uri="******************": URI of data source
-user="": Username
1075)internationalsos/riskratings % ./rr
Password is required
Usage of ./rr:
-continue=false: Dummy -- needed by endless
-debug=false: Enable debugging
-pass="": Password
-port=":8080": Listen port
-refresh=10: Cache refresh time, in minutes
-uri="******************": URI of data source
-user="": Username
1076)internationalsos/riskratings % ./rr -pass xyz -user abc -port :8123
2015/06/29 22:33:58 start listening on localhost:8123
flag provided but not defined: -pass
Usage of ./rr:
-continue=false: listen on open fd (after forking)
-socketorder="": previous initialization order - used when more than one listener was started
when starting more than one listener (most likely through goroutines) the initialization order is not determined. on restart this can result in the wrong socket being assigned to a server.
solution: pass original initialization order (by address) to the child.
this only affects users that start more than one server in one binary.
Solution shared by one of our devs
https://developpaper.com/using-endless-in-windows-undefined-syscall-sigusr1/
Inside: C:\Program Files\Go\src\syscall\types_windows.go
var signals = [...]string{
//Omit line n here....
/**Compatible with Windows start*/
16: "SIGUSR1",
17: "SIGUSR2",
18: "SIGTSTP",
/**Compatible with windows end*/
}
/**Compatible with Windows start*/
func Kill(...interface{}) {
return;
}
const (
SIGUSR1 = Signal(0x10)
SIGUSR2 = Signal(0x11)
SIGTSTP = Signal(0x12)
)
/**Compatible with windows end*/
Originally posted by @kevincobain2000 in #35 (comment)
For example on go1.5.1, the C:\Go\src\syscall\types_windows.go
file is missing, but
After you edit the C:\Go\src\syscall\ztypes_windows.go
file you receive 10 warnings:
dll_windows.go
1 warning in func (p *Proc) Call(...) (...) {...}
,
exec_windows.go
1 warning in func joinExeDirAndFName(...) (...) {...}
,
zsyscall_windows.go
8 warnings possible misuse of unsafe.Pointer
.
If it is enough to comment out the return keyword in the first two files, then it is not clear how to fix the last one.
Ideas?
https://cs.opensource.google/go/go/+/refs/tags/go1.5.1:src/syscall/
https://cs.opensource.google/go/go/+/refs/tags/go1.10:src/syscall/
go get -u github.com/fvbock/endless
..\github.com\fvbock\endless\endless.go:99: undefined: syscall.SIGUSR1
..\github.com\fvbock\endless\endless.go:100: undefined: syscall.SIGUSR2
..\github.com\fvbock\endless\endless.go:103: undefined: syscall.SIGTSTP
..\github.com\fvbock\endless\endless.go:107: undefined: syscall.SIGUSR1
..\github.com\fvbock\endless\endless.go:108: undefined: syscall.SIGUSR2
..\github.com\fvbock\endless\endless.go:111: undefined: syscall.SIGTSTP
..\github.com\fvbock\endless\endless.go:193: undefined: syscall.Kill
..\github.com\fvbock\endless\endless.go:243: undefined: syscall.Kill
..\github.com\fvbock\endless\endless.go:288: undefined: syscall.SIGUSR1
..\github.com\fvbock\endless\endless.go:289: undefined: syscall.SIGUSR2
..\github.com\fvbock\endless\endless.go:289: too many errors
func (c *Cmd) Run() error {
if err := c.Start(); err != nil {
return err
}
return c.Wait() // if execute the command fails,return error
}
I use endless in my project. In a production environment, we need to modify the configuration file frequently, and then restart the program for the new configuration to take effect.
But the configuration file often has syntax errors, making the program unable to restart successfully. We need to modify the configuration again and restart again, but endless output "Fork err: Another process already forked. Ignoring this one."
Hi Folks,
Is there a way to configure a new logger? I am thinking of using logrus to structure the logging in json format for a project, so if you have any suggestion, please recommend.
-Kartlee
if your work use this file, when you go build to make a exe, there will be
undefined: syscall.SIGUSR1
undefined: syscall.SIGUSR2
undefined: syscall.SIGTSTP
undefined: syscall.SIGUSR1
undefined: syscall.SIGUSR2
undefined: syscall.SIGTSTP
undefined: syscall.SIGUSR1
undefined: syscall.SIGUSR2
undefined: syscall.SIGTSTP
undefined: syscall.Kill
too many errors
Because these constants are not available in Windows,
For normal operation, you can modify the file in GOROOT \ src \ syscall \ types_windows.go
Anyway, these constants are useless under windows
var signals = [...]string{ //It already exists, usually in line 67 of the document
// Omit some rows
/** change start */
16: "SIGUSR1",
17: "SIGUSR2",
18: "SIGTSTP",
/** change end */
}
/** change start */
func Kill(...interface{}) { //It doesn't exist, you can add it after last change
return;
}
const (
SIGUSR1 = Signal(0x10)
SIGUSR2 = Signal(0x11)
SIGTSTP = Signal(0x12)
)
/** change end */
Method from: https://learnku.com/articles/51696
I want to use LimitListener from "golang.org/x/net/netutil" for the Serve() function, but there is no way to do that. Normally, the Serve() function has a listener param.
...
l, err := net.Listen("tcp", addr)
if err != nil {
return
}
l = netutil.LimitListener(TcpKeepAliveListener(l), MAX_CONN_NUM)
server := &http.Server{Addr: addr, Handler: handler}
err = server.Serve(l)
Maybe we can use like this:
...
l, err := net.Listen("tcp", addr)
if err != nil {
return
}
l = netutil.LimitListener(TcpKeepAliveListener(l), MAX_CONN_NUM)
//server := &http.Server{Addr: addr, Handler: handler}
server := endless.NewServer(addr, handler)
err = server.Serve(l)
Sorry for my broken English.
This repo hasn't been updated since 2016. Is it officially abandoned? If so, can you mark it as archive, or put out a call for maintainers as a new issue and in the README?
Support Http2.0
The standard flag library is not a part of Go that everyone love and use. Personally I (and many other) like another flag library.
The problem is, with this library, we should use the standard flag library (LAME IMHO), or else, this won't work.
I think, using the flag library is not required at all, since the parent process, can simply change env for the child library.
And the other problem is, I want to change it to system fork (not execute with os.exec) and in that case I can not use the flag library, since the child is exactly like parent and there is no way to change child Arguments.
Hi,
This is a feature request to make the network="tcp"
parameter used here configurable. In our case we would like to use unix
as the value here.
Some Background:
I'm using endless in a couple of services and liking it very much 😄
But we are now working on a lite, offline version of our service stack, in which clients connect to a service on the same machine. Here we don't actually need TCP sockets for communication, but can instead use unix
sockets directly. Unfortunately I couldn't figure out a way to do this while still using endless.
The Go stdlib supports this out of the box, the only change required was:
listener, err := net.Listen("tcp", "localhost:8080")
to
listener, err := net.Listen("unix", "some-unix-socket.sock")
However, with endless, that part seems to be not configurable.
There seems to be same additional logic for listening on a file already, but only if the ENDLESS_CONTINUE
environment variable is set, which then sets isChild
to true. I didn't want to mess around in that direction to much.
Is there any chance something like this could be added to endless?
This signal prevents Unix shells from sending the process to the background via CTRL-Z
It would be a good idea if SIGTSTP is not intercepted by default.
Hey, thanks for this cool package!
Is this anyhow combinable with supervisor? I'm using it for regular restarts, log rotations, and so on. Only downside while deploying is that with supervisor I don't have graceful restarts...
Any experiences with such a stack?
On restart, the program forks a new copy and closes the old copy, leaving the new one parentless/daemonized. How would you use this with init, upstart, systemd and others? If you can't, what do you do about crashes - restarts and reporting, and also logging?
../github.com/fvbock/endless/endless.go:380: srv.SetKeepAlivesEnabled undefined (type *endlessServer has no field or method SetKeepAlivesEnabled)
When I used go get to get this.
PS C:\Users\xxx\Workspace\xxx-xxx> go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\xxx\AppData\Local\go-build
set GOENV=C:\Users\xxx\AppData\Roaming\go\env
set GOEXE=.exe
set GOEXPERIMENT=
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=C:\Users\xxx\go\pkg\mod
set GONOPROXY=.corp.example.com
set GONOSUMDB=.corp.example.com
set GOOS=windows
set GOPATH=C:\Users\xxx\go
set GOPRIVATE=*.corp.example.com
set GOPROXY=https://proxy.golang.com.cn,direct
set GOROOT=C:\Program Files\Go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.18.1
set GCCGO=gccgo
set GOAMD64=v1
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\Users\xxx\Workspace\xxx-xxx\go.mod
set GOWORK=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\xxx\AppData\Local\Temp\go-build219240513=/tmp/go-build -gno-record-gcc-switches
a process is running , then i modified some code ,then, i try to build it again. and it shows the error msg above.
i know the reason, but, i mean , how to use the endless with the right way.
I want to make a struct like this:
type App struct {
Server *endless.endlessServer
Router *gin.Engine
}
func (a *App) Initialize(config *config.Config) {
a.Server = endless.NewServer(fmt.Sprintf(":%d", a.Config.Server.Port), a.Router)
}
Using this struct
: https://github.com/fvbock/endless/blob/master/endless.go#L72
But it's give me error:
cannot refer to unexported name *endless.endlessServer
How can i that endless
reference in my own App
struct?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.