cybozu-go / well Goto Github PK

View Code? Open in Web Editor NEW

125.0 22.0 14.0 203 KB

Go framework for well-behaving commands

Home Page: https://github.com/cybozu-go/well/wiki/Tutorial

License: MIT License

Go 100.00%

well's Introduction

Go Command Framework

Deprecated: This project has come to an end and will not receive any update.

This is a framework to create well-behaving commands.

Features

Context-based goroutine management.
Signal handlers.
Graceful stop/restart for any kind of network servers.
Logging options.
Enhanced http.Server.
Ultra fast UUID-like ID generator.
Activity tracking.
Support for systemd socket activation.
Support for github.com/spf13/cobra.

Requirements

Go 1.11 or later, though the code is expected to work with Go 1.7.

Specifications

Commands using this framework implement these external specifications:

Command-line options

-logfile FILE

Output logs to FILE instead of standard error.
-loglevel LEVEL

Change logging threshold to LEVEL. Default is info.
LEVEL is one of critical, error, warning, info, or debug.
-logformat FORMAT

Change log formatter. Default is plain.
FORMAT is one of plain, logfmt, or json.

Signal Handlers

SIGUSR1

If -logfile is specified, this signal make the program reopen the log file to cooperate with an external log rotation program.

On Windows, this is not implemented.
SIGINT and SIGTERM

These signals cancel the context of the global environment, and hence goroutines registered with the environment. Usually this will result in graceful stop of network servers, if any.

On Windows, only SIGINT is handled.
SIGHUP

This signal is used to restart network servers gracefully. Internally, the main (master) process restarts its child process. The PID of the master process thus will not change.

There is one limitation: the location of log file cannot be changed by graceful restart. To change log file location, the server need to be (gracefully) stopped and started.

On Windows, this is not implemented.
SIGPIPE

The framework changes the way Go handles SIGPIPE slightly. If a program using this framework receives SIGPIPE when writing to stdout or stderr, the program exits with status code 2. See #15 for details.

Environment variables

REQUEST_ID_HEADER

The value of this variable is used as HTTP header name. The HTTP header is used to track activities across services. The default header name is "X-Cybozu-Request-ID".
CYBOZU_LISTEN_FDS

This is used internally for graceful restart.
CANCELLATION_DELAY_SECONDS

After SIGINT or SIGTERM received, the signal handler waits for the seconds before cancelling the context. The default value is 5 sec.

Usage

Read Tutorial, the design notes and API documents.

A wiki page for cobra users is also available.

Real world examples

Pull requests are welcome to add your project to this list!

well's People

Contributors

Stargazers

Watchers

Forkers

mstoykov mattn joonhochoi windbg zoetrope justforkin satoru-takeuchi m4i stanxii jjakob doytsujin itchyny luis-pinto-fanduel

well's Issues

HTTPS server fail to handle request

HTTPS server cannot handle http request with well with cods:

serv := &well.HTTPServer{
		Server: &http.Server{
			Addr:    config.listenAddr,
			Handler: h,
		},
	}

err = serv.ListenAndServeTLS("server.crt", "server.key")

Error is as below:

2019-05-21T02:43:15.273506Z kaz-ubuntu topolvm-hook info: "http2: panic serving 127.0.0.1:58100: interface conversion: *http.http2responseWriter is not well.StdResponseWriter: missing method Hijack"
2019-05-21T02:43:15.275443Z kaz-ubuntu topolvm-hook info: "goroutine 9 [running]:"
2019-05-21T02:43:15.275449Z kaz-ubuntu topolvm-hook info: "net/http.(*http2serverConn).runHandler.func1(0xc00000e5b8, 0xc0000b4faf, 0xc0002d4a80)"
2019-05-21T02:43:15.275452Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/h2_bundle.go:5651 +0x16b"
2019-05-21T02:43:15.275454Z kaz-ubuntu topolvm-hook info: "panic(0x1152ba0, 0xc0002bc450)"
2019-05-21T02:43:15.275456Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/runtime/panic.go:522 +0x1b5"
2019-05-21T02:43:15.275460Z kaz-ubuntu topolvm-hook info: "github.com/cybozu-go/well.(*HTTPServer).ServeHTTP(0xc00036c000, 0x144da40, 0xc00000e5b8, 0xc0001a4700)"
2019-05-21T02:43:15.275463Z kaz-ubuntu topolvm-hook info: "\t/home/kazuhito/go/src/github.com/cybozu-go/topolvm/vendor/github.com/cybozu-go/well/http.go:119 +0x7e"
2019-05-21T02:43:15.275466Z kaz-ubuntu topolvm-hook info: "net/http.serverHandler.ServeHTTP(0xc000362410, 0x144da40, 0xc00000e5b8, 0xc0001a4700)"
2019-05-21T02:43:15.275468Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/server.go:2774 +0xa8"
2019-05-21T02:43:15.275471Z kaz-ubuntu topolvm-hook info: "net/http.initNPNRequest.ServeHTTP(0xc000086a80, 0xc000362410, 0x144da40, 0xc00000e5b8, 0xc0001a4700)"
2019-05-21T02:43:15.275474Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/server.go:3323 +0x8d"
2019-05-21T02:43:15.275476Z kaz-ubuntu topolvm-hook info: "net/http.(*http2serverConn).runHandler(0xc0002d4a80, 0xc00000e5b8, 0xc0001a4700, 0xc0001f8740)"
2019-05-21T02:43:15.275479Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/h2_bundle.go:5658 +0x89"
2019-05-21T02:43:15.275481Z kaz-ubuntu topolvm-hook info: "created by net/http.(*http2serverConn).processHeaders"
2019-05-21T02:43:15.275483Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/h2_bundle.go:5392 +0x4f4"
2019-05-21T02:43:25.942018Z kaz-ubuntu topolvm-hook info: "http2: panic serving 127.0.0.1:58218: interface conversion: *http.http2responseWriter is not well.StdResponseWriter: missing method Hijack"
2019-05-21T02:43:25.942081Z kaz-ubuntu topolvm-hook info: "goroutine 25 [running]:"
2019-05-21T02:43:25.942084Z kaz-ubuntu topolvm-hook info: "net/http.(*http2serverConn).runHandler.func1(0xc0002f8000, 0xc00034dfaf, 0xc000231380)"
2019-05-21T02:43:25.942087Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/h2_bundle.go:5651 +0x16b"
2019-05-21T02:43:25.942090Z kaz-ubuntu topolvm-hook info: "panic(0x1152ba0, 0xc0001620c0)"
2019-05-21T02:43:25.942092Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/runtime/panic.go:522 +0x1b5"
2019-05-21T02:43:25.942094Z kaz-ubuntu topolvm-hook info: "github.com/cybozu-go/well.(*HTTPServer).ServeHTTP(0xc00036c000, 0x144da40, 0xc0002f8000, 0xc0001a4600)"
2019-05-21T02:43:25.942097Z kaz-ubuntu topolvm-hook info: "\t/home/kazuhito/go/src/github.com/cybozu-go/topolvm/vendor/github.com/cybozu-go/well/http.go:119 +0x7e"
2019-05-21T02:43:25.942100Z kaz-ubuntu topolvm-hook info: "net/http.serverHandler.ServeHTTP(0xc000362410, 0x144da40, 0xc0002f8000, 0xc0001a4600)"
2019-05-21T02:43:25.942102Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/server.go:2774 +0xa8"
2019-05-21T02:43:25.942105Z kaz-ubuntu topolvm-hook info: "net/http.initNPNRequest.ServeHTTP(0xc000160380, 0xc000362410, 0x144da40, 0xc0002f8000, 0xc0001a4600)"
2019-05-21T02:43:25.942107Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/server.go:3323 +0x8d"
2019-05-21T02:43:25.942109Z kaz-ubuntu topolvm-hook info: "net/http.(*http2serverConn).runHandler(0xc000231380, 0xc0002f8000, 0xc0001a4600, 0xc00030e260)"
2019-05-21T02:43:25.942112Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/h2_bundle.go:5658 +0x89"
2019-05-21T02:43:25.942114Z kaz-ubuntu topolvm-hook info: "created by net/http.(*http2serverConn).processHeaders"
2019-05-21T02:43:25.942117Z kaz-ubuntu topolvm-hook info: "\t/usr/local/go/src/net/http/h2_bundle.go:5392 +0x4f4"
2019-05-21T02:43:34.493591Z kaz-ubuntu topolvm-hook warning: "well: got signal" signal="terminated"

Do not specify NextProtos for ListenAndServeTLS

coreos/go-systemd#191

dummy for add a new label and all Neco related repos

Data race

go install -race ./...
$GOHOME/bin/restart

WARNING: DATA RACE
Read at 0x00c420174680 by goroutine 14:
  os.(*file).close()
      /usr/local/go/src/os/file_unix.go:136 +0xb0
  os.(*File).Close()
      /usr/local/go/src/os/file_unix.go:132 +0x55
  os/exec.(*Cmd).closeDescriptors()
      /usr/local/go/src/os/exec/exec.go:262 +0x67
  os/exec.(*Cmd).Wait()
      /usr/local/go/src/os/exec/exec.go:447 +0x2bd
  github.com/cybozu-go/cmd.(*Graceful).runMaster.func2()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/graceful_unix.go:170 +0x38

Previous write at 0x00c420174680 by goroutine 13:
  os.(*file).close()
      /usr/local/go/src/os/file_unix.go:143 +0x124
  os.(*File).Close()
      /usr/local/go/src/os/file_unix.go:132 +0x55
  github.com/cybozu-go/cmd.copyLog.func1()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/graceful_unix.go:209 +0x3e
  github.com/cybozu-go/cmd.copyLog()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/graceful_unix.go:223 +0x471

Goroutine 14 (running) created at:
  github.com/cybozu-go/cmd.(*Graceful).runMaster()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/graceful_unix.go:171 +0x3c3
  github.com/cybozu-go/cmd.(*Graceful).(github.com/cybozu-go/cmd.runMaster)-fm()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/graceful_unix.go:106 +0x55
  github.com/cybozu-go/cmd.(*Environment).Go.func1()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/env.go:132 +0xa4

Goroutine 13 (finished) created at:
  github.com/cybozu-go/cmd.(*Graceful).runMaster()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/graceful_unix.go:162 +0x347
  github.com/cybozu-go/cmd.(*Graceful).(github.com/cybozu-go/cmd.runMaster)-fm()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/graceful_unix.go:106 +0x55
  github.com/cybozu-go/cmd.(*Environment).Go.func1()
      /home/ymmt/go/src/github.com/cybozu-go/cmd/env.go:132 +0xa4
==================
Found 1 data race(s)

[Go 1.8] [HTTPServer] Use standard methods for graceful shutdown

In Go 1.8, net/http has new methods to force or graceful shutdown of servers.
Use them for Go 1.8+.

https://go-review.googlesource.com/#/c/32329/2/src/net/http/server.go

Does not cancel context on termination signal

here you go.. try to terminate this with ctrl+c

package main

import (
	"context"
	"github.com/cybozu-go/well"
	"time"
)

func main() {
	env := well.NewEnvironment(context.Background())
	env.Go(func(ctx context.Context) error {
		for {
			select {
			case <-ctx.Done():
				return nil
			default:
			}
			time.Sleep(10 * time.Second)
		}
	})

	env.Stop()
	env.Wait()
}

SIGPIPE from journald prevents restart

Go does not ignore SIGPIPE for fd 1 & 2 (stdout & stderr).
This can be a problem when programs written in Go run under systemd.

Quote from https://golang.org/pkg/os/signal/

If the program has not called Notify to receive SIGPIPE signals, then the behavior depends on the file descriptor number. A write to a broken pipe on file descriptors 1 or 2 (standard output or standard error) will cause the program to exit with a SIGPIPE signal. A write to a broken pipe on some other file descriptor will take no action on the SIGPIPE signal, and the write will fail with an EPIPE error.

A typical systemd service configuration looks like:

[Service]
Type=simple
ExecStart=/path/to/go/program
Restart=on-failure

Services configured like this have its stdout and stderr connected with systemd-journald.
Journald is infamous for raising SIGPIPE when it restarts. If this happens, Go programs that do not install SIGPIPE handlers will die because of the aforementioned specification.

Worse, systemd does not restart the service when it dies for SIGPIPE despite Restart=on-failure.
This is because the default value of SuccessExitStatus includes SIGPIPE and therefore systemd considers that the program exited normally.

http: remote IP is never logged

well/http.go

Lines 142 to 145 in 2b24859

 ip, _, err := net.SplitHostPort(r.RemoteAddr) 

 if err != nil { 

 fields[log.FnRemoteAddress] = ip 

 }

Shouldn't this be if err == nil? Otherwise remote IP will never be logged.

Graceful restart

In #3, I studied that systemd socket activation is not perfect to implement graceful restart.

Instead, we should implement it by ourselves purely in Go.
As the framework already provides graceful stop mechanism, this is as easy as:

The master process creates listeners, and pass them to a child process.
The child process uses listeners to accept connections until it gets SIGINT/SIGTERM.
When SIGHUP is sent to the master, it sends SIGTERM to the child, sleeps a while, and starts another child.
The child stops accepting new connections immediately, waits for active connections to close, then exits.
To write logs safely, the master process need to work as a log server for children.
- The master creates a pipe and set its write end to the children's stderr.
- Children ignore log file configurations to send logs via stderr.
- The master reads lines from the pipe and forwards them to the log file.

When journald restarts, program should exit abnormally.

In #13, we just ignored SIGPIPE upon journald restarts.

As a consequence, programs using this framework will not fail
on journald restart but now they may stack when writing logs to
stdout/stderr pipes.

I don't know the exact reason but as systemd & journald is so flaky,
and SIGPIPE is treated as normal exit condition by systemd, we need
to catch SIGPIPE on stdout/stderr and exit abnormally, say, with
exit condition 2.

Fix SIGPIPE handling, take 3

In #13, we just ignored SIGPIPE from stdout and stderr.
The result was process hangup after journald crashed and restarted.
Ignoring SIGPIPE and ignoring EPIPE from stdout and stderr connected to journald
will eventually blocks the process if we continue to write to stdout or stderr.

In #15, we handled SIGPIPE and exit with status code 2.
This worked, but we started to get SIGPIPEs from file descriptors other than stdout
and stderr, causing the programs using this framework crash unexpectedly often
due to broken network connections.

We rethink how to resolve these and those issues related to Go and systemd-journald.

Ignore SIGPIPE completely just as in #13, but only when the program runs as a systemd service.
The default behavior of Go that raises SIGPIPE only for stdout and stderr is reasonable if the program runs outside of systemd.
Handle EPIPE and exit with status code 5 in the logging package.
We should ignore SIGPIPE for systemd services but should not ignore EPIPE from stdout and stderr.

Support for systemd.socket activation

The framework already provides support for graceful shutdown.

With this feature, our Go programs can be gracefully restarted with the help of systemd.
https://www.freedesktop.org/software/systemd/man/systemd.socket.html

	ip, _, err := net.SplitHostPort(r.RemoteAddr)
	if err != nil {
	fields[log.FnRemoteAddress] = ip
	}