Non-command-line configuration about goworker HOT 13 CLOSED

https://github.com/benmanns/goworker/pull/38/files

from goworker.

I vote this is needed. It would make goworker much more usable to have configuration outside of command line flags.

from goworker.

+1 for this. My app has quite a few (non-goworker-related) configs per-environment that I like to keep in external config files...having to also manage per-environment upstart scripts to manage my Redis host URI's is a bit of a hassle.

from goworker.

@benmanns, any comments on this? Would you accept such a PR?

from goworker.

@twpayne Would configuration stored in environment variables work for your situation?

E.g.

QUEUES=high,medium,low
INTERVAL=0.5
CONCURRENCY=10
./worker

This sort of configuration has been requested in #8, and I'm looking to find the best way to handle it.

from goworker.

In my case my Goworker service shares a number of configuration parameters with other services. I want to have a single configuration file shared by all services to keep it DRY. I suppose I could write a wrapper script that read my configuration file, set environment variables and then execed the Goworker service, but it's rather ugly. It would be much nicer if my Goworker service could read the configuration file itself.

from goworker.

I would second the idea that it's better to allow setting directly, not via environment vars. It would be nice to set the options directly and let the user figure out their own way of storing that data...so the user could (at their choice) read a config file, read env vars, or whatever, then init Goworker with those options.

from goworker.

What would you think about an interface for fetching configuration values? Something like

type Configurator interface {
    Get(key string) (*string, error)
}

that could be initialized and passed to goworker. Then we could have predefined goworker.FlagConfigurator or goworker.EnvConfigurator, and you can create your own JSONConfigurator or EtcdConfigurator, etc.

from goworker.

This is definitely a step in the right direction, but it's not a technique that I've seen elsewhere in Go code. For example, it removes all type checking by forcing the the configuration to be an implicit map[string]string. Normally things are explicitly configured by passing arguments to some sort of constructor.

Another possibility (not sure if it's a good one):

// Example config from command-line flags
func main() {
  goworker.AddFlags()
  flag.Parse()
  goworker.Run(goworker.ConfigFromFlags())
}

// Example config from environment variables
func main() {
  goworker.Run(goworker.ConfigFromEnv())
}

// Example config from explicit config struct
func main() {
  config := goworker.Config{
    queues: []string{"high", "medium", "low"},
  }
  goworker.Run(&config)
}

from goworker.

Agreed. What about a "configurator" interface that looks more like:

type Configurator interface {
    Get() Config
}

type Config struct {
    Queues []string
    URI string
    Interval float64
    // ...
}

I'm trying to avoid requiring everyone who relies on the flag-based configuration to have to add the

goworker.AddFlags()
flag.Parse()
goworker.Run(goworker.ConfigFromFlags())

block when they upgrade.

from goworker.

Is this a case where 100% backwards-compatibility is required? (Aside: I think we're getting hit by Go's primitive package support here). A couple of thoughts:

• Some people might not want goworker to add flags: these might conflict with application-specific flags, or (worse) be silently ignored (but still appear in -help) if the user uses a different Configurator.
• One way to work around the backwards-incompatible change of goworker.AddFlags() and friends is to fail fast and loud if goworker is not configured. Just adding a single mandatory argument to the existing goworker.Work() function means that this will be caught immediately at compile time. Users of goworker will go "Ug, what changed?", quickly find what they need to change (because the compiler will point them straight at it), and move on.

from goworker.

+1 on this. I love this library for what it does, but it's doing way too much behind-the-scenes magic for me to feel comfortable. I don't want to have to specify queue name on the command line, for example, because it will never and should never change.

Is there a reason besides a small backwards-incompatible change for why you'd like to optimize for zero-configuration if the user would like to use CLI flags? Does it matter that much for non-trivial applications (which I imagine will want some way of customizing their configuration anyways instead of relying on CLI flags)?

It seems non-idiomatic for a package to add its own mandatory CLI flags (that show up in "--help" output, etc.). That behavior should be completely optional, in my opinion.

from goworker.

The Sidekiq equivalent of this project uses a relatively nice and certainly much more flexible method of configuration: https://github.com/jrallison/go-workers

Could be taken as inspiration for a similar feature for goworker.

from goworker.