svengreb / wand Goto Github PK
View Code? Open in Web Editor NEWA simple and powerful toolkit for Mage.
License: MIT License
A simple and powerful toolkit for Mage.
License: MIT License
Currently the Run
method of the Runner
interface allows to run a command, but does not return its output. This can be blocking when running commands like go env GOBIN
to get the path to the GOBIN
environment variable.
To support such uses cases, a new RunOut(Task) (string, error)
method will be added to the Runner
interface that runs a command and returns its output.
This is issue supersedes #78 which documents how the official deprecation of gobin
in favor of the new Go 1.16 go install pkg@version
syntax feature should have been handled for this project. The idea was to replace the gobin
task runner with a one that leverages bingo, a project similar to gobin
, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that bingo
uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the go install
command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects.
go install
is still not perfectSupport for the new go install
features, which allow to install commands without affecting the main
module, have already been added in #71 as an alternative to gobin
, but one significant problem is still not addressed: install module/package executables globally without overriding already installed executables of different versions.
Since go install
will always place compiled binaries in the path defined by go env GOBIN
, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since go install
still messes up the local user environment.
go install
task runnerThe solution is to implement a custom Runner
that uses go install
under the hood, but places the compiled executable in a custom cache directory instead of go env GOBIN
. The runner will check if the executable already exists, installs it if not so, and executes it afterwards.
The concept of storing dependencies locally on a per-project basis is well-known from the node_modules
directory of the Node package manager npm. Storing executables in a cache directory within the repository (not tracked by Git) allows to use go install
mechanisms while not affect the global user environment and executables stored in go env GOBIN
. The runner will achieve this by changing the GOBIN
environment variable to the custom cache directory during the execution of go install
. This way it bypasses the need for “dirty hacks“ while using a custom output path.
The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages.
Note that the runner dynamically runs executables based on the given task so Validate() error
will be a NOOP.
The solution described above will work totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again:
go run
command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed.go install
aware of the -o
flag like the go build
command which is the only reason why the custom runner will be implemented.Because the new custom task runner dynamically runs executables based on the given task, the Bootstrap
method of the Wand
reference implementation Elder
will additionally allow to pass Go module import paths, optionally including a version suffix (pkg@version
), to install executables from Go module-based main
packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task in JetBrains IDEs.
The method will also ensure that the local cache directory exists and will create a .gitignore
file that includes ignore pattern for the cache directory.
As of Go 1.17 the go run
command can finally run in module-aware mode while not “polluting“ the current module in the working directory, if there is one (go.mod
file present) 🎉
This finally allows to run commands on-the-fly of Go main
module packages without installing them or without changing dependencies of the current module!
To support this feature with wand a new task.GoModule
will be implemented in a new golang/run
package.
It can be run using a command runner that handles tasks of kind KindGoModule
so mainly gotool.Runner
.
The new golang/run.Task
will be customizable through the following functions:
WithArgs(...string) run.Option
— sets additional arguments to pass to the command.WithEnv(map[string]string) run.Option
— sets the task specific environment.WithModulePath(string) run.Option
— sets the module import path.WithModuleVersion(*semver.Version) run.Option
— sets the module version.Next to the new task the gotool.Runner
will be extended with a new WithCache(bool)
runner option to toggle the usage of the local cache directory in the root directory of the module. The runner will be made “smart“ in the way that it either…
golang.Runner
, which runs go install pkg@version
to leverage Go 1.16‘s feature, and execute it afterwards. This is the current default behavior of this runner which will be used when WithCache(true)
is used.golang.Runner
, using the new golang/run
package task, so that it can run go run pkg@version <args>
instead. This is the new “smart“ behavior of the runner which will be used when WithCache(false)
(default) is used.The new default behavior will be to not use a local cache so that caching will be a opt-in. This decision was made because native support for running commands on-the-fly should always be preferred to custom logic which is what the local cache directory and gotool.Runner
purpose is.
Warning
Note that the minimum Go version for task runners, the newgolang/run
task and the Elder wand will be increased to1.17.0
since this version initially introducedgo run
support in module-aware mode!
This will be enforced through a build constraint (go:build go1.17
).
The Elder
reference implementation will also adapt to this new feature by…
*elder.Elder.Bootstrap(...string) []error
method! As of wand version 0.9.0
it will be a no-op and will be removed in version 0.10.0
. To install executables anyway the new *elder.Elder.CacheExecutables error
method should be used instead. To ensure that the wand is properly initialized and operational the *elder.Elder.Validate(..task.Runner) []error
method is the way to go. A warning message will be printed when the method is called to ensure that users adapt accordionally.*elder.Elder.CacheExecutables(...string) error
method which allows to pass paths of Go modules that should be explicitly installed to the local cache directory. This method is a kind of workaround for the, now deprecated, *elder.Elder.Bootstrap(...string) []error
method to allows users to still cache command executables locally.*elder.Elder.Validate() error
method to *elder.Elder.Validate(...task.Runner) []error
method which allows users to ensure that the wand is properly initialized and operational. Optionally command runner can be passed that will be validated while passing nothing will validate all currently supported runners.To support the go env
command of the Go toolchain, a new Task
will be implemented in a new env
package that can be used through a Go toolchain Runner
.
The task will be customizable through the following functions:
WithEnv(env map[string]string) env.Option
— sets the task specific environment.WithEnvVars(envVars ...string) env.Option
— sets the names of the target environment variables.WithExtraArgs(extraArgs ...string) env.Option
— sets additional arguments to pass to the command.The mvdan.cc/gofumpt Go module provides the gofumpt
command, a tool that enforces a stricter format than gofmt
and provides additional rules, while being backwards compatible. It is a modified fork of gofmt
so it can be used as a drop-in replacement.
To configure and run the gofumpt
command, a new task.GoModule
will be implemented in a new gofumpt package that can be run using the gobin command runner or any other command runner that handles tasks of kind KindGoModule
.
The task will be customizable through the following functions:
WithEnv(map[string]string) gofumpt.Option
— sets the task specific environment.WithExtraArgs(...string) gofumpt.Option
— sets additional arguments to pass to the command.WithExtraRules(bool) gofumpt.Option
— indicates whether gofumpt
‘s extra rules should be enabled. See the repository documentation for a listing of available rules.WithListNonCompliantFiles(bool) gofumpt.Option
— indicates whether files, whose formatting are not conform to the style guide, are listed.WithModulePath(string) gofumpt.Option
— sets the module import path.WithModuleVersion(*semver.Version) gofumpt.Option
— sets the module version.WithPaths(...string) gofumpt.Option
— sets the paths to search for Go source files. By default all directories are scanned recursively starting from the current working directory.WithReportAllErrors(bool) gofumpt.Option
— indicates whether all errors should be printed instead of only the first 10 on different lines.WithSimplify(bool) gofumpt.Option
— indicates whether code should be simplified.The “elder“ reference implementation will provide a new Gofumpt
method.
When the WithRunnerQuiet
option of the golang
runner or WithQuiet
option of the gotool
runner is set to true
, the task passed to their Run
method will run twice. This is caused by a missing return statement when the execution finishes with a error
of value nil
1 2.
To fix this bug both code blocks must return early with nil
instead of keep going on in the code flow.
To allow to compose, manipulate and read spell incantation options after the initial creation, two new types will be added for the spell packages:
spell.Options
— A interface
type as a generic representation for spell.Incantation
options.spell.Mixin
— A interface
type that allows to compose functions that process spell.Options
of spell.Incantation
s.
Apply(Options) (Options, error)
— applies generic spell.Options
to spell.Incantation
options.Update to tmpl-go
version 0.11.0
which…
golangci-lint
running errors due to revive
s unknown time-equal
rule.revive
linter rule package-comments
.This will also include changes required for any linter matches.
See the full tmpl-go
version 0.10.1
changelog for all details.
To use the Go toolchain, also known as the go
command, a new caster, introduced in #14, will be implemented.
To unify the handling of errors in the cast package, a new ErrCast
struct
type will also be implemented.
The Validate
function of the new caster will return an error of type *cast.ErrCast
when the go
binary executable does not exists at the configured path or when it is also not available in the executable search paths of the current environment.
Currently the Install
method of the gobin
task runner sets the environment of the command that gets executed initially to os.Environ()
, but overrides it later on with custom variables configured through the WithEnv(map[string]string)
option.
This results in errors when running the method in minimal environments like containers:
build cache is required, but could not be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined
This can be fixed by ensuring that the inherited OS environment is prepended before applying custom environment variables.
Also to improve the debugging process the combined output (stdout
+ stderr
) should be included in the error when the command execution fails.
As of Go version 1.16 go install $pkg@$version
allows to install commands without affecting the main
module. Additionally commands like go build
and go test
no longer modify go.mod
and go.sum
files by default but report an error if a module requirement or checksum needs to be added or updated (as if the -mod=readonly
flag were used).
This can be used as alternative to the already existing gobin
runner.
To support the go install
command of the Go toolchain, a new Task
will be implemented in a new install package that can be used through a Go toolchain Runner
.
The task will be customizable through the following functions:
WithEnv(env map[string]string) install.Option
— sets the task specific environment.WithModulePath(path string) install.Option
— sets the module import path.WithModuleVersion(version *semver.Version) install.Option
— sets the module version.The github.com/golangci/golangci-lint Go module provides the golangci-lint
commands, a fast, parallel runner for dozens of Go linters Go that uses caching, supports YAML configurations and has integrations with all major IDEs. The source code for the golangci-lint
command can be found in the golangci/golangci-lint repository.
To configure and run the golangci-lint
command, a new spell.Incantation
will be implemented in a new golangcilint package that can be casted using the gobin caster or any other spell caster that handles spell incantations of kind KindGoModule
.
The spell incantation will be customizable through the following functions:
WithArgs(args ...string) golangcilint.Option
— sets additional arguments to pass to the golangci-lint
module command.WithEnv(env map[string]string) golangcilint.Option
— sets the spell incantation specific environment.WithModulePath(path string) golangcilint.Option
— sets the golangci-lint
module command import path. Defaults to golangcilint.DefaultGoModulePath
.WithModuleVersion(version *semver.Version) golangcilint.Option
— sets the golangci-lint
module version. Defaults to golangcilint.DefaultGoModuleVersion
.WithVerboseOutput(verbose bool) golangcilint.Option
— indicates whether the output should be verbose.Write the initial project documentation that includes…
Currently most tasks provide a TaskName
package constant that contains the name of the task, but this is not an idiomatic and consistent way. To make sure that this information is part of the API, a new Name() string
method will be added to the Task
interface.
To also improve the way how RunerExec
handles the name of the executable file, a new ExecName() string
method will be added to the Exec
interface.
Most of the GoModule
tasks using an outdated default Go module version so the following tasks will be updated and adjusted to the currently latest versions:
github.com/svengreb/wand/pkg/task/gofumpt
task currently uses version v0.1.1
and will be updated to version 0.2.0
by…-r
flag which has been removed in favor of gofmt -r
.-s
flag (WithSimplify
option) as it is always enabled.github.com/svengreb/wand/pkg/task/goimports
task currently uses version v0.1.0
and will be updated to version 0.1.7
.github.com/svengreb/wand/pkg/task/golangcilint
task currently uses version v1.39.0
and will be updated to version 1.43.0
. The configuration has already been updated in #104.In #10 two types were implemented to derive and handle version information of a Git repository:
pkg/vcs/git.Version
— A struct
type that stores version information and metadata derived from a Git repository.pkg/vcs/git.deriveVersion
— A func
type that derives version information and metadata from a Git repository.The logic has been extracted into the pkg/vcs/git package of the github.com/svengreb/golib module and therefore both types will be replaced with this new module.
When installing a Go executable from within a Go module directory using the go install
command, it is installed into the Go executable search path that is defined through the GOBIN
environment variable and can also be shown and modified using the go env
command. Even though the executable gets installed globally, the go.mod
file will be updated to include the installed packages since this is the default behavior of the go get
command when running in module mode.
Next to this problem, the installed executable will also overwrite any executable of the same module/package that was installed already, but maybe from a different version. Therefore only one version of a executable can be installed at a time which makes it impossible to work on different projects that use the same tool but with different versions.
The local installation of executables built from Go modules/packages has always been a somewhat controversial point which unfortunately, partly for historical reasons, does not offer an optimal and user-friendly solution up to now. The go
command is a fantastic toolchain that provides many great features one would expect to be provided out-of-the-box from a modern and well designed programming language without the requirement to use a third-party solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging utilities and many more.
Unfortunately the way the go install
command of Go versions less or equal to 1.15 handles the installation of an Go module/package executable is still not optimal.
The general problem of tool dependencies is a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community with discussions like golang/go#30515, golang/go#25922 and golang/go#27653 to improve this essential feature, but they‘ve been around for quite a long time without a solution that works without introducing breaking changes and most users and the Go team agree on.
Luckily, this topic was finally picked up for the next upcoming Go release version 1.16 and gh-golang/go#40276 introduces a way to install executables in module mode outside a module. The release note preview also already includes details about this change and how installation of executables from Go modules will be handled in the future.
Beside the great news and anticipation about an official solution for the problem the usage of a workaround is almost inevitable until Go 1.16 is finally released.
The official Go wiki provides a section on “How can I track tool dependencies for a module?” that describes a workaround that tracks tool dependencies. It allows to use the Go module logic by using a file like tools.go
with a dedicated tools
build tag that prevents the included module dependencies to be picked up included for normal executable builds. This approach works fine for non-main packages, but CLI tools that are only implemented in the main
package can not be imported in such a file.
In order to tackle this problem, a user from the community created gobin, an experimental, module-aware command to install/run main packages.
It allows to install or run main-package commands without “polluting“ the go.mod
file by default. It downloads modules in version-aware mode into a binary cache path within the systems cache directory.
It prevents problems due to already globally installed executables by placing each version in its own directory. The decision to use a cache directory instead of sub-directories within the GOBIN
path keeps the system clean.
gobin is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also members of the core Go team that have contributed to the project and the chance is high that the changes for Go 1.16 were influenced or partially ported from it.
It is currently the best workaround to…
GOMOD
environment variable (see go env GOMOD
) that is initialized automatically with the path to the go.mod
file in the current working directory.go.mod
file.See gobin‘s FAQ page in the repository wiki for more details about the project.
To allow to manage the tool dependency problem, wand, will use gobin
through a new caster that prevents the “pollution“ of the project go.mod
file and allows to…
gobin
itself into GOBIN
(go env GOBIN
).KindGoModule
by installing the executable globally into the dedicated gobin
cache.The github.com/oligot/go-mod-upgrade Go module provides the go-mod-upgrade
command, a too to update outdated Go module dependencies interactively.
To configure and run the go-mod-upgrade
command, a new task.GoModule
will be implemented in a new gomodupgrade
package. It can be be run using a command runner that handles tasks of kind KindGoModule
.
The task will be customizable through the following functions:
WithEnv(map[string]string) gomodupgrade.Option
— sets the task specific environment.WithExtraArgs(...string) gomodupgrade.Option
— sets additional arguments to pass to the command.WithModulePath(string) gomodupgrade.Option
— sets the module import path.WithModuleVersion(*semver.Version) gomodupgrade.Option
— sets the module version.The Elder
reference implementation will provide a new GoModUpgrade
method.
In #52 a task for the github.com/markbates/pkger Go module was added, a tool for embedding static files into Go binaries.
The issue also includes the “Official Static Assets Embedding“ section which mentions that the task might be removed later on again as soon as Go 1.16 will be released as it comes with toolchain support for embedding static assets (files) through the embed
package. Also see markbates/pkger#114 for more details about the project future of pkger
.
The pkger
package will be removed and the //go:embed
directive should be used instead.
To clean paths in a filesystem, like application specific output directories, a new GoCode
spell incantation will be implemented which can be used without a caster.
The spell incantation will provide the following methods:
Clean() ([]string, error)
— removes the configured paths. It returns an error of type *spell.ErrGoCode
for any error that occurs during the execution of the Go code.The spell incantation will be customizable through the following functions:
WithLimitToAppOutputDir(limitToAppOutputDir bool) clean.Option
— indicates whether only paths within the configured application output directory should be allowed.WithPaths(paths ...string) clean.Option
— sets the paths to remove. Note that only paths within the configured application output directory are allowed when WithLimitToAppOutputDir
is enabled.With #14 the “abstract“ wand API was introduced with a naming scheme is inspired by the fantasy novel “Harry Potter“ that was used to to define interfaces.
The main motivation was to create a matching naming to the overall “magic“ topic and the actual target project Mage, but in retrospect this is way too abstract and confusing.
The goal of this ticket is to…
The basic mindset of the API will remain partially the same, but it will be designed around the concept of tasks and the ways to run them.
🅸 task.Runner
is a new base interface that runs a command with parameters in a specific environment. It can be compared to the current 🅸 cast.Caster
interface, but provides a cleaner method set accepting the new 🅸 task.Task
interface.
Handles() task.Kind
— returns the supported task kind.Run(task.Task) error
— runs a command.Validate() error
— validates the runner.The new 🅸 task.RunnerExec
interface is a specialized task.Runner
and serves as an abstract representation for a command or action, in most cases a (binary) executable of external commands or Go module main
packages, that provides corresponding information like the path to the executable. It can be compared to the current BinaryCaster
interface, but also comes with a cleaner method set and a more appropriate name.
FilePath() string
— returns the path to the (binary) command executable.🅸 task.Task
is the new interface that is scoped for Mage “target“ usage. It can be compared to the current 🅸 spell.Incantation
interface, but provides a smaller method set without Formula() []string
.
Kind() task.Kind
— returns the task kind.Options() task.Options
— returns the task options.The new 🅸 task.Exec
interface is a specialized task.Task
and serves as an abstract task for an executable command. It can be compared to the current Binary
interface, but also comes with the new BuildParams() []string
method that enables a more flexible usage by exposing the parameters for command runner like task.RunnerExec
and also allows to compose with other tasks. See the Wikipedia page about the anatomy of a shell CLI for more details about parameters.
BuildParams() []string
— builds the parameters for a command runner where parameters can consist of options, flags and arguments.Env() map[string]string
— returns the task specific environment.The new 🅸 task.GoModule
interface is a specialized task.Exec
for a executable Go module command. It can be compared to the current spell.GoModule
interface and the method set has not changed except a renaming of the GoModuleID() *project.GoModuleID
to the more appropriate name ID() *project.GoModuleID
. See the official Go module reference documentation for more details about Go modules.
ID() *project.GoModuleID
— returns the identifier of a Go module.The following listing shows the new name concept and how the current API components can be mapped to the upcoming changes:
main
packages. The current API component that can be compared to runners is 🅸 cast.Caster
and its specialized interfaces.spell.Incantation
and its specialized interfaces.Even though the API will be changed quite heavily, the basic usage almost won't change.
→ A task.Task
can only be run through a task.Runner
!
Currently a spell.Incantation
is passed to a cast.Caster
in order to run it, in most cases a (binary) executable of a command that uses the Formula() []string
method of spell.Incantation
to pass the result as parameters.
The new API works the same: A task.Task
is passed to a task.Runner
that calls the BuildParams() []string
method when the runner is specialized for (binary) executable of commands.
The current documentation is mainly scoped on the technical details, but lacks more user-friendly sections about topics like the way how to implement own API components, how to compose the “elder“ reference implementation or usage examples for single or monorepo project layouts
Most of the current sections will be rewritten or removed entirely while new sections will provide more user-friendly guides about how to…
Some examples will be added, that are linked and documented in the user guides described above, to show how to…
Update to tmpl-go
version 0.10.0
which…
golangci-lint
's default excluded issues — this prevents that explicitly enabled rules are not ignored due to the default set of excluded issues.ci-go
workflow — this improves the workflow execution time.See the full tmpl-go
version 0.10.0
changelog for all details.
The default way to use the wand API, with its casters and spells, will be the reference implementation “elder“.
It will provide a way to use all wand spells and additionally comes with helper methods to bootstrap a project, validate all casters and simplify logging for process exits:
Bootstrap() error
— runs initialization tasks to ensure the wand is operational. This includes the installation of configured caster like cast.BinaryCaster
that can handle spell incantations of kind spell.KindGoModule
.Clean(appName string, opts ...clean.Option) ([]string, error)
— a spell.GoCode
to remove configured filesystem paths, e.g. output data like artifacts and reports from previous development, test, production and distribution builds. It returns paths that have been cleaned along with an error of type *spell.ErrGoCode
when an error occurred during the execution of the Go code. When any error occurs it will be of type *app.ErrApp
or *cast.ErrCast
. See the clean package for all available options.ExitPrintf(code int, verb nib.Verbosity, format string, args ...interface{})
— simplifies the logging for process exits with a suitable nib.Verbosity
.GetAppConfig(name string) (app.Config, error)
— returns an application configuration. An empty application configuration is returned along with an error of type *app.ErrApp
when there is no configuration in the store for the given name.GetProjectMetadata() project.Metadata
— returns metadata of the project.GoBuild(appName string, opts ...build.Option)
— casts the spell incantation for the build
command of the Go toolchain. When any error occurs it will be of type *app.ErrApp
or *cast.ErrCast
. See the build package for all available options.Goimports(appName string, opts ...goimports.Option) error
— casts the spell incantation for the golang.org/x/tools/cmd/goimports Go module command that allows to update Go import lines, add missing ones and remove unreferenced ones. It also formats code in the same style as gofmt
command so it can be used as a replacement. When any error occurs it will be of type *app.ErrApp
or *cast.ErrCast
.goimports
see the module documentation. The source code of goimports
is available in the GitHub repository.GolangCILint(appName string, opts ...golangcilint.Option) error
— casts the spell incantation for the github.com/golangci/golangci-lint/cmd/golangci-lint Go module command, a fast, parallel runner for dozens of Go linters Go that uses caching, supports YAML configurations and has integrations with all major IDEs. When any error occurs it will be of type *app.ErrApp
or *cast.ErrCast
. See the golangcilint package for all available options.golangci-lint
see the module documentation and the official website. The source code of golangci-lint
is available in the GitHub repository.GoTest(appName string, opts ...spellGoTest.Option) error
— casts the spell incantation for the test
command of the Go toolchain. When any error occurs it will be of type *app.ErrApp
or *cast.ErrCast
. See the test package for all available options.Gox(appName string, opts ...spellGox.Option) error
— casts the spell incantation for the github.com/mitchellh/gox Go module command, a dead simple, no frills Go cross compile tool that behaves a lot like the standard Go toolchain build
command. When any error occurs it will be of type *app.ErrApp
or *cast.ErrCast
. See the gox package for all available options.gox
see the module documentation. The source code of gox
is available in the GitHub repository.RegisterApp(name, displayName, pathRel string) error
— creates and stores a new application configuration. Note that the package path must be relative to the project root directory!Validate() error
— ensures that all casters are properly initialized and available. It returns an error of type *cast.ErrCast when the validation of any of the supported casters fails.New(opts ...Option) (*Elder, error)
— creates a new elder wand.runtime/debug
package. The absolute path to the root directory is automatically set based on the current working directory. Note that the working directory must be set manually when the “magefile“ is not placed in the root directory by pointing Mage to it:
-d <PATH>
option to set the directory from which “magefiles“ are read (defaults to .
).-w <PATH>
option to set the working directory where “magefiles“ will run (defaults to value of -d
flag).The wand will be customizable through the following functions:
WithGobinCasterOptions(opts ...castGobin.Option) elder.Option
— sets “gobin“ caster options.WithGoToolchainCasterOptions(opts ...castGoToolchain.Option) elder.Option
— sets Go toolchain caster options.WithNib(n nib.Nib) elder.Option
— sets the log-level based line printer for human-facing messages.WithProjectOptions(opts ...project.Option) elder.Option
— sets project options.Like described in the /apps
directory documentation of the tmpl-go template repository, wand also aims to support the monorepo layout.
In order to manage multiple applications, their information and metadata will be recorded in a configuration store where each entry will be identified by a unique ID, usually the name of the application. The pkg/app
package will provide two interfaces and an unexported struct that implements it that can be used through the exported NewStore() Store
function.
pkg/app.Config
— A struct
type that holds information and metadata of an application.pkg/app.Store
— A storage that provides methods to record application configurations:
Add(*Config)
— Adds a application configuration.Get(string) (*Config, error)
— Returns the application configuration for the given name or nil along with an error when not stored.appStore
— A storage for application configurations.NewStore() Store
— Creates a new store for application configurations.Update to tmpl-go
version 0.12.0
which…
This will also include changes required for any linter matches.
See the full tmpl-go
version 0.12.0
changelog for all details.
As of Go 1.18 the debug.ReadBuildInfo function does not work for Mage executables anymore because the way how module information is stored changed. Therefore the fields of the returned debug.Module type only has zero values, including the module path. The debug.Module.Version field has a default value ((devel)
) which is not Semver compatible and causes the parsing to fail. The change in Go 1.18 also came with the new debug/buildinfo
package which allows to read the information from compiled binaries while the runtime/debug.ReadBuildInfo
function returns information from within the running binary. Both are not suitable anymore which is also described in the Go 1.18 version
command release notes:
The underlying data format of the embedded build information can change with new
go
releases, so an older version ofgo
may not handle the build information produced with a newer version ofgo
. To read the version information from a binary built withgo
1.18, use thego
version command and thedebug/buildinfo
package fromgo
1.18+.
To get the required module information that was previously provided by the runtime/debug package the official golang.org/x/mod/modfile package will be used instead that provides the implementation for a parser and formatter for go.mod
files 1. This allows to safely get the module path without the need to depend on runtime/dynamic logic that might change in future Go versions.
Note that this change will also increase the minimum Go version from 1.17
to 1.19
!
There are currently no OS specific tests so the test
workflow job should only run on Linux. The additional run on Windows is currently unnecessary and only wastes build minutes because each minute is calculated 2x when running on Windows. See the section about “Minute multipliers“ in the official GitHub Action Pricing documentation.
Next to this it also helps to improve the performance since workflows running on Windows are currently taking the most time.
The github.com/markbates/pkger Go module provides the pkger
command, a tool for embedding static files into Go binaries.
To configure and run the pkger
command, a new task.GoModule
will be implemented in a new pkger package that can be run using the gobin command runner or any other command runner that handles tasks of kind KindGoModule
.
The task will be customizable through the following functions:
WithEnv(env map[string]string) pkger.Option
— sets the task specific environment.WithExtraArgs(extraArgs ...string) pkger.Option
— sets additional arguments to pass to the command.WithIncludes(includes ...string) pkger.Option
— adds the relative paths of files and directories that should be included.pkger
itself when used within any of the packages of the target Go module.WithModulePath(path string) pkger.Option
— sets the module import path.WithModuleVersion(version *semver.Version) pkger.Option
— sets the module version.The “elder“ reference implementation will provide a new Pkger
method including the handling of the “monorepo“ workaround.
Please note that the pkger project might be superseded and discontinued due to the official Go toolchain support for embedding static assets (files) that will most probably be released with Go version 1.16.
Please see the official draft document and markbates/pkger#114 for more details.
pkger tries to mimic the Go standard library and the way how the Go toolchain handles modules, but is therefore also affected by its problems and edge cases.
When the pkger
command is used from the root of a Go module repository, the directory where the go.mod
file is located, and there is no valid Go source file, the command will fail because it internally uses the same logic like the list
command of the Go toolchain (go list
).
Therefore a “dummy“ Go source file may need to be created as a workaround. This is mostly only required for repositories that use a “monorepo“ layout where one or more main
packages are placed in a subdirectory relative to the root directory, e.g. apps
or cmd
. For repositories where the root directory already has a Go package, that does not contain any build constraints/tags, or uses a “library“ layout, a “dummy“ file is probably not needed.
The new Pkger
method of the “elder“ reference implementation will handle the creation of a temporary “dummy“ file that will be deleted automatically when the tasks finishes in order to avoid the need for the user to add such a file to the repository and commit it into the VCS.
Please see markbates/pkger#109 and markbates/pkger#121 for more details.
In #79 a new ExecName() string
method has been added to the Exec
interface to improve the way how RunnerExec
handles the name of the executable file, but it turns out that this is not useful at all and also breaks the separation of concern between the Task
and Runner
interfaces.
Therefore the method will be removed again.
The GitHub action workflows using the actions/checkout
action to fetch the repository that triggered the workflow. However, by default only the history of the latest commit is fetched which results in errors when wand tries to extract repository metadata information like the amount of commits ahead of the latest commit. As an example this can be seen when running the bootstrap
command in the test
job of the ci-go
workflow which fails with an object not found
error because the history only contains a single commit.
To fix this problem action/checkout
provides an option to fetch all history for all tags and branches which will be used to prevent errors like this in the pipeline.
The golang.org/x/tools/cmd/goimports Go module allows to update Go import lines, adding missing ones and removing unreferenced ones. It also formats code in the same style as gofmt so it can be used as a replacement. The source code for the goimports
command can be found in the golang/tools repository.
To configure and run the goimports
command, a new spell.Incantation
will be implemented in a new goimports package that can be casted using the gobin caster or any other spell caster that handles spell incantations of kind KindGoModule
.
The spell incantation will be customizable through the following functions:
WithEnv(env map[string]string) goimports.Option
— sets the spell incantation specific environment.WithExtraArgs(extraArgs ...string) goimports.Option
— sets additional arguments to pass to the goimports
command.WithListNonCompliantFiles(listNonCompliantFiles bool) goimports.Option
— indicates whether files, whose formatting are not conform to the style guide, are listed.WithLocalPkgs(localPkgs ...string) goimports.Option
— sets local packages whose imports will be placed after 3rd-party packages.WithModulePath(path string) goimports.Option
— sets the goimports
module import path. Defaults to goimports.DefaultGoModulePath
.WithModuleVersion(version *semver.Version) goimports.Option
— sets the goimports
module version. Defaults to goimports.DefaultGoModuleVersion
.WithPaths(paths ...string) goimports.Option
— sets the paths to search for Go source files. By default all directories are scanned recursively starting from the current working directory.WithPersistedChanges(persistChanges bool) goimports.Option
— indicates whether results are written to the source files instead of standard output.WithReportAllErrors(reportAllErrors bool) goimports.Option
— indicates whether all errors should be printed instead of only the first 10 on different lines.WithVerboseOutput(verbose bool) goimports.Option
— indicates whether the output should be verbose.To support common use cases for debugging and production optimization, some spell mixins will be implemented in the golang package:
MixinImproveDebugging
— A struct
type that adds linker flags to improve the debugging of binary artifacts. This includes the disabling of inlining and all compiler optimizations tp improve the compatibility for debuggers.all
prefix for —gcflags
parameters to make sure all packages are affected. If you disabled the all
prefix on purpose you need to handle this conflict on your own, e.g. by creating more than one binary artifact each with different build options.MixinImproveEscapeAnalysis
— A struct
type that will add linker flags to improve the escape analysis of binary artifacts.all
prefix for —gcflags
parameters to make sure only the target package is affected, otherwise reports for (traverse) dependencies would be included as well. If you enabled the all
prefix on purpose you need to handle this conflict on your own, e.g. by creating more than one binary artifact each with different build options.MixinStripDebugMetadata
— A struct
type that will add linker flags to strip debug information from binary artifacts. This will include DWARF tables needed for debuggers, but keeps annotations needed for stack traces so panics are still readable. It will also shrink the file size and memory overhead as well as reducing the chance for possible security related problems due to enabled development features or debug information leaks.all
prefix for —gcflags
parameters to make sure all packages are affected. If you disabled the all
prefix on purpose you need to handle this conflict on your own, e.g. by creating more than one binary artifact each with different build options.MixinInjectBuildTimeVariableValues
— A struct
type that will inject build—time values through the —X
linker flags to populate e.g. application metadata variables.map[string]string
of key/value pairs to inject to variables at build—time. The key must be the path to the variable in form of <IMPORT_PATH>.<VARIABLE_NAME>
, e.g. pkg/internal/support/app.version
. The value is the actual value that will be assigned to the variable, e.g. the application version.*project.GoModuleID
will store partial information about the target Go module to inject the key/value pairs from the data map into.The wand API is inspired by the fantasy novel “Harry Potter“ and uses an abstract view to define interfaces. The main motivation to create a matching naming to the overall “magic“ topic and the actual target project Mage. This might be too abstract for some, but is kept understandable insofar as it should allow everyone to use the "task" API and to derive their own tasks from it.
cast.Caster
— A interface
type that casts a spell.Incantation
using a command for a specific spell.Kind
:
Cast(spell.Incantation) error
— casts a spell incantation.Handles() spell.Kind
— returns the spell kind that can be casted.Validate() error
— validates the caster command.cast.BinaryCaster
— A interface
type that composes cast.Caster
to run commands using a binary executable:
GetExec() string
— returns the path to the binary executable of the command.spell.Incantation
— A interface
type that is the abstract representation of parameters for a command or action:
Formula() []string
— returns all parameters of a spell.Kind() Kind
— returns the Kind of a spell.Options() interface{}
— return the options of a spell.cast.Binary
— A interface
type that composes cast.Caster
for commands which are using a binary executable:
Env() map[string]string
— returns additional environment variables.cast.GoCode
— A interface
type that composes cast.Caster
for actions that can be casted without a cast.Caster
:
Cast() (interface{}, error)
— casts itself.cast.GoModule
— A interface
type that composes cast.Binary
for commands that are compiled from a Go module:
GoModuleID() *project.GoModuleID
— returns the identifier of a Go module.spell.Kind
— A struct
type that defines the kind of a spell.The API components can be roughly translated to their purpose:
cast.Caster
→ a executable commandspell.Kind
can be handled by this caster. It could be executed without parameters (spell.Incantation
), but in most cases needs at least one parameter.
cast.BinaryCaster
→ a composed cast.Caster
to run commands using a binary executable.spell.Incantation
), but would not have any effect im many cases.spell.Incantation
→ the parameters of a executable commandspell.GoCode
a incantation cannot be used alone but must be passed to a cast.Caster
that is able to handle the spell.Kind
of this incantation.
spell.Binary
→ a composed spell.Incantation
to run commands that are using binary executable.spell.GoCode
→ a composed spell.Incantation
for pure Go code instead of a (binary) executable command.os
from the Go standard library. It has been designed this way to also allow such tasks to be handled by the incantation API.spell.GoModule
→ a composed spell.Binary
to run binary commands managed by a Go module, in other words executables installed in GOBIN
or received via go get
.path@version
) in order to download and run the executable.The github.com/mitchellh/gox Go module provides the gox
command, a dead simple, no frills Go cross compile tool that behaves a lot like the standard Go toolchain build
command.
To configure and run the gox
command, a new spell.Incantation
will be implemented in a new gox package that can be casted using the gobin caster or any other spell caster that handles spell incantations of kind KindGoModule
.
The spell incantation will be customizable through the following functions:
WithEnv(env map[string]string) gox.Option
— sets the spell incantation specific environment.WithGoCmd(goCmd string) gox.Option
— sets the path to the Go toolchain executable.WithOutputTemplate(outputTemplate string) gox.Option
— sets the name template for cross-compile platform targets. Defaults to gox.DefaultCrossCompileBinaryNameTemplate
.WithGoOptions(goOpts ...spellGo.Option) gox.Option
— sets shared Go toolchain command options.WithGoBuildOptions(goBuildOpts ...spellGoBuild.Option) gox.Option
— sets options for the Go toolchain build
command.WithModulePath(path string) gox.Option
— sets the gox
module command import path. Defaults to gox.DefaultGoModulePath
.WithModuleVersion(version *semver.Version) gox.Option
— sets the gox
module version. Defaults to gox.DefaultGoModuleVersion
.WithVerboseOutput(verbose bool) gox.Option
— indicates whether the output should be verbose.Update to tmpl-go
version 0.5.0 (including version 0.4.0) that…
The project currently only uses GitHub Action workflows for CI but not Mage to automate tasks for itself though.
Following the “dogfooding“ concept Mage will finally be added to the repository, using wand as toolkit through the Elder
wand reference implementation.
In #15 some parts of the wand API have been implemented in form of spell incantations, kinds and casters, inspired by the fantasy novel “Harry Potter“ as an abstract view to define interfaces. In #9 and #11 the API implementations for an application configuration store as well as project and VCS repository metadata were introduced.
These implementations will be usable in a combined form via the main wand API that will consist of the following types:
wand.Wand
— A interface
type that manages a project and its applications and stores their metadata. Applications are registered using a unique name and the stored metadata can be received based on this name:
GetAppConfig(appName string) (app.Config, error)
— returns an application configuration.GetProjectMetadata() project.Metadata
— returns the project metadata.RegisterApp(name, displayName, pathRel string) error
— registers a new application.wand.ctxKey
— A struct
type that serves as context key used to wrap a wand.Wand
.wand.GetCtxKey() interface{}
— A func
type that returns the key used to wrap a wand.Wand
.wand.WrapCtx(parentCtx context.Context, wand Wand) context.Context
— A func
type that wraps the given wand.Wand
into the parent context. Use wand.GetCtxKey() interface{}
to receive the key used to wrap the wand.Wand
.Superseded by #87
As of Go 1.16 gobin
has officially been deprecated since go install $pkg@$version
allows to install commands without affecting the main
module. See myitcv/gobin#103 for more details about the gobin
deprecation and #22 for more information about the gobin
task runner.
Support for the new go install
features have already been added in #71 as an alternative to gobin
, but this method still lacks an essential feature: install module/package executables globally without overriding already installed executables of different versions.
Since go install
will always place compiled binaries in $GOBIN
(or whatever is set for go env GOBIN
), any already existing binary with the same name will be replaced. So installing a module command with two different versions is still not possible with go install
and still messes up the local environment.
This and other well-known problems are tackled by bingo, a project similar to gobin
. It comes with many great features and allows to manage development tools on a per-module basis, almost like the mechanism of Node‘s node_modules
directory.
Replacing the task runner that uses the deprecated gobin
project with a one that uses bingo
under the hood will bring back the features missing from go install
while also adding many more goodies to simplify and improve development setups.
Currently some functions that create a Task
require an app.Config
struct, but most tasks do not use the data in any way. To improve the code quality and simplify the internal usage of tasks these parameters will be removed as well as the field from the structs that implement the Task
interfaces.
To run the go build
command of the Go toolchain, a new spell.Incantation
will be implemented in a new build package that can be used through a Go toolchain caster.
The spell incantation will be configurable through the following functions:
WithBinaryArtifactName(name string) build.Option
— sets the name for the binary build artifact.WithCrossCompileTargetPlatforms(platforms ...string) build.Option
— sets the names of cross-compile platform targets.WithFlags(flags ...string) build.Option
— sets additional flags to pass to the Go build
command along with the base Go flags.WithGoOptions(goOpts ...spellGo.Option) build.Option
— sets shared Go toolchain commands options.WithOutputDir(dir string) build.Option
— sets the output directory, relative to the project root, for compilation artifacts.To unify further implementations for the Go toolchain, a new struct
type will also be implemented in the golang package that stores global/shared Go toolchain options that are shared between multiple Go toolchain commands:
WithAsmFlags(asmFlags ...string) golang.Option
— sets flags to pass on each go tool asm
invocation.WithRaceDetector(enableRaceDetector bool) golang.Option
— indicates if the race detector should be enabled.WithTrimmedPath(enableTrimPath bool) golang.Option
— indicates if all file system paths should be removed from the resulting executable.WithEnv(env map[string]string) golang.Option
— adds or overrides Go toolchain command specific environment variables.WithFlags(flags ...string) golang.Option
— sets additional Go toolchain command flags.WithFlagsPrefixAll(flagsPrefixAll bool) golang.Option
— indicates if the values of -asmflags
and -gcflags
should be prefixed with the all=
pattern in order to apply to all packages.WithGcFlags(gcFlags ...string) golang.Option
— sets flags to pass on each go tool compile
invocation.WithLdFlags(ldFlags ...string) golang.Option
— sets flags to pass on each go tool link
invocation.WithMixins(mixins ...spell.Mixin) golang.Option
— sets spell.Mixin
s that can be applied by option consumers.WithTags(tags ...string) golang.Option
— sets Go toolchain tags.A new CompileFormula(opts ...Option) []string
function will be provided that can be used to compile the formula for these options.
To run the go test
command of the Go toolchain, a new spell.Incantation
will be implemented in a new test package that can be used through a Go toolchain caster.
The spell incantation will be customizable through the following functions:
WithBlockProfileOutputFileName(blockProfileOutputFileName string) test.Option
— sets the file name for the Goroutine blocking profile file.WithCoverageProfileOutputFileName(coverageProfileOutputFileName string) test.Option
— sets the file name for the test coverage profile file.WithCPUProfileOutputFileName(cpuProfileOutputFileName string) test.Option
— sets the file name for the CPU profile file.WithBlockProfile(withBlockProfile bool) test.Option
— indicates if the tests should be run with a Goroutine blocking profiling.WithCoverageProfile(withCoverageProfile bool) test.Option
— indicates if the tests should be run with coverage profiling.WithCPUProfile(withCPUProfile bool) test.Option
— indicates if the tests should be run with CPU profiling.WithFlags(flags ...string) test.Option
— sets additional flags that are passed to the Go "test" command along with the shared Go flags.WithGoOptions(goOpts ...spellGo.Option) test.Option
— sets shared Go toolchain command options.WithMemProfile(withMemProfile bool) test.Option
— indicates if the tests should be run with memory profiling.WithMemoryProfileOutputFileName(memoryProfileOutputFileName string) test.Option
— sets the file name for the memory profile file.WithMutexProfile(withMutexProfile bool) test.Option
— indicates if the tests should be run with mutex profiling.WithMutexProfileOutputFileName(mutexProfileOutputFileName string) test.Option
— sets the file name for the mutex profile file.WithOutputDir(outputDir string) test.Option
— sets the output directory, relative to the project root, for reports like coverage or benchmark profiles.WithoutCache(withoutCache bool) test.Option
— indicates if the tests should be run without test caching that is enabled by Go by default.WithPkgs(pkgs ...string) test.Option
— sets the list of packages to test.WithTraceProfile(withTraceProfile bool) test.Option
— indicates if the tests should be run with trace profiling.WithTraceProfileOutputFileName(traceProfileOutputFileName string) test.Option
— sets the file name for the execution trace profile file.WithVerboseOutput(withVerboseOutput bool) test.Option
— indicates if the test output should be verbose.Bootstrap the basic project setup, structure and development workflow from version 0.3.0 of the “tmpl-go“ template repository.
Additionally specific assets like the repository hero image will be replaced and documentations like the README and GitHub issue/PR templates will be adjusted.
Currently most Task
creation functions [1]1 [2]2 [3]3 [4]4 require a Wand
as parameter which is not used but blocks the internal usage for task runners. Therefore these parameters will be removed. When necessary, it can be added individually later on or can be reintroduced through a dedicated function with extended parameters to cover different use cases.
https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/gofumpt#New ↩
https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/goimports#New ↩
https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/build#New ↩
https://pkg.go.dev/github.com/svengreb/[email protected]/pkg/task/golang/install#New ↩
In #9 the store and configuration for applications has been implemented. In wand applications are not standalone but part of a project which in turn is stored in a repository of a VCS like Git. In case of wand this can also be a monorepo to manage multiple applications, but there is always only a single project which all these applications are part of.
To store project and VCS repository information, some of the newly implemented packages will provide the following types:
pkg/project.Metadata
— A struct
type that stores information and metadata of a project.pkg/project.GoModuleID
— A struct
type that stores partial information to identify a Go module.pkg/vcs.Kind
— A struct
type that defines the kind of a pkg/vcs.Repository
.pkg/vcs.Repository
— A interface
type to represents a VCS repository that provides methods to receive repository information:
Kind() Kind
— returns the repository pkg/vcs.Kind
.DeriveVersion() error
— derives the repository version based on the pkg/vcs.Kind
.Version() interface{}
— returns the repository version.pkg/vcs/git.Git
— A struct
type that implements pkg/vcs.Repository
to represent a Git repository.pkg/vcs/git.Version
— A struct
type that stores version information and metadata derived from a Git repository.pkg/vcs/none.None
— A struct
type that implements pkg/vcs.Repository
to represent a nonexistent repository.Update to tmpl-go
version 0.9.0
which…
golangci-lint
version 1.43.0
— new linters are introduced and configurations of already supported ones are improved or added.1.17
.ci
workflow has been optimized by splitting it into new ci-go
and ci-node
workflows.tmpl
template repository version 0.10.0
.See the full tmpl-go
version 0.9.0
changelog for all details.
Before switching the GitHub repository visibility to “public“ a few adjustments need to be made.
This includes a change for referenced SVG images in documentations, like e.g. the repository hero, to use the URLs for public repositories instead of the ones that allow to resolve the files in private repositories.
-https://github.com/svengreb/wand/blob/main/assets/images/repository-hero.svg?raw=true
+https://raw.githubusercontent.com/svengreb/wand/main/assets/images/repository-hero.svg?sanitize=true
Update to "tmpl-go" version 0.7.0 which comes with updates to GitHub Actions and Node development dependencies.
In #1 this repository was bootstrapped from the "tmpl-go" template repository, but the copyright start year was not adjusted to the project's actual year 2019.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.