svengreb / wand Goto Github PK

Currently some functions that create a Task require an app.Config struct, but most tasks do not use the data in any way. To improve the code quality and simplify the internal usage of tasks these parameters will be removed as well as the field from the structs that implement the Task interfaces.

Update to tmpl-go version 0.10.0 which…

1. disables golangci-lint's default excluded issues — this prevents that explicitly enabled rules are not ignored due to the default set of excluded issues.
2. caches Go dependencies and build outputs in ci-go workflow — this improves the workflow execution time.

See the full tmpl-go version 0.10.0 changelog for all details.

Update to tmpl-go version 0.11.0 which…

1. fixed golangci-lint running errors due to revives unknown time-equal rule.
2. disabled the revive linter rule package-comments.

This will also include changes required for any linter matches.
See the full tmpl-go version 0.10.1 changelog for all details.

In #10 two types were implemented to derive and handle version information of a Git repository:

• 🆃 pkg/vcs/git.Version — A struct type that stores version information and metadata derived from a Git repository.
• 🆃 pkg/vcs/git.deriveVersion — A func type that derives version information and metadata from a Git repository.

The logic has been extracted into the pkg/vcs/git package of the github.com/svengreb/golib module and therefore both types will be replaced with this new module.

To run the go build command of the Go toolchain, a new spell.Incantation will be implemented in a new build package that can be used through a Go toolchain caster.
The spell incantation will be configurable through the following functions:

• WithBinaryArtifactName(name string) build.Option — sets the name for the binary build artifact.
• WithCrossCompileTargetPlatforms(platforms ...string) build.Option — sets the names of cross-compile platform targets.
• WithFlags(flags ...string) build.Option — sets additional flags to pass to the Go build command along with the base Go flags.
• WithGoOptions(goOpts ...spellGo.Option) build.Option — sets shared Go toolchain commands options.
• WithOutputDir(dir string) build.Option — sets the output directory, relative to the project root, for compilation artifacts.

To unify further implementations for the Go toolchain, a new struct type will also be implemented in the golang package that stores global/shared Go toolchain options that are shared between multiple Go toolchain commands:

• WithAsmFlags(asmFlags ...string) golang.Option — sets flags to pass on each go tool asm invocation.
• WithRaceDetector(enableRaceDetector bool) golang.Option — indicates if the race detector should be enabled.
• WithTrimmedPath(enableTrimPath bool) golang.Option — indicates if all file system paths should be removed from the resulting executable.
• WithEnv(env map[string]string) golang.Option — adds or overrides Go toolchain command specific environment variables.
• WithFlags(flags ...string) golang.Option — sets additional Go toolchain command flags.
• WithFlagsPrefixAll(flagsPrefixAll bool) golang.Option — indicates if the values of -asmflags and -gcflags should be prefixed with the all= pattern in order to apply to all packages.
• WithGcFlags(gcFlags ...string) golang.Option — sets flags to pass on each go tool compile invocation.
• WithLdFlags(ldFlags ...string) golang.Option — sets flags to pass on each go tool link invocation.
• WithMixins(mixins ...spell.Mixin) golang.Option — sets spell.Mixins that can be applied by option consumers.
• WithTags(tags ...string) golang.Option — sets Go toolchain tags.

A new CompileFormula(opts ...Option) []string function will be provided that can be used to compile the formula for these options.

In #1 this repository was bootstrapped from the "tmpl-go" template repository, but the copyright start year was not adjusted to the project's actual year 2019.

Most of the GoModule tasks using an outdated default Go module version so the following tasks will be updated and adjusted to the currently latest versions:

1. mvdan.cc/gofumpt — The github.com/svengreb/wand/pkg/task/gofumpt task currently uses version v0.1.1 and will be updated to version 0.2.0 by…
   1.1 removing the -r flag which has been removed in favor of gofmt -r.
   1.2 removing the -s flag (WithSimplify option) as it is always enabled.
2. golang.org/x/tools/cmd/goimports — The github.com/svengreb/wand/pkg/task/goimports task currently uses version v0.1.0 and will be updated to version 0.1.7.
3. github.com/golangci/golangci-lint/cmd/golangci-lint — The github.com/svengreb/wand/pkg/task/golangcilint task currently uses version v1.39.0 and will be updated to version 1.43.0. The configuration has already been updated in #104.

As of Go version 1.16 go install $pkg@$version allows to install commands without affecting the main module. Additionally commands like go build and go test no longer modify go.mod and go.sum files by default but report an error if a module requirement or checksum needs to be added or updated (as if the -mod=readonly flag were used).
This can be used as alternative to the already existing gobin runner.

To support the go install command of the Go toolchain, a new Task will be implemented in a new install package that can be used through a Go toolchain Runner.
The task will be customizable through the following functions:

• WithEnv(env map[string]string) install.Option — sets the task specific environment.
• WithModulePath(path string) install.Option — sets the module import path.
• WithModuleVersion(version *semver.Version) install.Option — sets the module version.

The github.com/markbates/pkger Go module provides the pkger command, a tool for embedding static files into Go binaries.

To configure and run the pkger command, a new task.GoModule will be implemented in a new pkger package that can be run using the gobin command runner or any other command runner that handles tasks of kind KindGoModule.

The task will be customizable through the following functions:

• WithEnv(env map[string]string) pkger.Option — sets the task specific environment.
• WithExtraArgs(extraArgs ...string) pkger.Option — sets additional arguments to pass to the command.
• WithIncludes(includes ...string) pkger.Option — adds the relative paths of files and directories that should be included.
  By default the paths will be detected by pkger itself when used within any of the packages of the target Go module.
• WithModulePath(path string) pkger.Option — sets the module import path.
• WithModuleVersion(version *semver.Version) pkger.Option — sets the module version.

The “elder“ reference implementation will provide a new Pkger method including the handling of the “monorepo“ workaround.

Official “Static Assets Embedding“

Please note that the pkger project might be superseded and discontinued due to the official Go toolchain support for embedding static assets (files) that will most probably be released with Go version 1.16.

Please see the official draft document and markbates/pkger#114 for more details.

“Monorepo“ Workaround

pkger tries to mimic the Go standard library and the way how the Go toolchain handles modules, but is therefore also affected by its problems and edge cases.
When the pkger command is used from the root of a Go module repository, the directory where the go.mod file is located, and there is no valid Go source file, the command will fail because it internally uses the same logic like the list command of the Go toolchain (go list).
Therefore a “dummy“ Go source file may need to be created as a workaround. This is mostly only required for repositories that use a “monorepo“ layout where one or more main packages are placed in a subdirectory relative to the root directory, e.g. apps or cmd. For repositories where the root directory already has a Go package, that does not contain any build constraints/tags, or uses a “library“ layout, a “dummy“ file is probably not needed.

The new Pkger method of the “elder“ reference implementation will handle the creation of a temporary “dummy“ file that will be deleted automatically when the tasks finishes in order to avoid the need for the user to add such a file to the repository and commit it into the VCS.

Please see markbates/pkger#109 and markbates/pkger#121 for more details.

There are currently no OS specific tests so the test workflow job should only run on Linux. The additional run on Windows is currently unnecessary and only wastes build minutes because each minute is calculated 2x when running on Windows. See the section about “Minute multipliers“ in the official GitHub Action Pricing documentation.

Next to this it also helps to improve the performance since workflows running on Windows are currently taking the most time.

Superseded by #87

As of Go 1.16 gobin has officially been deprecated since go install $pkg@$version allows to install commands without affecting the main module. See myitcv/gobin#103 for more details about the gobin deprecation and #22 for more information about the gobin task runner.

Support for the new go install features have already been added in #71 as an alternative to gobin, but this method still lacks an essential feature: install module/package executables globally without overriding already installed executables of different versions.
Since go install will always place compiled binaries in $GOBIN (or whatever is set for go env GOBIN), any already existing binary with the same name will be replaced. So installing a module command with two different versions is still not possible with go install and still messes up the local environment.

This and other well-known problems are tackled by bingo, a project similar to gobin. It comes with many great features and allows to manage development tools on a per-module basis, almost like the mechanism of Node‘s node_modules directory.
Replacing the task runner that uses the deprecated gobin project with a one that uses bingo under the hood will bring back the features missing from go install while also adding many more goodies to simplify and improve development setups.

Before switching the GitHub repository visibility to “public“ a few adjustments need to be made.
This includes a change for referenced SVG images in documentations, like e.g. the repository hero, to use the URLs for public repositories instead of the ones that allow to resolve the files in private repositories.

-https://github.com/svengreb/wand/blob/main/assets/images/repository-hero.svg?raw=true
+https://raw.githubusercontent.com/svengreb/wand/main/assets/images/repository-hero.svg?sanitize=true

Update to tmpl-go version 0.5.0 (including version 0.4.0) that…

1. …introduces the initial project documentation.
2. …updates golangci-lint to the currently latest version 1.32.0 which introduces new linters like errorlint., tparallel. and wrapcheck.
3. …updates to "tmpl" version 0.7.0 (#25 #34).
   This includes…
   • …a new configuration file for automated dependency updates and security alerts with Dependabot. Next to update configurations for the CI/CD GitHub action workflow and Yarn/NPM dependencies, the file has been extended to support Go modules.
   • …updates to the latest Node.js package dependency & GitHub Action versions.
   • …a change of the NPM package name to use a namespace which helps to prevent collisions with already existing NPM packages like tmpl.

In #79 a new ExecName() string method has been added to the Exec interface to improve the way how RunnerExec handles the name of the executable file, but it turns out that this is not useful at all and also breaks the separation of concern between the Task and Runner interfaces.
Therefore the method will be removed again.

Bootstrap the basic project setup, structure and development workflow from version 0.3.0 of the “tmpl-go“ template repository.
Additionally specific assets like the repository hero image will be replaced and documentations like the README and GitHub issue/PR templates will be adjusted.

Update to tmpl-go version 0.12.0 which…

1. updated to the tmpl template repository version 0.11.0.

This will also include changes required for any linter matches.
See the full tmpl-go version 0.12.0 changelog for all details.

Update to tmpl-go version 0.8.0 which updates golangci-lint to version 1.39.0 and the tmpl repository version 0.9.0.

In #52 a task for the github.com/markbates/pkger Go module was added, a tool for embedding static files into Go binaries.
The issue also includes the “Official Static Assets Embedding“ section which mentions that the task might be removed later on again as soon as Go 1.16 will be released as it comes with toolchain support for embedding static assets (files) through the embed package. Also see markbates/pkger#114 for more details about the project future of pkger.

The pkger package will be removed and the //go:embed directive should be used instead.

Update to "tmpl-go" version 0.6.0 which reduces Dependabot PR noise for the NPM package ecosystem.

Currently most tasks provide a TaskName package constant that contains the name of the task, but this is not an idiomatic and consistent way. To make sure that this information is part of the API, a new Name() string method will be added to the Task interface.

To also improve the way how RunerExec handles the name of the executable file, a new ExecName() string method will be added to the Exec interface.

In #9 the store and configuration for applications has been implemented. In wand applications are not standalone but part of a project which in turn is stored in a repository of a VCS like Git. In case of wand this can also be a monorepo to manage multiple applications, but there is always only a single project which all these applications are part of.
To store project and VCS repository information, some of the newly implemented packages will provide the following types:

• 🆃 pkg/project.Metadata — A struct type that stores information and metadata of a project.
• 🆃 pkg/project.GoModuleID — A struct type that stores partial information to identify a Go module.
• 🆃 pkg/vcs.Kind — A struct type that defines the kind of a pkg/vcs.Repository.
• 🅸 pkg/vcs.Repository — A interface type to represents a VCS repository that provides methods to receive repository information:
  • Kind() Kind — returns the repository pkg/vcs.Kind.
  • DeriveVersion() error — derives the repository version based on the pkg/vcs.Kind.
  • Version() interface{} — returns the repository version.
• 🆃 pkg/vcs/git.Git — A struct type that implements pkg/vcs.Repository to represent a Git repository.
• 🆃 pkg/vcs/git.Version — A struct type that stores version information and metadata derived from a Git repository.
• 🆃 pkg/vcs/none.None — A struct type that implements pkg/vcs.Repository to represent a nonexistent repository.

Currently most Task creation functions [¹]¹ [²]² [³]³ [⁴]⁴ require a Wand as parameter which is not used but blocks the internal usage for task runners. Therefore these parameters will be removed. When necessary, it can be added individually later on or can be reintroduced through a dedicated function with extended parameters to cover different use cases.

To support common use cases for debugging and production optimization, some spell mixins will be implemented in the golang package:

• 🆂 MixinImproveDebugging — A struct type that adds linker flags to improve the debugging of binary artifacts. This includes the disabling of inlining and all compiler optimizations tp improve the compatibility for debuggers.
  Note that this mixin will add the all prefix for —gcflags parameters to make sure all packages are affected. If you disabled the all prefix on purpose you need to handle this conflict on your own, e.g. by creating more than one binary artifact each with different build options.
• 🆂 MixinImproveEscapeAnalysis — A struct type that will add linker flags to improve the escape analysis of binary artifacts.
  Note that this mixin removes the all prefix for —gcflags parameters to make sure only the target package is affected, otherwise reports for (traverse) dependencies would be included as well. If you enabled the all prefix on purpose you need to handle this conflict on your own, e.g. by creating more than one binary artifact each with different build options.
• 🆂 MixinStripDebugMetadata — A struct type that will add linker flags to strip debug information from binary artifacts. This will include DWARF tables needed for debuggers, but keeps annotations needed for stack traces so panics are still readable. It will also shrink the file size and memory overhead as well as reducing the chance for possible security related problems due to enabled development features or debug information leaks.
  Note that this mixin will add the all prefix for —gcflags parameters to make sure all packages are affected. If you disabled the all prefix on purpose you need to handle this conflict on your own, e.g. by creating more than one binary artifact each with different build options.
• 🆂 MixinInjectBuildTimeVariableValues — A struct type that will inject build—time values through the —X linker flags to populate e.g. application metadata variables.
  It will store a map[string]string of key/value pairs to inject to variables at build—time. The key must be the path to the variable in form of <IMPORT_PATH>.<VARIABLE_NAME>, e.g. pkg/internal/support/app.version. The value is the actual value that will be assigned to the variable, e.g. the application version.
  A field of type *project.GoModuleID will store partial information about the target Go module to inject the key/value pairs from the data map into.

To clean paths in a filesystem, like application specific output directories, a new GoCode spell incantation will be implemented which can be used without a caster.

The spell incantation will provide the following methods:

• Clean() ([]string, error) — removes the configured paths. It returns an error of type *spell.ErrGoCode for any error that occurs during the execution of the Go code.

The spell incantation will be customizable through the following functions:

• WithLimitToAppOutputDir(limitToAppOutputDir bool) clean.Option — indicates whether only paths within the configured application output directory should be allowed.
• WithPaths(paths ...string) clean.Option — sets the paths to remove. Note that only paths within the configured application output directory are allowed when WithLimitToAppOutputDir is enabled.

The github.com/oligot/go-mod-upgrade Go module provides the go-mod-upgrade command, a too to update outdated Go module dependencies interactively.

To configure and run the go-mod-upgrade command, a new task.GoModule will be implemented in a new gomodupgrade package. It can be be run using a command runner that handles tasks of kind KindGoModule.

The task will be customizable through the following functions:

• WithEnv(map[string]string) gomodupgrade.Option — sets the task specific environment.
• WithExtraArgs(...string) gomodupgrade.Option — sets additional arguments to pass to the command.
• WithModulePath(string) gomodupgrade.Option — sets the module import path.
• WithModuleVersion(*semver.Version) gomodupgrade.Option — sets the module version.

The Elder reference implementation will provide a new GoModUpgrade method.

This is issue supersedes #78 which documents how the official deprecation of gobin in favor of the new Go 1.16 go install pkg@version syntax feature should have been handled for this project. The idea was to replace the gobin task runner with a one that leverages bingo, a project similar to gobin, that comes with many great features and also allows to manage development tools on a per-module basis. The problem is that bingo uses some non-default and nontransparent mechanisms under the hood and automatically generates files in the repository without the option to disable this behavior. It does not make use of the go install command but relies on custom dependency resolution mechanisms, making it prone to future changes in the Go toolchain and therefore not a good choice for the maintainability of projects.

go install is still not perfect

Support for the new go install features, which allow to install commands without affecting the main module, have already been added in #71 as an alternative to gobin, but one significant problem is still not addressed: install module/package executables globally without overriding already installed executables of different versions.
Since go install will always place compiled binaries in the path defined by go env GOBIN, any already existing executable with the same name will be replaced. It is not possible to install a module command with two different versions since go install still messes up the local user environment.

The Workaround: Hybrid go install task runner

The solution is to implement a custom Runner that uses go install under the hood, but places the compiled executable in a custom cache directory instead of go env GOBIN. The runner will check if the executable already exists, installs it if not so, and executes it afterwards.

The concept of storing dependencies locally on a per-project basis is well-known from the node_modules directory of the Node package manager npm. Storing executables in a cache directory within the repository (not tracked by Git) allows to use go install mechanisms while not affect the global user environment and executables stored in go env GOBIN. The runner will achieve this by changing the GOBIN environment variable to the custom cache directory during the execution of go install. This way it bypasses the need for “dirty hacks“ while using a custom output path.

The only known disadvantage is the increased usage of storage disk space, but since most Go executables are small in size anyway, this is perfectly acceptable compared to the clearly outweighing advantages.

Note that the runner dynamically runs executables based on the given task so Validate() error will be a NOOP.

Upcoming Changes

The solution described above will work totally fine, but is still not a clean solution that uses the Go toolchain without any special logic so as soon as the following changes are made to the Go toolchain (Go 1.17 or later), the custom runner will be removed again:

• golang/go/issues#42088 — tracks the process of adding support for the Go module syntax to the go run command. This will allow to let the Go toolchain handle the way how compiled executable are stored, located and executed.
• golang/go#44469 — tracks the process of making go install aware of the -o flag like the go build command which is the only reason why the custom runner will be implemented.

Further Adjustments

Because the new custom task runner dynamically runs executables based on the given task, the Bootstrap method of the Wand reference implementation Elder will additionally allow to pass Go module import paths, optionally including a version suffix (pkg@version), to install executables from Go module-based main packages into the local cache directory. This way the local development environment can be set up, for e.g. by running it as startup task in JetBrains IDEs.
The method will also ensure that the local cache directory exists and will create a .gitignore file that includes ignore pattern for the cache directory.

The golang.org/x/tools/cmd/goimports Go module allows to update Go import lines, adding missing ones and removing unreferenced ones. It also formats code in the same style as gofmt so it can be used as a replacement. The source code for the goimports command can be found in the golang/tools repository.

To configure and run the goimports command, a new spell.Incantation will be implemented in a new goimports package that can be casted using the gobin caster or any other spell caster that handles spell incantations of kind KindGoModule.

The spell incantation will be customizable through the following functions:

• WithEnv(env map[string]string) goimports.Option — sets the spell incantation specific environment.
• WithExtraArgs(extraArgs ...string) goimports.Option — sets additional arguments to pass to the goimports command.
• WithListNonCompliantFiles(listNonCompliantFiles bool) goimports.Option — indicates whether files, whose formatting are not conform to the style guide, are listed.
• WithLocalPkgs(localPkgs ...string) goimports.Option — sets local packages whose imports will be placed after 3rd-party packages.
• WithModulePath(path string) goimports.Option — sets the goimports module import path. Defaults to goimports.DefaultGoModulePath.
• WithModuleVersion(version *semver.Version) goimports.Option — sets the goimports module version. Defaults to goimports.DefaultGoModuleVersion.
• WithPaths(paths ...string) goimports.Option — sets the paths to search for Go source files. By default all directories are scanned recursively starting from the current working directory.
• WithPersistedChanges(persistChanges bool) goimports.Option — indicates whether results are written to the source files instead of standard output.
• WithReportAllErrors(reportAllErrors bool) goimports.Option — indicates whether all errors should be printed instead of only the first 10 on different lines.
• WithVerboseOutput(verbose bool) goimports.Option — indicates whether the output should be verbose.

The project currently only uses GitHub Action workflows for CI but not Mage to automate tasks for itself though.
Following the “dogfooding“ concept Mage will finally be added to the repository, using wand as toolkit through the Elder wand reference implementation.

To support the go env command of the Go toolchain, a new Task will be implemented in a new env package that can be used through a Go toolchain Runner.
The task will be customizable through the following functions:

• WithEnv(env map[string]string) env.Option — sets the task specific environment.
• WithEnvVars(envVars ...string) env.Option — sets the names of the target environment variables.
• WithExtraArgs(extraArgs ...string) env.Option — sets additional arguments to pass to the command.

To use the Go toolchain, also known as the go command, a new caster, introduced in #14, will be implemented.
To unify the handling of errors in the cast package, a new ErrCast struct type will also be implemented.

The Validate function of the new caster will return an error of type *cast.ErrCast when the go binary executable does not exists at the configured path or when it is also not available in the executable search paths of the current environment.

Write the initial project documentation that includes…

1. …an overview of the project features.
2. …information about the project motivation:
   1. “Why should I use Mage…“
   2. “…and why wand?“
3. …the project design decisions and how to use it:
   1. The overall wording and inspiration.
   2. A basic overview of the API.
   3. An introduction to the “elder“ reference implementation.
4. …information about how to contribute to this project.

To run the go test command of the Go toolchain, a new spell.Incantation will be implemented in a new test package that can be used through a Go toolchain caster.
The spell incantation will be customizable through the following functions:

• WithBlockProfileOutputFileName(blockProfileOutputFileName string) test.Option — sets the file name for the Goroutine blocking profile file.
• WithCoverageProfileOutputFileName(coverageProfileOutputFileName string) test.Option — sets the file name for the test coverage profile file.
• WithCPUProfileOutputFileName(cpuProfileOutputFileName string) test.Option — sets the file name for the CPU profile file.
• WithBlockProfile(withBlockProfile bool) test.Option — indicates if the tests should be run with a Goroutine blocking profiling.
• WithCoverageProfile(withCoverageProfile bool) test.Option — indicates if the tests should be run with coverage profiling.
• WithCPUProfile(withCPUProfile bool) test.Option — indicates if the tests should be run with CPU profiling.
• WithFlags(flags ...string) test.Option — sets additional flags that are passed to the Go "test" command along with the shared Go flags.
• WithGoOptions(goOpts ...spellGo.Option) test.Option — sets shared Go toolchain command options.
• WithMemProfile(withMemProfile bool) test.Option — indicates if the tests should be run with memory profiling.
• WithMemoryProfileOutputFileName(memoryProfileOutputFileName string) test.Option — sets the file name for the memory profile file.
• WithMutexProfile(withMutexProfile bool) test.Option — indicates if the tests should be run with mutex profiling.
• WithMutexProfileOutputFileName(mutexProfileOutputFileName string) test.Option — sets the file name for the mutex profile file.
• WithOutputDir(outputDir string) test.Option — sets the output directory, relative to the project root, for reports like coverage or benchmark profiles.
• WithoutCache(withoutCache bool) test.Option — indicates if the tests should be run without test caching that is enabled by Go by default.
• WithPkgs(pkgs ...string) test.Option — sets the list of packages to test.
• WithTraceProfile(withTraceProfile bool) test.Option — indicates if the tests should be run with trace profiling.
• WithTraceProfileOutputFileName(traceProfileOutputFileName string) test.Option — sets the file name for the execution trace profile file.
• WithVerboseOutput(withVerboseOutput bool) test.Option — indicates if the test output should be verbose.

The github.com/golangci/golangci-lint Go module provides the golangci-lint commands, a fast, parallel runner for dozens of Go linters Go that uses caching, supports YAML configurations and has integrations with all major IDEs. The source code for the golangci-lint command can be found in the golangci/golangci-lint repository.

To configure and run the golangci-lint command, a new spell.Incantation will be implemented in a new golangcilint package that can be casted using the gobin caster or any other spell caster that handles spell incantations of kind KindGoModule.

The spell incantation will be customizable through the following functions:

• WithArgs(args ...string) golangcilint.Option — sets additional arguments to pass to the golangci-lint module command.
• WithEnv(env map[string]string) golangcilint.Option — sets the spell incantation specific environment.
• WithModulePath(path string) golangcilint.Option — sets the golangci-lint module command import path. Defaults to golangcilint.DefaultGoModulePath.
• WithModuleVersion(version *semver.Version) golangcilint.Option — sets the golangci-lint module version. Defaults to golangcilint.DefaultGoModuleVersion.
• WithVerboseOutput(verbose bool) golangcilint.Option — indicates whether the output should be verbose.

The mvdan.cc/gofumpt Go module provides the gofumpt command, a tool that enforces a stricter format than gofmt and provides additional rules, while being backwards compatible. It is a modified fork of gofmt so it can be used as a drop-in replacement.

To configure and run the gofumpt command, a new task.GoModule will be implemented in a new gofumpt package that can be run using the gobin command runner or any other command runner that handles tasks of kind KindGoModule.

The task will be customizable through the following functions:

• WithEnv(map[string]string) gofumpt.Option — sets the task specific environment.
• WithExtraArgs(...string) gofumpt.Option — sets additional arguments to pass to the command.
• WithExtraRules(bool) gofumpt.Option — indicates whether gofumpt‘s extra rules should be enabled. See the repository documentation for a listing of available rules.
• WithListNonCompliantFiles(bool) gofumpt.Option — indicates whether files, whose formatting are not conform to the style guide, are listed.
• WithModulePath(string) gofumpt.Option — sets the module import path.
• WithModuleVersion(*semver.Version) gofumpt.Option — sets the module version.
• WithPaths(...string) gofumpt.Option — sets the paths to search for Go source files. By default all directories are scanned recursively starting from the current working directory.
• WithReportAllErrors(bool) gofumpt.Option — indicates whether all errors should be printed instead of only the first 10 on different lines.
• WithSimplify(bool) gofumpt.Option — indicates whether code should be simplified.

The “elder“ reference implementation will provide a new Gofumpt method.

When the WithRunnerQuiet option of the golang runner or WithQuiet option of the gotool runner is set to true, the task passed to their Run method will run twice. This is caused by a missing return statement when the execution finishes with a error of value nil ^{1 2}.

To fix this bug both code blocks must return early with nil instead of keep going on in the code flow.

As of Go 1.18 the debug.ReadBuildInfo function does not work for Mage executables anymore because the way how module information is stored changed. Therefore the fields of the returned debug.Module type only has zero values, including the module path. The debug.Module.Version field has a default value ((devel)) which is not Semver compatible and causes the parsing to fail. The change in Go 1.18 also came with the new debug/buildinfo package which allows to read the information from compiled binaries while the runtime/debug.ReadBuildInfo function returns information from within the running binary. Both are not suitable anymore which is also described in the Go 1.18 version command release notes:

The underlying data format of the embedded build information can change with new go releases, so an older version of go may not handle the build information produced with a newer version of go. To read the version information from a binary built with go 1.18, use the go version command and the debug/buildinfo package from go 1.18+.

To get the required module information that was previously provided by the runtime/debug package the official golang.org/x/mod/modfile package will be used instead that provides the implementation for a parser and formatter for go.mod files ¹. This allows to safely get the module path without the need to depend on runtime/dynamic logic that might change in future Go versions.

Note that this change will also increase the minimum Go version from 1.17 to 1.19!

Currently the Install method of the gobin task runner sets the environment of the command that gets executed initially to os.Environ(), but overrides it later on with custom variables configured through the WithEnv(map[string]string) option.

This results in errors when running the method in minimal environments like containers:

build cache is required, but could not be located: GOCACHE is not defined and neither $XDG_CACHE_HOME nor $HOME are defined

This can be fixed by ensuring that the inherited OS environment is prepended before applying custom environment variables.
Also to improve the debugging process the combined output (stdout + stderr) should be included in the error when the command execution fails.

Go Executable Installation

When installing a Go executable from within a Go module directory using the go install command, it is installed into the Go executable search path that is defined through the GOBIN environment variable and can also be shown and modified using the go env command. Even though the executable gets installed globally, the go.mod file will be updated to include the installed packages since this is the default behavior of the go get command when running in module mode.

Next to this problem, the installed executable will also overwrite any executable of the same module/package that was installed already, but maybe from a different version. Therefore only one version of a executable can be installed at a time which makes it impossible to work on different projects that use the same tool but with different versions.

History & Future

The local installation of executables built from Go modules/packages has always been a somewhat controversial point which unfortunately, partly for historical reasons, does not offer an optimal and user-friendly solution up to now. The go command is a fantastic toolchain that provides many great features one would expect to be provided out-of-the-box from a modern and well designed programming language without the requirement to use a third-party solution: from compiling code, running unit/integration/benchmark tests, quality and error analysis, debugging utilities and many more.
Unfortunately the way the go install command of Go versions less or equal to 1.15 handles the installation of an Go module/package executable is still not optimal.

The general problem of tool dependencies is a long-time known issue/weak point of the current Go toolchain and is a highly rated change request from the Go community with discussions like golang/go#30515, golang/go#25922 and golang/go#27653 to improve this essential feature, but they‘ve been around for quite a long time without a solution that works without introducing breaking changes and most users and the Go team agree on.
Luckily, this topic was finally picked up for the next upcoming Go release version 1.16 and gh-golang/go#40276 introduces a way to install executables in module mode outside a module. The release note preview also already includes details about this change and how installation of executables from Go modules will be handled in the future.

The Workaround

Beside the great news and anticipation about an official solution for the problem the usage of a workaround is almost inevitable until Go 1.16 is finally released.

The official Go wiki provides a section on “How can I track tool dependencies for a module?” that describes a workaround that tracks tool dependencies. It allows to use the Go module logic by using a file like tools.go with a dedicated tools build tag that prevents the included module dependencies to be picked up included for normal executable builds. This approach works fine for non-main packages, but CLI tools that are only implemented in the main package can not be imported in such a file.

In order to tackle this problem, a user from the community created gobin, an experimental, module-aware command to install/run main packages.
It allows to install or run main-package commands without “polluting“ the go.mod file by default. It downloads modules in version-aware mode into a binary cache path within the systems cache directory.
It prevents problems due to already globally installed executables by placing each version in its own directory. The decision to use a cache directory instead of sub-directories within the GOBIN path keeps the system clean.

gobin is still in an early development state, but has already received a lot of positive feedback and is used in many projects. There are also members of the core Go team that have contributed to the project and the chance is high that the changes for Go 1.16 were influenced or partially ported from it.
It is currently the best workaround to…

1. …prevent the Go toolchain to pick up the GOMOD environment variable (see go env GOMOD) that is initialized automatically with the path to the go.mod file in the current working directory.
2. …install module/package executables globally without “polluting“ the go.mod file.
3. …install module/package executables globally without overriding already installed executables of different versions.

See gobin‘s FAQ page in the repository wiki for more details about the project.

The Go Module Caster

To allow to manage the tool dependency problem, wand, will use gobin through a new caster that prevents the “pollution“ of the project go.mod file and allows to…

1. …install gobin itself into GOBIN (go env GOBIN).
2. …cast any spell incantation of kind KindGoModule by installing the executable globally into the dedicated gobin cache.

The default way to use the wand API, with its casters and spells, will be the reference implementation “elder“.
It will provide a way to use all wand spells and additionally comes with helper methods to bootstrap a project, validate all casters and simplify logging for process exits:

• Bootstrap() error — runs initialization tasks to ensure the wand is operational. This includes the installation of configured caster like cast.BinaryCaster that can handle spell incantations of kind spell.KindGoModule.
• Clean(appName string, opts ...clean.Option) ([]string, error) — a spell.GoCode to remove configured filesystem paths, e.g. output data like artifacts and reports from previous development, test, production and distribution builds. It returns paths that have been cleaned along with an error of type *spell.ErrGoCode when an error occurred during the execution of the Go code. When any error occurs it will be of type *app.ErrApp or *cast.ErrCast. See the clean package for all available options.
• ExitPrintf(code int, verb nib.Verbosity, format string, args ...interface{}) — simplifies the logging for process exits with a suitable nib.Verbosity.
• GetAppConfig(name string) (app.Config, error) — returns an application configuration. An empty application configuration is returned along with an error of type *app.ErrApp when there is no configuration in the store for the given name.
• GetProjectMetadata() project.Metadata — returns metadata of the project.
• GoBuild(appName string, opts ...build.Option) — casts the spell incantation for the build command of the Go toolchain. When any error occurs it will be of type *app.ErrApp or *cast.ErrCast. See the build package for all available options.
• Goimports(appName string, opts ...goimports.Option) error — casts the spell incantation for the golang.org/x/tools/cmd/goimports Go module command that allows to update Go import lines, add missing ones and remove unreferenced ones. It also formats code in the same style as gofmt command so it can be used as a replacement. When any error occurs it will be of type *app.ErrApp or *cast.ErrCast.
  See the goimports package for all available options. For more details about goimports see the module documentation. The source code of goimports is available in the GitHub repository.
• GolangCILint(appName string, opts ...golangcilint.Option) error — casts the spell incantation for the github.com/golangci/golangci-lint/cmd/golangci-lint Go module command, a fast, parallel runner for dozens of Go linters Go that uses caching, supports YAML configurations and has integrations with all major IDEs. When any error occurs it will be of type *app.ErrApp or *cast.ErrCast. See the golangcilint package for all available options.
  For more details about golangci-lint see the module documentation and the official website. The source code of golangci-lint is available in the GitHub repository.
• GoTest(appName string, opts ...spellGoTest.Option) error — casts the spell incantation for the test command of the Go toolchain. When any error occurs it will be of type *app.ErrApp or *cast.ErrCast. See the test package for all available options.
• Gox(appName string, opts ...spellGox.Option) error — casts the spell incantation for the github.com/mitchellh/gox Go module command, a dead simple, no frills Go cross compile tool that behaves a lot like the standard Go toolchain build command. When any error occurs it will be of type *app.ErrApp or *cast.ErrCast. See the gox package for all available options.
  For more details about gox see the module documentation. The source code of gox is available in the GitHub repository.
• RegisterApp(name, displayName, pathRel string) error — creates and stores a new application configuration. Note that the package path must be relative to the project root directory!
  It returns an error of type *app.ErrApp when the application path is not relative to the project root directory, when it is not a subdirectory of it or when any other error occurs.
• Validate() error — ensures that all casters are properly initialized and available. It returns an error of type *cast.ErrCast when the validation of any of the supported casters fails.
• New(opts ...Option) (*Elder, error) — creates a new elder wand.
  The module name is determined automatically using the runtime/debug package. The absolute path to the root directory is automatically set based on the current working directory. Note that the working directory must be set manually when the “magefile“ is not placed in the root directory by pointing Mage to it:
  • -d <PATH> option to set the directory from which “magefiles“ are read (defaults to .).
  • -w <PATH> option to set the working directory where “magefiles“ will run (defaults to value of -d flag).
    If any error occurs it will be of type *cast.ErrCast or *project.ErrProject.

The wand will be customizable through the following functions:

• WithGobinCasterOptions(opts ...castGobin.Option) elder.Option — sets “gobin“ caster options.
• WithGoToolchainCasterOptions(opts ...castGoToolchain.Option) elder.Option — sets Go toolchain caster options.
• WithNib(n nib.Nib) elder.Option — sets the log-level based line printer for human-facing messages.
• WithProjectOptions(opts ...project.Option) elder.Option — sets project options.

Update to tmpl-go version 0.9.0 which…

1. updated to golangci-lint version 1.43.0 — new linters are introduced and configurations of already supported ones are improved or added.
2. updated the Go module to Go 1.17.
3. optimized the GitHub action workflows for Go and Node — the ci workflow has been optimized by splitting it into new ci-go and ci-node workflows.
4. updated to the tmpl template repository version 0.10.0.

See the full tmpl-go version 0.9.0 changelog for all details.

Currently the Run method of the Runner interface allows to run a command, but does not return its output. This can be blocking when running commands like go env GOBIN to get the path to the GOBIN environment variable.

To support such uses cases, a new RunOut(Task) (string, error) method will be added to the Runner interface that runs a command and returns its output.

With #14 the “abstract“ wand API was introduced with a naming scheme is inspired by the fantasy novel “Harry Potter“ that was used to to define interfaces.
The main motivation was to create a matching naming to the overall “magic“ topic and the actual target project Mage, but in retrospect this is way too abstract and confusing.

The goal of this ticket is to…

• rewrite the API to make it way easier to use.
• use a “normal“ naming scheme.
• improve all documentations to be more user-scoped and provide guides and examples.

New API Concept

The basic mindset of the API will remain partially the same, but it will be designed around the concept of tasks and the ways to run them.

Command Runner

🅸 task.Runner is a new base interface that runs a command with parameters in a specific environment. It can be compared to the current 🅸 cast.Caster interface, but provides a cleaner method set accepting the new 🅸 task.Task interface.

• 🅼 Handles() task.Kind — returns the supported task kind.
• 🅼 Run(task.Task) error — runs a command.
• 🅼 Validate() error — validates the runner.

The new 🅸 task.RunnerExec interface is a specialized task.Runner and serves as an abstract representation for a command or action, in most cases a (binary) executable of external commands or Go module main packages, that provides corresponding information like the path to the executable. It can be compared to the current BinaryCaster interface, but also comes with a cleaner method set and a more appropriate name.

• 🅼 FilePath() string — returns the path to the (binary) command executable.

Tasks

🅸 task.Task is the new interface that is scoped for Mage “target“ usage. It can be compared to the current 🅸 spell.Incantation interface, but provides a smaller method set without Formula() []string.

• 🅼 Kind() task.Kind — returns the task kind.
• 🅼 Options() task.Options — returns the task options.

The new 🅸 task.Exec interface is a specialized task.Task and serves as an abstract task for an executable command. It can be compared to the current Binary interface, but also comes with the new BuildParams() []string method that enables a more flexible usage by exposing the parameters for command runner like task.RunnerExec and also allows to compose with other tasks. See the Wikipedia page about the anatomy of a shell CLI for more details about parameters.

• 🅼 BuildParams() []string — builds the parameters for a command runner where parameters can consist of options, flags and arguments.
• 🅼 Env() map[string]string — returns the task specific environment.

The new 🅸 task.GoModule interface is a specialized task.Exec for a executable Go module command. It can be compared to the current spell.GoModule interface and the method set has not changed except a renaming of the GoModuleID() *project.GoModuleID to the more appropriate name ID() *project.GoModuleID. See the official Go module reference documentation for more details about Go modules.

• 🅼 ID() *project.GoModuleID — returns the identifier of a Go module.

New API Naming Scheme

The following listing shows the new name concept and how the current API components can be mapped to the upcoming changes:

1. Runner — A component that runs a command with parameters in a specific environment, in most cases a (binary) executable of external commands or Go module main packages. The current API component that can be compared to runners is 🅸 cast.Caster and its specialized interfaces.
2. Tasks — A component that is scoped for Mage “target“ usage in order to run a action. The current API component that can be compared to tasks is 🅸 spell.Incantation and its specialized interfaces.

API Usage

Even though the API will be changed quite heavily, the basic usage almost won't change.

→ A task.Task can only be run through a task.Runner!

Currently a spell.Incantation is passed to a cast.Caster in order to run it, in most cases a (binary) executable of a command that uses the Formula() []string method of spell.Incantation to pass the result as parameters.
The new API works the same: A task.Task is passed to a task.Runner that calls the BuildParams() []string method when the runner is specialized for (binary) executable of commands.

Improved Documentations

The current documentation is mainly scoped on the technical details, but lacks more user-friendly sections about topics like the way how to implement own API components, how to compose the “elder“ reference implementation or usage examples for single or monorepo project layouts

User Guide

Most of the current sections will be rewritten or removed entirely while new sections will provide more user-friendly guides about how to…

• use or compose the “elder“ reference implementation.
• build own tasks and runners using the new API.
• structure repositories independent of the layout, single or “monorepo“.

Usage Examples

Some examples will be added, that are linked and documented in the user guides described above, to show how to…

• use or compose the “elder“ reference implementation.
• build own tasks and runners using the new API.
• structure repositories independent of the layout, single or “monorepo“.

In #15 some parts of the wand API have been implemented in form of spell incantations, kinds and casters, inspired by the fantasy novel “Harry Potter“ as an abstract view to define interfaces. In #9 and #11 the API implementations for an application configuration store as well as project and VCS repository metadata were introduced.
These implementations will be usable in a combined form via the main wand API that will consist of the following types:

• 🅸 wand.Wand — A interface type that manages a project and its applications and stores their metadata. Applications are registered using a unique name and the stored metadata can be received based on this name:
  • GetAppConfig(appName string) (app.Config, error) — returns an application configuration.
  • GetProjectMetadata() project.Metadata — returns the project metadata.
  • RegisterApp(name, displayName, pathRel string) error — registers a new application.
• 🆃 wand.ctxKey — A struct type that serves as context key used to wrap a wand.Wand.
• 🅵 wand.GetCtxKey() interface{} — A func type that returns the key used to wrap a wand.Wand.
• 🅵 wand.WrapCtx(parentCtx context.Context, wand Wand) context.Context — A func type that wraps the given wand.Wand into the parent context. Use wand.GetCtxKey() interface{} to receive the key used to wrap the wand.Wand.

The wand API is inspired by the fantasy novel “Harry Potter“ and uses an abstract view to define interfaces. The main motivation to create a matching naming to the overall “magic“ topic and the actual target project Mage. This might be too abstract for some, but is kept understandable insofar as it should allow everyone to use the "task" API and to derive their own tasks from it.

• 🅸 cast.Caster — A interface type that casts a spell.Incantation using a command for a specific spell.Kind:
  • Cast(spell.Incantation) error — casts a spell incantation.
  • Handles() spell.Kind — returns the spell kind that can be casted.
  • Validate() error — validates the caster command.
• 🅸 cast.BinaryCaster — A interface type that composes cast.Caster to run commands using a binary executable:
  • GetExec() string — returns the path to the binary executable of the command.
• 🅸 spell.Incantation — A interface type that is the abstract representation of parameters for a command or action:
  • Formula() []string — returns all parameters of a spell.
  • Kind() Kind — returns the Kind of a spell.
  • Options() interface{} — return the options of a spell.
• 🅸 cast.Binary — A interface type that composes cast.Caster for commands which are using a binary executable:
  • Env() map[string]string — returns additional environment variables.
• 🅸 cast.GoCode — A interface type that composes cast.Caster for actions that can be casted without a cast.Caster:
  • Cast() (interface{}, error) — casts itself.
• 🅸 cast.GoModule — A interface type that composes cast.Binary for commands that are compiled from a Go module:
  • GoModuleID() *project.GoModuleID — returns the identifier of a Go module.
• 🆃 spell.Kind — A struct type that defines the kind of a spell.

The API components can be roughly translated to their purpose:

• cast.Caster → a executable command
  It validates the command and defines which spell.Kind can be handled by this caster. It could be executed without parameters (spell.Incantation), but in most cases needs at least one parameter.
  • cast.BinaryCaster → a composed cast.Caster to run commands using a binary executable.
    It ensures that the executable file exists and stores information like the path. It could also be executed without parameters (spell.Incantation), but would not have any effect im many cases.
• spell.Incantation → the parameters of a executable command
  It assemble all parameters based on the given options and ensures the they are correctly formatted for the execution in a shell environment. Except for special incantations like spell.GoCode a incantation cannot be used alone but must be passed to a cast.Caster that is able to handle the spell.Kind of this incantation.
  • spell.Binary → a composed spell.Incantation to run commands that are using binary executable.
    It can inject or override environment variables in the shell environment in which the the command will be run.
  • spell.GoCode → a composed spell.Incantation for pure Go code instead of a (binary) executable command.
    It can “cast itself“, e.g. to simply delete a directory using packages like os from the Go standard library. It has been designed this way to also allow such tasks to be handled by the incantation API.
  • spell.GoModule → a composed spell.Binary to run binary commands managed by a Go module, in other words executables installed in GOBIN or received via go get.
    It requires the module identifier (path@version) in order to download and run the executable.

The github.com/mitchellh/gox Go module provides the gox command, a dead simple, no frills Go cross compile tool that behaves a lot like the standard Go toolchain build command.

To configure and run the gox command, a new spell.Incantation will be implemented in a new gox package that can be casted using the gobin caster or any other spell caster that handles spell incantations of kind KindGoModule.

The spell incantation will be customizable through the following functions:

• WithEnv(env map[string]string) gox.Option — sets the spell incantation specific environment.
• WithGoCmd(goCmd string) gox.Option — sets the path to the Go toolchain executable.
• WithOutputTemplate(outputTemplate string) gox.Option — sets the name template for cross-compile platform targets. Defaults to gox.DefaultCrossCompileBinaryNameTemplate.
• WithGoOptions(goOpts ...spellGo.Option) gox.Option — sets shared Go toolchain command options.
• WithGoBuildOptions(goBuildOpts ...spellGoBuild.Option) gox.Option — sets options for the Go toolchain build command.
• WithModulePath(path string) gox.Option — sets the gox module command import path. Defaults to gox.DefaultGoModulePath.
• WithModuleVersion(version *semver.Version) gox.Option — sets the gox module version. Defaults to gox.DefaultGoModuleVersion.
• WithVerboseOutput(verbose bool) gox.Option — indicates whether the output should be verbose.

Like described in the /apps directory documentation of the tmpl-go template repository, wand also aims to support the monorepo layout.
In order to manage multiple applications, their information and metadata will be recorded in a configuration store where each entry will be identified by a unique ID, usually the name of the application. The pkg/app package will provide two interfaces and an unexported struct that implements it that can be used through the exported NewStore() Store function.

• 🆃 pkg/app.Config — A struct type that holds information and metadata of an application.
• 🅸 pkg/app.Store — A storage that provides methods to record application configurations:
  • Add(*Config) — Adds a application configuration.
  • Get(string) (*Config, error) — Returns the application configuration for the given name or nil along with an error when not stored.
• 🆃 appStore — A storage for application configurations.
• 🅵 NewStore() Store — Creates a new store for application configurations.

Update to "tmpl-go" version 0.7.0 which comes with updates to GitHub Actions and Node development dependencies.

As of Go 1.17 the go run command can finally run in module-aware mode while not “polluting“ the current module in the working directory, if there is one (go.mod file present) 🎉
This finally allows to run commands on-the-fly of Go main module packages without installing them or without changing dependencies of the current module!

To support this feature with wand a new task.GoModule will be implemented in a new golang/run package.
It can be run using a command runner that handles tasks of kind KindGoModule so mainly gotool.Runner.

The new golang/run.Task will be customizable through the following functions:

• WithArgs(...string) run.Option — sets additional arguments to pass to the command.
• WithEnv(map[string]string) run.Option — sets the task specific environment.
• WithModulePath(string) run.Option — sets the module import path.
• WithModuleVersion(*semver.Version) run.Option — sets the module version.

Next to the new task the gotool.Runner will be extended with a new WithCache(bool) runner option to toggle the usage of the local cache directory in the root directory of the module. The runner will be made “smart“ in the way that it either…

• installing the executable through a golang.Runner, which runs go install pkg@version to leverage Go 1.16‘s feature, and execute it afterwards. This is the current default behavior of this runner which will be used when WithCache(true) is used.
• pass the task to a golang.Runner, using the new golang/run package task, so that it can run go run pkg@version <args> instead. This is the new “smart“ behavior of the runner which will be used when WithCache(false) (default) is used.

The new default behavior will be to not use a local cache so that caching will be a opt-in. This decision was made because native support for running commands on-the-fly should always be preferred to custom logic which is what the local cache directory and gotool.Runner purpose is.

Warning
Note that the minimum Go version for task runners, the new golang/run task and the Elder wand will be increased to 1.17.0 since this version initially introduced go run support in module-aware mode!
This will be enforced through a build constraint (go:build go1.17).

The Elder reference implementation will also adapt to this new feature by…

1. deprecating the *elder.Elder.Bootstrap(...string) []error method! As of wand version 0.9.0 it will be a no-op and will be removed in version 0.10.0. To install executables anyway the new *elder.Elder.CacheExecutables error method should be used instead. To ensure that the wand is properly initialized and operational the *elder.Elder.Validate(..task.Runner) []error method is the way to go. A warning message will be printed when the method is called to ensure that users adapt accordionally.
2. providing a new *elder.Elder.CacheExecutables(...string) error method which allows to pass paths of Go modules that should be explicitly installed to the local cache directory. This method is a kind of workaround for the, now deprecated, *elder.Elder.Bootstrap(...string) []error method to allows users to still cache command executables locally.
3. changing the signature of the *elder.Elder.Validate() error method to *elder.Elder.Validate(...task.Runner) []error method which allows users to ensure that the wand is properly initialized and operational. Optionally command runner can be passed that will be validated while passing nothing will validate all currently supported runners.

The GitHub action workflows using the actions/checkout action to fetch the repository that triggered the workflow. However, by default only the history of the latest commit is fetched which results in errors when wand tries to extract repository metadata information like the amount of commits ahead of the latest commit. As an example this can be seen when running the bootstrap command in the test job of the ci-go workflow which fails with an object not found error because the history only contains a single commit.

To fix this problem action/checkout provides an option to fetch all history for all tags and branches which will be used to prevent errors like this in the pipeline.

To allow to compose, manipulate and read spell incantation options after the initial creation, two new types will be added for the spell packages:

• 🅸 spell.Options — A interface type as a generic representation for spell.Incantation options.
• 🅸 spell.Mixin — A interface type that allows to compose functions that process spell.Options of spell.Incantations.
  • Apply(Options) (Options, error) — applies generic spell.Options to spell.Incantation options.