exercism / go-test-runner Goto Github PK

View Code? Open in Web Editor NEW

14.0 13.0 17.0 157 KB

License: GNU Affero General Public License v3.0

Go 44.89% Shell 3.38% Dockerfile 0.68% Jupyter Notebook 51.05%

exercism-tooling exercism-test-runner community-contributions-paused

go-test-runner's Introduction

Exercism's Go Test Runner

This is Exercism's test runner for the Go track.

Executing the Test Runner

The test runner requires 2 parameters:

input_dir: the path containing the solution to test
output_dir: the output path for the test results

Local Development

go run . testrunner/testdata/practice/passing outdir

Run the package tests

go test ./...

Run the linter

Linting (and testing) is performed in a github action workflow - test.yml. You can install golangci-lint locally and then run:

golangci-lint run ./...

Interactive Debug / REPL

The original AST parsing code was developed using a Jupyter interactive Go REPL thanks to the gophernotes project. Consult the gophernotes docs for installation instructions. Once installed, you should be able to view, run, and modify the provided debug code "live" without constantly recompiling:

# assuming python3 with notebook installed via pip3, ymmv
python3 -m notebook ast_debug.ipynb

Docker

A docker container is used to run the test runner against submitted exercises. To build the container locally, execute the following from the repository root directory:

docker build -t exercism/go-test-runner .

Run the test runner in the container by passing in the slug name, and absolute paths to the exercise (solution) and a writeable tmp directory. These directories should be mounted as volumes:

docker run --network none --read-only -v $(pwd)/testrunner/testdata/practice/gigasecond:/solution -v /tmp:/tmp exercism/go-test-runner gigasecond /solution /tmp

For troubleshooting / debug you can name the container, run it in interactive mode, and detach from it using:

docker run --name exercism-go-test-runner -d -i --network none --read-only -v $(pwd)/testrunner/testdata/practice/gigasecond:/solution -v /tmp:/tmp exercism/go-test-runner gigasecond /solution /tmp
# You can then access the container as follows:
docker exec -it $(docker ps -q --filter name=exercism-go-test-runner) /bin/sh

External Go packages

Some extra Go packages that are not part of the standard library are downloaded when the docker image is built. This allows students to use these external packages in their solutions.

The list of external packages and their versions is in external-packages/go.mod.

To add or remove a package from the list of external packages supported:

Add/remove the corresponding import from external-packages/deps.go
Run go mod tidy inside the external-packages directory
Commit deps.go along with the changes to go.mod and go.sum produced by go mod tidy.

Note: The Go version declared in the go.mod file of the external-packages module should be the same as the version in the go.mod file of the exercises students download. This is because the Go version in the go.mod file affects the indirect dependencies that are downloaded, and consequently the go.sum file that is generated. A student can have a go.mod file declaring only supported dependencies, but if the Go version in that go.mod is different from the Go version in external-packages/go.mod, their go.sum may include more dependencies than external-packages/go.sum, which means they won't be able to run the solution.

Subtests

The test runner is responsible for returning the test_code field, which should be a copy of the test code corresponding to each test result.

For top-level tests, the AST is used to return the function code directly. For tests containing subtests, additional processing is required. To ease the burden of advanced AST processing on unstructured / non deterministic test code, subtests should adhere to the following specification. If a test employs subtests, do not mix it with test or other code outside of the Run() call.

Subtests not meeting the spec will be treated as top-level tests, with the entire test function code being returned for every subtest.
Assertions/outputs made outside of the Run() call will not be included in the result JSON because the "parent" tests are removed from the results if subtests are present. (Parent test reports were confusing to students because they did not include any assertion or fmt.Println output.)

At some point, we may implement a static analyzer which warns the exercise submitter when they commit subtests not meeting the specification.

Subtest Format Specification

The specification is annotated in the comments of the following example test:

func TestParseCard(t *testing.T) {
  // There can be additional code here, it will be shown for all subtests.
  // If the code here includes assignments, the test data variable below needs to be called "tests".

  tests := []struct {
    name string // The name field is required
    card string
    want int
  }{
    // The relevant test data will be parsed out individually for each subtest
    {
      // spaces are fine in a test name, but do not mix them with underscores
      // - because the extraction code won't be able to find it
      name: "parse queen",
      card: "queen",
      want: 10,
    },
    // For example, this element will be parsed for `TestParseCard/parse_king`
    {
      name: "parse king",
      card: "king",
      want: 10,
    },
  }

  // There can be additional code here, it will be shown for all subtests.

  // The contents of the function literal will be extracted as the test code
  for _, tt := range tests {
    // The Run() call must be the first statement in the for loop
    t.Run(tt.name, func(t *testing.T) {
      // This code block will be pulled into the resulting test_code field
      if got := ParseCard(tt.card); got != tt.want {
        t.Errorf("ParseCard(%s) = %d, want %d", tt.card, got, tt.want)
      }
    })
  }

  // There can be additional code here, it will be shown for all subtests.
}

The test code above will result in the following test_code field, corresponding to the test named TestParseCard/parse_queen:

tt := struct {
  name string
  card string
  want int
}{
  name: "parse queen",
  card: "queen",
  want: 10,
}
if got := ParseCard(tt.card); got != tt.want {
  t.Errorf("ParseCard(%s) = %d, want %d", tt.card, got, tt.want)
}

Providing Additional Testing Flags

Exercises can supply additional flags that will be included when the test runner executes the go test command. This is done via the .meta/config.json file of the exercise. See example below.

{
  // ...
  "custom": {
    "testingFlags": ["-race"]
  }
}

Currently, only the flag -race is supported. If more flags should be allowed in the future, they first need to be added to the allowedTestingFlags list in testrunner/execute.go.

Assigning Task Ids

For concept exercises, the output of the test runner can contain task ids for the different test cases. These ids are used on the website to associate the test result with the corresponding task in the exercise. This leads to a great user experience for the students.

The test runner output will only contain task ids if they were explicitly enabled in the .meta/config.json file for the exercise as shown below (otherwise task ids are omitted):

{
  // ...
  "custom": {
    "taskIdsEnabled": true
  }
}

Note that this flag should only ever be set for concept exercises. Task ids are currently not supported for practice exercise.

There are two ways the test runner can assign task ids for concept exercises.

Implicit Task Id Assignment

If "taskIdsEnabled": true was set but there are no explicit task ids found in the test file, the test runner will automatically assign task ids. It assumes each parent test corresponds to one task and will assign task id 1 to the first parent test and its sub-tests, task id 2 to the next one etc.

For most concept exercises, we have this 1 to 1 relationship between tests and tasks. This implicit assignment means we do not need to add anything to the test files to get task ids in the test runner output. We only need to set the type in the config as shown above.

You can test this locally end-to-end via go run . testrunner/testdata/concept/conditionals outdir.

Explicit Task Id Assignment

If the implicit system would lead to wrong task ids, they can be set manually via a comment in the following format that is added for the test function:

// testRunnerTaskID=1
func TestSomething(t *testing.T) {
  // ...
}

Sub-tests automatically get the task id from their parent, they don't need any explicit assignment.

You can test this locally end-to-end via go run . testrunner/testdata/concept/conditionals-with-task-ids outdir.

Explicit task id assignment will only take effect if an explicit task id was found on every parent test in the test file. Otherwise no task ids will be set at all.

Finding the task id is robust again other comments before or after or in the same line as the testRunnerTaskID comment, see the conditionals-with-task-ids test file for examples.

Known limitations

Besides what is mentioned in the open issues, the test runner has the following limitations currently.

The test runner assumes all test functions func Test... can be found in one file.
The cases_test.go file is ignored when extracting the code for the test.
Sub-tests need to follow a certain format, see details above.

go-test-runner's People

Contributors

Stargazers

Watchers

Forkers

ccare sebito91 erikschierboom exercism-bot moshegood andrerfcsantos jmrunkle junedev faisalafroz mhmmdd pedreviljoen eklatzer pilukarts h3n4l dreig meoconbatu bramvanberkel

go-test-runner's Issues

Allow to enable race detector for certain exercises

Currently, we don't use the race detector in the test runner. As @bitfield pointed out here exercism/go#2021 it would be beneficial for some exercise to run it.

Here some initial thoughts the topic:

with the newest configlet release, exercises can have custom keys in the config, we could use that to turn the race detector on for specific exercises in case it is too slow to leave it on all the time
we need to double check that we don't run into the test runner timeout when the race detector is on for this exercise
we need to check whether the race detector works with the alpine image that we use in the test runner now (@andrerfcsantos might have done this already, not sure though)

All test failures should always be reported

I am solving Cars Assemble and I was really confused that there are no tests for the third function, CalculateProductionRatePerMinute. Here is the test output of a partial solution, 6 passed, 2 failed:

Then, after I implemented the second function correctly, suddenly the tests for CalculateProductionRatePerMinute appeared. The results are now 8 tests passed, 6 failed = 6 more tests than the last time?

I didn't see anything that would explain this behavior in the test file itself https://github.com/exercism/go/blob/main/exercises/concept/cars-assemble/cars_assemble_test.go that's why I decided to open an issue in this repo, not in the exercises repo. But I might we wrong.

Version 3 format does not work well when not all tests are executed

Two problems as shown in the screenshot:

Only the first test is listed and has the exercise linked because the others were not executed yet. (Go stops at the panic and there are no good alternative to the panic where the code still compiles and we don't do a lot of tricks.)
The counter at the top thinks the tasks that don't have tests listed are completed.

No compilation errors are shown

I ran the following code:

package go1a

// Describe takes an amount and returns a description.
func Describe(amount int) string {
	return ""
}

func Units(amount int) (int, int, int, int) {
	grosses := amount / 144
	amount %= 144
	
	scores := amount / 20
	amount %= 20
	
	dozens := amount / 12
	amount %= 12
	
	return (grosses, scores, dozens, amount)
}

This has a syntax error, but it didn't show up in the test output. Looking at the test results JSON, it has captured the compile error, but in the test itself and not in the top-level message property (see the spec):

{
	"status": "error",
	"tests": [
		{
			"name": "build",
			"status": "error",
			"expected": "",
			"actual": "",
			"message": "./go_1_a.go:18:17: syntax error: unexpected comma, expecting )"
		}
	]
}

For this reason., the compilation error message is not shown as the output. The test runner should output the compilation error in the top-level message property.

Investigate whether Scanln can be removed

There is this bit in the test runner code: https://github.com/exercism/go-test-runner/blob/main/testrunner/extract.go#L34

It means that if you run the test runner outside of the docker container with the command from the README go run . testrunner/testdata/practice/passing outdir then you need to press enter a lot of times to actually get the output json file. We don't yet understand why it works correctly in the production setting. We want to find out how this works to decide whether the Scanln part can be removed or guarded by some flag to activate it only if needed.

Replace `print` commands without final newline

The commands print, fmt.Print, fmt.Printf and maybe others currently destroy the output of go test --json as they have no final newline.

We should report that to be fixed in go test but for the time being we will replace these commands in students solutions with println / fmt.Println.

My current suggestion would be to add some sed commands to the test runner before executing the tests. This could also be done using the AST library to manipulate the AST if someone wants to dig into that.

Here the replace table I'd recommend to avoid having to remove/add imports:

print -> println
fmt.Print -> fmt.Println
fmt.Printf(...) -> fmt.Println(fmt.Sprintf(...))

Integrate Go Test Runner in Exercism

The Go test runner is ready to be integrated in exercism.

See the Readme for details.

Stack overflow is not properly reported on the website

In the gross store exercises. Using the following code will generate a blank error message on the website:

func GetItem(bill map[string]int, item string) (int, bool) {
qty , ok := GetItem(bill, item)
return qty, ok
}

Redundant empty line in extracted test code

If the extraction of the individual test from a list of test cases worked out, the result has a redundant empty line after the struct definition part:

Update to Go 1.18

Go 1.18 is a very anticipated release of the language due to the inclusion of generics, among other things. We should update the tooling of the Go track to support solutions using Go 1.18.

Go 1.18 release notes

This update should be similar to the upgrade done for 1.17:

Tracking issue: exercism/go#2129

Test order should not be randomized

I noticed that on every test run I am getting a slightly different test order. This might be fine for practice exercise (or not, I think that depends on the track), but it's not fine for concept exercises IMO. Those are meant to be solved task by task, so I always expect the test failures to be ordered by task. The tests for this exercise are ordered by task. It's the test runner that randomizes their order.

Run 1

Run 2

PS: There is a different issue visible on those screenshot (no tests for CalculateProductionRatePerMinute). I will debug/report that separately.

Skip benchmark tests by using `go test -short`

I have an issue addressed by exercism/go#1608. Once that goes in we just need to pass the -short option to skip those tests when running on the website. There is no real reason we cannot start doing that immediately.

potential BUG: bank-account exercise, timeout issue

local go version: go1.20.6 linux/amd64
exercise: Bank Account in Go

I may have found an infrastructure bug.
My tests timed out on the server, but couldn't find where I was messing up even when comparing to community solutions. I requested mentoring, and bernot-dev replied and mentioned that he thinks it's an issue with infrastructure. We both run it and pass tests locally.

Let me know what other information I may provide.

Unsure if this will give you access, Direct mentoring link to my attempt on this exercise, with source copied below.

my code, copied from iteration 5

package account
import (
	"sync"
)
// Define the Account type here.
type Account struct {
	balance int64
	active  bool
	mu      sync.RWMutex
}
func Open(amount int64) *Account {
	if amount < 0 {
		return nil
	}
	return &Account{
		balance: amount,
		active:  true,
		mu:      sync.RWMutex{},
	}
}
// Balance returns the balance of the account
func (a *Account) Balance() (int64, bool) {
	a.mu.RLock()
	defer a.mu.RUnlock()
	b, ok := a.balance, a.active
	// fail early in lieu of returning invalid state on API
	if b > 0 && !ok {
		panic("an inactive account has nonzero balance")
	} else if b < 0 {
		panic("account found with negative balance")
	}
	return b, ok
}
func (a *Account) Deposit(amount int64) (int64, bool) {
	a.mu.Lock()
	defer a.mu.Unlock()
	b, ok := a.balance, a.active
	if !ok || b+amount < 0 {
		return b, false
	}
	b += amount
	a.balance = b
	return b, true
}
func (a *Account) Close() (int64, bool) {
	a.mu.Lock()
	defer a.mu.Unlock()
	b, ok := a.balance, a.active
	if !ok {
		return 0, false
	}
	a.balance, a.active = 0, false
	return b, true
}

Update of Go test runner

@iHiD Latest commit updates the Go test runner to capture compile error messages.

@ccare can you update the Go test runner? Let me know if you want version tags, PRs, or some other workflow for updating the test runner.

Extracting code for the specific test case does not work even when it conforms to the requirements

Even though some exercise tests seems to conform to the spec described here https://github.com/exercism/go-test-runner#subtest-format-specification, the test code on the website includes all test cases instead only the one that is currently tested.
Example where it is not working: Card Tricks Exercise

Example where the test case extraction is working correctly: Cars Assemble

Test code extraction for exercises with separate file for test cases

For practice exercises that have a generator, the list of test cases is written into a separate file cases_test.go currently because this is easier to do for the generator then to somehow insert it into an existing test file.

The issue is that the test runner currently can't handle this separate file. The content of that file is ignored which leads to sub-optimal output on the website. The extracted code does not include the actual test case.

The current output e.g. for hamming looks like this:

What we would like to have instead is:

var tt = struct {
	s1          string
	s2          string
	want        int
	expectError bool
}{
	description: "long identical strands",
	s1:          "GGACTGAAATCTG",
	s2:          "GGACTGAAATCTG",
	want:        0,
	expectError: false,
}

got, err := Distance(tc.s1, tc.s2)

switch {
// the usual code here
}

This be be doable by AST parsing the cases file and inserting the test cases AST part into the Test function so the result mimics the usual structure that the test runner can handle correctly, see https://github.com/exercism/go-test-runner#subtest-format-specification.

Related: exercism/go#1872

No proper error message for old version of Lasagna

The lasagna exercise was changed to use a constant instead of a function for OvenTime.

Then I run the the old code (shown below) against the new tests, the general "an error occured" message shows up instead of an error message.

package lasagna

// OvenTime returns the expected baking time in minutes.
func OvenTime() int {
    return 40
}

// RemainingOvenTime returns the remaining minutes based on the `actual` minutes already in the oven.
func RemainingOvenTime(actual int) int {
	return OvenTime() - actual
}

// PreparationTime calculates the time needed to prepare the lasagna based on the amount of layers.
func PreparationTime(numberOfLayers int) int {
	return numberOfLayers *2
}

// ElapsedTime calculates the total time needed to create and bake a lasagna.
func ElapsedTime(numberOfLayers, actualMinutesInOven int) int {
	return PreparationTime(numberOfLayers) + actualMinutesInOven
}

Separate error message from console output

According to the test runner interface there is a distinction to be made between the Message (error message in case the test failed) and Output (the output of the test including user output).

When parsing the output of go test --json we currently write all lines marked with "Action":"output" into the Output field. Instead we should additionally check the "Output" if it has the following pattern and write that in the Message field.

Sample of line to be written to the Message field:

{"Time":"2020-01-03T14:05:53.978508+01:00","Action":"output","Package":"github.com/exercism/go-test-runner/tests/5","Test":"TestSample/first_test","Output":"        sample_test.go:30: Output should be output: first test, got output: third test\n"}

Suggestion is to use a regex to search for the prefix ^\t+(.+?)_test\.go:/d+: (or similar) and use these line(s) to fill Message in the test runner output.

Issue with special characters in output message

Quotes in the output message are not displayed correctly anymore. Might be related to #41 (not sure how though).

Example (see last line)

# lasagna [lasagna.test]
./lasagna_test.go:39:14: undefined: RemainingOvenTime
FAIL	lasagna [build failed]
&#39;/usr/local/go/bin/go test --short --json .&#39; returned exit code 2: exit status 2

Upgrade to version 3 spec

If possible, this test runner should be updated to version 3 of the test runner interface specification. In version 3, one additional feature is enabled: the ability to link individual tests to tasks. This allows the website to show which tests belong to which tasks.

The way tests are linked to tasks is via an (optional) task_id field, which is an integer that matches the number of the task as defined in the exercise's instructions.md file (note: the instructions start at index 1).

This is an example of a test in the results.json file:

{
  "name": "Expected oven time in minutes",
  "status": "pass",
  "task_id": 1,
  "test_code": "Assert.Equal(40, Lasagna.ExpectedMinutesInOven());"
}

You are completely free in how to implement this. Some options are:

Add metadata to a test that the test runner can then discover while running the tests (e.g. an attribute or annotation)
Define a test name/task id mapping (e.g. in the exercise's .meta/config.json file)
Any other option you can think of...

Let me know if there are any questions.

Upgrade to version 3 spec

This is an example of a test in the results.json file:

{
  "name": "Expected oven time in minutes",
  "status": "pass",
  "task_id": 1,
  "test_code": "Assert.Equal(40, Lasagna.ExpectedMinutesInOven());"
}

You are completely free in how to implement this. Some options are:

Add metadata to a test that the test runner can then discover while running the tests (e.g. an attribute or annotation)
Define a test name/task id mapping (e.g. in the exercise's .meta/config.json file)
Any other option you can think of...

Let me know if there are any questions.

Test runner does not handle stack overflow correctly

Solution ec84829229e74e1da9828f67823637c9 to 'Tree Building' is not correct, but the test runner shows a pass. Running the tests on my machine produces:

runtime: goroutine stack exceeds 1000000000-byte limit
runtime: sp=0xc020480380 stack=[0xc020480000, 0xc040480000]
fatal error: stack overflow

runtime stack:
runtime.throw({0x1123e73?, 0x11eb3e0?})
	/usr/local/go/src/runtime/panic.go:992 +0x71
runtime.newstack()
	/usr/local/go/src/runtime/stack.go:1101 +0x5cc
runtime.morestack()
	/usr/local/go/src/runtime/asm_amd64.s:547 +0x8b

goroutine 8 [running]:
runtime.assertE2I2(0x1108a40?, {0x10fefa0?, 0xc008b85110?})
	/usr/local/go/src/runtime/iface.go:456 +0x78 fp=0xc020480390 sp=0xc020480388 pc=0x100a8f8
fmt.(*pp).handleMethods(0xc008b8ed00, 0x0?)
	/usr/local/go/src/fmt/print.go:590 +0xdc fp=0xc0204805e0 sp=0xc020480390 pc=0x109ddbc
fmt.(*pp).printArg(0xc008b8ed00, {0x10fefa0?, 0xc008b85110}, 0x73)
	/usr/local/go/src/fmt/print.go:709 +0x693 fp=0xc020480680 sp=0xc0204805e0 pc=0x109eb93
fmt.(*pp).doPrintf(0xc008b8ed00, {0x1122146, 0x5}, {0xc020480810?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020480778 sp=0xc020480680 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020480810, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc0204807d0 sp=0xc020480778 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc020480840 sp=0xc0204807d0 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8ec30, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020480a90 sp=0xc020480840 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8ec30, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020480c78 sp=0xc020480a90 pc=0x109edba
fmt.(*pp).printValue(0xc008b8ec30, {0x10fefa0?, 0xc008b850f8?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020480e60 sp=0xc020480c78 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8ec30, {0x10fefa0?, 0xc008b850f8}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020480f00 sp=0xc020480e60 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8ec30, {0x1122146, 0x5}, {0xc020481090?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020480ff8 sp=0xc020480f00 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020481090, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020481050 sp=0xc020480ff8 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc0204810c0 sp=0xc020481050 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8eb60, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020481310 sp=0xc0204810c0 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8eb60, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc0204814f8 sp=0xc020481310 pc=0x109edba
fmt.(*pp).printValue(0xc008b8eb60, {0x10fefa0?, 0xc008b850e0?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc0204816e0 sp=0xc0204814f8 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8eb60, {0x10fefa0?, 0xc008b850e0}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020481780 sp=0xc0204816e0 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8eb60, {0x1122146, 0x5}, {0xc020481910?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020481878 sp=0xc020481780 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020481910, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc0204818d0 sp=0xc020481878 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc020481940 sp=0xc0204818d0 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8ea90, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020481b90 sp=0xc020481940 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8ea90, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020481d78 sp=0xc020481b90 pc=0x109edba
fmt.(*pp).printValue(0xc008b8ea90, {0x10fefa0?, 0xc008b850c8?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020481f60 sp=0xc020481d78 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8ea90, {0x10fefa0?, 0xc008b850c8}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020482000 sp=0xc020481f60 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8ea90, {0x1122146, 0x5}, {0xc020482190?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc0204820f8 sp=0xc020482000 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020482190, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020482150 sp=0xc0204820f8 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc0204821c0 sp=0xc020482150 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e9c0, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020482410 sp=0xc0204821c0 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e9c0, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc0204825f8 sp=0xc020482410 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e9c0, {0x10fefa0?, 0xc008b850b0?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc0204827e0 sp=0xc0204825f8 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e9c0, {0x10fefa0?, 0xc008b850b0}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020482880 sp=0xc0204827e0 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e9c0, {0x1122146, 0x5}, {0xc020482a10?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020482978 sp=0xc020482880 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020482a10, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc0204829d0 sp=0xc020482978 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc020482a40 sp=0xc0204829d0 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e8f0, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020482c90 sp=0xc020482a40 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e8f0, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020482e78 sp=0xc020482c90 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e8f0, {0x10fefa0?, 0xc008b85098?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020483060 sp=0xc020482e78 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e8f0, {0x10fefa0?, 0xc008b85098}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020483100 sp=0xc020483060 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e8f0, {0x1122146, 0x5}, {0xc020483290?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc0204831f8 sp=0xc020483100 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020483290, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020483250 sp=0xc0204831f8 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc0204832c0 sp=0xc020483250 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e820, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020483510 sp=0xc0204832c0 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e820, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc0204836f8 sp=0xc020483510 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e820, {0x10fefa0?, 0xc008b85080?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc0204838e0 sp=0xc0204836f8 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e820, {0x10fefa0?, 0xc008b85080}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020483980 sp=0xc0204838e0 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e820, {0x1122146, 0x5}, {0xc020483b10?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020483a78 sp=0xc020483980 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020483b10, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020483ad0 sp=0xc020483a78 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc020483b40 sp=0xc020483ad0 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e750, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020483d90 sp=0xc020483b40 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e750, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020483f78 sp=0xc020483d90 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e750, {0x10fefa0?, 0xc008b85068?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020484160 sp=0xc020483f78 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e750, {0x10fefa0?, 0xc008b85068}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020484200 sp=0xc020484160 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e750, {0x1122146, 0x5}, {0xc020484390?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc0204842f8 sp=0xc020484200 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020484390, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020484350 sp=0xc0204842f8 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc0204843c0 sp=0xc020484350 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e680, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020484610 sp=0xc0204843c0 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e680, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc0204847f8 sp=0xc020484610 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e680, {0x10fefa0?, 0xc008b85050?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc0204849e0 sp=0xc0204847f8 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e680, {0x10fefa0?, 0xc008b85050}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020484a80 sp=0xc0204849e0 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e680, {0x1122146, 0x5}, {0xc020484c10?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020484b78 sp=0xc020484a80 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020484c10, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020484bd0 sp=0xc020484b78 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc020484c40 sp=0xc020484bd0 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e5b0, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020484e90 sp=0xc020484c40 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e5b0, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020485078 sp=0xc020484e90 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e5b0, {0x10fefa0?, 0xc008b85038?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020485260 sp=0xc020485078 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e5b0, {0x10fefa0?, 0xc008b85038}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020485300 sp=0xc020485260 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e5b0, {0x1122146, 0x5}, {0xc020485490?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc0204853f8 sp=0xc020485300 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020485490, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020485450 sp=0xc0204853f8 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc0204854c0 sp=0xc020485450 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e4e0, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020485710 sp=0xc0204854c0 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e4e0, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc0204858f8 sp=0xc020485710 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e4e0, {0x10fefa0?, 0xc008b85020?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020485ae0 sp=0xc0204858f8 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e4e0, {0x10fefa0?, 0xc008b85020}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020485b80 sp=0xc020485ae0 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e4e0, {0x1122146, 0x5}, {0xc020485d10?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020485c78 sp=0xc020485b80 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020485d10, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020485cd0 sp=0xc020485c78 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc020485d40 sp=0xc020485cd0 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e410, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020485f90 sp=0xc020485d40 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e410, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020486178 sp=0xc020485f90 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e410, {0x10fefa0?, 0xc008b85008?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020486360 sp=0xc020486178 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e410, {0x10fefa0?, 0xc008b85008}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020486400 sp=0xc020486360 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e410, {0x1122146, 0x5}, {0xc020486590?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc0204864f8 sp=0xc020486400 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020486590, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020486550 sp=0xc0204864f8 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc0204865c0 sp=0xc020486550 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e340, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020486810 sp=0xc0204865c0 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e340, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc0204869f8 sp=0xc020486810 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e340, {0x10fefa0?, 0xc008b84ff0?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020486be0 sp=0xc0204869f8 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e340, {0x10fefa0?, 0xc008b84ff0}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020486c80 sp=0xc020486be0 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e340, {0x1122146, 0x5}, {0xc020486e10?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc020486d78 sp=0xc020486c80 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020486e10, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020486dd0 sp=0xc020486d78 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc020486e40 sp=0xc020486dd0 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e270, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020487090 sp=0xc020486e40 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e270, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020487278 sp=0xc020487090 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e270, {0x10fefa0?, 0xc008b84fd8?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020487460 sp=0xc020487278 pc=0x10a048e
fmt.(*pp).printArg(0xc008b8e270, {0x10fefa0?, 0xc008b84fd8}, 0x73)
	/usr/local/go/src/fmt/print.go:712 +0x74c fp=0xc020487500 sp=0xc020487460 pc=0x109ec4c
fmt.(*pp).doPrintf(0xc008b8e270, {0x1122146, 0x5}, {0xc020487690?, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:1026 +0x288 fp=0xc0204875f8 sp=0xc020487500 pc=0x10a1448
fmt.Sprintf({0x1122146, 0x5}, {0xc020487690, 0x2, 0x2})
	/usr/local/go/src/fmt/print.go:219 +0x59 fp=0xc020487650 sp=0xc0204875f8 pc=0x109b979
tree.Node.String(...)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:245
tree.(*Node).String(0x0?)
	<autogenerated>:1 +0xb9 fp=0xc0204876c0 sp=0xc020487650 pc=0x10f4239
fmt.(*pp).handleMethods(0xc008b8e1a0, 0xe038?)
	/usr/local/go/src/fmt/print.go:626 +0x30b fp=0xc020487910 sp=0xc0204876c0 pc=0x109dfeb
fmt.(*pp).printValue(0xc008b8e1a0, {0x11064e0?, 0xc00000e038?, 0x0?}, 0x73, 0x1)
	/usr/local/go/src/fmt/print.go:723 +0xda fp=0xc020487af8 sp=0xc020487910 pc=0x109edba
fmt.(*pp).printValue(0xc008b8e1a0, {0x10fefa0?, 0xc008b84fc0?, 0x203002?}, 0x73, 0x0)
	/usr/local/go/src/fmt/print.go:865 +0x17ae fp=0xc020487ce0 sp=0xc020487af8 pc=0x10a048e
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1457 +0x35f

goroutine 1 [chan receive]:
testing.(*T).Run(0xc0001104e0, {0x1124ef7?, 0x993eb0364f35?}, 0x112ce98)
	/usr/local/go/src/testing/testing.go:1458 +0x37a
testing.runTests.func1(0xc0001104e0?)
	/usr/local/go/src/testing/testing.go:1810 +0x6e
testing.tRunner(0xc0001104e0, 0xc000074cd8)
	/usr/local/go/src/testing/testing.go:1410 +0x102
testing.runTests(0xc000108140?, {0x11fad40, 0x2, 0x2}, {0x1270108?, 0x40?, 0x1203180?})
	/usr/local/go/src/testing/testing.go:1808 +0x457
testing.(*M).Run(0xc000108140)
	/usr/local/go/src/testing/testing.go:1690 +0x5d9
main.main()
	_testmain.go:55 +0x1aa

goroutine 6 [chan receive]:
testing.(*T).Run(0xc000110680, {0x11229f6?, 0x105f7dc?}, 0xc000058530)
	/usr/local/go/src/testing/testing.go:1458 +0x37a
tree.TestMakeTreeSuccess(0x0?)
	/Users/john/git/bitfield/exercism/users/regul4rj0hn/go/tree-building/tree_building_test.go:250 +0xf9
testing.tRunner(0xc000110680, 0x112ce98)
	/usr/local/go/src/testing/testing.go:1410 +0x102
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1457 +0x35f

`status: error` is being returned without `message`

Reference chat: https://exercism-team.slack.com/archives/CAS0EK88H/p1635353715044200

Output from Test Runner:

{"status"=>"error", "version"=>2, "tests"=>[{"name"=>"TestNameValid", "status"=>"pass", "test_code"=>"func TestNameValid(t *testing.T) {\n\tn := New().getName(t, false)\n\tif !namePat.MatchString(n) {\n\t\tt.Errorf(`Invalid robot name %q, want form \"AA###\".`, n)\n\t}\n}", "message"=>"\n=== RUN   TestNameValid\n\n--- PASS: TestNameValid (0.00s)\n"}, {"name"=>"TestNameSticks", "status"=>"pass", "test_code"=>"func TestNameSticks(t *testing.T) {\n\tr := New()\n\tn1 := r.getName(t, false)\n\tn2 := r.getName(t, true)\n\tif n2 != n1 {\n\t\tt.Errorf(`Robot name changed.  Now %s, was %s.`, n2, n1)\n\t}\n}", "message"=>"\n=== RUN   TestNameSticks\n\n--- PASS: TestNameSticks (0.00s)\n"}, {"name"=>"TestSuccessiveRobotsHaveDifferentNames", "status"=>"pass", "test_code"=>"func TestSuccessiveRobotsHaveDifferentNames(t *testing.T) {\n\tn1 := New().getName(t, false)\n\tn2 := New().getName(t, false)\n\tif n1 == n2 {\n\t\tt.Errorf(`Robots with same name.  Two %s's.`, n1)\n\t}\n}", "message"=>"\n=== RUN   TestSuccessiveRobotsHaveDifferentNames\n\n--- PASS: TestSuccessiveRobotsHaveDifferentNames (0.00s)\n"}, {"name"=>"TestResetName", "status"=>"pass", "test_code"=>"func TestResetName(t *testing.T) {\n\tr := New()\n\tn1 := r.getName(t, false)\n\tr.Reset()\n\tif r.getName(t, false) == n1 {\n\t\tt.Errorf(`Robot name not cleared on reset.  Still %s.`, n1)\n\t}\n}", "message"=>"\n=== RUN   TestResetName\n\n--- PASS: TestResetName (0.00s)\n"}, {"name"=>"TestMultipleNames", "status"=>"pass", "test_code"=>"// Test 1000 naems are unique - this should run reasonably quickly even with a sub-optimal solution\n// (e.g. pick a random name, then pick a new name if it's been seen before)\nfunc TestMultipleNames(t *testing.T) {\n\t// Test uniqueness for new robots.\n\tfor i := len(seen); i <= 1000; i++ {\n\t\tNew().getName(t, false)\n\t}\n}", "message"=>"\n=== RUN   TestMultipleNames\n\n--- PASS: TestMultipleNames (0.00s)\n"}, {"name"=>"TestCollisions", "status"=>"skip", "test_code"=>"// TestCollisions tests if unique names are generated by creating new robots until all names are used.\nfunc TestCollisions(t *testing.T) {\n\t// Remove the next line to make this test run\n\tt.Skip(\"skipping test as it can take a long time to run if solution is sub-optimal.\")\n\n\t// Test uniqueness for new robots.\n\tfor i := len(seen); i <= lotsOfNames; i++ {\n\t\tNew().getName(t, false)\n\t}\n\n\t// Test that names aren't recycled either.\n\t// Note that this runs till names are exhausted.\n\tr := New()\n\tfor i := len(seen); i < maxNames; i++ {\n\t\tr.Reset()\n\t\tr.getName(t, false)\n\t}\n\n\t// Test that name exhaustion is handled more or less correctly.\n\t_, err := New().Name()\n\tif err == nil {\n\t\tt.Fatalf(\"should return error if namespace is exhausted\")\n\t}\n}", "message"=>"\n=== RUN   TestCollisions\n\n    robot_name_test.go:82: skipping test as it can take a long time to run if solution is sub-optimal.\n\n--- SKIP: TestCollisions (0.00s)\n"}]}

The status is error, but there is no corrosponding message. Should the status by fail? Or has something validly eroneous happened which has led to an error but not attached a message?

Golang: Remaining test case are stopped when adding `fmt.Printf` for console log purpose

Using Printf function from fmt has stopping remaining running test case and allowed for submit which lead to false positive case.

Link: https://exercism.org/tracks/go/exercises/need-for-speed

Do not include parent test in test results when using sub-tests

The recommendation for Go test cases is usually to use table-based tests. This is also how the samples defined in the tests directory are defined. However, running the test runner on these results also results in the parent (root) test being included in the test results, which is problematic when trying to display what went wrong for which input. The test runner should omit the parent/root test from the test results output.

As an example, consider the second example's test suite:

package sample

import (
	"testing"
)

var tests = []struct {
	name string
	fail bool
}{
	{
		name: "first test",
		fail: false,
	},
	{
		name: "second test",
		fail: true,
	},
	{
		name: "third test",
		fail: true,
	},
}

func TestSample(t *testing.T) {
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if tt.fail {
				t.Fail()
			}
		})
	}
}

If we run the test runner on these tests using go test --json ./tests/2 | go run cmd/testoutput/main.go > tests/2/results.json, we get the following output:

{
	"status": "fail",
	"tests": [
		{
			"name": "TestSample",
			"status": "fail",
			"expected": "",
			"actual": "",
			"message": "\n=== RUN   TestSample\n\n--- FAIL: TestSample (0.00s)\n"
		},
		{
			"name": "TestSample/first_test",
			"status": "pass",
			"expected": "",
			"actual": "",
			"message": "\n=== RUN   TestSample/first_test\n\n    --- PASS: TestSample/first_test (0.00s)\n"
		},
		{
			"name": "TestSample/second_test",
			"status": "fail",
			"expected": "",
			"actual": "",
			"message": "\n=== RUN   TestSample/second_test\n\n    --- FAIL: TestSample/second_test (0.00s)\n"
		},
		{
			"name": "TestSample/third_test",
			"status": "fail",
			"expected": "",
			"actual": "",
			"message": "\n=== RUN   TestSample/third_test\n\n    --- FAIL: TestSample/third_test (0.00s)\n"
		}
	]
}

We can see that besides the three sub-tests, there is also an entry for the parent test:

{
    "name": "TestSample",
    "status": "fail",
    "expected": "",
    "actual": "",
    "message": "\n=== RUN   TestSample\n\n--- FAIL: TestSample (0.00s)\n"
},

This entry should be omitted from the test results output.

Go - Tests passing for solutions that don't compile

I noticed that for some solutions I'm mentoring, there's the little icon at the top saying the tests passed but the solutions don't even compile. I noticed this on the Go track on 2 different exercises, there might be happening in more.

@exercism/go
@ErikSchierboom

Unspecific error messages if the code does not compile

In the editor bug reports a lot of people wrote they saw '/usr/local/go/bin/go test --json .' returned exit code 2: exit status 2 as error message which wasn't helpful for them. Is there something that could be improved?

I also see the generic "an error occurred" for certain solutions.

I will post some examples below.
My feeling is this happens when the (test) code does not compile.

The master branch will be renamed to main

In line with our new org-wide policy, the master branch of this repo will be renamed to main. All open PRs will be automatically repointed.

GitHub will show you a notification about this when you look at this repo after renaming:

In case it doesn't, this is the command it suggests:

git branch -m master main
git fetch origin
git branch -u origin/main main

You may like to update the primary branch on your forks too, which you can do under Settings->Branches and clicking the pencil icon on the right-hand-side under Default Branch:

We will post a comment below when this is done. We expect it to happen within the next 12 hours.

Output in case of a test fail

Hi, I've recently pick up Go on Exercism and one by one I try to go through all exercises. The web editor is great for this purpose but what's bother me, and I think also slows down quite a bit, are tests which failed. I feel like the messages are not clear enough and in some tests one doesn't even know what exactly failed.

An example of such test would be following output - in this case the "failure" description is completely missing and by single test case it would be helpful if there is also got key or something like this, which would hold returned value:

func TestSetItem(t *testing.T) {
	type args struct {
		slice	[]int
		index	int
		value	int
	}
	tests := []struct {
		name	string
		args	args
		want	[]int
	}{
		{
			name:	"Overwrite an existing item",
			args: args{
				slice:	[]int{5, 2, 10, 6, 8, 7, 0, 9},
				index:	4,
				value:	1,
			},
			want:	[]int{5, 2, 10, 6, 1, 7, 0, 9},
		},
		{
			name:	"Overwrite first item",
			args: args{
				slice:	[]int{5, 2, 10, 6, 8, 7, 0, 9},
				index:	0,
				value:	8,
			},
			want:	[]int{8, 2, 10, 6, 8, 7, 0, 9},
		},
		{
			name:	"Overwrite last item",
			args: args{
				slice:	[]int{5, 2, 10, 6, 8, 7, 0, 9},
				index:	7,
				value:	8,
			},
			want:	[]int{5, 2, 10, 6, 8, 7, 0, 8},
		},
		{
			name:	"Index out of bounds",
			args: args{
				slice:	[]int{5, 2, 10, 6, 8, 7, 0, 9},
				index:	8,
				value:	8,
			},
			want:	[]int{5, 2, 10, 6, 8, 7, 0, 9, 8},
		},
		{
			name:	"Negative index",
			args: args{
				slice:	[]int{5, 2, 10, 6, 8, 7, 0, 9},
				index:	-1,
				value:	8,
			},
			want:	[]int{5, 2, 10, 6, 8, 7, 0, 9, 8},
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			if got := SetItem(tt.args.slice, tt.args.index, tt.args.value); !slicesEqual(got, tt.want) {
				t.Errorf("SetItem(slice:%v, index:%v, value:%v) = %v, want %v",
					tt.args.slice, tt.args.index, tt.args.value, got, tt.want)
			}
		})
	}
}

=======

=== RUN   TestSetItem

--- FAIL: TestSetItem (0.00s)

If I'm missing something or read provided output wrong please let me know.

Text overflow issues

In some cards, the text goes out of the div which looks bad.

`elons-toys` getting error fetching go-cmp

Here is the error message:

go: github.com/google/[email protected]: Get "https://proxy.golang.org/github.com/google/go-cmp/@v/v0.5.3.mod": dial tcp: lookup proxy.golang.org on 10.0.2.3:53: dial udp 10.0.2.3:53: connect: network is unreachable
'/usr/local/go/bin/go test --short --json .' returned exit code 1: exit status 1

This occurs even when copy-pasting the exemplar solution into the web editor and running the tests.

Errors should not hide all test case info

When there's an error in the code, e.g. when running an incomplete solution of Gopher's Gorgeous Lasagna, this error will be shown rather than the usual Exercism task test view. This makes it harder to approach the problems task-by-task, as well as making it hard to solve them in a TDD-inspired style.

If there's an error, I'd prefer it to still show which tasks have been completed and which task the error is associated with.

This can sometimes lead to not being able to solve exercises in order. See exercism/go#1755

Code snippet output for subtests sometimes includes the whole test table

The README of the test runner says that if you conform to a certain structure, the code that is shown on the website for a certain sub test only includes the parts that are relevant for that subtest. See https://github.com/exercism/go-test-runner#subtest-format-specification

However, there seem to be quite a lot of cases where the code seemingly conforms to the specification given in the README but the subtests show the whole test table on the website which is confusing for the students.

Here is one example, I will try to add more later on.

This is TestFirstTurn from the BlackJack exercise:
https://github.com/exercism/go/blob/b8cccc8969db842b591f762e5426381a64880adc/exercises/concept/blackjack/blackjack_test.go#L175-L385

On the website the FirstTurn subtest code contains the full table of test cases (it is the same no matter whether the test failed or passed):

Resources for writing the test runner

This repository defines the common rules for all test runners:
https://github.com/exercism/automated-tests
There are basically 2 files in there, one defining things about docker, the other how the test runner interface should look like (input, output)

Looking at how other languages like C#, Ruby, JS, etc. have implemented their test runners can also help.

As far as the docker is concerned, the Dockerfile in the go-analyzer repo can be an additional help.

This repository currently only contains a proof of concept on how to convert the output of go test --json in the json output defined in the automated-tests repository above. (interface is probably not yet implemented 100%).

Steps I think need to be done

Building a complete script that runs go test --json and then transforms the output according to the defined interface.
Wrap that script into a docker container according the docker file rules.

Feel free to reorganise, rewrite as you see fit.