Currently get-tested outputs JSON:

$ get-tested my-project.cabal
["9.8.2", "9.10.1"]

It was nice if there was an option to print that in a more shell-friendly format, i.e. one version per line, unquoted:

$ get-tested --format=shell my-project.cabal
9.8.2
9.10.1

Then one could do something like for v in $(get-tested --shell); do cabal test all --with-compiler "ghc-${v}"; done...

We should encourage people to use the exact-version syntax, and not use version ranges. For instance

tested-with: GHC == { 8.6.3, 8.4.4, 8.2.2, 8.0.2, 7.10.3, 7.8.4, 7.6.3, 7.4.2 }

or

tested-with: GHC == 8.6.3, GHC == 8.4.4, GHC == 8.2.2, GHC == 8.0.2,
             GHC == 7.10.3, GHC == 7.8.4, GHC == 7.6.3, GHC == 7.4.2

This frees us from having to maintain a whitelist.

To support the * automagic * nature of this plugin, it'd be cool to additionally support base version bounds for generating CI. While this is convenient (don't have to specify a Tested-With clause), it's also great for bounds checking.

A base constraint with no upper bound would "opt in" to be tested, automatically, once new versions of GHC are released. This would alert library authors to breaking changes in their CI much more quickly than if they were to opt-in.

Likewise, a base constraint with no lower bound would have lots of old GHCs to test with. This would help prevent problems where a feature of a new GHC is used and the CI suite doesn't manually test old enough versions of GHC.

But, IMO, the best part about this would be that it would take into account the information that cabal actually uses when providing dependencies. If I say tested-with: GHC == 8.10 || == 9.0 || == 9.2 || == 9.4, but I've got base < 5, then cabal will happily try to build my library with GHC 8.2 or 9.6. Both are likely to break.

Is it possible to have a check mode, that tells you whether a given GHC version is in the get-tested stanza of a Cabal file or not?
What I mean is something like this:

$ get-tested get-tested.cabal
["9.10.1"]
$ get-tested check get-tested.cabal 9.10.1; echo $?
0
$ get-tested check get-tested.cabal 9.8.2; echo $?
1
$ get-tested check get-tested.cabal 9.8.2 9.10.1; echo $?
1

I know I can work around that (e.g. get-tested get-tested.cabal | jq --raw-output '.[]' | grep -q '9.10.1'), but that's rather hacky...

It would be great if this package was published on hackage. Then it would be automatically be picked up and packaged in nixpkgs.

Then you can load it into your own dimension. That's useful if you want to add other dimensions to your matrix. I think you can't actually do that at all with get-tested today since it emits the entire matrix definition.

That might also simplify things since you wouldn't need the OS flags to emit those parts of the matrix. More composable.

I mistyped the name of the cabal file in my CI setup, and I got this error message:

get-tested: mleh!
CallStack (from HasCallStack):
  error, called at app/Main.hs:28:15 in main:Main
Error: Process completed with exit code 1.

This is fun, but not helpful ;) a better error message would have been something like mleh! File foo.cabal could not be found!

I tried to use get-tested here: turion/rhine#201

But unfortunately, the list of versions is always empty. I reproduced locally:

$ ./get-tested --ubuntu --macos rhine/rhine.cabal 
matrix={"include":[]}

The corresponding section is:

tested-with:
  GHC == 8.8.4,
  GHC == 8.10.7,
  GHC == 9.0.2

Nix (at least in both in nixpkgs and haskell.nix) usually represents GHC versions as e.g. ghc865/ghc8107/ghc944. Would you be open to a convenience flag that allows to emit the GHC versions in this format, or do you consider this out of scope for this project?

Hi! Nice to meet you! First of all thanks for creating this tool and GH Action ❤️

I'm trying to push for the adoption of this GH Action at @stackbuilders for some of the repos we maintain. I was having a slightly unrelated conversation with regards to the macOS runners that GitHub offers, and wanted to test the macos-14 one since that uses Apple Silicon (macos-latest is based on macos-12 which is currently x86-64 based AFAIK). But I noticed that this tool doesn't currently support that, and that it has the *-latest versions hardcoded.

I think it would be nice to support choosing the runner version for each OS. Adding a new input to the action that allows one to set an explicit list of the OS runners one wants to support could be a possible approach.

I am willing to try to work on this, unless you prefer doing that. Let me know if you have any ideas/preferences with regards to how you would like to see this reflected in the action.

It would be cool if there was a Github action that one could simply use like:

jobs:
  get-tested:
    name: Generate matrix
    steps:
    -uses: Kleidukos/get-tested/v1

  build:
    needs: get-tested
    strategy:
      matrix: ${{ fromJSON(needs.get-tested.outputs.matrix) }}
...

That way, one saves on code to write, and dependabot could automatically bump versions.

I think it was nice if get-tested supported the extraction of the tested-with stanzas of multiple Cabal files; In particular those listed as packages in a cabal.project file. What I have in mind is something similar to the following:

1. Make the Cabal file an optional input on the CLI.
2. If no Cabal file was given, look for a cabal.project file in the current directory and its parents.
3. Resolve the Cabal files in the packages stanza of the cabal.project file.
4. Extract the get-tested stanzas of the packages.
5. Output the union of the collected stanzas. Or output a separate entry for each file (e.g. <file>: <listed versions>)

I think most of the work can be done by using cabal-install-parsers, but the real question is if that is useful feature.
Personally I think it would be useful for mono-repos like https://github.com/dhall-lang/dhall-haskell to keep the tested-with stanzas in sync with what was actually tested in the CI.

I often find myself debugging why it doesn't work as intended on Github Actions, which is cumbersome. It would be great if there was a verbose mode, or if get-tested would always produce debug output when used as an action. E.g. when I look at this:

https://github.com/turion/time-domain/actions/runs/9665355746/job/26662232443?pr=21#step:2:134

I can't really see what the output of the run was. If it could just echo that, it would already be helpful.