mithro / actions-includes Goto Github PK

Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.

License: Apache License 2.0

Python 94.51% Makefile 4.33% Dockerfile 1.14% Shell 0.02%

actions-includes's People

Contributors

Stargazers

Watchers

Forkers

drdanz awoimbee antmicro asford bigwheel tdsmith trent-abc jcourteau shakahl

actions-includes's Issues

No installation instructions

Can you please add (1) prereqs and (2) installation instructions to the README?

I assumed I could use poetry add --dev actions-includes. However, when I tried this, I got a SolverProblemError "Because actions-includes (0.0.post117) requires Python >=3.8." (I'm still using Python 3.7, so maybe I can't use actions-includes.)

Doesn't handle "toJSON" function

I have this in my main src action (not in the include file) and it causes an error:

        env:
          GITHUB_CONTEXT: ${{ toJSON(github) }}

Error dump is:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__main__.py", line 25, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 1032, in main
    out_data = expand_workflow(current_action, to_path, insert_check)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 917, in expand_workflow
    data = yaml_load(current_workflow, '\n'.join(workflow_data))
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 818, in yaml_load
    return yaml.load(yaml_data, Loader=RoundTripLoaderWithExp)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/main.py", line 1071, in load
    return loader._constructor.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 122, in get_single_data
    return self.construct_document(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 132, in construct_document
    for _dummy in generator:
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1617, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 730, in construct_mapping
    return RoundTripConstructor.construct_mapping(self, node, maptyp, deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1500, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 155, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 197, in construct_non_recursive_object
    for _dummy in generator:
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1617, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 730, in construct_mapping
    return RoundTripConstructor.construct_mapping(self, node, maptyp, deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1500, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 155, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 197, in construct_non_recursive_object
    for _dummy in generator:
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1617, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 730, in construct_mapping
    return RoundTripConstructor.construct_mapping(self, node, maptyp, deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1500, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 155, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 197, in construct_non_recursive_object
    for _dummy in generator:
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1609, in construct_yaml_seq
    data.extend(self.construct_rt_sequence(node, data))
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1341, in construct_rt_sequence
    ret_val.append(self.construct_object(child, deep=deep))
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 155, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 197, in construct_non_recursive_object
    for _dummy in generator:
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1617, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 730, in construct_mapping
    return RoundTripConstructor.construct_mapping(self, node, maptyp, deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1500, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 155, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 197, in construct_non_recursive_object
    for _dummy in generator:
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1617, in construct_yaml_map
    self.construct_mapping(node, data, deep=True)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 730, in construct_mapping
    return RoundTripConstructor.construct_mapping(self, node, maptyp, deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 1500, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 155, in construct_object
    data = self.construct_non_recursive_object(node)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 190, in construct_non_recursive_object
    data = constructor(self, node)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 687, in construct_expression
    v = exp.parse(v)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/expressions.py", line 1346, in parse
    return simplify(exp[3:-2].strip())
  File "/usr/local/lib/python3.9/site-packages/actions_includes/expressions.py", line 1301, in simplify
    o = tokens_eval(tokenizer(exp), context)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/expressions.py", line 373, in tokens_eval
    assert not isinstance(t, list), t
AssertionError: [<class 'exp.ToJSONF'>, Value(github)]

Support for processing entire folder or wildcard

Rather than using

python -m actions_includes ./.github/src/my-action.yml ./.github/workflows/my-action.yml

individually for each action, it would be nice if it supported folders, such as:

python -m actions_includes ./.github/src ./.github/workflows

or wildcards:

python -m actions_includes ./.github/src/*.yml ./.github/workflows/*.yml

Figure out a good syntax for "workflow level job includes"

Unique step ids in included plans?

Wow. This project is amazing. 👏 👏 👏

Is is possible to include unique step ids in the expanded plan, so that step outputs in includes: template?

As a motivating example... https://github.com/peter-evans/create-or-update-comment recommends a two-step workflow to find and update a PR comment:

    - name: Find Comment
      uses: peter-evans/find-comment@v1
      id: fc
      with:
        issue-number: ${{ github.event.pull_request.number }}
        comment-author: 'github-actions[bot]'
        body-includes: Build output

    - name: Create or update comment
      uses: peter-evans/create-or-update-comment@v1
      with:
        comment-id: ${{ steps.fc.outputs.comment-id }}
        issue-number: ${{ github.event.pull_request.number }}
        body: |
          Build output
          ${{ steps.build.outputs.build-log }}
        edit-mode: replace

Here steps.fc.outputs.commend-id and steps.id.outputs.build-log are used in template expansions to pass step outputs into the resulting comment. This would be an ideal application of actions-includes, because you may want to run a few templated actions each outputting a different comment. This is a bit hard though, because you don't have access to unique ids per step.

I've done a workaround, where each unique required step is a different input variable of the form:

name: build
description: Build target and post comment.
inputs:
  target:
    required: True
  build-id:
    required: True
  comment-id:
    required: True
runs:
  using: "includes"
  steps:
    - id: ${{ inputs.build-id }}
      run: build ${{ inputs.target }}
    - id: ${{inputs.comment-id}}
      uses: peter-evans/find-comment@v1
      with:
        issue-number: ${{ github.event.pull_request.number }}
        comment-author: 'github-actions[bot]'
        body-includes: Build output ${{ inputs.target }}

    - uses: peter-evans/create-or-update-comment@v1
      with:
        comment-id: ${{ steps[inputs.comment-id].outputs.comment-id }}
        issue-number: ${{ github.event.pull_request.number }}
        body: |
          Build output ${{ inputs.target }}
          ${{ steps[inputs.build-id].outputs.stdout }}
        edit-mode: replace

However this feels a little clunky as the includer needs to specify a bunch of unique step ids like:

jobs:
  build:
    - name: base
      includes: /build
      with:
        target: base
        build-id: build-base
        comment-id: build-base
    - name: app
      includes: /build
      with:
        target: app
        build-id: build-app
        comment-id: build-app
    - name: app2
      includes: /build
      with:
        target: app2
        build-id: build-app2
        comment-id: build-app2

Not too crazy, but am I missing some logic that would allow me to express these unique ids and access their outputs in the expanded workflow?

Support for `includes-script` in custom places

It would be nice to be able to include scripts in custom places with custom tags.

For example, it would be nice if a script could be inserted using the command tag in when using the nick-invision/retry action

comment_handling attribute gone in ruamel?

  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/parser.py", line 637, in parse_block_mapping_value
    return self.parse_block_node_or_indentless_sequence()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/parser.py", line 351, in parse_block_node_or_indentless_sequence
    return self.parse_node(block=True, indentless_sequence=True)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/parser.py", line 414, in parse_node
    if self.loader and self.loader.comment_handling is None:
AttributeError: 'RoundTripLoaderWithExp' object has no attribute 'comment_handling'
Source of .github/workflows/test.actions-ifexpands.yml is ../../tests/workflows/actions-ifexpands.yml found at mithro/actions-includes/tests/workflows/actions-ifexpands.yml@d3712aa248266971c5aa89b44ff4e5044f259e8d

Make sure includes work with standard composite actions

It would be great if you could just include an standard composite action. It may actually already work and just need to add a test to actually confirm that is the case.

Latest version is broken

non existing include_action referenced:

actions-includes/actions_includes/__init__.py

Line 963 in bb6134d

'uses': include_action,

line 954: set only if running in this repo
line 963: used

Improve documentation

Mention that the script add check step which makes sure the yaml file is up to date.
Mention how the evaluation stuff works and where you need the ${{ }}.
Better examples...

If clause not properly transpiled using workflow includes

if: needs.detect-changes.outputs.trigger == 'true'

is being transpiled to

if: true

Upload the tool to PyPi

It would be good to publish the tool to PyPi.

Check against the YAML Schemas

Actions - SchemaStore/schemastore#850 & https://json.schemastore.org/github-action
Workflows - SchemaStore/schemastore#853 & https://json.schemastore.org/github-workflow

Add a "find .yaml workflows with includes and expand" mode

This should make it easy for people to add the GitHub action to their repository.

Requires running from a top-level git repository

When using the docker container run command to execute, it gives this error:

fatal: not a git repository (or any parent up to mount point /github)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__main__.py", line 25, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/site-packages/actions_includes/__init__.py", line 999, in main
    git_root_output = subprocess.check_output(
  File "/usr/local/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'rev-parse', '--show-toplevel']' returned non-zero exit status 128.

if you run it from a folder that is not a git repository. Would be nice to remove the git dependency here. As a yml processor it should just be talking the input src files and outputting the resulting workflow files regardless of whether I'm in a git checkout repo or not.

Simple yet another alternative yq eval 'explode(.)'

https://stackoverflow.com/questions/67368724/share-same-steps-for-different-github-actions-jobs/67391185#67391185

TL;DR

I solved my problem with shell tool yq

yq eval 'explode(.)' file.yml

Long answer

GitHub Workflow description in YAML does not support anchors.
There are several workarounds => anyway they come to building-editing workflow yaml from source.
So I suggest yet another one make-workflows.sh based on YAML tool yq.

USAGE

Move your workflows to .github/*.src.yml
Put make-workflows.sh to directory .github/
(optional) Copy or link pre-commit.sh to .git/hooks/pre-commit
Like ln -s ../../.github/pre-commit.sh .git/hooks/pre-commit

File make-workflows.sh

#!/usr/bin/env bash
set -euo pipefail

## The script expands '*.src.yml' from $1(default: script's directory) 
## to $2 (default:subdirectory 'workflows') with corresponding name '*.yml'
## Main goal is to dereference YAML anchors.
## Deals only with Git cached/indexed files
## Set -x to debug

script_dir=$(dirname $(realpath "$0"))
dir_from=${1:-${script_dir}}
dir_to=${2:-workflows}
cd $dir_from

edited=
for f in $(git status -s -- \*.src.yml | sed 's,^.. ,,') ;do
    readonly out=$(echo $f | sed s,.src.yml\$,.yml,)
    readonly wout=$dir_to/$out
    readonly tempout=$(mktemp)
    trap "rm -f $tempout" EXIT
    echo >>$tempout "## DO NOT EDIT"
    echo >>$tempout "## Generated from $f with $(basename $0)"
    echo >>$tempout ""
    yq eval 'explode(.)' $f >>$tempout
    if ! diff -q $wout $tempout &>/dev/null ;then
        mv $tempout $wout
        edited+="'$out' "
    fi
done

if [[ -n "$edited" ]] 
then echo >&2 "make-workflows: these files were edited: $edited"
else echo >&2 "make-workflows: everything is up to date"
fi

File pre-commit.sh

#!/usr/bin/env bash
set -euo pipefail

gitroot=$(git rev-parse --show-toplevel)

cd $gitroot
./.github/make-workflows.sh
git add .github/workflows

Auto replace full expanded expressions in strings

Current if you have

- name: 📤 Publish ${{ inputs.type }} to Test PyPI

and you set inputs.type: wheels you end up with

- name: 📤 Publish ${{ 'wheels' }} to Test PyPI

It would be nicer if you ended up with;

- name: 📤 Publish wheels to Test PyPI

Generate a warning that actions-includes is deprecated / abandoned

As suggested by @awoimbee at #45 (comment)

Fix comments lining up from included actions

They need to be indented more.

Make mithro/actions-includes@main work with private workflows

Executing a check at the beginning of generated workflow doesn't work with private workflows.

Would it be possible to either support private repositories or add possibility to turn off this step?

includes-script in included actions

It would be nice to be able to use includes-script in actions included with includes.

For example:

.github/workflows-src/workflow.yml

jobs:
  job1:
    steps:
    - includes: /an_action

.github/actions/an_action/action.yml (or maybe .github/actions-src/an_action/action.yml)

runs:
  using: "includes"
  steps:
    - name: An action
      includes-script: script.py

.github/actions/an_action/script.py (or maybe .github/actions-src/an_action/script.py)

    print('Hello world')

would result in this workflow:

.github/workflows-src/workflow.yml

jobs:
  job1:
    steps:
    - name: An action
      shell: python
      run: |
         print('Hello world')

Convert action to use docker

The tool needs;

Latest Python version to get order preserving dicts.
python3-yaml installed.

These dependencies mean it would work well as a docker action.

Add test which checks action doesn't run if yaml file is out of date.

UnicodeEncodeError: 'charmap' codec can't encode character '\u23f0' in position 328: character maps to <undefined>_

I have a very simple test here:
https://github.com/devkeydet/actions-includes-test/blob/main/.github/workflows-src/convertme.yml

I installed actions-includes using pip. I run the following command:

python -m actions_includes ./.github/workflows-src/convertme.yml ./.github/workflows/converted.yml

I get the following error:
_Expanding ./.github/workflows-src/convertme.yml into ./.github/workflows/converted.yml
Expanding workflow file from: C:\github\actions-includes-test.github\workflows-src\convertme.yml
to: .github\workflows\converted.yml
Loading yaml file C:\github\actions-includes-test.github\workflows-src\convertme.yml with contents md5 of 7ce0775b6ab05a87deb53ae9d298df80
get_action_data: C:\github\actions-includes-test.github\workflows-src\convertme.yml ./actions/basic C:\github\actions-includes-test\actions\basic
Including: C:\github\actions-includes-test\actions\basic/action.yml
Loading yaml file C:\github\actions-includes-test\actions\basic/action.yml with contents md5 of bb475f262af8e4df34241a188a4f451f
Traceback (most recent call last):
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 197, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\Marc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\actions_includes_main.py", line 25, in
sys.exit(main())
File "C:\Users\Marc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\actions_includes_init.py", line 1035, in main
f.write(out_data)
File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.1776.0_x64__qbz5n2kfra8p0\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u23f0' in position 328: character maps to

Support actions outputs

I tried to include an action which would produce an output, but these instructions were ignored by the script.

Add tests for the GitHub expression evaluator / simplifier code

The tool does some nice stuff around if an input makes a step's if statement a static value, then it removes the step (if: false) or if: condition when if: true.

Enable modes of operation

The tool should have multiple modes of operations;

Update all
Check (and potentially Update) single
Manual expand

Typo in repo description

preprosing -> preprocessing

Find all users and get them to migrate off actions-includes

Given this project is mostly abandoned, it would be good to notify any users and get them to migrate onto a better supported solution.

Why does the included action need to be action.yml?

I have an include file within .github/includes/my-action.yml

When using

    steps:
      - includes: my-action

it tries to find .github/includes/actions/my-action/action.yml file, which doesn't exist.

If I try to use:

    steps:
      - includes: .github/includes/my-action

Then it tries to find .github/includes/my-action/action.yml file, which doesn't exist.

If I try to use:

    steps:
      - includes: .github/includes/my-action.yml

Then it tries to find .github/includes/my-action.yml/action.yml file, which doesn't exist.

In all cases, it seems to have this action.yml filename hardcoded somewhere. Would be nice if I could just point to a specific file and not make assumptions about folder structure or file names.

Output revision information about included file into output

It would be good if the tool would include information about the revision of the included files in the output.