Support different failure levels, similar to the Gradle IntelliJ Plugin about intellij-platform-plugin-verifier-action HOT 11 CLOSED

Hey @peterlin741 - thank you for PR #12 for the quick fix!

I just ran two executions on a sample plugin of mine that is in a known failure state right now; both showed failures as expected.

For my own reference

• Before: https://github.com/ChrisCarini/sample-intellij-plugin/runs/1362984283?check_suite_focus=true
• After: https://github.com/ChrisCarini/sample-intellij-plugin/runs/1363028588?check_suite_focus=true

Would you mind also doing a test, since my failure does not have extra characters that the removal of the $ would catch? You can do this by temporarily changing the workflow file to below:

uses: ChrisCarini/intellij-platform-plugin-verifier-action@d3de73c8c612421670001960de64e97916093571

Once you report back that it's fixed for you, I'll go ahead and release a new version of the action (and update the latest tag as well).

Thank you again! Really appreciate the contribution (and, of course, raising this issue)!

from intellij-platform-plugin-verifier-action.

@peterlin741 - Nice catch! I totally forgot to do that! I just updated the README.md (where the sample docs are pulled from) and just changed everything to reference the latest tag. Figure that way I won't have to make future-updates to the README.md when new versions are released.

from intellij-platform-plugin-verifier-action.

Hi @peterlin741 - as mentioned in the Slack thread (the one you linked above), would you mind sharing an execution that showed as passed when it should have failed?

Thank you for opening this issue! Hopefully we can get to the bottom of the root cause and make the needed improvements!

from intellij-platform-plugin-verifier-action.

Hi Chris, Thanks for taking a look! I can't bulk share the logs, since this is proprietary code, but I can give you more background.

• IntelliJ 2020.2 refactored and introduced several new libraries, which were incompatible with versions beforehand. Running my 2020.2 compatible plugin branch against 2019.3 should fail, but the status is displayed as passing in the github action status

• Some output from the Verify Plugin on IntelliJ Platforms step include:

2020-11-03T16:08:44 [main] INFO  verification - Finished 1 of 2 verifications (in 26.1 s): IC-193.5233.102 against com.google.gct.core:20.12.1-SNAPSHOT: 21 compatibility problems, some of which may be caused by missing optional dependencies. 66 usages of deprecated API. 4 usages of experimental API. 1 usage of internal API
2020-11-03T16:08:47 [main] INFO  verification - Finished 2 of 2 verifications (in 28.4 s): IU-193.5233.102 against com.google.gct.core:20.12.1-SNAPSHOT: 25 compatibility problems, some of which may be caused by missing optional dependency. 66 usages of deprecated API. 4 usages of experimental API. 1 usage of internal API
Plugin com.google.gct.core:20.12.1-SNAPSHOT against IC-193.5233.102: 21 compatibility problems, some of which may be caused by missing optional dependencies. 66 usages of deprecated API. 4 usages of experimental API. 1 usage of internal API
Missing dependencies: 

• Looking through the source code in entrypoint.sh, you do check for the above:

# Validate the log; fail if we find compatibility problems.
if (grep -E -q "^Plugin (.*) against .*: .* compatibility problems?$" "$VERIFICATION_OUTPUT_LOG"); then
  error_wall
elif egrep -q "^The following files specified for the verification are not valid plugins:$" "$VERIFICATION_OUTPUT_LOG"; then
  error_wall
fi

• I created a temp file with the output I provided above and tried the grep statements, without the -q quiet flag. It looks like the grep statement works when I remove the$ at the end of the regex, since $ indicates the end of a line, and my project's logs include other warnings after the match.

skellige% grep -E "^Plugin (.*) against .*: .* compatibility problems?$" temp.txt
skellige% grep -E "^Plugin (.*) against .*: .* compatibility problems?" temp.txt 
Plugin com.google.gct.core:20.12.1-SNAPSHOT against IC-193.5233.102: 21 compatibility problems, some of which may be caused by missing optional dependencies. 66 usages of deprecated API. 4 usages of experimental API. 1 usage of internal API

from intellij-platform-plugin-verifier-action.

Hi @peterlin741 - thank you for digging in and figuring this out! You provided the information I was hoping to see from the logs (I didn't want bulk logs, just specific to this GH action, which shouldn't be printing any proprietary things afaik).

From the sounds of it, this could easily be fixed by removing the $ from the regex, correct? I'm almost wondering if it makes sense to (a) do this quick fix, or (b) hold off for some overall improvements on the errors checked (based on what you provided), or, (c) a combination of both (a) and (b). Let me know your thoughts, I'm happy with any of the options.

from intellij-platform-plugin-verifier-action.

hi @ChrisCarini, no problem! That's correct, removing the$ from the regex should fix it. I think (c) a combination of the above is probably best, but I would definitely like the quick fix. If this is broken for me, it is probably currently broken for most users using the latest version of the IntelliJ Plugin Verifier, specifically checking for compatibility issues. If users are using the plugin verifier, their project is most likely older and also has other warnings, breaking the regex.

This short term solution fixes my immediate problem. I don't necessarily require the long term solution as of now, but here are my thoughts: I haven't looked too closely at the implementation details/changes required, but I think in the long term, it'll be easier in terms of maintainability for you to follow the enum class in the source code, since new versions of the IntelliJ Plugin Verifier may change the output format, causing breaking changes, and require additional debugging. I think adding a list of failure level inputs will also be useful, because users will be able to create separate github actions, for checking the status of different failures, but that's just my two cents.

from intellij-platform-plugin-verifier-action.

hi @ChrisCarini, everything looks good on my end!

Testing Notes:

• IntelliJ 2020.2 refactored several libraries, which were incompatible with previous versions
• Tested a 2020.2 compatible version of my plugin, which should only work with IntelliJ 2020.2 and IntelliJ EAP
• Tested a 2019.3 compatible version of my plugin, which should only work with IntelliJ 2019.3 and IntelliJ 2020.1
• I tested both the community and ultimate editions

Test 1

• Running a 2020.2 compatible plugin against 2020.2 and EAP succeeds as expected. I do not see any compatibility issues in the logs.

Test 2

• Running a 2020.2 compatible plugin against 2019.3 and 2020.1 fails as expected. I see the expected failure statement in the logs:

Plugin com.google.gct.core:20.12.1-SNAPSHOT against IC-193.5233.102: 21 compatibility problems, some of which may be caused by missing optional dependencies. 66 usages of deprecated API. 4 usages of experimental API. 1 usage of internal API

Test 3

• Running a 2019.3 compatible plugin against 2019.3 and 2020.1 succeeds as expected. I do not see any compatibility issues in the logs.

• yaml IDE versions same as screenshot in Test 2

Test 4

• Running a 2019.3 compatible plugin against 2020.2 and EAP fails as expected. I see the expected failure statement in the logs:

Plugin com.google.gct.core:20.12.1-SNAPSHOT against IU-202.6397.94: 19 compatibility problems. 5 usages of scheduled for removal API and 147 usages of deprecated API. 4 usages of experimental API. 2 usages of internal API

• yaml IDE versions same as screenshot in Test 1

from intellij-platform-plugin-verifier-action.

Great! Thank you again for the detail @peterlin741 ! I've just released v1.0.4 with your change, and also moved the latest tag to point to this version/commit as well!

I'm going to leave this issue open, and the remaining work to be accomplished for this issue is to get the remaining strings added into the action.

from intellij-platform-plugin-verifier-action.

thanks @ChrisCarini! Don't forget to update the sample docs on the homepage to v1.0.4

from intellij-platform-plugin-verifier-action.

@all-contributors please add @peterlin741 for bug, idea, userTesting

from intellij-platform-plugin-verifier-action.

@ChrisCarini

I've put up a pull request to add @peterlin741! 🎉

from intellij-platform-plugin-verifier-action.