Comments (3)
The issue seems to be related to this where more recent versions of dbt output to stdout messages like Partial parse save file not found. Starting full parse.
which get inadvertently written to the SQLFluff output. So the result of sqlfluff lint
becomes e.g.
�[0m12:43:09 Partial parse save file not found. Starting full parse.
[{"filepath": "models/adyen/fact_adyen_dispute_transaction_report.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 15, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}]}, {"filepath": "models/adyen/fact_adyen_payment_accounting.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 14, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}, {"line_no": 16, "line_pos": 34, "code": "L009", "description": "Files must end with a single trailing newline."}]}]
instead of
[{"filepath": "models/adyen/fact_adyen_dispute_transaction_report.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 15, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}]}, {"filepath": "models/adyen/fact_adyen_payment_accounting.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 14, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}, {"line_no": 16, "line_pos": 34, "code": "L009", "description": "Files must end with a single trailing newline."}]}]
My workaround has been to pipe the results of the sqlfluff lint
to grep '^\['
to exclude any unwanted rows added to the output by dbt, e.g.
# Lint changed files if the mode is lint
if [[ "${SQLFLUFF_COMMAND:?}" == "lint" ]]; then
echo '::group:: Running sqlfluff 🐶 ...'
# Allow failures now, as reviewdog handles them
set +Eeuo pipefail
lint_results="sqlfluff-lint.json"
# shellcheck disable=SC2086,SC2046
sqlfluff lint \
--format json \
$(if [[ "x${SQLFLUFF_CONFIG}" != "x" ]]; then echo "--config ${SQLFLUFF_CONFIG}"; fi) \
$(if [[ "x${SQLFLUFF_DIALECT}" != "x" ]]; then echo "--dialect ${SQLFLUFF_DIALECT}"; fi) \
$(if [[ "x${SQLFLUFF_PROCESSES}" != "x" ]]; then echo "--processes ${SQLFLUFF_PROCESSES}"; fi) \
$(if [[ "x${SQLFLUFF_RULES}" != "x" ]]; then echo "--rules ${SQLFLUFF_RULES}"; fi) \
$(if [[ "x${SQLFLUFF_EXCLUDE_RULES}" != "x" ]]; then echo "--exclude-rules ${SQLFLUFF_EXCLUDE_RULES}"; fi) \
$(if [[ "x${SQLFLUFF_TEMPLATER}" != "x" ]]; then echo "--templater ${SQLFLUFF_TEMPLATER}"; fi) \
$(if [[ "x${SQLFLUFF_DISABLE_NOQA}" != "x" ]]; then echo "--disable-noqa ${SQLFLUFF_DISABLE_NOQA}"; fi) \
$changed_files |
grep '^\[' \
>> "$lint_results"
sqlfluff_exit_code="${PIPESTATUS[0]}"
echo "sqlfluff_exit_code = $sqlfluff_exit_code"
echo "echo and cat lint_results start"
echo "echo lint_results = $lint_results"
cat "$lint_results"
echo "echo and cat lint_results end"
echo "::set-output name=sqlfluff-results::$(cat <"$lint_results" | jq -r -c '.')" # Convert to a single line
echo "::set-output name=sqlfluff-exit-code::${sqlfluff_exit_code}"
set -Eeuo pipefail
echo '::endgroup::'
echo '::group:: Running reviewdog 🐶 ...'
# Allow failures now, as reviewdog handles them
set +Eeuo pipefail
lint_results_rdjson="sqlfluff-lint.rdjson"
cat "$lint_results" |
jq -r -f "${SCRIPT_DIR}/to-rdjson.jq" \
>> "$lint_results_rdjson"
echo "echo and cat lint_results_rdjson start"
echo "echo lint_results_rdjson = $lint_results_rdjson"
cat "$lint_results_rdjson"
cat "$lint_results"
echo "echo and cat lint_results_rdjson end"
cat <"$lint_results_rdjson" |
reviewdog -f=rdjson \
-name="sqlfluff-lint" \
-reporter="${REVIEWDOG_REPORTER}" \
-filter-mode="${REVIEWDOG_FILTER_MODE}" \
-fail-on-error="${REVIEWDOG_FAIL_ON_ERROR}" \
-level="${REVIEWDOG_LEVEL}"
reviewdog_return_code="${PIPESTATUS[1]}"
echo "::set-output name=sqlfluff-results-rdjson::$(cat <"$lint_results_rdjson" | jq -r -c '.')" # Convert to a single line
echo "::set-output name=reviewdog-return-code::${reviewdog_return_code}"
set -Eeuo pipefail
echo '::endgroup::'
exit $sqlfluff_exit_code
Possibly these problems have gone unnoticed by some because this GitHub Action's tests utilize dbt-bigquery==1.0.0 which was released in December 2021, and these stdout messages may have only started happening with a more recent dbt-bigquery release.
from action-sqlfluff.
Seeing this as well and was also experiencing it with running SQLFluff lint directly.
So was doing a similar grep before piping into the annotations.json
from action-sqlfluff.
a better way to resolve (i.e. remove) Partial parse save file not found. Starting full parse.
is to set partial_parse: False
in the profiles.yml
file.
We're using a dummy profiles.yml
being on dbt Cloud, but having:
config:
partial_parse: False
allows sqlfluff_command: "lint"
to run without issue.
from action-sqlfluff.
Related Issues (20)
- Parse error when running on latest version HOT 7
- Can't get changed files. HOT 3
- SQLfluff lint fails, but workflow passed
- Wrong JSON output with tee and dbt templater HOT 1
- SqlFluff action causes Deprecation Warning on GitHub HOT 1
- Current version of sqlfluff is 1.4.5 HOT 2
- reviewdog outputs raw format instead of code suggestion HOT 1
- I get an error for my github action even after I fix all the linting issues HOT 1
- Enhancement: enable --show-lint-violations for sqlfluff fix HOT 1
- SQLFluff failing due to ImportError HOT 4
- SQLFluff failing due to ModuleNotFoundError: No module named 'pytz' HOT 4
- Action doesn't pick up space in paths HOT 3
- fatal: --unshallow on a complete repository does not make sense HOT 4
- SQLFluff not leaving comments on GitHub Pull Requests on lint mode HOT 1
- sqlfluff v2.0 HOT 2
- action-sqlfluff GH action outputs raw format instead of inline code suggestion HOT 1
- Alternate working directory configuration not respected (dbt_project.yml) HOT 4
- Security Policy violation Branch Protection HOT 2
- Security Policy violation SECURITY.md HOT 149
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from action-sqlfluff.