Giter Club home page Giter Club logo

Comments (3)

kieronellis avatar kieronellis commented on June 12, 2024

The issue seems to be related to this where more recent versions of dbt output to stdout messages like Partial parse save file not found. Starting full parse. which get inadvertently written to the SQLFluff output. So the result of sqlfluff lint becomes e.g.

�[0m12:43:09  Partial parse save file not found. Starting full parse.
[{"filepath": "models/adyen/fact_adyen_dispute_transaction_report.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 15, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}]}, {"filepath": "models/adyen/fact_adyen_payment_accounting.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 14, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}, {"line_no": 16, "line_pos": 34, "code": "L009", "description": "Files must end with a single trailing newline."}]}]

instead of

[{"filepath": "models/adyen/fact_adyen_dispute_transaction_report.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 15, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}]}, {"filepath": "models/adyen/fact_adyen_payment_accounting.sql", "violations": [{"line_no": 13, "line_pos": 1, "code": "L036", "description": "Select targets should be on a new line unless there is only one select target."}, {"line_no": 13, "line_pos": 1, "code": "L044", "description": "Query produces an unknown number of result columns."}, {"line_no": 13, "line_pos": 7, "code": "L001", "description": "Unnecessary trailing whitespace."}, {"line_no": 14, "line_pos": 1, "code": "L003", "description": "Expected 1 indentation, found 0 [compared to line 03]"}, {"line_no": 16, "line_pos": 34, "code": "L009", "description": "Files must end with a single trailing newline."}]}]

My workaround has been to pipe the results of the sqlfluff lint to grep '^\[' to exclude any unwanted rows added to the output by dbt, e.g.


# Lint changed files if the mode is lint
if [[ "${SQLFLUFF_COMMAND:?}" == "lint" ]]; then
  echo '::group:: Running sqlfluff 🐶 ...'
  # Allow failures now, as reviewdog handles them
  set +Eeuo pipefail
  lint_results="sqlfluff-lint.json"
  # shellcheck disable=SC2086,SC2046
  sqlfluff lint \
      --format json \
      $(if [[ "x${SQLFLUFF_CONFIG}" != "x" ]]; then echo "--config ${SQLFLUFF_CONFIG}"; fi) \
      $(if [[ "x${SQLFLUFF_DIALECT}" != "x" ]]; then echo "--dialect ${SQLFLUFF_DIALECT}"; fi) \
      $(if [[ "x${SQLFLUFF_PROCESSES}" != "x" ]]; then echo "--processes ${SQLFLUFF_PROCESSES}"; fi) \
      $(if [[ "x${SQLFLUFF_RULES}" != "x" ]]; then echo "--rules ${SQLFLUFF_RULES}"; fi) \
      $(if [[ "x${SQLFLUFF_EXCLUDE_RULES}" != "x" ]]; then echo "--exclude-rules ${SQLFLUFF_EXCLUDE_RULES}"; fi) \
      $(if [[ "x${SQLFLUFF_TEMPLATER}" != "x" ]]; then echo "--templater ${SQLFLUFF_TEMPLATER}"; fi) \
      $(if [[ "x${SQLFLUFF_DISABLE_NOQA}" != "x" ]]; then echo "--disable-noqa ${SQLFLUFF_DISABLE_NOQA}"; fi) \
      $changed_files |
    grep '^\[' \
    >> "$lint_results"
  sqlfluff_exit_code="${PIPESTATUS[0]}"
  echo "sqlfluff_exit_code = $sqlfluff_exit_code"

  echo "echo and cat lint_results start"
  echo "echo lint_results = $lint_results"
  cat "$lint_results"
  echo "echo and cat lint_results end"

  echo "::set-output name=sqlfluff-results::$(cat <"$lint_results" | jq -r -c '.')" # Convert to a single line
  echo "::set-output name=sqlfluff-exit-code::${sqlfluff_exit_code}"

  set -Eeuo pipefail
  echo '::endgroup::'

  echo '::group:: Running reviewdog 🐶 ...'
  # Allow failures now, as reviewdog handles them
  set +Eeuo pipefail

  lint_results_rdjson="sqlfluff-lint.rdjson"
  cat "$lint_results" |
    jq -r -f "${SCRIPT_DIR}/to-rdjson.jq" \
    >> "$lint_results_rdjson"
  echo "echo and cat lint_results_rdjson start"
  echo "echo lint_results_rdjson = $lint_results_rdjson"
  cat "$lint_results_rdjson"
  cat "$lint_results"
  echo "echo and cat lint_results_rdjson end"

  cat <"$lint_results_rdjson" |
    reviewdog -f=rdjson \
      -name="sqlfluff-lint" \
      -reporter="${REVIEWDOG_REPORTER}" \
      -filter-mode="${REVIEWDOG_FILTER_MODE}" \
      -fail-on-error="${REVIEWDOG_FAIL_ON_ERROR}" \
      -level="${REVIEWDOG_LEVEL}"
  reviewdog_return_code="${PIPESTATUS[1]}"

  echo "::set-output name=sqlfluff-results-rdjson::$(cat <"$lint_results_rdjson" | jq -r -c '.')" # Convert to a single line
  echo "::set-output name=reviewdog-return-code::${reviewdog_return_code}"

  set -Eeuo pipefail
  echo '::endgroup::'

  exit $sqlfluff_exit_code

Possibly these problems have gone unnoticed by some because this GitHub Action's tests utilize dbt-bigquery==1.0.0 which was released in December 2021, and these stdout messages may have only started happening with a more recent dbt-bigquery release.

from action-sqlfluff.

rossserven avatar rossserven commented on June 12, 2024

Seeing this as well and was also experiencing it with running SQLFluff lint directly.

So was doing a similar grep before piping into the annotations.json

from action-sqlfluff.

rossserven avatar rossserven commented on June 12, 2024

a better way to resolve (i.e. remove) Partial parse save file not found. Starting full parse. is to set partial_parse: False in the profiles.yml file.

We're using a dummy profiles.yml being on dbt Cloud, but having:

config:
    partial_parse: False

allows sqlfluff_command: "lint" to run without issue.

from action-sqlfluff.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.