Giter Club home page Giter Club logo

liquibase-bigquery's Introduction

Liquibase Build and Test Quality Gate Status

Liquibase helps millions of developers track, version, and deploy database schema changes. It will help you to:

  • Control database schema changes for specific versions
  • Eliminate errors and delays when releasing databases
  • Automatically order scripts for deployment
  • Easily rollback changes
  • Collaborate with tools you already use

This repository contains the main source code for Liquibase Open Source. For more information about the product, see the Liquibase website.

Liquibase Automation and Integrations

Liquibase Open Source has built-in support for a variety of databases. Databases that are not part of Liquibase Open Source require extensions that you can download for free. Here is the full list of supported databases.

Liquibase can be integrated with Maven, Ant, Gradle, Spring Boot, and other CI/CD tools. For a full list, see Liquibase Tools & Integrations. You can use Liquibase with GitHub Actions, Spinnaker, and many different workflows.

Install and Run Liquibase

System Requirements

Liquibase system requirements can be found on the Download Liquibase page.

An H2 in-memory database example for CLI

  1. Download and run the appropriate installer.
  2. Make sure to add Liquibase to your PATH.
  3. Copy the included examples directory to the needed location.
  4. Open your CLI and navigate to your examples/sql or examples/xml directory.
  5. Start the included H2 database with the liquibase init start-h2 command.
  6. Run the liquibase update command.
  7. Run the liquibase history command to see what has executed!

See also how to get started with Liquibase in minutes or refer to our Installing Liquibase documentation page for more details.

Documentation

Visit the Liquibase Documentation website to find the information on how Liquibase works.

Courses

Learn all about Liquibase by taking our free online courses at Liquibase University.

Want to help?

Want to file a bug or improve documentation? Excellent! Read up on our guidelines for contributing!

Contribute code

Use our step-by-step instructions for contributing code to the Liquibase open source project.

Join the Liquibase Community

Earn points for your achievements and contributions, collect and show off your badges, add accreditations to your LinkedIn. Learn more about the pathway to Legend and benefits. Enjoy being part of the community!

Liquibase Extensions

Provide more database support and features for Liquibase.

License

Liquibase Open Source is licensed under the Apache 2.0 License.

Liquibase Pro has additional features and support and is commercially licensed.

LIQUIBASE is a registered trademark of Liquibase Inc.

Liquibase Forum

Liquibase Blog

Get Support & Advanced Features

Publish Release Manual Trigger to Sonatype

  1. When a PO (Product Owner) or a Team Leader navigates to Publish a release from here -> https://github.com/liquibase/liquibase/releases/, the workflow from /workflow/release-published.yml job is triggered.
  2. When a release is triggered, the workflow file will stop after Setup step and an email will be sent out to the list of approvers mentioned in job manual_trigger_deployment. You can click on the link and perform anyone of the options mentioned in description.
  3. A minimum of 2 approvers are needed in order for the other jobs such as deploy_maven, deploy_javadocs, publish_to_github_packages, etc to be executed.
  4. When you view the GitHub PR, make sure to verify the version which is being published. It should say something like Deploying v4.20.0 to sonatype

liquibase-bigquery's People

Contributors

bimission avatar danipiario avatar dependabot[bot] avatar ewojtach avatar filipelautert avatar github-actions[bot] avatar ismailsimsek avatar jandroav avatar karticn-google avatar kevin-atx avatar kushnirykoleh avatar liquibot avatar nvoxland avatar pavlotytarchuk avatar r2-lf avatar sayalim0412 avatar suryaaki2 avatar tatibeans avatar tylorjg avatar vitaliimak avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

liquibase-bigquery's Issues

Liquibase biguqery 4.28.0 unable to find valid certificate path

Community Note

  • Please vote on this issue by adding a ๐Ÿ‘ reaction
    to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
    not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Debug Output

Caused by: liquibase.exception.DatabaseException: Connection could not be created to jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<project_id>;Location=EU;OAuthType=3;DefaultDataset=himanshu_local_cdc;Timeout=3600; with driver com.simba.googlebigquery.jdbc.Driver.  [Simba][BigQueryJDBCDriver](100004) HttpTransport IO error : Error getting subject token from metadata server: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
	at liquibase.ext.bigquery.database.BigqueryConnection.openConn(BigqueryConnection.java:106)
	at liquibase.ext.bigquery.database.BigqueryConnection.open(BigqueryConnection.java:94)
	at liquibase.database.ConnectionServiceFactory.create(ConnectionServiceFactory.java:32)
	... 30 more

Panic Output

NA

Steps to Reproduce

  1. Use Liquibase 4.28.0 in github action as container
  2. Download bigquery jdbc jars and liquibase-bigquery jar and copy it inside liquibase/lib
  3. Use github action google-github-actions/auth@v2 to setup WIF, this service account has bigquery admin permissions
  4. Finally run liquibase using command below
liquibase \
            --driver=com.simba.googlebigquery.jdbc.Driver \
            --url="jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<project_id>;Location=EU;OAuthType=3;DefaultDataset=himanshu_local_cdc;Timeout=3600;" \
            --changeLogFile="changelog.xml" \
            --searchPath="./liquibase/cdc-tables" \
            --log-level debug \
            update
  1. I have also tried using encrypt=false;trustServerCertificate=true; as suggested on post here even thought it was for sqlserver but error was on the same lines but still same issue.

Expected Behavior

Liquibase changeset should have got executed

Actual Behavior

Error below

Caused by: liquibase.exception.DatabaseException: Connection could not be created to jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=<project_id>Location=EU;OAuthType=3;DefaultDataset=himanshu_local_cdc;Timeout=3600; with driver com.simba.googlebigquery.jdbc.Driver.  [Simba][BigQueryJDBCDriver](100004) HttpTransport IO error : Error getting subject token from metadata server: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target.
	at liquibase.ext.bigquery.database.BigqueryConnection.openConn(BigqueryConnection.java:106)
	at liquibase.ext.bigquery.database.BigqueryConnection.open(BigqueryConnection.java:94)
	at liquibase.database.ConnectionServiceFactory.create(ConnectionServiceFactory.java:32)
	... 30 more

References

  • #0000

EXEC_JOB_EXECUTION_ERR when using CASE statements in Stored Procedures

When creating Stored Procedures containing CASE statements like below, Liquibase throws liquibase.exception.DatabaseException and fails with EXEC_JOB_EXECUTION_ERR

Sample Stored Procedure

Note: All the referenced tables, udfs are already in place.

--liquibase formatted sql
--changeset gcp:1 runOnChange: true

CREATE OR REPLACE PROCEDURE `project.dataset.sp`(load_dt DATE)
BEGIN

  BEGIN TRANSACTION;

 --Full Refresh Table truncate
  TRUNCATE TABLE `project.dataset.sp`;

  --Insert into structured table
  INSERT INTO `project.dataset.sp` (
  SELECT
    `project.dataset.sp.udf_std_str_to_str`(epvid_evt_id) as event_id,
    `project.dataset.sp.udf_std_str_to_str`(epvid_vend_sbsy) as vendor_subsidiary,
    CASE WHEN epvid_vers_id LIKE '%-%' THEN NULL ELSE `project.dataset.sp.udf_std_str_to_str`(epvid_vers_id) END as version_id,
    `project.dataset.sp.udf_std_str_to_str`(epvid_itm_nbr) as item_nbr,
    `project.dataset.sp.udf_std_str_to_date`(epvid_trn_dte,'%Y-%m-%d') as transaction_date,
    `project.dataset.sp.udf_std_str_to_str`(epvid_rgn) as region_nbr,
    `project.dataset.sp.udf_std_str_to_str`(epvid_cur_itm_nbr) as current_item_nbr,
    epvid_act_itsl_dlr as actual_sale_dlr,
    epvid_act_itsl_unt as actual_sale_units,
    epvid_act_itmd_dlr as actual_markdown_dlr,
    epvid_act_itgm_dlr as actual_gross_margin_dlr,
    `project.dataset.sp.udf_std_str_to_str`(epvid_crt_opr_id) as create_operator_id,
    `project.dataset.sp.udf_std_str_to_date`(epvid_crt_dte,'%Y-%m-%d') as create_date,
    `project.dataset.sp.udf_std_str_to_datetime`(epvid_lst_mnt_tsmp,'%Y-%m-%d %H:%M:%E*S') as last_maintenance_timestamp,
    `project.dataset.sp.udf_std_str_to_str`(epvid_lst_opr_id) as last_op_id,
    `project.dataset.sp.udf_std_str_to_str`(epvid_lst_trn_cde) as last_trnsctn_code,
    `project.dataset.sp.udf_std_str_to_str`(epvid_grp_id) as grp_id,
    epvid_out_stk_ind as out_of_stock_ind,
    epvid_nbr_str_out as number_of_stores_out,
    `project.dataset.sp.udf_std_str_to_str`(epvid_pvt_itlb_ind) as pvt_lbl_ind,
    file_ingested_date,
    "dag_struct_load_evt_item_daily" as dag_id,
    current_datetime as created_datetime,
    current_datetime as updated_datetime
  FROM `otherproject.dataset.sp.event_item_daily`
  WHERE file_ingested_date = load_dt);

  COMMIT TRANSACTION;

  --Roll back transaction
  EXCEPTION WHEN ERROR THEN
  ROLLBACK TRANSACTION;

  RAISE USING MESSAGE = FORMAT("Error Message - %s, Error Statement - %s, Error Stack - %s", @@error.message, @@error.statement_text, @@error.formatted_stack_trace);

END;

Tested by removing CASE statement with below and after that the creation of stored procedure succeeds.
if(epvid_vers_id like '%-%', null, `project.dataset.sp.udf_std_str_to_str`(epvid_vers_id)) as version_id

Version Details

JDBC Simba Driver Version: 1.3.3.1004
Liquibase BigQuery Version: 4.17.0

Output

Running Changeset: sp.sql::1::google
SEVERE [liquibase.changelog] ChangeSet sp.sql::1::google encountered an exception.
SEVERE [liquibase.integration] Migration failed for changeset sp.sql::1::gcp:
     Reason: liquibase.exception.DatabaseException: [Simba][BigQueryJDBCDriver](100032) Error executing query job. Message: EXEC_JOB_EXECUTION_ERR [Failed SQL: (100032) 

Steps to Reproduce

  1. Create any Stored Procedure with CASE statement in it.
  2. Run liquibase update

Expected Behavior

The stored procedure would've have been created.

Actual Behavior

Liquibase is throwing exception mentioned above.

Community Note

  • Please vote on this issue by adding a ๐Ÿ‘ reaction
    to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
    not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

generateChangeLog creates an invalid changelog for array fields

Community Note

  • Please vote on this issue by adding a ๐Ÿ‘ reaction
    to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
    not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Debug Output

https://gist.github.com/gabrielmdeal/85ba1470346861ef87779027556218ff

Also part of the gist is Array_Repro_Table.json, and the changelog.json that liquibase generateChangeLog creates.

Panic Output

N/A

Steps to Reproduce

TLDR: ARRAY and STRUCT column types don't seem to work with liquibase generateChangeLog. I can't find documentation explaining if it is possible to manually create JSON for these types of columns. Nor can I find documentation that says these types are unsupported.

  1. Create multi-region datasets named Gabriel_Array and Gabriel_Array_Target
  2. Get Array_Repro_Table.json from the gist above.
  3. bq mk Gabriel_Array.Array_Repro_Table Array_Repro_Table.json
  4. Run liquibase generateChangeLog on Gabriel_Array
  5. Run liquibase update on Gabriel_Array_Target

I ran into a similar issue with a STRUCT column.

Expected Behavior

One of these:

  • The Array_Repro_Table is created in Gabriel_Array_Target. ๐ŸŽ‰
  • liquibase generateChangeLog fails with an error saying that arrays and structs are not supported for Big Query.
  • The documentation says that arrays and structs are not supported.

Actual Behavior

liquibase update fails with this error: liquibase.exception.DatabaseException: [Simba][BigQueryJDBCDriver](100032) Error executing query job. Message: Syntax error: Expected "<" but got "(" at [1:72] [Failed SQL: (100032) CREATE TABLE Gabriel_Array_Target.Array_Repro_Table (Array_Column ARRAY(65535))] See the gist for the full log.

Workaround

If I use --changeLogFile=changelog.bigquery.sql, then liquibase generateChangeLog still generates an invalid changelog.sql. But I am able to manually fix the SQL and then successfully run liquibase update.

BigQuery DATABASECHANGELOG is not supported in datasets named non-lowercase

Environment

Liquibase Version: 4.13.0 (latest now)

Liquibase Integration & Version: CLI

Liquibase Extension(s) & Version: liquibase-bigquery-4.13.0

Database Vendor & Version: Google BigQuery

Operating System Type & Version: Windows 10 21H2

Infrastructure Type/Provider: local

Description

Liquibase doesn't work properly with BigQuery, when dataset for DATABASECHANGELOG table is not in lowercase.

  • PascalCase doesn't work (eg Changelog dataset, which is a valid name in BigQuery)
  • underscores doesn't work (eg _changelog, which is a valid name too and useful to hide the dataset in BigQuery UI)

Most operations with DATABASECHANGELOG seems to work fine (changesets are applied and correctly excluded during the next liquibase update call). But there is one exception that breaks the CLI usage: when Liquibase tries to check if DATABASECHANGELOG table exists in the init phase - the code transforms dataset names to lowercase (Changeset -> changeset) and escapes underscores (_changelog -> \_changelog). Dataset names are case-sensitive in BigQuery, so JDBC driver returns null when DATABASECHANGELOG exists and Liquibase always tries to re-create this table, breaking the usage of liquibase update command.

Found workarounds for lowercasing:

  • Use lowercase for Liquibase dataset name (eg liquibaseSchemaName=changelog in configuration)
  • Set liquibase.preserveSchemaCase=true property, but this requires an update to https://github.com/liquibase/liquibase-bigquery to set backtick (`) as escape character, because current extension uses double quotes ("), which breaks queries

It makes sense to update BigQuery extension documentation (https://docs.liquibase.com/install/tutorials/bigquery.html) with these limitations, so people don't stuck before BigQuery support is better.

I'm not sure how to fix support for underscores, as there is a private method in JdbcDatabaseSnapshot class that adds escaping.

Steps To Reproduce

  1. Install Liquibase with BigQuery extension
  2. Create any changelog, can be a simple SQL version with a single changeset
--liquibase formatted sql
--changeset author:1
CREATE SCHEMA Test;
  1. Add Liquibase properties file with BigQuery configuration and PascalCase dataset name for Liquibase tables
changeLogFile=changelog.sql
liquibase.command.driver=com.simba.googlebigquery.jdbc.Driver
liquibase.command.url=jdbc:bigquery://https://googleapis.com/bigquery/v2:443;ProjectId={ProjectId};OAuthType=1;
liquibaseSchemaName=Changelog
  1. Create Changelog dataset in your BigQuery project
  2. Run liquibase update - it successfully creates DATABASECHANGELOG table in Changelog dataset and applies changesets
  3. Run liquibase update again

Actual Behavior

Second update call fails with JDBC driver error, because it tries to create DATABASECHANGELOG table, that already exists. This can be inspected by liquibase update-sql call that displays CREATE TABLE DATABASECHANGELOG ... statement.

Expected/Desired Behavior

Second update call success and Liquibase doesn't try to re-create existing DATABASECHANGELOG table.

Error creating DATABASECHANGELOG table

Hi,

Iยดm trying to execute some changelogs and Iยดve de following error when liquidbase tries to create its tables:

[2023-11-23 10:47:49] INFO [liquibase.command] Command execution complete
[2023-11-23 10:47:49] SEVERE [liquibase.integration] [Simba]BigQueryJDBCDriver Error executing query job. Message: Type not found: VARCHAR at [1:36] [Failed SQL: (100032) CREATE TABLE DATABASECHANGELOG (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED datetime NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10))]


"code": 400,
"errors": [
***
"domain": "global",
"location": "q",
"locationType": "parameter",
"message": "Type not found: VARCHAR at [1:36]",
"reason": "invalidQuery"
***
],
"message": "Type not found: VARCHAR at [1:36]",
"status": "INVALID_ARGUMENT"


liquibase.exception.CommandExecutionException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: [Simba]BigQueryJDBCDriver Error executing query job. Message: Type not found: VARCHAR at [1:36] [Failed SQL: (100032) CREATE TABLE DATABASECHANGELOG (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED datetime NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10))]
at liquibase.command.CommandScope.execute(CommandScope.java:237)
at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:55)
at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:24)
at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
at picocli.CommandLine.execute(CommandLine.java:2170)

Iยดm using the liquidbase github action plugin with the following options:

is it maybe liquidbase library github action plugin is using?

Many thanks

Type not found: VARCHAR

Caused by: java.sql.SQLException: [Simba]BigQueryJDBCDriver Error executing query job. Message: Type not found: VARCHAR at [1:41]

should be STRING

Caused by: liquibase.exception.DatabaseException: [Simba]BigQueryJDBCDriver Error executing query job. Message: Type not found: VARCHAR at [1:41] [Failed SQL: (100032) CREATE TABLE mydb.DATABASECHANGELOG (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED datetime NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10))]

Unexpected error running Liquibase: [Simba][BigQueryJDBCDriver](100032) Error executing query job. Message: Table "DATABASECHANGELOGLOCK" must be qualified with a dataset (e.g. dataset.table). [Failed SQL: (100032) CREATE TABLE DATABASECHANGELOGLOCK (ID INT, LOCKED BOOLEAN, LOCKGRANTED datetime, LOCKEDBY STRING(255))]


Issues on GitHub are intended to be related to bugs or feature requests with provider codebase,
so we recommend using our other community resources instead of asking here ๐Ÿ‘.

I am trying to run the following command on liquibase 4.21.1 and the liquibase-bigquery extension is also 4.21.1.

liquibase update --changelog-file="/db/changelog/changelog-master.yaml" --url="jdbc:bigquery://https://googleapis.com/bigquery/v2:443/my_bq_db;ProjectId=my-gcp-project;OAuthType=0;[email protected];OAuthPvtKeyPath=/gcloud/my-gcp-project-61e489b85654.json" --driver="com.simba.googlebigquery.jdbc42.Driver" --classpath="/SimbaJDBCDriverforGoogleBigQuery42_1.3.3.1004/GoogleBigQueryJDBC42.jar"

this is complaining with the following error:

Unexpected error running Liquibase: [Simba]BigQueryJDBCDriver Error executing query job. Message: Table "DATABASECHANGELOGLOCK" must be qualified with a dataset (e.g. dataset.table). [Failed SQL: (100032) CREATE TABLE DATABASECHANGELOGLOCK (ID INT, LOCKED BOOLEAN, LOCKGRANTED datetime, LOCKEDBY STRING(255))]

  • Caused by: 400 Bad Request
    POST https://bigquery.googleapis.com/bigquery/v2/projects/my-gcp-project/queries
    {
    "code": 400,
    "errors": [
    {
    "domain": "global",
    "message": "Table "DATABASECHANGELOGLOCK" must be qualified with a dataset (e.g. dataset.table).",
    "reason": "invalid"
    }
    ],
    "message": "Table "DATABASECHANGELOGLOCK" must be qualified with a dataset (e.g. dataset.table).",
    "status": "INVALID_ARGUMENT"
    }

I was following the following page: https://contribute.liquibase.com/extensions-integrations/directory/database-tutorials/bigquery/

Was wondering if there's a workaround for this?

Liquibase connecting to non primary replica

Community Note

  • Please vote on this issue by adding a ๐Ÿ‘ reaction
    to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
    not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Debug Output

N/A

Panic Output

N/A

Steps to Reproduce

  1. Use Liquibase v4.27.0.
  2. Have a dataset set to Multi-region EU.
  3. Have a dataset replica set up in the US.
  4. Do not specify the Location in the URL
  5. Run a Liquibase command

Expected Behavior

The expectation is that it would connect to the EU location without needing to specify it in the URL.

Actual Behavior

Liquibase is connecting to the US region even though is it the secondary replica which is read-only.

{
  "code": 400,
  "errors": [
    {
      "domain": "global",
      "message": "The dataset replica of the cross region dataset '<host>:<dataset>' in region 'US' is read-only because it's not the primary replica.",
      "reason": "invalid"
    }
  ],
  "message": "The dataset replica of the cross region dataset '<host>:<dataset>' in region 'US' is read-only because it's not the primary replica.",
  "status": "INVALID_ARGUMENT"
}

References

  • #0000

generateChangeLog with a regional dataset and a view causes "Dataset X:X was not found in location US"

Community Note

  • Please vote on this issue by adding a ๐Ÿ‘ reaction
    to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
    not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Debug Output

https://gist.github.com/gabrielmdeal/f82a75154fefbade72061d95c6d6fd2d

The gist also contains the Repro_Table.json and Repro_View.sql files that are referenced below.

Panic Output

N/A

Steps to Reproduce

TLDR: liquibase generateChangeLog fails if my dataset is regional and contains a view. It works for a multi-region dataset.

  1. Install:
  2. Create a Big Query dataset named Gabriel_Liquibase in the us-central1 region. Not multi-region.
  3. Create a liquibase service account with the BigQuery Admin, BigQuery Data Owner and BigQuery Metadata Viewer roles. (I know this is overkill, but I knee-jerk added more roles when I ran into this error.)
  4. gcloud iam service-accounts keys create "liquibase-service-account-credentials.json" --iam-account "[email protected]"
  5. Use the JSON and SQL from the above gist to create a table and a view:
    bq mk Gabriel_Liquibase.Repro_Table Repro_Table.json
    bq mk --use_legacy_sql=false --view "$(cat Repro_View.sql)" Gabriel_Liquibase.Repro_View
    
  6. Try to generate a change log via liquibase generateChangeLog --url 'jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=redacted-project-id;DefaultDataset=Gabriel_Liquibase;OAuthType=0;OAuthServiceAcctEmail=liquibase@redacted-project-id.iam.gserviceaccount.com;OAuthPvtKeyPath=liquibase-service-account-credentials.json' --logLevel DEBUG --changeLogFile changelog.json --driver com.simba.googlebigquery.jdbc.Driver

Expected Behavior

A changelog is created with my table and view.

Actual Behavior

The changelog is not created. liquibase generateChangeLog fails with the below error (see gist for full output). If I use console.cloud.google.com to run the SQL from the error, I do get the expected view definition.

The liquibase generateChangeLog command succeeds if I delete my view.

[2023-05-16 21:39:35] SEVERE [liquibase.integration] Error executing SQL SELECT view_definition FROM Gabriel_Liquibase.INFORMATION_SCHEMA.VIEWS WHERE table_name='Repro_View' AND table_schema='Gabriel_Liquibase' AND table_catalog='redacted-project-id';: [Simba][BigQueryJDBCDriver](100032) Error executing query job. Message: Not found: Dataset redacted-project-id:Gabriel_Liquibase was not found in location US
  - Caused by: 404 Not Found
POST https://bigquery.googleapis.com/bigquery/v2/projects/redacted-project-id/queries
{
  "code": 404,
  "errors": [
    {
      "domain": "global",
      "message": "Not found: Dataset redacted-project-id:Gabriel_Liquibase was not found in location US",
      "reason": "notFound"
    }
  ],
  "message": "Not found: Dataset redacted-project-id:Gabriel_Liquibase was not found in location US",
  "status": "NOT_FOUND"
}

Syntax error because the SQL TABLE name is enclosed in double quotes.

Dockerfile

FROM liquibase/liquibase:4.13.0
USER 0:0
COPY liquibase.docker.properties /liquibase/
COPY key.json /liquibase/
COPY changelog/ /liquibase/changelog/
RUN apt update && apt install curl unzip -y
RUN curl -SsL https://storage.googleapis.com/simba-bq-release/jdbc/SimbaJDBCDriverforGoogleBigQuery42_1.2.23.1027.zip -o SimbaJDBCDriverforGoogleBigQuery.zip \
 && unzip SimbaJDBCDriverforGoogleBigQuery.zip -d /liquibase/lib/ \
 && curl -SsL https://github.com/liquibase/liquibase-bigquery/releases/download/liquibase-bigquery-4.13.0/liquibase-bigquery-4.13.0.jar -o /liquibase/lib/liquibase-bigquery-4.13.0.jar

liquibase.docker.properties

classpath: /liquibase/changelog
url: jdbc:bigquery://https://www.googleapis.com/bigquery/v2:443;ProjectId=[ProjectId];OAuthType=0;OAuthServiceAcctEmail=[ServiceAccount];OAuthPvtKeyPath=/liquibase/key.json
changeLogFile: changelog.sql
username:
password:
databaseChangeLogLockTableName: [ProjectId].[DataSetName].databasechangeloglock
databaseChangeLogTableName: [ProjectId].[DataSetName].databasechangelog

changelog/changelog.sql

-- liquibase formatted sql

-- changeset liquibase:1
CREATE TABLE test_table (test_id INT, test_column VARCHAR, PRIMARY KEY (test_id))

Then execute this command.

docker build -t liquibase:0.0.1 . --no-cache=true
docker run --rm liquibase:0.0.1 update --log-level=fine

Then an error occurs.

Caused by: liquibase.exception.DatabaseException: [Simba][BigQueryJDBCDriver](100032) Error executing query job. Message: Syntax error: Unexpected string literal "[ProjectId].[DataSetName]...." at [1:14] [Failed SQL: (100032) CREATE TABLE "[ProjectId].[DataSetName].databasechangeloglock" (ID INT NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED datetime, LOCKEDBY STRING(255))]

I get an error when creating the DATABASECHANGELOGLOCK table.
The cause of the error seems to be that the TABLE name is enclosed in double quotes.
Why do I get this error?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.