Giter Club home page Giter Club logo

terraform-provider-keboola's People

Contributors

halama avatar jestey-pbp avatar mavdyushkin avatar nyazdanfar-pbp avatar plmwong avatar wtran-pbp avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

terraform-provider-keboola's Issues

Transfer ownership to `paybyphone` organisation

  1. Transfer Github repository to paybyphone organisation
  2. Re-configure or remove Travis CI build - which currently runs under my account
  3. Contact Terraform Registry about transfer or de-listing of the Provider

Columns in gooddata_writer_table should be a hashset

The collection of columns within the gooddata_writer_table resource should be tracked as a HashSet of column definitions rather than a list/array. They are not guaranteed to be in a certain order, so when they are updated, often the difference calculated is much larger because re-orderings show up as differences.

Add encrypted value (project secure) as a Data Source

It might be handy to be able to create an encrypted password or secured value in Keboola, and be able to reference its value through a Data Source, in a similar way one would create and manage public/private keys in AWS terraform.

keboola_transformation phase not in correct format

The phase attribute for keboola_transformation in the Keboola API can be a quoted string, e.g.

phase: "\"1\""

Even though the phase can be also be set to the value without quotes:

phase: "1"

The API apparently normalises it to the value with added quotes, or the UI adds the quotes (have not confirmed which).

The provider should probably be consistent with what the UI saves as, to prevent terraform from identifying changes in phase all the time.

Integration tests not running due to change in Keboola access tokens

Keboola has changed their policy of issuing access tokens, so that tokens that 'users' create are time limited, and carry a different token prefix.

Since non-user access tokens do not have access to certain resources (e.g. they cannot manage access tokens themselves), those are not appropriate for running the integration tests as part of the travis-ci build.

Will need to figure out another solution or workaround. In the meantime, integration tests have been turned off by removing the setting TF_ACC from the defined environment variables of the travis-ci build.

Panic when deleting and recreating Snowflake Writer Tables

Getting a panic on an interface conversion when applying changes that require removing and recreating the Snowflake Writer.

Re-running the same apply command succeeds since at that point the .tfstate recognises it only needs to perform a resource creation.

panic: interface conversion: interface {} is *schema.Set, not []interface {}
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe:
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: goroutine 511 [running]:
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola.resourceKeboolaSnowflakeWriterTablesUpdate(0xc04226e8c0, 0xd92680, 0xc042366430, 0x24, 0x138b320)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/go/src/github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola/resource_keboola_snowflake_writer_tables.go:178 +0x1a5d
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc04233f490, 0xc042864af0, 0xc0422b5ee0, 0xd92680, 0xc042366430, 0x1, 0xc0423a5b90, 0xc0423a5b90)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/go/src/github.com/hashicorp/terraform/helper/schema/resource.go:233 +0x260
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc04233fab0, 0xc042864aa0, 0xc042864af0, 0xc0422b5ee0, 0x180000, 0x0, 0xc7d000)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/go/src/github.com/hashicorp/terraform/helper/schema/provider.go:283 +0xab
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc0424d3f60, 0xc0422b5e60, 0xc042516b30, 0x0, 0x0)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/go/src/github.com/hashicorp/terraform/plugin/resource_provider.go:527 +0x5e
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: reflect.Value.call(0xc0420433e0, 0xc042156d90, 0x13, 0xdeed1b, 0x4, 0xc04245df20, 0x3, 0x3, 0x0, 0x0, ...)
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/tools/go/src/reflect/value.go:434 +0x90c
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: reflect.Value.Call(0xc0420433e0, 0xc042156d90, 0x13, 0xc04245df20, 0x3, 0x3, 0x0, 0x0, 0xc0424dee60)
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/tools/go/src/reflect/value.go:302 +0xab
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: net/rpc.(*service).call(0xc04239b8c0, 0xc042077770, 0xc0423efb70, 0xc042144400, 0xc04243a4e0, 0xc6de00, 0xc0422b5e60, 0x16, 0xc6de40, 0xc042516b30, ...)
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/tools/go/src/net/rpc/server.go:381 +0x149
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: created by net/rpc.(*Server).ServeCodec
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe:     C:/tools/go/src/net/rpc/server.go:475 +0x372

Support limited updates on storage tables

There are some limited fields that can be updated on storage tables without requiring a full destroy/recreate, such as indexed columns. We should add support for this.

GoodData writer 'Read' fails when the writer is set to do Full load

When a GoodData writer has been configured to do a Full load to GoodData instead of an Incremental, then a terraform plan or terraform refresh will fail with an error about unmarshaling a boolean in to an integer. So there will be a type mismatch between what is in the JSON response when the GoodData writer is set to Full.

Incremental Load property for gooddata table should be integer

The incrementalLoad property for the gooddata_writer_table is not a boolean type as originally assumed, it is actually an integer indicating the number of days worth of data to use when doing incremental data writes to GoodData.

The type should be changed, and the name of the configuration property possibly renamed to better reflect what the property controls.

Panic when running under Ubuntu

2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: panic: runtime error: invalid memory address or nil pointer dereference
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xb567f4]
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: goroutine 148 [running]:
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola.resourceKeboolaGoodDataWriterRead(0xc42022ca80, 0xd0af00, 0xc42010f8a0, 0xc42022ca80, 0x0)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola/resource_keboola_gooddata_writer.go:135 +0x194
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc42021e230, 0xc4201ae5a0, 0xd0af00, 0xc42010f8a0, 0xc42027f208, 0x4bf401, 0xbe3c20)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/resource.go:355 +0x167
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc42021efc0, 0xc4201ae550, 0xc4201ae5a0, 0x3, 0xc4201ccfc0, 0x1a)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc420478f80, 0xc42010faa0, 0xc42010fb70, 0x0, 0x0)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xd69691, 0x4, 0xc42037af18, 0x3, 0x3, 0xc4201c62c0, 0xc420125b80, ...)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/reflect/value.go:447 +0x969
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.Call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xc4205aff18, 0x3, 0x3, 0xc420125cc0, 0xc420125ce0, 0xc420125d00)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/reflect/value.go:308 +0xa4
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: net/rpc.(*service).call(0xc4201c6f00, 0xc420472410, 0xc42048a7e0, 0xc42048a7f0, 0xc4201ca680, 0xc4203dec00, 0xbe3be0, 0xc42010faa0, 0x16, 0xbe3c20, ...)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/net/rpc/server.go:384 +0x14e
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: created by net/rpc.(*Server).ServeCodec
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/net/rpc/server.go:480 +0x43a
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: panic: runtime error: invalid memory address or nil pointer dereference
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xb791f1]
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola:
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: goroutine 164 [running]:
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola.resourceKeboolaTransformBucketRead(0xc42021f490, 0xd0af00, 0xc42010f8a0, 0xc42021f490, 0x0)
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola/resource_keboola_transformation_bucket.go:54 +0x1a1
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc42021e1c0, 0xc420472780, 0xd0af00, 0xc42010f8a0, 0xc42027f2e8, 0x4bf401, 0xbe3c20)
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/resource.go:355 +0x167
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc42021efc0, 0xc420472730, 0xc420472780, 0x3, 0xc42048ec00, 0x1a)
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc420478f80, 0xc4203162e0, 0xc4203163b0, 0x0, 0x0)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola:         /mnt/c/go/src/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xd69691, 0x4, 0xc42037cf18, 0x3, 0x3, 0xc4201c75c0, 0x0, ...)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/reflect/value.go:447 +0x969
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.Call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xc42005b718, 0x3, 0x3, 0x0, 0x0, 0x0)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/reflect/value.go:308 +0xa4
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: net/rpc.(*service).call(0xc4201c6f00, 0xc420472410, 0xc42048a7e0, 0xc42048a7f0, 0xc4201ca680, 0xc4203ded80, 0xbe3be0, 0xc4203162e0, 0x16, 0xbe3c20, ...)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/net/rpc/server.go:384 +0x14e
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: created by net/rpc.(*Server).ServeCodec
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola:         /usr/lib/go-1.10/src/net/rpc/server.go:480 +0x43a
2018-07-12T11:15:18.296-0700 [DEBUG] plugin: plugin process exited: path=/home/pwong/windows/source/repos/data/keboola-standard/terraform-provider-keboola

Inclusion in Terraform Registry

Hi there โ€” we're experimenting with Keboola and are interested in leveraging this community provider.

Unfortunately, I'm having trouble using it locally, as Terraform cannot find this provider. It looks like this provider needs to be registered in the Terraform Registry. Would one of the maintainers be able to publish this provider? It seems like there's a "publish" button at the top right of this page https://registry.terraform.io/ to do so.

Otherwise, is there another way to use this provider without it being included in the Terraform registry?

Support FTP Extractor resource

Internally, we require support for the keboola.ex-ftp extractor application.

Example config (https://connection.keboola.com/v2/storage/components/keboola.ex-ftp/configs/495213212):

{
   "id":"495213212",
   "name":"Example FTP server",
   "description":"Notes from Example",
   "created":"2019-04-09T20:27:06+0200",
   "creatorToken":{
      "id":147218,
      "description":"[email protected]"
   },
   "version":25,
   "changeDescription":"Table exampleTransactionCount configuration edited manually",
   "isDeleted":false,
   "configuration":{
      "parameters":{
         "host":"transfer.does.not.exist.com",
         "port":22,
         "connectionType":"SFTP",
         "username":"Keboola",
         "#password":"KBC::ProjectSecure::redacted",
         "#privateKey":"KBC::ProjectSecure::redacted"
      }
   },
   "rowsSortOrder":[

   ],
   "rows":[
      {
         "id":"496020352",
         "name":"exampleTransactionCount",
         "description":"",
         "configuration":{
            "parameters":{
               "onlyNewFiles":true,
               "path":"*.csv"
            },
            "processors":{
               "after":[
                  {
                     "definition":{
                        "component":"keboola.processor-move-files"
                     },
                     "parameters":{
                        "direction":"tables",
                        "addCsvSuffix":true,
                        "folder":"exampleTransactionCount"
                     }
                  },
                  {
                     "definition":{
                        "component":"keboola.processor-create-manifest"
                     },
                     "parameters":{
                        "delimiter":",",
                        "enclosure":"\"",
                        "incremental":true,
                        "primary_key":[

                        ],
                        "columns_from":"header"
                     }
                  },
                  {
                     "definition":{
                        "component":"keboola.processor-skip-lines"
                     },
                     "parameters":{
                        "lines":1
                     }
                  },
                  {
                     "definition":{
                        "component":"keboola.processor-skip-lines"
                     },
                     "parameters":{
                        "lines":2,
                        "direction_from":"bottom"
                     }
                  }
               ]
            }
         },
         "isDisabled":false,
         "version":4,
         "created":"2019-04-12T22:14:24+0200",
         "creatorToken":{
            "id":147218,
            "description":"[email protected]"
         },
         "changeDescription":"Table exampleTransactionCount configuration edited manually",
         "state":{
            "component":{
               "ex-ftp-state":{
                  "newest-timestamp":1555632870,
                  "last-timestamp-files":[
                     "ExampleTransactions.csv"
                  ]
               }
            },
            "storage":{
               "input":{
                  "tables":[

                  ]
               }
            }
         }
      }
   ],
   "state":{

   },
   "currentVersion":{
      "created":"2019-04-12T22:25:01+0200",
      "creatorToken":{
         "id":147218,
         "description":"[email protected]"
      },
      "changeDescription":"Table exampleTransactionCount configuration edited manually"
   }
}

keboola_storage_table.indexedColumns are being dropped if they are not part of primaryKey

Having an issue where any column which is sent as part of indexedColumns, but not part of primaryKey, is being dropped from the config. So the next time terraform plan is executed, it thinks that the indexedColumns need to be updated again.

  primaryKey = [
    "FOO",
  ]

  indexedColumns = [
    "FOO",
    "BAR",
  ]

terraform plan indicates that the resource needs to be altered to add BAR back in to indexedColumns.

When storage table has just been created, GoodData writer sometimes cannot find it

When a storage table has just been created, and it is depended on by a GoodData Writer change. Sometimes when the writer change is being performed by terraform, an error is thrown where it cannot find the table:

Table out.c-main.lot_dimension does not exist or is not accessible with the token

Running the terraform apply again will succeed, so there may just be a timing/race condition issue happening.

Support Snowflake writer update

resourceKeboolaSnowflakeWriterUpdate needs to be implemented.

It should allow for updating the name and description of the Snowflake writer, without requiring recreation.

Update to support latest version of terraform

Currently this provider is not compliant with the latest version of terraform. For one, terraform decided to settle of requiring snake case (e.g. foo_bar) for schema elements and disallowing camel casing (e.g. fooBar). So at least that will need to be changed, which will be a breaking change to all existing configurations. There may be other changes as well.

Add more DEBUG level logging

Hunting down issues is quite difficult at the moment, with the lack of detailed DEBUG and TRACE level logging.

Support PartialState where appropriate

A few of the resources have multiple API calls/steps to properly provision them. Which means there is a chance that one or more calls will fail and a resource will only be partially created or updated (or possibly only partially deleted).

Need to use terraforms support for partial states so that we can properly recover from these types of errors.

Changed behavior in Storage API for config row detail

We're fixing an issue in our API, where component configuration row detail (call to /v2/storage/components/{component_id}/configs/{config_id}/rows/{row_id}) currently returns all rows instead of the exact row.

We have a release ready (see related change in Storage API client for PHP and Apiary docs), that fixes this. The endpoint will return only one row based on the supplied id.

We've checked and most of the calls to this API are from this terraform provider, so we'd like to coordinate the change with a change in your API. Could you change your API accordingly and let us know if the change causes any trouble please?

fix build for terraform 0.15 & go 1.16

Hi ๐Ÿ‘‹
thanks for this great provider!
I was wondering if I could submit a PR with fixing the build that is terraform 0.15 & go 1.16 compatible.
I managed to fix all the imports and build works fine, the provider binary too.

Would you be interested?

When Snowflake writer is updated, it loses association with the snowflake credentials

When the keboola_snowflake_writer is updated (i.e. for column changes or whatever), then it loses its snowflake credentials, meaning that it will fail when it is actually used. At the moment, we've had to manually select credentials again in the UI. The update request (PUT) might need to be changed to always send the snowflake credentials so they don't get dropped.

Orchestration resource doesn't support 'Active'

The orchestration resource lacks an active flag (true/false), which means orchestrations can't be enabled or disabled without using the UI.

We should add this in to the Orchestration struct

{  
   "id":458432432,
   "name":"Some stuff",
   ...
   "active":true,

`datatypes` should be marked as 'omitempty'

After updating a GoodData User Management configuration through terraform, the following error occurs when executing the application:

message: Unrecognized option "datatypes" under "container.storage.input.tables.0"
exceptionId: docker-b4e5a5f169c9078248a857c14b4fab8b

This appears to the be caused by the JSON being pushed up as configuration containing the datatypes attribute, even though it is not used for this type of resource.

Should add 'omitempty' so that when it is not set, it is not sent.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.