paybyphone / terraform-provider-keboola Goto Github PK
View Code? Open in Web Editor NEWTerraform Custom Provider for Keboola Connection (KBC)
License: Other
Terraform Custom Provider for Keboola Connection (KBC)
License: Other
paybyphone
organisationThe 'dimension' attribute, which applies to a column in a keboola_gooddata_writer_table
configuration is missing, so GoodData doesn't know which columns map to which data dimensions in the GoodData project.
The collection of columns
within the gooddata_writer_table
resource should be tracked as a HashSet of column definitions rather than a list/array. They are not guaranteed to be in a certain order, so when they are updated, often the difference calculated is much larger because re-orderings show up as differences.
Instead of using a JSON document for configuring the keboola_postgresql_writer
database settings, we should consider porting this over to a proper configuration map.
It might be handy to be able to create an encrypted password or secured value in Keboola, and be able to reference its value through a Data Source, in a similar way one would create and manage public/private keys in AWS terraform.
The phase
attribute for keboola_transformation
in the Keboola API can be a quoted string, e.g.
phase: "\"1\""
Even though the phase can be also be set to the value without quotes:
phase: "1"
The API apparently normalises it to the value with added quotes, or the UI adds the quotes (have not confirmed which).
The provider should probably be consistent with what the UI saves as, to prevent terraform from identifying changes in phase
all the time.
The incremental
attribute is not being refreshed from the remote API when a terraform plan
or refresh
is performed. Possibly the attribute is not being mapped correctly on the Read
function.
Keboola has changed their policy of issuing access tokens, so that tokens that 'users' create are time limited, and carry a different token prefix.
Since non-user access tokens do not have access to certain resources (e.g. they cannot manage access tokens themselves), those are not appropriate for running the integration tests as part of the travis-ci build.
Will need to figure out another solution or workaround. In the meantime, integration tests have been turned off by removing the setting TF_ACC
from the defined environment variables of the travis-ci build.
Getting a panic
on an interface conversion when applying changes that require removing and recreating the Snowflake Writer.
Re-running the same apply
command succeeds since at that point the .tfstate
recognises it only needs to perform a resource creation.
panic: interface conversion: interface {} is *schema.Set, not []interface {}
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe:
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: goroutine 511 [running]:
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola.resourceKeboolaSnowflakeWriterTablesUpdate(0xc04226e8c0, 0xd92680, 0xc042366430, 0x24, 0x138b320)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/go/src/github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola/resource_keboola_snowflake_writer_tables.go:178 +0x1a5d
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/hashicorp/terraform/helper/schema.(*Resource).Apply(0xc04233f490, 0xc042864af0, 0xc0422b5ee0, 0xd92680, 0xc042366430, 0x1, 0xc0423a5b90, 0xc0423a5b90)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/go/src/github.com/hashicorp/terraform/helper/schema/resource.go:233 +0x260
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/hashicorp/terraform/helper/schema.(*Provider).Apply(0xc04233fab0, 0xc042864aa0, 0xc042864af0, 0xc0422b5ee0, 0x180000, 0x0, 0xc7d000)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/go/src/github.com/hashicorp/terraform/helper/schema/provider.go:283 +0xab
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Apply(0xc0424d3f60, 0xc0422b5e60, 0xc042516b30, 0x0, 0x0)
2018-07-11T18:12:23.667-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/go/src/github.com/hashicorp/terraform/plugin/resource_provider.go:527 +0x5e
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: reflect.Value.call(0xc0420433e0, 0xc042156d90, 0x13, 0xdeed1b, 0x4, 0xc04245df20, 0x3, 0x3, 0x0, 0x0, ...)
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/tools/go/src/reflect/value.go:434 +0x90c
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: reflect.Value.Call(0xc0420433e0, 0xc042156d90, 0x13, 0xc04245df20, 0x3, 0x3, 0x0, 0x0, 0xc0424dee60)
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/tools/go/src/reflect/value.go:302 +0xab
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: net/rpc.(*service).call(0xc04239b8c0, 0xc042077770, 0xc0423efb70, 0xc042144400, 0xc04243a4e0, 0xc6de00, 0xc0422b5e60, 0x16, 0xc6de40, 0xc042516b30, ...)
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/tools/go/src/net/rpc/server.go:381 +0x149
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: created by net/rpc.(*Server).ServeCodec
2018-07-11T18:12:23.668-0700 [DEBUG] plugin.terraform-provider-keboola.exe: C:/tools/go/src/net/rpc/server.go:475 +0x372
There are some limited fields that can be updated on storage tables without requiring a full destroy/recreate, such as indexed columns. We should add support for this.
The phase
property is missing from the keboola_transformation
resource, so it's not possible to re-order the execution of transformations within the same bucket.
When a GoodData writer has been configured to do a Full load to GoodData instead of an Incremental, then a terraform plan
or terraform refresh
will fail with an error about unmarshaling a boolean in to an integer. So there will be a type mismatch between what is in the JSON response when the GoodData writer is set to Full.
The incrementalLoad
property for the gooddata_writer_table
is not a boolean type as originally assumed, it is actually an integer indicating the number of days worth of data to use when doing incremental data writes to GoodData.
The type should be changed, and the name of the configuration property possibly renamed to better reflect what the property controls.
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: panic: runtime error: invalid memory address or nil pointer dereference
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xb567f4]
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola:
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: goroutine 148 [running]:
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola.resourceKeboolaGoodDataWriterRead(0xc42022ca80, 0xd0af00, 0xc42010f8a0, 0xc42022ca80, 0x0)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola/resource_keboola_gooddata_writer.go:135 +0x194
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc42021e230, 0xc4201ae5a0, 0xd0af00, 0xc42010f8a0, 0xc42027f208, 0x4bf401, 0xbe3c20)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/resource.go:355 +0x167
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc42021efc0, 0xc4201ae550, 0xc4201ae5a0, 0x3, 0xc4201ccfc0, 0x1a)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc420478f80, 0xc42010faa0, 0xc42010fb70, 0x0, 0x0)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xd69691, 0x4, 0xc42037af18, 0x3, 0x3, 0xc4201c62c0, 0xc420125b80, ...)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/reflect/value.go:447 +0x969
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.Call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xc4205aff18, 0x3, 0x3, 0xc420125cc0, 0xc420125ce0, 0xc420125d00)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/reflect/value.go:308 +0xa4
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: net/rpc.(*service).call(0xc4201c6f00, 0xc420472410, 0xc42048a7e0, 0xc42048a7f0, 0xc4201ca680, 0xc4203dec00, 0xbe3be0, 0xc42010faa0, 0x16, 0xbe3c20, ...)
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/net/rpc/server.go:384 +0x14e
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: created by net/rpc.(*Server).ServeCodec
2018-07-12T11:15:18.283-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/net/rpc/server.go:480 +0x43a
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: panic: runtime error: invalid memory address or nil pointer dereference
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0xb791f1]
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola:
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: goroutine 164 [running]:
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola.resourceKeboolaTransformBucketRead(0xc42021f490, 0xd0af00, 0xc42010f8a0, 0xc42021f490, 0x0)
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/plmwong/terraform-provider-keboola/plugin/providers/keboola/resource_keboola_transformation_bucket.go:54 +0x1a1
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc42021e1c0, 0xc420472780, 0xd0af00, 0xc42010f8a0, 0xc42027f2e8, 0x4bf401, 0xbe3c20)
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/resource.go:355 +0x167
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc42021efc0, 0xc420472730, 0xc420472780, 0x3, 0xc42048ec00, 0x1a)
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2018-07-12T11:15:18.293-0700 [DEBUG] plugin.terraform-provider-keboola: github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc420478f80, 0xc4203162e0, 0xc4203163b0, 0x0, 0x0)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: /mnt/c/go/src/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xd69691, 0x4, 0xc42037cf18, 0x3, 0x3, 0xc4201c75c0, 0x0, ...)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/reflect/value.go:447 +0x969
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: reflect.Value.Call(0xc4201a3020, 0xc4203fe2e0, 0x13, 0xc42005b718, 0x3, 0x3, 0x0, 0x0, 0x0)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/reflect/value.go:308 +0xa4
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: net/rpc.(*service).call(0xc4201c6f00, 0xc420472410, 0xc42048a7e0, 0xc42048a7f0, 0xc4201ca680, 0xc4203ded80, 0xbe3be0, 0xc4203162e0, 0x16, 0xbe3c20, ...)
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/net/rpc/server.go:384 +0x14e
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: created by net/rpc.(*Server).ServeCodec
2018-07-12T11:15:18.294-0700 [DEBUG] plugin.terraform-provider-keboola: /usr/lib/go-1.10/src/net/rpc/server.go:480 +0x43a
2018-07-12T11:15:18.296-0700 [DEBUG] plugin: plugin process exited: path=/home/pwong/windows/source/repos/data/keboola-standard/terraform-provider-keboola
Hi there โ we're experimenting with Keboola and are interested in leveraging this community provider.
Unfortunately, I'm having trouble using it locally, as Terraform cannot find this provider. It looks like this provider needs to be registered in the Terraform Registry. Would one of the maintainers be able to publish this provider? It seems like there's a "publish" button at the top right of this page https://registry.terraform.io/ to do so.
Otherwise, is there another way to use this provider without it being included in the Terraform registry?
Internally, we require support for the keboola.ex-ftp
extractor application.
Example config (https://connection.keboola.com/v2/storage/components/keboola.ex-ftp/configs/495213212):
{
"id":"495213212",
"name":"Example FTP server",
"description":"Notes from Example",
"created":"2019-04-09T20:27:06+0200",
"creatorToken":{
"id":147218,
"description":"[email protected]"
},
"version":25,
"changeDescription":"Table exampleTransactionCount configuration edited manually",
"isDeleted":false,
"configuration":{
"parameters":{
"host":"transfer.does.not.exist.com",
"port":22,
"connectionType":"SFTP",
"username":"Keboola",
"#password":"KBC::ProjectSecure::redacted",
"#privateKey":"KBC::ProjectSecure::redacted"
}
},
"rowsSortOrder":[
],
"rows":[
{
"id":"496020352",
"name":"exampleTransactionCount",
"description":"",
"configuration":{
"parameters":{
"onlyNewFiles":true,
"path":"*.csv"
},
"processors":{
"after":[
{
"definition":{
"component":"keboola.processor-move-files"
},
"parameters":{
"direction":"tables",
"addCsvSuffix":true,
"folder":"exampleTransactionCount"
}
},
{
"definition":{
"component":"keboola.processor-create-manifest"
},
"parameters":{
"delimiter":",",
"enclosure":"\"",
"incremental":true,
"primary_key":[
],
"columns_from":"header"
}
},
{
"definition":{
"component":"keboola.processor-skip-lines"
},
"parameters":{
"lines":1
}
},
{
"definition":{
"component":"keboola.processor-skip-lines"
},
"parameters":{
"lines":2,
"direction_from":"bottom"
}
}
]
}
},
"isDisabled":false,
"version":4,
"created":"2019-04-12T22:14:24+0200",
"creatorToken":{
"id":147218,
"description":"[email protected]"
},
"changeDescription":"Table exampleTransactionCount configuration edited manually",
"state":{
"component":{
"ex-ftp-state":{
"newest-timestamp":1555632870,
"last-timestamp-files":[
"ExampleTransactions.csv"
]
}
},
"storage":{
"input":{
"tables":[
]
}
}
}
}
],
"state":{
},
"currentVersion":{
"created":"2019-04-12T22:25:01+0200",
"creatorToken":{
"id":147218,
"description":"[email protected]"
},
"changeDescription":"Table exampleTransactionCount configuration edited manually"
}
}
Need a new resource for CSV Import Extractor component in Keboola.
Having an issue where any column which is sent as part of indexedColumns
, but not part of primaryKey
, is being dropped from the config. So the next time terraform plan
is executed, it thinks that the indexedColumns
need to be updated again.
primaryKey = [
"FOO",
]
indexedColumns = [
"FOO",
"BAR",
]
terraform plan
indicates that the resource needs to be altered to add BAR
back in to indexedColumns
.
Have a thought about this.
Snowflake Extractor no longer enjoys having driver in the database configuration. It's possible that writer still requires it. But who knows. Keboola knows. Ask Keboola?
Most of the KBC resources should be importable just by using the basic passthrough function ImportStatePassthrough
(refer: https://github.com/hashicorp/terraform/blob/master/helper/schema/resource_importer.go).
Importer: &schema.ResourceImporter{
State: schema.ImportStatePassthrough,
},
When a storage table has just been created, and it is depended on by a GoodData Writer change. Sometimes when the writer change is being performed by terraform, an error is thrown where it cannot find the table:
Table out.c-main.lot_dimension does not exist or is not accessible with the token
Running the terraform apply
again will succeed, so there may just be a timing/race condition issue happening.
resourceKeboolaSnowflakeWriterUpdate
needs to be implemented.
It should allow for updating the name
and description
of the Snowflake writer, without requiring recreation.
Currently this provider is not compliant with the latest version of terraform. For one, terraform decided to settle of requiring snake case (e.g. foo_bar
) for schema elements and disallowing camel casing (e.g. fooBar). So at least that will need to be changed, which will be a breaking change to all existing configurations. There may be other changes as well.
Hunting down issues is quite difficult at the moment, with the lack of detailed DEBUG and TRACE level logging.
A few of the resources have multiple API calls/steps to properly provision them. Which means there is a chance that one or more calls will fail and a resource will only be partially created or updated (or possibly only partially deleted).
Need to use terraforms support for partial states so that we can properly recover from these types of errors.
expiresIn
is a relative value (relative to "now"), so it will always keep changing, causing it to mark the resource as requiring an update.
We're fixing an issue in our API, where component configuration row detail (call to /v2/storage/components/{component_id}/configs/{config_id}/rows/{row_id}
) currently returns all rows instead of the exact row.
We have a release ready (see related change in Storage API client for PHP and Apiary docs), that fixes this. The endpoint will return only one row based on the supplied id.
We've checked and most of the calls to this API are from this terraform provider, so we'd like to coordinate the change with a change in your API. Could you change your API accordingly and let us know if the change causes any trouble please?
Hi ๐
thanks for this great provider!
I was wondering if I could submit a PR with fixing the build that is terraform 0.15 & go 1.16 compatible.
I managed to fix all the imports and build works fine, the provider binary too.
Would you be interested?
When the keboola_snowflake_writer
is updated (i.e. for column changes or whatever), then it loses its snowflake credentials, meaning that it will fail when it is actually used. At the moment, we've had to manually select credentials again in the UI. The update request (PUT) might need to be changed to always send the snowflake credentials so they don't get dropped.
The orchestration
resource lacks an active
flag (true/false), which means orchestrations can't be enabled or disabled without using the UI.
We should add this in to the Orchestration struct
{
"id":458432432,
"name":"Some stuff",
...
"active":true,
The PUT
call for updating the snowflake db configuration (for a Snowflake writer) should be pointing at the Storage API instead of the Syrup API.
Currently terraform import is not supported on all resources. This feature would allow greater flexibility for those that are moving from an unmanaged to a terraform managed keboola instance.
https://www.terraform.io/docs/extend/resources.html#importers
After updating a GoodData User Management configuration through terraform, the following error occurs when executing the application:
message: Unrecognized option "datatypes" under "container.storage.input.tables.0"
exceptionId: docker-b4e5a5f169c9078248a857c14b4fab8b
This appears to the be caused by the JSON being pushed up as configuration containing the datatypes
attribute, even though it is not used for this type of resource.
Should add 'omitempty' so that when it is not set, it is not sent.
As changes are being made to the provider, it needs to be properly versioned.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.