consider adding support for terraform provider google beta about pulumi-gcp HOT 18 CLOSED

Let's try to come up with a plan for this in M20 - there is an increasing amount of functionality in both beta and 2.0.0 that is not available in the main provider.

from pulumi-gcp.

HI @pbzdyl! Thanks for expressing concern here: I agree this is important - switching to the beta provider for everything is the course of action we're intending to take. Unfortunately, the beta provider was introduced as v2.0.0, with a variety of breaking changes upstream (many shared with the production-services-only provider), and we need to get pulumi/pulumi#2389 out to upgrade to v2.0.0 of either. However, that versioning work has been brought back in scope for M21, thus it should be possible to move this back into M21 also.

from pulumi-gcp.

I spoke briefly with @danawillow last week about both magic modules and the beta split. Both are things we want to look into further.

Medium/long term, generating the Pulumi provider directly from magic modules is very appealing. It sounded like maybe there is some await logic that is not yet schematized that would be important to us fully autogenerating these - but we will be tracking this.

Nearer term, with the beta split, we will need to decide how to represent that in the Pulumi gcp package. There are at least two options:

1. Create two providers @pulumi/gcp and @pulumi/gcp-beta
2. Project both TF providers into a single Pulumi provider and put all resources from the beta provider under a .beta. namespace.

Of these (2) feel like a much better user experience. This is essentially what we do for Kubernetes today, and it works really well. (We actually overlap all versions into a single namespaced API structure). The only question will be about the versioning of the two Terraform providers. We would need to build a single Pulumi provider from two different versions of the two different Terraform providers, which may be challenging to reliably version in lock-step. This will depend a bit on the versioning strategy used across these in the Terraform GCP providers.

If for some reason that is not possible, (1) is a viable fallback option.

from pulumi-gcp.

If it's truly a strict superset, I vote for saving ourselves the overhead of yet another package to manage, and just standardize on this as the new GCP.

from pulumi-gcp.

This is now merged to master, and assuming a successful run with the examples repo overnight, I'll push out a new release in the morning.

from pulumi-gcp.

As it stands right now, the only difference between dependencies of the "stable" and "beta" Terraform providers for GCP is google.golang.org/api/compute/v0.beta:

• Stable revision: 31fc25cd84cd7812131ff616db6bf91ae0f6cb98 (2018-10-24T23:56:32Z)
• Beta revision: 087779f1d2c96357d4c45bd04c4d10d7b5f22736 (2018-08-29T00:05:35Z)

This is somewhat counter-intuitive in that one would expect the dependencies of the beta provider to be newer.

I guess we'll see what happens in regards to dependencies as resources get removed from the stable provider and make a decision then.

from pulumi-gcp.

We would greatly appreciate the availability of the beta provider, as of yet we are stuck due to the lack of it.

Our use case is the following: We require our k8s clusters to be private, e.g. using the google beta provider and terraform we used to configure the cluster like this

private_cluster_config = {
    enable_private_endpoint = true
    enable_private_nodes    = true
    master_ipv4_cidr_block  = "..."
}

These options are not available as of now in pulumi gcp, I found out that there are the following two properties instead ...

privateCluster: true,
masterIpv4CidrBlock: "..."

... but the master IP of the cluster is publicly available once it is created, not really sure if this is possible to do until the beta provider is available in pulumi-gcp.

As a side note, thank you all for this awesome tool, really enjoying it so far ❤️

from pulumi-gcp.

The situation here is slightly different to how we originally understood it. Rather than being a separate provider with some additional resources, the Google Beta provider (guide here) is interchangeable with the main Google provider, but exposes additional resources and properties for beta products or features.

The Google beta provider makes heavy use of the unfortunate feature of govendor which allows taking a dependency on different packages from the same repository at different revisions. Consequently using the approach of dep, it is not possible to create a working vendor tree for the beta provider, since using the later revision for beta features introduces breaking changes in other resources.

Our options, if we want to adopt the beta provider, then:

• Use an alternative tool to dep to generate the vendor tree. It feels like govendor would be the most straightforward since the information is already contained upstream. This isn't particularly satisfying as we'll also need to satisfy the pulumi-terraform versions and so forth. Other options here include please.build and bazel however, both of which allow the required workflow, but with a significant maintenance overhead.
• Maintain a fork of the Google beta repository which depends on the latest-vendored (for all packages) version of the SDK, and fixes up the breakage. Right now this feels like it would be less work.

As Terraform switches to Go modules, this problem may go away if each SDK component is versioned as an independent module, or if the upstream is fixed to use a single revision of the SDK. I'm not sure we can rely on that happening as part of addressing this issue, however.

Update: if we're prepared to depend on an unreleased version for the 2.0.0 branch, the switch to Go modules has already been made.

from pulumi-gcp.

Work on this is now under way - pulumi/scripts#34 is the first step in allowing our build process to work for the Google beta provider.

from pulumi-gcp.

Now that gomod-override is working, we have a few options for the beta provider:

1. Adopt that as the backing for pulumi-gcp. This is my preference overall, even though it will result in breaking changes more frequently (though only in beta features, one would hope). The major benefit here is that it will result in a better workflow for users who find themselves wanting a beta feature (or worse, property) halfway through development - no new package would need installing, and there would be no need to potentially migrate resources between providers. A pull request is open for this at #86.

2. Bring up a new provider named pulumi-gcp-beta.

@lukehoban, @joeduffy - is there a significant preference for one of these options vs the other?

from pulumi-gcp.

I think (1) is a better user experience, but I worry about the breaking changes at least until we have a solution to pulumi/pulumi#2389 (basically - today breaking changes in the provider binary are difficult for users to adapt to cleanly). I'm also just not 100% sure about the planned strategy for versioning these two providers upstream yet - for example, will the beta provider always be purely a superset - or will they start getting out-of-sync (the non-beta provider ships with some delay shipping the beta provider)?

Also - since the beta provider hasn't shipped 2.0.0 yet, I'm not sure about jumping over to it in our main @pulumi/gcp yet.

(2) feels a little safer given unknowns for now - with the option to move back to (1) once we gain more experience with how these will version. Thoughts?

from pulumi-gcp.

From what I can tell, the beta provider will always be a strict superset of the non-beta one - based on the instructions for switching between them. The changes we’d pick up from 1.2 to 2.0 are the same ones we’d pick up when the regular provider goes to 2.0 (there hasn’t been a release in some time, so I assume there is preparation for that ongoing).

If we go with a new provider, what URNs and so forth should we use? “gcp” also or something more like “gcp-beta”? How would we recommend people migrate provisioned resources between the regular and beta provider if they need access to beta properties?

from pulumi-gcp.

Okay - I'm fine with standarizing on mapping @pulumi/gcp to google-beta.

Given that though, I'd suggest we should not make this transition until they release 2.0.0, since we'd prefer to map released versions of upstream, but have a dependency on the Go Modules work they've done in the 2.0.0 tree. Thoughts? If so, let's push this to M21 for tracking upstream.

from pulumi-gcp.

Right now we're tracking master of terraform-provider-google-beta in #86, which has not been released but is destined to be v2.0.0 from what I can tell. v1.2.0 (the latest beta release) is not solvable with dep as it has multiple revisions of different packages in the GCP SDK itself (thanks to govendor). I'll try to keep #86 up to date during the course of M21 so that we can release whenever upstream does - the bulk of the work was done in pulumi/scripts#34 and pulumi/scripts#35.

from pulumi-gcp.

This is dependent on solving the issue of versioning breaking changes for providers (pulumi/pulumi#2389), which doesn't look like it will be done until M22. Consequently I'm going to move this to M22. The other preconditions are all met:

• Upstream has released v2.0.0 in both 'beta' and 'production' flavours
• We have gomod-override and have verified it works correctly for this provider

from pulumi-gcp.

This is quite serious limitation for my usage as I need some of the google beta provider features (e.g. https://www.terraform.io/docs/providers/google/r/service_networking_connection.html). Would it be possible to just switch everything to google-beta provider instead of waiting for something that will be released much later?

from pulumi-gcp.

Hi all, the work on which this depends has been completed, but is not yet widely circulated enough to release a provider which such sweeping breaking changes. I'm going to move this to M22, but with the intention that it will be released as soon as there is confidence that it won't break everyone in hard-to-discover ways. The actual work is done in #106, for people who want to build it on their own until a release date.

from pulumi-gcp.

That's amazing. Thank you 🙌

from pulumi-gcp.