Plan always showing changes when using multiple NAT pool addresses about terraform-provider-packetfabric HOT 19 CLOSED

Sorry, I should have included this initially. We're using for_each to create multiple BGP sessions, and a dynamic block within each BGP session resource to configure the pool prefixes.

resource "packetfabric_cloud_router_bgp_session" "cr_bgp_gcp" {
  circuit_id = packetfabric_cloud_router.cr.id

  for_each = {
    for index, conn in var.gcp_connections :
    conn.set_identifier => conn
  }

  bfd_interval   = each.value.bgp_session.enable_bfd ? local.default_bfd_interval : 0
  bfd_multiplier = each.value.bgp_session.enable_bfd ? local.default_bfd_multiplier : 0
  connection_id  = packetfabric_cloud_router_connection_google.crc[each.key].id
  md5            = each.value.bgp_session.md5
  med            = each.value.bgp_session.med
  orlonger       = each.value.bgp_session.orlonger

  l3_address     = each.value.bgp_session.pf_router_peer_address
  remote_asn     = each.value.bgp_session.remote_asn
  remote_address = each.value.bgp_session.remote_router_peer_address

  dynamic "nat" {
    for_each = each.value.bgp_session.nat != null ? [each.value.bgp_session.nat] : []

    content {
      direction       = nat.value.direction
      nat_type        = nat.value.nat_type
      pre_nat_sources = nat.value.pre_nat_sources
      pool_prefixes   = nat.value.pool_prefixes
    }
  }

  dynamic "prefixes" {
    for_each = each.value.bgp_session.allowed_prefixes

    content {
      prefix     = prefixes.value.prefix
      type       = prefixes.value.type
      match_type = prefixes.value.match_type
    }
  }

  lifecycle {
    ignore_changes = [
      md5 # Avoid storing MD5 authentication key in TF config, if possible.
    ]
  }
}

Using terraform 1.5.7 and PF provider v1.7.0.

from terraform-provider-packetfabric.

I tried re-building the provider with the following patch bobmshannon@4fe11cf and the issue went away. I am not sure if this change introduces any unintended side effects though.

from terraform-provider-packetfabric.

I've spent some time trying to replicate the problem but without much success so far.

Could you attach the relevant pieces of your .tf file so I can try to debug the issue?

from terraform-provider-packetfabric.

Thanks for getting back to me on this. Would it be possible to also get the variable definition for gcp_connections with any information you want obscured replaced with private ips, asn, or empty strings?

from terraform-provider-packetfabric.

Here are the redacted values populated for one of the impacted packetfabric_cloud_router_bgp_session in our state file:

resource "packetfabric_cloud_router_bgp_session" "cr_bgp_gcp" {
    address_family   = "v4"
    as_prepend       = 0
    bfd_interval     = 1000
    bfd_multiplier   = 5
    circuit_id       = "PF-L3-CUST-<<SNIPPED>>"
    connection_id    = "PF-L3-CON-<<SNIPPED>>"
    id               = "<<SNIPPED>>"
    l3_address       = "aaa.bbb.cc.dd/29"
    local_preference = 0
    md5              = "<<SNIPPED>>"
    med              = 2000
    multihop_ttl     = 1
    orlonger         = false
    remote_address   = "eee.fff.gg.hh/29"
    remote_asn       = 16550

    nat {
        direction       = "input"
        nat_type        = "overload"
        pool_prefixes   = [
            "185.161.2.96/27",
        ]
        pre_nat_sources = [
            "10.0.0.0/8",
            "100.64.0.0/10",
        ]
    }

    prefixes {
        as_prepend       = 0
        local_preference = 0
        match_type       = "exact"
        med              = 0
        prefix           = "x.yyy.0.0/14"
        type             = "out"
    }
    prefixes {
        as_prepend       = 0
        local_preference = 0
        match_type       = "orlonger"
        med              = 0
        prefix           = "10.0.0.0/8"
        type             = "in"
    }
    prefixes {
        as_prepend       = 0
        local_preference = 0
        match_type       = "orlonger"
        med              = 0
        prefix           = "100.64.0.0/10"
        type             = "in"
    }
}

When we run a plan, the following changes are proposed:

  ~ resource "packetfabric_cloud_router_bgp_session" "cr_bgp_gcp" {
        id               = "<<snipped>>"
        # (15 unchanged attributes hidden)

      - nat {
          - direction       = "input" -> null
          - nat_type        = "overload" -> null
          - pool_prefixes   = [
              - "185.161.2.96/27",
            ] -> null
          - pre_nat_sources = [
              - "100.64.0.0/10",
              - "10.0.0.0/8",
            ] -> null
        }
      + nat {
          + direction       = "input"
          + nat_type        = "overload"
          + pool_prefixes   = [
              + "185.161.2.96/27",
            ]
          + pre_nat_sources = [
              + "10.0.0.0/8",
              + "100.64.0.0/10",
            ]
        }

        # (3 unchanged blocks hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

It looks like its detecting changes to the nat block but on manual inspection there doesn't appear to actually be any meaningful changes.

I know you asked for the full value of the local.gcp_connections variable which is somewhat harder to get but if needed I can certainly try to share it here just let me know.

from terraform-provider-packetfabric.

Great, thanks! Our current theory is the backend is sorting the prefixes and this is cause terraform to see the prefixes as changed, but so far I haven't been able to verify this.

from terraform-provider-packetfabric.

There's a BGP session on one of our accounts actively experiencing this we could grant you access to in order to help reproduce the issue. If that's a path we want to explore we'd have to re-connect through support though.

from terraform-provider-packetfabric.

One other thing I'll add re: sorting, based on my reports, it looks like changes in ordering for both pre_nat_sources and pool_prefixes appear to lead to this issue.

from terraform-provider-packetfabric.

If you order the prefixes before calling the apply does that fix the issue? This was our original theory, but I wasn't able to reproduce it with a simplistic example for some reason. I'm still in the process of trying to reproduce it with the additional information you have provided.

from terraform-provider-packetfabric.

OK! I think I have reproduced the issue with this configuration (snippet):

resource "packetfabric_cloud_router_bgp_session" "cr_bgp1" {
    provider       = packetfabric
    circuit_id     = packetfabric_cloud_router.cr1.id
    connection_id  = packetfabric_cloud_router_connection_aws.crc1.id
    remote_asn     = 64535
    remote_address = "10.0.247.41/30" # AWS side
    l3_address     = "10.0.247.42/30" # PF side

    address_family   = "v4"
    as_prepend       = 1
    bfd_interval     = 1000
    bfd_multiplier   = 5
    local_preference = 0
    med              = 2000
    multihop_ttl     = 1
    orlonger         = false

    nat {
        direction       = "input"
        nat_type        = "overload"
        pool_prefixes   = [
            "10.161.2.96/27",
            "10.101.2.96/27",
            "10.191.2.96/27",
        ]
        pre_nat_sources = [
            "10.0.0.0/8",
            "10.64.0.0/19",
            "10.33.0.0/19"
        ]
    }
    prefixes {
        as_prepend       = 1
        local_preference = 0
        match_type       = "exact"
        med              = 0
        prefix           = "10.33.0.0/19"
        type             = "out"
    }
    prefixes {
        as_prepend       = 1
        local_preference = 0
        match_type       = "orlonger"
        med              = 0
        prefix           = "10.0.0.0/8"
        type             = "in"
    }
    prefixes {
        as_prepend       = 1
        local_preference = 0
        match_type       = "orlonger"
        med              = 0
        prefix           = "10.64.0.0/19"
        type             = "in"
    }
}

and rerunning apply it always says something like:

  # packetfabric_cloud_router_bgp_session.cr_bgp1 will be updated in-place
  ~ resource "packetfabric_cloud_router_bgp_session" "cr_bgp1" {
        id               = "<<xxx>>"
        # (14 unchanged attributes hidden)

      - nat {
          - direction       = "input" -> null
          - nat_type        = "overload" -> null
          - pool_prefixes   = [
              - "10.101.2.96/27",
              - "10.161.2.96/27",
              - "10.191.2.96/27",
            ] -> null
          - pre_nat_sources = [
              - "10.0.0.0/8",
              - "10.64.0.0/19",
              - "10.33.0.0/19",
            ] -> null
        }
      + nat {
          + direction       = "input"
          + nat_type        = "overload"
          + pool_prefixes   = [
              + "10.161.2.96/27",
              + "10.101.2.96/27",
              + "10.191.2.96/27",
            ]
          + pre_nat_sources = [
              + "10.0.0.0/8",
              + "10.64.0.0/19",
              + "10.33.0.0/19",
            ]
        }

      - prefixes {
          - as_prepend       = 0 -> null
          - local_preference = 0 -> null
          - match_type       = "orlonger" -> null
          - med              = 0 -> null
          - prefix           = "10.0.0.0/8" -> null
          - type             = "in" -> null
        }
      - prefixes {
          - as_prepend       = 0 -> null
          - local_preference = 0 -> null
          - match_type       = "orlonger" -> null
          - med              = 0 -> null
          - prefix           = "10.64.0.0/19" -> null
          - type             = "in" -> null
        }
      + prefixes {
          + as_prepend       = 1
          + local_preference = 0
          + match_type       = "orlonger"
          + med              = 0
          + prefix           = "10.0.0.0/8"
          + type             = "in"
        }
      + prefixes {
          + as_prepend       = 1
          + local_preference = 0
          + match_type       = "orlonger"
          + med              = 0
          + prefix           = "10.64.0.0/19"
          + type             = "in"
        }

        # (1 unchanged block hidden)
    }

I'll start working on a fix for this, just let me know if this is a different issue.

from terraform-provider-packetfabric.

Yeah, that looks like it might be it. 👍

from terraform-provider-packetfabric.

It seemed like it was tied to the ordering but even if the .tf file has the prefixes in the same order it doesn't seem to make a difference.

from terraform-provider-packetfabric.

Nice! I was actually going the other way from Set to List for prefixes and added some sorting for the nat pieces, but I'll try this fix instead. Thanks!

from terraform-provider-packetfabric.

I tried that change from list to set, but I still get the same error:

      - nat {
          - direction       = "input" -> null
          - nat_type        = "overload" -> null
          - pool_prefixes   = [
              - "10.101.2.96/27",
              - "10.161.2.96/27",
              - "10.191.2.96/27",
            ] -> null
          - pre_nat_sources = [
              - "10.0.0.0/8",
              - "10.33.0.0/19",
              - "10.64.0.0/19",
            ] -> null
        }
      + nat {
          + direction       = "input"
          + nat_type        = "overload"
          + pool_prefixes   = [
              + "10.161.2.96/27",
              + "10.101.2.96/27",
              + "10.191.2.96/27",
            ]
          + pre_nat_sources = [
              + "10.0.0.0/8",
              + "10.64.0.0/19",
              + "10.33.0.0/19",
            ]
        }

from terraform-provider-packetfabric.

Hmm, I wonder if the sorting/order of these fields coming from the backend is undefined, and when I re-ran the plan with the change, the order coming from the backend happened to match what was in the state file by chance.

I was thinking that https://developer.hashicorp.com/terraform/plugin/sdkv2/schemas/schema-types#typeset would handle these as unordered sets of collections properly assuming that's the issue we're running into here:

TypeSet implements set behavior and is used to represent an unordered collection of items, meaning that their ordering specified does not need to be consistent, and the ordering itself has no impact on the behavior of the resource.

from terraform-provider-packetfabric.

From what I can tell, the backed always returns the lists sorted and this was different from the order in the config based on the snippet provided. What did work for me was to convert the "prefixes" to list and then make sure the hcl had the prefixes (also for the pool, etc) sorted.

It's really strange set worked for you because I saw the opposite occur. When I sorted the prefixes like 1,2,3 the set reordered them in the plan to say something like 3,1,2, but changing to a List left the order alone. I also experimented with sorting the pool_prefixes and pre_nat_sources but it didn't seem to make any difference.

Thanks for all the feedback! Hopefully it won't be too much longer to get a fix

from terraform-provider-packetfabric.

My mistake: when trying out the patch I checked out fresh, but forgot to change the OS_ARCH in the Makefile. Now though, I get: Error: Status: 400, body: {"message": "nat: pre_nat_sources and pool_prefixes must be set when nat_type is overload."}

This maybe due to the extra code that is usually needed for the provider to marshal / unmarshal the TypeSet vs TypeList

from terraform-provider-packetfabric.

I added the set to list conversion and it does seem to be working, so I will start a PR for it shortly.

Thanks finding this! It may be worthwhile to go thru the rest of our provider and make a similar change.

from terraform-provider-packetfabric.

Thanks, once a new release is cut, we'll definitely upgrade and start using the new provider version 👍

from terraform-provider-packetfabric.