jeffgiroux / f5_terraform Goto Github PK

folder: GCP/HA_via_api

Intermittently, the custom account admin account isn't created.

symptoms:

can't login with sshkey or password after creation. have to use the "admin" account which should be disabled.

[admin@mydemo-f5vm01:Active:Standalone] ~ # cat /var/log/cloud/onboard.log | grep -A 20 Updating
Updating admin account
Syntax Error: "bash" unknown property
Copying SSH key
Admin account updated

code:

https://github.com/JeffGiroux/f5_terraform/blob/master/GCP/HA_via_api/onboard.tpl#L407

# Create admin account and password
echo "Updating admin account"
if [[ $admin_username != "admin" ]]; then
  tmsh create auth user $admin_username password "$admin_password" shell bash partition-access add { all-partitions { role admin } };
else
  tmsh modify auth user admin password "$admin_password";
fi

I don't see the difference from mine, but I haven't hit this, maybe something in 15.1?
https://github.com/vinnie357/bigip-bash-onboard-templates/blob/master/gcp/onboard.sh#L100

Updating BIG-IP and F5 toolchain RPM versions

• BIG-IP version = 16.1.2.1
• Declarative Onboarding (DO) = 1.28
• Application Services (AS3) = 3.35
• Telemetry Streaming (TS) = 1.27
• Cloud Failover (CFE) = 1.10

Examples to update:

• Autoscale via LB
• HA via API
• Standalone
• Infrastructure

Updating BIG-IP and F5 toolchain RPM versions

• BIG-IP version = 16.1.2.1
• Declarative Onboarding (DO) = 1.28
• Application Services (AS3) = 3.35
• Telemetry Streaming (TS) = 1.27
• Cloud Failover (CFE) = 1.10
  Updating BIG-IP and F5 toolchain RPM versions

Examples to update:

• Autoscale via LB
• HA via API
• HA via LB
• Standalone
• Infrastructure

Repo: https://github.com/JeffGiroux/f5_terraform/tree/main/GCP/HA_via_api

Issue where the longer GCP regions e.g. Canada northamerica-northeast1 when pulled in via the GCP metadata server to create the BigIP name break the 64 char limit.
Example is mydemo123-bigip1-28a4.northamerica-northeast1-a.c.th-mysandbox-lab-56c8bd.internal

Use a variable to manage the name:

1. Add variable domain_name to variables with a default of example.com
2. Add variable domain_name to terraform.tfvars
3. Add domain_name = var.domain_name to bigip.tf for locals f5_onboard1 and f5_onboard2.
4. Update f5_onboard.tmpl 3 lines below to leverage the variable:

Line 153: http://169.254.169.254/computeMetadata/v1/instance/name
Line 214: ui.advisory.text: '{{{ HOST_NAME }}}.${domain_name}'
Line 218: hostname: '{{{ HOST_NAME }}}.${domain_name}'

BIG-IP virtual instances licensed with BYOL in Google Cloud can lose management connectivity after a reboot.

Workaround

1. Re-apply management routes manually
2. Or...add mgmt-route.sh script to run each boot by adding an entry to /config/startup

echo "/config/cloud/mgmt-route.sh &" >> /config/startup

This script is already included with the repo and built during onboarding by the onboard.tpl code. It will delete the mgmt route and re-add it at each reboot.

Reference

https://support.f5.com/csp/article/K85730674

Also see F5 Product Development produced GDM templates for similar issue.
F5Networks/f5-google-gdm-templates#33
F5Networks/f5-google-gdm-templates#45

I found some things that need to be changed in the startup_script:

• In line 169 the "pre_onboard_enabled" throws an error when the machine is starting
  In the official repository of F5Networks (https://github.com/F5Networks/terraform-gcp-bigip-module/blob/main/startup-script.tpl) those commands executed in the beginning of the script before the the process MCPD starts

• There is no need for the "post_onboard_enabled" because there is no need to save the config once things has changed

Hope this is helpful
Regards,
Daniel Y

The current GCP repo is configured to deploy BIG-IP devices with 3 NICs. The repo is statically set and currently not doing any dynamic interface. Therefore, some manual edits are needed in a few spots to make a 2 NIC deployment. You will need to edit onboard.tpl and bigip.tf.

Modify onboard.tpl

1. Remove the TMSH commands to prevent creation of the 'internal' NIC associated to INT2ADDRESS

"tmsh create net vlan internal interfaces add { 1.2 } mtu 1460"
"tmsh create net self self_internal address $${INT2ADDRESS}/32 vlan internal allow-service add { tcp:4353 udp:1026 }"
"tmsh create net route int_gw_interface network $${INT2GATEWAY}/32 interface internal"
"tmsh create net route int_rt network $${INT2NETWORK}/$${INT2MASK} gw $${INT2GATEWAY}"

2. Remove lines that reference INT2NETWORK

INT2NETWORK=$(/bin/ipcalc -n $INT2ADDRESS $INT2MASK | cut -d= -f2)
echo "INT2NETWORK=$INT2NETWORK" >> /config/cloud/interface.config

4. Save file

**Note: If you are doing an HA deployment then it is VERY important to make sure the external NIC has "allow-service add { tcp:4353 udp:1026 }" so cluster HA creation is formed. Replace this...

"tmsh create net self self_external address $${INT1ADDRESS}/32 vlan external"

With this...

"tmsh create net self self_external address $${INT1ADDRESS}/32 vlan external allow-service add { tcp:4353 udp:1026 }"

Modify bigip.tf

1. Locate the resource "google_compute_instance" for the F5 VM(s)
2. Delete the block for network_interface that points to var.intVPC

  network_interface {
    network    = var.intVpc
    subnetwork = var.intSubnet
  }

4. Save file

Now, you can run 'terraform apply' to deploy a 2-NIC BIG-IP device.

Hi Jeff, we are deploying the templates on GCP using terraform version v0.13.5.
BYOL image: f5-bigip-15-1-0-4-0-0-6-byol-all-modules-2boot-loc-00618231635
BIGIP VEs are created successfully, however, the onborad scripts were not being exceuted.

I have re-deployed several times (3 times), still got the same results. Do you know what could be possible cause?

Hi Jeff
I'm using the HA_via_api in GCP, and the DO and CFE modules are working, but the AS3 module doesn't ever apply to the f5. No error message that I can see either. I'm working with Turo out of Finland. Thanks

The script is working fine for deployment however instances are not getting the azure managed dns name for the instances because of that i can not ping instances using dns name.

the local host file has entry to resolve the mgmt ip address on 127.0.0.1 instance of local domain name .

2 fixes are needed

1. Assign the a valid hostname e.g xx.southeastasia.cloudapp.azure.com which should be coming from Azure managed DNS
2. Hostfile should be having only valid entry to map the mgmt ip address with the valid hostname

1. https://github.com/JeffGiroux/f5_terraform/tree/main/AWS/HA_via_api#byol-licensing

In the "f5_onboard.tmpl", add the "myLicense" block under the "Common" declaration (example here)

• "myLicense" should be "My_License" per https://github.com/F5Networks/f5-aws-cloudformation-v2/blob/main/examples/failover/bigip-configurations/runtime-init-conf-3nic-byol-instance01.yaml#:~:text=mgmtDhcpEnabled%3A%20false-,My_License,-%3A

2. https://github.com/JeffGiroux/f5_terraform/tree/main/AWS/HA_via_api#installation-example

• Perhaps recommend if modifying the "f5_password" variable that it must be a complex password conforming to F5 password secure restrictions. Changing the password to something like "admin" will cause DO and AS3 functions to fail.

in f5_terraform/AWS/Infrastructure-only/main.tf

hardcoded version no longer works. Id just let it go to default

User requires that the BIG-IP in Azure deploy with an encrypted disk at instance boot up and that this is encrypted with the customer managed key.

1. Update README
2. Add steps for encryption process

on HA_via_lb azure template after running DO, its perfectly running on instance 1 but the script doesnt execute on instance 2:

null_resource.f5vm02_DO: Provisioning with 'local-exec'...
null_resource.f5vm02_DO (local-exec): Executing: ["/bin/sh" "-c" "#!/bin/bash\ncurl -k -X POST https://x.x.x.x/mgmt/shared/declarative-onboarding -u user:password -d @vm02_do_data.json\nx=1; while [ $x -le 30 ]; do STATUS=$(curl -s -k -X GET https://52.224.75.44/mgmt/shared/declarative-onboarding/task -u f5pocbdb:33F5PoCdbd33); if ( echo $STATUS | grep "OK" ); then break; fi; sleep 10; x=$(( $x + 1 )); done\nsleep 10\n"]
null_resource.f5vm02_DO (local-exec): % Total % Received % Xferd Average Speed Time Time Time Current
null_resource.f5vm02_DO (local-exec): Dload Upload Total Spent Left Speed
null_resource.f5vm02_DO (local-exec): 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
null_resource.f5vm02_DO (local-exec): 0 2846 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
null_resource.f5vm02_DO (local-exec): 100 2846 0 0 100 2846 0 2719 0:00:01 0:00:01 --:--:-- 2720

null_resource.f5vm02_DO: Still creating... [10s elapsed]

Instance 1 has this behavior that is the correct:

null_resource.f5vm01_DO: Provisioning with 'local-exec'...
null_resource.f5vm01_DO (local-exec): Executing: ["/bin/sh" "-c" "#!/bin/bash\ncurl -k -X POST https://52.226.54.105/mgmt/shared/declarative-onboarding -u f5pocbdb:33F5PoCdbd33 -d @vm01_do_data.json\nx=1; while [ $x -le 30 ]; do STATUS=$(curl -s -k -X GET https://52.226.54.105/mgmt/shared/declarative-onboarding/task -u f5pocbdb:33F5PoCdbd33); if ( echo $STATUS | grep "OK" ); then break; fi; sleep 10; x=$(( $x + 1 )); done\nsleep 10\n"]
null_resource.f5vm01_DO (local-exec): % Total % Received % Xferd Average Speed Time Time Time Current
null_resource.f5vm01_DO (local-exec): Dload Upload Total Spent Left Speed
null_resource.f5vm01_DO (local-exec): 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
null_resource.f5vm01_DO (local-exec): 0 2846 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
null_resource.f5vm01_DO (local-exec): 100 4859 100 2013 100 2846 1938 2741 0:00:01 0:00:01 --:--:-- 2741
null_resource.f5vm01_DO (local-exec): {"id":"b5a70bf6-81c9-4ebc-8946-888861da9446","selfLink":"https://localhost/mgmt/shared/declarative-onboarding/task/b5a70bf6-81c9-4ebc-8946-888861da9446","result"

Upon changing the VM variables the terraform update fails with the following error:

│ Error: Error when reading or editing TargetInstance: googleapi: Error 400: The target_instance resource 'ZONE:northamerica-northeast1-a/PROJECT:977287114090/TARGET_INSTANCE:myf5demo-ti-c389' is already being used by 'REGION:northamerica-northeast1/PROJECT:977287114090/FORWARDING_RULE:myf5demo-forwarding-rule-c389', resourceInUseByAnotherResource

Workaround: To manually delete the forwarding rules using the target instances. And, re-run terraform apply works.

Need troubleshooting info documented. Example...

1. Make sure device has outbound internet access
2. Make sure proper VPC and firewall rules are in place
3. Make sure service account has correct permissions
4. Review onboard logs in /var/log/cloud directory

and so on...

Background

Currently the mgmt NIC swap happens during onboard. It moves mgmt from default eth0 to eth1. This is to facilitate Google LB requests which can only send to eth0 of a virtual machine. Therefore, the BIG-IP performs a NIC swap during onboard. Google numbers the NICs on a VM like eth0, eth1, eth2, and so on. Metadata ordering is also the same and starts with interface/0, interface1, interface2, and so on.

Requested Scenario

User needs to have the mgmt NIC moved to the last NIC of the BIG-IP virtual machine. Using the 3nic example, they want mgmt on the third NIC. That means code changes are needed to move mgmt to eth2. As noted in the "Background" section, the current NIC swap moves mgmt to eth1.

Change 1 - onboard.tmpl "sys db" values

Modify this line for NIC swap in onboard.tmpl. Currently the script moves it to eth1. Update to eth2.

Current code is...

tmsh modify sys db provision.managementeth value eth1

To move mgmt to the 3rd NIC (ex. eth2)...

tmsh modify sys db provision.managementeth value eth2

Change 2 - onboard.tmpl "metadata" values

Modify multiple lines where metadata NIC retrieval happens. The mgmt NIC and the last NIC each have three lines to update. The script currently collects info on array position /instance/network-interfaces/1/ for the mgmt NIC. This data is saved to the interface.config file later to be used by TMSH commands to set IPs and routes. Update to /instance/network-interfaces/2/.

Current code is...

echo "MGMTADDRESS=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/1/ip" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "MGMTMASK=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/1/subnetmask" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "MGMTGATEWAY=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/1/gateway" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT1ADDRESS=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/0/ip" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT1MASK=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/0/subnetmask" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT1GATEWAY=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/0/gateway" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT2ADDRESS=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/2/ip" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT2MASK=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/2/subnetmask" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT2GATEWAY=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/2/gateway" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config

Swap the values for MGMT (use /interface/2)...

echo "MGMTADDRESS=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/2/ip" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "MGMTMASK=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/2/subnetmask" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "MGMTGATEWAY=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/2/gateway" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config

Swap the values for INT2 (use /interface/1)...

echo "INT2ADDRESS=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/1/ip" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT2MASK=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/1/subnetmask" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config
echo "INT2GATEWAY=$(curl -s -f --retry 10 "$${COMPUTE_BASE_URL}/instance/network-interfaces/1/gateway" -H 'Metadata-Flavor: Google')" >> /config/cloud/interface.config

Change 3 - bigip.tf "network_interface" blocks

Modify the BIG-IP virtual machine resource. Currently the network_interface blocks place mgmt NIC second in order which makes it eth1. It goes eth0, eth1, eth2. Move it to the third location so it becomes eth2.

Current code is...

  network_interface {
    network    = var.extVpc
    subnetwork = var.extSubnet
    access_config {
    }
  }
  network_interface {
    network    = var.mgmtVpc
    subnetwork = var.mgmtSubnet
    access_config {
    }
  }
  network_interface {
    network    = var.intVpc
    subnetwork = var.intSubnet
  }

Move mgmt to last location...

  network_interface {
    network    = var.extVpc
    subnetwork = var.extSubnet
    access_config {
    }
  }
  network_interface {
    network    = var.intVpc
    subnetwork = var.intSubnet
  }
  network_interface {
    network    = var.mgmtVpc
    subnetwork = var.mgmtSubnet
    access_config {
    }
  }

The template variables show example.com as the dns_suffix variable. This is for example purposes only. If this dns suffix is not modified in terraform.tfvars (or variables.tf), then the device hostname will be unreachable because the VPC is unaware of domain example.com.

Fix

Make sure you update the dns_suffix variable with your correct dns suffix from the VPC network. This will allow the onboarding to properly name the BIG-IP devices with hostname.dns_suffix which matches your environment and will result in successful DNS resolution of hostnames.

Tip

dig -x <internal IP>

...against the gcp dns server to find the dns suffix.

Or...hop on cli of another instance in same VPC and run...

uname -a

...pick your favorite linux way

Updating BIG-IP and F5 toolchain RPM versions

• BIG-IP version = 16.1.2.1
• Declarative Onboarding (DO) = 1.28
• Application Services (AS3) = 3.35
• Telemetry Streaming (TS) = 1.27
• Cloud Failover (CFE) = 1.10

Examples to update:

• Autoscale via LB
• HA via API
• HA via LB
• Standalone
• Infrastructure

Other changes:

• GCP availability zones
• ssh key format documentation

The bigip.tf files in this repo call the BIG-IP module created by the F5 cloud team. The module uses a data "template_file" and this provider is deprecated and causing terraform initialize to fail. Issues have been open with the main provider to update.

Incompatible provider version
Provider registry.terraform.io/hashicorp/template v2.2.0 does not have a package available for your current platform, darwin_arm64.

workaround

1. Perform terraform init
2. it will fail with above error
3. then navigate to hidden .terraform folder, find module, locate tf file with offending data "template_file" code
4. comment all lines references the "template_file"
5. run "terraform init" again

Sometimes a deployment will fail during BIB-IP onboarding phase. Need to use new runtime init version to workaround Azure IMDS issues and API throttling. See existing issue on ARM BIG-IP v2 templates.

F5Networks/f5-azure-arm-templates-v2#15