Giter Club home page Giter Club logo

f5-azure-saca's Introduction

F5 & Azure Secure Cloud Computing Architecture

Introduction

Moving to the Cloud can be tough. The Department of Defense (DoD) has requirements to protect the Defense Information System Networks (DISN) and DoD Information Networks (DoDIN), even for workloads residing in a Cloud Service Provider (CSP). Per the SCCA Functional Requirements Document, the purpose of SCCA is to provide a barrier of protection between the DISN and commercial cloud services used by the DoD.

“It specifically addresses attacks originating from mission applications that reside within the Cloud Service Environment (CSE) upon both the DISN infrastructure and neighboring tenants in a multi-tenant environment. It provides a consistent CSP independent level of security that enables the use of commercially available Cloud Service Offerings (CSO) for hosting DoD mission applications operating at all DoD Information System Impact Levels (i.e. 2, 4, 5, & 6).” * https://dl.dod.cyber.mil/wp-content/uploads/cloud/pdf/SCCA_FRD_v2-9.pdf

This solution uses Terraform to launch a Single Tiered or Three Tier deployment of three NIC cloud-focused BIG-IP VE cluster(s) (Active/Standby) in Microsoft Azure. This is the standard cloud design where the BIG-IP VE instance is running with three interfaces, where both management and data plane traffic is segregated.

The BIG-IP VEs have the following features / modules enabled:

Prerequisites

  • Important: When you configure the admin password for the BIG-IP VE in the template, you cannot use the character #. Additionally, there are a number of other special characters that you should avoid using for F5 product user accounts. See K2873 for details.
  • This template requires a service principal, one will be created in the provided script at ./prepare/setupAzureGovVars_local.sh.
    • Important For gov cloud deployments its important to run this script to prepare your environment, whether local or Azure Cloud CLI based. There are extra env variables that ned to be passed by TF to Gov Cloud Regions.
  • This deployment will be using the Terraform Azurerm provider to build out all the neccessary Azure objects. Therefore, Azure CLI is required. for installation, please follow this Microsoft link
  • If this is the first time to deploy the F5 image, the subscription used in this deployment needs to be enabled to programatically deploy. For more information, please refer to Configure Programatic Deployment
  • You need to set your region and log in to azure ahead of time, the scripts will map your authenitcation credentials and create a service principle, so you will not need to hardcode any credentials in the files.

Important configuration notes

  • All variables are configured in variables.tf
  • MOST STIG / SRG configurations settings have been addressed in the Declarative Onboarding and Application Services templates used in this example.
  • An Example application is optionally deployed with this template. The example appliation includes several apps running in docker on the host:
    • Juiceshop on port 3000
    • F5 Demo app by Eric Chen on ports 80 and 443
    • rsyslogd with PimpMyLogs on port 808
    • Note Juiceshop and PimpMyLogs URLS are part of the terraform output when deployed.
  • All Configuration should happen at the root level; auto.tfvars or variables.tf.

PAYG versus BYOL Settings

  • For PAYG deployments the variables image_name and product need to be configured accordingly, default values are set for PAYG.

  • Example: image_name = f5-bigip-virtual-edition-1g-best-hourly and product = f5-big-ip-best

  • For BYOL deployments the variables image_name, product, and licenses need to be configured accordingly.

  • Example: image_name = f5-big-all-2slot-byol, product = f5-big-ip-byol, and licenses = appropriate licenses.

Variables

Requirements

Name Version
terraform ~> 0.13

Providers

Name Version
azurerm n/a

Inputs

Name Description Type Default
projectPrefix REQUIRED: Prefix to prepend to all objects created, minus Windows Jumpbox string "ccbad9e7"
adminUserName REQUIRED: Admin Username for All systems string "xadmin"
adminPassword REQUIRED: Admin Password for all systems string "pleaseUseVault123!!"
location REQUIRED: Azure Region: usgovvirginia, usgovarizona, etc. For a list of available locations for your subscription use az account list-locations -o table string "usgovvirginia"
region Azure Region: US Gov Virginia, US Gov Arizona, etc string "US Gov Virginia"
deploymentType REQUIRED: This determines the type of deployment; one tier versus three tier: one_tier, three_tier string "three_tier"
deployDemoApp OPTIONAL: Deploy Demo Application with Stack. Recommended to show functionality. Options: deploy, anything else. string "deploy"
sshPublicKey OPTIONAL: ssh public key for instances string ""
sshPublicKeyPath OPTIONAL: ssh public key path for instances string "/mykey.pub"
cidr REQUIRED: VNET Network CIDR string "10.90.0.0/16"
subnets REQUIRED: Subnet CIDRs map(string)
{
"application": "10.90.10.0/24",
"external": "10.90.1.0/24",
"inspect_ext": "10.90.4.0/24",
"inspect_int": "10.90.5.0/24",
"internal": "10.90.2.0/24",
"management": "10.90.0.0/24",
"vdms": "10.90.3.0/24",
"waf_ext": "10.90.6.0/24",
"waf_int": "10.90.7.0/24"
}
f5_mgmt F5 BIG-IP Management IPs. These must be in the management subnet. map(string)
{
"f5vm01mgmt": "10.90.0.4",
"f5vm02mgmt": "10.90.0.5",
"f5vm03mgmt": "10.90.0.6",
"f5vm04mgmt": "10.90.0.7"
}
f5_t1_ext Tier 1 BIG-IP External IPs. These must be in the external subnet. map(string)
{
"f5vm01ext": "10.90.1.4",
"f5vm01ext_sec": "10.90.1.11",
"f5vm02ext": "10.90.1.5",
"f5vm02ext_sec": "10.90.1.12"
}
f5_t1_int Tier 1 BIG-IP Internal IPs. These must be in the internal subnet. map(string)
{
"f5vm01int": "10.90.2.4",
"f5vm01int_sec": "10.90.2.11",
"f5vm02int": "10.90.2.5",
"f5vm02int_sec": "10.90.2.12"
}
f5_t3_ext Tier 3 BIG-IP External IPs. These must be in the waf external subnet. map(string)
{
"f5vm03ext": "10.90.6.4",
"f5vm03ext_sec": "10.90.6.11",
"f5vm04ext": "10.90.6.5",
"f5vm04ext_sec": "10.90.6.12"
}
f5_t3_int Tier 3 BIG-IP Internal IPs. These must be in the waf internal subnet. map(string)
{
"f5vm03int": "10.90.7.4",
"f5vm03int_sec": "10.90.7.11",
"f5vm04int": "10.90.7.5",
"f5vm04int_sec": "10.90.7.12"
}
internalILBIPs REQUIRED: Used by One and Three Tier. Azure internal load balancer ips, these are used for ingress and egress. map(string) {}
ilb01ip REQUIRED: Used by One and Three Tier. Azure internal load balancer ip, this is used as egress, must be in internal subnet. string "10.90.2.10"
ilb02ip REQUIRED: Used by Three Tier only. Azure waf external load balancer ip, this is used as egress, must be in waf_ext subnet. string "10.90.6.10"
ilb03ip REQUIRED: Used by Three Tier only. Azure waf external load balancer ip, this is used as ingress, must be in waf_ext subnet. string "10.90.6.13"
ilb04ip REQUIRED: Used by Three Tier only. Azure waf external load balancer ip, this is used as ingress, must be in inspect_external subnet. string "10.90.4.13"
app01ip OPTIONAL: Example Application used by all use-cases to demonstrate functionality of deploymeny, must reside in the application subnet. string "10.90.10.101"
ips01ext Example IPS private ips string "10.90.4.4"
ips01int n/a string "10.90.5.4"
ips01mgmt n/a string "10.90.0.8"
winjumpip REQUIRED: Used by all use-cases for RDP/Windows Jumpbox, must reside in VDMS subnet. string "10.90.3.98"
linuxjumpip REQUIRED: Used by all use-cases for SSH/Linux Jumpbox, must reside in VDMS subnet. string "10.90.3.99"
instanceType BIGIP Instance Type, DS5_v2 is a solid baseline for BEST string "Standard_DS5_v2"
jumpinstanceType Be careful which instance type selected, jump boxes currently use Premium_LRS managed disks string "Standard_B2s"
appInstanceType Demo Application Instance Size string "Standard_DS3_v2"
image_name REQUIRED: BIG-IP Image Name. 'az vm image list --output table --publisher f5-networks --location [region] --offer f5-big-ip --all' Default f5-bigip-virtual-edition-1g-best-hourly is PAYG Image. For BYOL use f5-big-all-2slot-byol string "f5-bigip-virtual-edition-1g-best-hourly"
product REQUIRED: BYOL = f5-big-ip-byol, PAYG = f5-big-ip-best string "f5-big-ip-best"
bigip_version REQUIRED: BIG-IP Version. Note: verify available versions before using as images can change. string "14.1.400000"
licenses BIGIP Setup Licenses are only needed when using BYOL images map(string)
{
"license1": "",
"license2": "",
"license3": "",
"license4": ""
}
hosts n/a map(string)
{
"host1": "f5vm01",
"host2": "f5vm02",
"host3": "f5vm03",
"host4": "f5vm04"
}
dns_server REQUIRED: Default is set to Azure DNS. string "168.63.129.16"
asm_policy REQUIRED: ASM Policy. Examples: https://github.com/f5devcentral/f5-asm-policy-templates. Default: OWASP Ready Autotuning string "https://raw.githubusercontent.com/f5devcentral/f5-asm-policy-templates/master/owasp_ready_template/owasp-auto-tune-v1.1.xml"
ntp_server n/a string "time.nist.gov"
timezone n/a string "UTC"
onboard_log n/a string "/var/log/startup-script.log"
tags Environment tags for objects map(string)
{
"application": "f5app",
"costcenter": "f5costcenter",
"environment": "f5env",
"group": "f5group",
"owner": "f5owner",
"purpose": "public"
}

Outputs

Name Description
DemoApplication_443 Public IP for applications. Https for example app, RDP for Windows Jumpbox, SSH for Linux Jumpbox
rSyslogdHttp_8080 Public IP for applications. Https for example app, RDP for Windows Jumpbox, SSH for Linux Jumpbox
tier_one One Tier Outputs: VM IDs, VM Mgmt IPs, VM External Private IPs
tier_three Three Tier Outputs: VM IDs, VM Mgmt IPs, VM External Private IPs

Deployment

For deployment you can do the traditional terraform commands or use the provided scripts.

terraform init
terraform plan
terraform apply

OR

./demo.sh

Docker

There is also a dockerfile provided, use make [options] to build as needed.

make build
make shell || make azure || make gov

Destruction

For destruction / tear down you can do the trafitional terraform commands or use the provided scripts.

terraform destroy

OR

./cleanup.sh

Docker

make destroy || make revolution

Development

Outline any requirements to setup a development environment if someone would like to contribute. You may also link to another file for this information.

# test pre commit manually
pre-commit run -a -v

f5-azure-saca's People

Contributors

0xhiteshpatel avatar billchurch avatar chen23 avatar f5rob avatar jahender avatar kyle-hoyer avatar mikej81 avatar mikeoleary avatar milazzom avatar vinnie357 avatar vtog avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

f5-azure-saca's Issues

Cannot deploy V1

When trying to deploy the following error is given.
{'code':'DeploymentFailed','message':'At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.','details':[{'code':'Conflict','message':'{\r\n \'status\': \'Failed\',\r\n \'error\': {\r\n \'code\': \'ResourceDeploymentFailure\',\r\n \'message\': \'The resource operation completed with terminal provisioning state 'Failed'.\',\r\n \'details\': [\r\n {\r\n \'code\': \'VMExtensionProvisioningError\',\r\n \'message\': \'VM has reported a failure when processing extension 'SetupAutomationScript'. Error message: \\\'Enable failed: processing file downloads failed: failed to download file[0]: failed to download file: unexpected status code: got=404 expected=200\\\'.\'\r\n }\r\n ]\r\n }\r\n}'}]}

Have the files been relocated and the does the script need to be updated?

Cannot reach jmpbox

The only way for me to reach into this environment after it's deployed is by adding a new public ip, and attaching it to -mgmt3 nic, followed by adding an inbound rule to the security group.
If that's not it, what is the intended traffic flow to get to either the bigip managemnt console or the jump box(es)?

DNS prefix used as part of Windows Jump Box Name

error

Attached is a picture of the error with a DNS prefix that is 5 characters long. it made the computer name 16 characters long and it generated an error. We should test the length and truncate.

These two lines in the firewall/bigip.tf use bigip_regkey, these will need updating too:

These two lines in the firewall/bigip.tf use bigip_regkey, these will need updating too:

bigip_regKey = var.licenses["license1"] != "" ? var.licenses["license1"] : ""

bigip_regKey = var.licenses["license2"] != "" ? var.licenses["license2"] : ""

Originally posted by @sewalshmsft in #124 (comment)

governmentCloudRegion needs updated description

"parameters": { "governmentCloudRegion": { "defaultValue": true, "metadata": { "description": "Type of authentication to use on the Virtual Machine, password based authentication or key based authentication." }, "type": "bool" }

This is part of the commercial deployment script. If it does belong here, what does it do?

files not found for Custom Script extension when choosing PUA

When I deploy SACAv2 with PUA, I get the following error:

    "code": "VMExtensionProvisioningError",
    "message": "VM has reported a failure when processing extension 'start'. Error message: \"Enable failed: processing file downloads failed: failed to download file[5]: failed to download file: unexpected status code: got=404 expected=200\"."

Looking further it looks like these fileUris may have broken links ready to update

"https://raw.githubusercontent.com/billchurch/f5-pua/master/bin/build_pua.sh",
"https://raw.githubusercontent.com/Mikej81/f5-azure-saca/SACAv2/3NIC%201Tier%20HA%20with%20PUA/dataSource.php"

Deployment failed - External links valid?

xxxx-bigip0/start failed to deploy. Attempting a commercial deployment to non-gov cloud space directly from GitHub page.

Azure Status message:

{
    "status": "Failed",
    "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The resource operation completed with terminal provisioning state 'Failed'.",
        "details": [
            {
                "code": "VMExtensionProvisioningError",
                "message": "VM has reported a failure when processing extension 'start'. Error message: \"Enable failed: processing file downloads failed: failed to download file[6]: failed to download file: unexpected status code: actual=404 expected=200\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "
            }
        ]
    }
}

bigip2 failed with a more verbose error:

{
    "status": "Failed",
    "error": {
        "code": "ResourceDeploymentFailure",
        "message": "The resource operation completed with terminal provisioning state 'Failed'.",
        "details": [
            {
                "code": "VMExtensionProvisioningError",
                "message": "VM has reported a failure when processing extension 'start'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nnfo:  Installing package at path: /var/config/rest/downloads/f5-appsvcs-3.16.0-6.noarch.rpm \n2020-07-20T22:35:27.190Z info:  Saving config. \n2020-07-20T22:35:31.540Z info:  Waiting for device to be active. \n2020-07-20T22:35:31.979Z info:  Device onboard complete. \n2020-07-20T22:35:32.383Z info:  Onboard finished. \n2020-07-20T22:35:32.570Z info:  /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/network.js called with /usr/bin/f5-rest-node /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/network.js --output /var/log/cloud/azure/network.log --wait-for ONBOARD_DONE --host 192.168.1.9 --port 443 -u svc_user --password-url file:///config/cloud/.passwd --password-encrypted --default-gw 192.168.3.1 --vlan name:external,nic:1.1 --vlan name:internal,nic:1.2 --self-ip name:self_2nic,address:192.168.2.5,vlan:external --self-ip name:self_3nic,address:192.168.3.5,vlan:internal --log-level info \n2020-07-20T22:35:32.579Z info:  Waiting for ONBOARD_DONE \n2020-07-20T22:35:33.581Z info:  Network setup starting. \n2020-07-20T22:35:33.583Z info:  Initializing BIG-IP. \n2020-07-20T22:35:33.598Z info:  This is a BIG-IP \n2020-07-20T22:35:35.572Z info:  Waiting for device to be ready. \n2020-07-20T22:35:36.078Z info:  Waiting for BIG-IP to be ready. \n2020-07-20T22:35:36.491Z info:  BIG-IP is ready. \n2020-07-20T22:35:36.492Z info:  Creating vlan external on interface 1.1   untagged \n2020-07-20T22:35:36.724Z info:  Creating vlan internal on interface 1.2   untagged \n2020-07-20T22:35:36.905Z info:  Creating self IP self_2nic with address 192.168.2.5/24 on vlan external allowing default \n2020-07-20T22:35:37.034Z info:  Creating self IP self_3nic with address 192.168.3.5/24 on vlan internal allowing default \n2020-07-20T22:35:37.084Z info:  Setting default gateway 192.168.3.1 \n2020-07-20T22:35:37.118Z info:  Saving config. \n2020-07-20T22:35:41.111Z info:  BIG-IP network setup complete. \n2020-07-20T22:35:41.112Z info:  Network setup finished. \n2020-07-20T22:35:42.628Z info:  /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/cluster.js called with /usr/bin/f5-rest-node /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/cluster.js --output /var/log/cloud/azure/cluster.log --log-level info --host 192.168.1.9 --port 443 -u svc_user --password-url file:///config/cloud/.passwd --password-encrypted --config-sync-ip 192.168.3.5 --join-group --device-group Sync --sync --remote-host 192.168.1.4 --remote-user svc_user --remote-password-url file:///config/cloud/.passwd \n2020-07-20T22:35:42.638Z info:  Cluster starting. \n2020-07-20T22:35:42.640Z info:  Initializing BIG-IP. \n2020-07-20T22:35:42.654Z info:  This is a BIG-IP \n2020-07-20T22:35:44.666Z info:  Waiting for device to be ready. \n2020-07-20T22:35:45.145Z info:  Waiting for BIG-IP to be ready. \n2020-07-20T22:35:45.562Z info:  BIG-IP is ready. \n2020-07-20T22:35:45.563Z info:  Setting config sync ip. \n2020-07-20T22:35:46.447Z info:  Joining group. \n2020-07-20T22:35:46.461Z info:  This is a BIG-IP \n2020-07-20T22:35:48.475Z info:  Waiting for device to be ready. \n2020-07-20T22:53:49.829Z info:  Device initialization failed tryUntil: max tries reached: tryUntil: max tries reached: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>401 Unauthorized</title>\n</head><body>\n<h1>Unauthorized</h1>\n<p>This server could not verify that you\nare authorized to access the document\nrequested.  Either you supplied the wrong\ncredentials (e.g., bad password), or your\nbrowser doesn't understand how to supply\nthe credentials required.</p>\n</body></html>\n \n2020-07-20T22:53:49.830Z info:  join cluster failed: tryUntil: max tries reached: tryUntil: max tries reached: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>401 Unauthorized</title>\n</head><body>\n<h1>Unauthorized</h1>\n<p>This server could not verify that you\nare authorized to access the document\nrequested.  Either you supplied the wrong\ncredentials (e.g., bad password), or your\nbrowser doesn't understand how to supply\nthe credentials required.</p>\n</body></html>\n \n\n[stderr]\n01020036:3: The requested system device (/Common/bigip1.eastus.cloudapp.azure.com) was not found.\n{\n  \"level\": \"error\",\n  \"message\": \"Cluster failed: tryUntil: max tries reached: tryUntil: max tries reached: <!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\">\\n<html><head>\\n<title>401 Unauthorized</title>\\n</head><body>\\n<h1>Unauthorized</h1>\\n<p>This server could not verify that you\\nare authorized to access the document\\nrequested.  Either you supplied the wrong\\ncredentials (e.g., bad password), or your\\nbrowser doesn't understand how to supply\\nthe credentials required.</p>\\n</body></html>\\n\",\n  \"label\": \"lib/bigIpOnboard.js\",\n  \"timestamp\": \"2020-07-20T22:53:49.832Z\"\n}\n2020-07-20T22:53:49.833Z error:  Cluster failed: tryUntil: max tries reached: tryUntil: max tries reached: <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>401 Unauthorized</title>\n</head><body>\n<h1>Unauthorized</h1>\n<p>This server could not verify that you\nare authorized to access the document\nrequested.  Either you supplied the wrong\ncredentials (e.g., bad password), or your\nbrowser doesn't understand how to supply\nthe credentials required.</p>\n</body></html>\n \n\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "
            }
        ]
    }
}

Are all the external links correct?

Licensing

Just a note: using a trial license didn't work for me. The primary symptom I experienced was the virtual machine extension creation never completing. When I swapped to a real registration key, everything worked fine.

Add troubleshooting info

If the deployment fails for some reason, it would help to know where to look to discover what the issue was (logfile locations, .json declarations, etc) A high-level overview of steps that this deployment performs could also help pinpoint a potential issue more quickly.

Template Validation Error after recent change

"additionalInfo": [
{
"type": "TemplateViolation",
"info": {
"lineNumber": 323,
"linePosition": 128,
"path": "variables.cloudPath"
}

Deployment template validation failed: 'The template variable 'cloudPath' is not valid: The provided arguments for template language function 'if' is not valid: all arguments should be of type 'boolean'. Please see https://aka.ms/arm-template-expressions#if for usage details.. Please see https://aka.ms/arm-template-expressions for usage details.'. (Code: InvalidTemplate)

I get this error every time I try to deploy the commercial, PAYG, 1 Tier version. This is the version we are using for Agility in 8 days. We deployed 60 instances of this 2 weeks ago with no issues. I think this is due to adding the Commercial or Govt option in the Template.

Update Default Value for Version to Patched Version

In variables.tf update version to "14.1.400000"

variable bigip_version {
type = string
description = "REQUIRED: BIG-IP Version, 14.1.2 for Compliance. Options: 12.1.502000, 13.1.304000, 14.1.206000, 15.0.104000, latest. Note: verify available versions before using as images can change."
default = "14.1.206000"
}

Create variables for Azure Service Fabric cluster endpoints

The ARM templates have values that are hard coded for Azure Government. The concatenated values in the failoverCmdArray variable and the commandToExecute parameter use the hard coded value of .cloudapp.usgovcloudapi.net which is the endpoint for Azure Government.

Suggest making a parameter for this that can take an input and has a default value.

Azure Government = *.cloudapp.usgovcloudapi.net
Azure Commercial = *.cloudapp.azure.com

Switching to variables for this value will also make the code portable to other environments
https://docs.microsoft.com/en-us/azure/azure-government/documentation-government-developer-guide

Commercial Cloud FQDN is hard-coded in the AzureDeploy.json

I noticed that the commercial cloud is hard coded in mutliple places in the template which produces the wrong VM name when deploying the template to MAG. I deployed the 1 Tier file but the 3 Tier file also looks wrong.

Correct URL Name should be ‘cloudapp.usgovcloudapi.net’ instead of ‘cloudapp.azure.com’. Probably need to have a GovCloud check or parameter to adjust the name for commercial or govcloud depending on deployment scenario.

Files:
https://github.com/f5devcentral/f5-azure-saca/blob/master/SACAv2/3NIC_1Tier_HA/azureDeploy.json
https://github.com/f5devcentral/f5-azure-saca/blob/master/SACAv2/3NIC_3Tier_HA/azureDeploy.json

Conflict error after deployment

Getting this error when deploying:
{ "id": "/subscriptions/a0b713be-1237-4769-8647-f0f281a998c9/resourceGroups/AW_ResGroup/providers/Microsoft.Resources/deployments/Microsoft.Template/operations/1A57531F9F9E51E9", "operationId": "1A57531F9F9E51E9", "properties": { "provisioningOperation": "Create", ### "provisioningState": "Failed", "timestamp": "2019-09-27T17:14:29.7262626Z", "duration": "PT50.775008S", "trackingId": "6cf492bd-7f97-487a-be25-134306fba701", "statusCode": "Conflict", "statusMessage": { "status": "Failed", "error": { "code": "ResourceDeploymentFailure", "message": "The resource operation completed with terminal provisioning state 'Failed'.", "details": [ { "code": "VMExtensionProvisioningError", "message": "VM has reported a failure when processing extension 'start'. Error message: "Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nabout to execute\nchecking mcpd\nmcpd ready\nloading verifyHash script\nLoading configuration...\n /config/verifyHash\nloaded verifyHash\nverifying /config/cloud/f5-cloud-libs.tar.gz\n/config/cloud/f5-cloud-libs.tar.gz is not valid\n\n[stderr]\nscript did not successfully complete, status:1\n/bin/sh: line 36: /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/util.sh: No such file or directory\n"." } ] } }, "targetResource": { "id": "/subscriptions/a0b713be-1237-4769-8647-f0f281a998c9/resourceGroups/AW_ResGroup/providers/Microsoft.Compute/virtualMachines/awdns-awf5vm1/extensions/start", "resourceType": "Microsoft.Compute/virtualMachines/extensions", "resourceName": "awdns-awf5vm1/start" } }}

Doesn't support multiple CIDR in Virtual Network

I have a customer that would like to use the three tier architecture. The external IPs will be public routable IP address space while everything after the first tier of BIG IPs will be private IP address space.

If I change var.cidr to a tuple ["10.0.1.0/24", "172.16.0.0/24"], the F5 configuration code in three_tier\firewall\bigip.tf lines 402/409 and three_tier\waf\bigip.tf lines 363/390 can't handle a tuple value and require a string.

I'm not fully versed on what those configuration items are doing.

AzureRM provider deprecating fields, update HCL to reflect.

Warning: "resource_group_name": [DEPRECATED] This field is no longer used and will be removed in the next major version of the Azure Provider

  on azure.tf line 42, in resource "azurerm_lb_backend_address_pool" "backend_pool":
  42: resource azurerm_lb_backend_address_pool backend_pool {

(and 7 more similar warnings elsewhere)

F5 VM deletes itself after deployment

After a successful deployment of the SACAv2 Azure Government Tier 1, the first f5 VM deletes itself automatically. I have tried deploying this 4x with the same result each time. Also, I cannot access the F5 frontend at all, it just times out. The management network is not the value I set it to be aswell. Any help would be great, thank you.

High percentage failure rate when doing bigiq deployment

When performing a deployment into Azure Commercial using the bigiq template, a high percentage of the time the deployment fails because the Custom Script Extension for the first, or second, or both F5s fails to complete.

The BIG-IQ instance being used to acquire licenses is hosted in Azure Government. Traffic coming from Commercial destined for Government must traverse the Internet.

The Custom Script Extension used for the bigiq deployment seem to be identical to the one used for the byol deployment. A majority of the files used by Custom Script Extension are hosted in Github. Should investigate hosting these files in Azure somewhere to reduce the latency caused by having to pull the files in from over the Internet. Given that we have a high success rate when performing a byol deployment, appears unlikely that the location of the files used by the Custom Script Extension is causing the bigiq failures.

Not sure where .tmpl files do or where they are hosted because they are sitting behind a CDN

included auto.admin.tfvars.example file has syntax errors

Variable definition for host host nics generates warning during terraform plan:

it's currently listed as:
f5vm01mgmt = "10.90.0.4"
f5vm01ext = "10.90.1.4"
f5vm01ext_sec = "10.90.1.11"
f5vm01int = "10.90.2.4"

02

f5vm02mgmt = "10.90.0.5"
f5vm02ext = "10.90.1.5"
f5vm02ext_sec = "10.90.1.12"
f5vm02int = "10.90.2.5"

three tier

03

f5vm03mgmt = "10.90.0.6"
f5vm03ext = "10.90.6.4"
f5vm03ext_sec = "10.90.6.11"
f5vm03int = "10.90.7.4"

04

f5vm04mgmt = "10.90.0.7"
f5vm04ext = "10.90.6.5"
f5vm04ext_sec = "10.90.6.12"
f5vm04int = "10.90.7.5"

Should probably be listed as:
f5_t1_ext = {
f5vm01ext = "10.90.1.4"
f5vm01ext_sec = "10.90.1.11"
f5vm02ext = "10.90.1.5"
f5vm02ext_sec = "10.90.1.12"
}

f5_t1_int = {
f5vm01int = "10.90.2.4"
f5vm01int_sec = "10.90.2.11"
f5vm02int = "10.90.2.5"
f5vm02int_sec = "10.90.2.12"
}

f5_t3_ext = {
f5vm03ext = "10.90.6.4"
f5vm03ext_sec = "10.90.6.11"
f5vm04ext = "10.90.6.5"
f5vm04ext_sec = "10.90.6.12"
}

f5_t3_int = {
f5vm03int = "10.90.7.4"
f5vm03int_sec = "10.90.7.11"
f5vm04int = "10.90.7.5"
f5vm04int_sec = "10.90.7.12"
}

i need to test this and ensure this works.

Virtual Server illegally shares destination address

I deployed the 3-NIC 1 Tier script in my personal Azure (commercial) space.
After deployment, I created a Shared Address List with both backend pool ips from the ext-alb loadBalancerBackEnd configuration. (192.168.2.10 and 192.168.2.11)

I created the vs, selected the Shared Address List, and set Service Port (80).
When clicking create, it fails like this:

01b90011:3: Virtual Server /Common/alb-test's Traffic Matching Criteria /Common/alb-test_VS_TMC_OBJ illegally shares destination address, source address, service port, and ip-protocol with Virtual Server /mgmt/mgmt_http/mgmt_http destination address, source address, service port.

What am I missing here?

included auto.admin.tfvars.example file has syntax errors

Subnet info is missing application subnet and generates errors during terraform plan. Should be replaced with below:

subnets = {
"management" = "10.90.0.0/24" #f5 management mgmtip
"external" = "10.90.1.0/24" #untrusted-virutal-network selip
"internal" = "10.90.2.0/24" #trusted-egress-virtual-network
"vdms" = "10.90.3.0/24" #management interfaces? or management devices that can access management interfaces?
"inspect_ext" = "10.90.4.0/24"
"inspect_int" = "10.90.5.0/24"
"waf_ext" = "10.90.6.0/24"
"waf_int" = "10.90.7.0/24"
"application" = "10.90.10.0/24"
}

Deployment in Commercial and Gov Fails

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n "status": "Failed",\r\n "error": {\r\n "code": "ResourceDeploymentFailure",\r\n "message": "The resource operation completed with terminal provisioning state 'Failed'.",\r\n "details": [\r\n {\r\n "code": "VMExtensionProvisioningError",\r\n "message": "VM has reported a failure when processing extension 'start'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nabout to execute\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nloading verifyHash script\ncannot validate signature of /config/verifyHash\n\n[stderr]\nData Input Error: The path \"/config/verifyHash\" is invalid. Check the valid paths with tmsh list sys global-settings file-whitelist-path-prefix file-blacklist-path-prefix file-blacklist-read-only-path-prefix\n/bin/sh: line 36: /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/util.sh: No such file or directory\n\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "\r\n }\r\n ]\r\n }\r\n}"},{"code":"Conflict","message":"{\r\n "status": "Failed",\r\n "error": {\r\n "code": "ResourceDeploymentFailure",\r\n "message": "The resource operation completed with terminal provisioning state 'Failed'.",\r\n "details": [\r\n {\r\n "code": "VMExtensionProvisioningError",\r\n "message": "VM has reported a failure when processing extension 'start'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=1\n[stdout]\nabout to execute\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nchecking mcpd\nmcpd not ready yet\nloading verifyHash script\ncannot validate signature of /config/verifyHash\n\n[stderr]\nData Input Error: The path \"/config/verifyHash\" is invalid. Check the valid paths with tmsh list sys global-settings file-whitelist-path-prefix file-blacklist-path-prefix file-blacklist-read-only-path-prefix\n/bin/sh: line 36: /config/cloud/azure/node_modules/@f5devcentral/f5-cloud-libs/scripts/util.sh: No such file or directory\n\"\r\n\r\nMore information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot "\r\n }\r\n ]\r\n }\r\n}"}]}

Declarative Onboarding isn't working with BYOL

I'm running basically the as-provided variables.tf, only changing the username/password and setting it to single tier deployment.

It looks like there's a bad character when outputting the declarative onboarding script when it executes as part of the Azure custom script extension at startup.

I've attached the startup log and the output JSONs in a zip to help you troubleshoot:
startup-script.zip

Add notes/clarity when using BYOL

I noticed the product and image name variables both need to be adjusted in the variable.tf file or there will be a mismatch and the deployment will fail. This isn't highlighted and assumes the user has the knowledge to make the change. I would recommend adding a note somewhere in the README stating that when changing the defaults to BYOL that those variables are updated accordingly.

Getting an error about missing variable

Should this line use bigip_regkey instead of license for the variable declaration that is getting passed to the DO?

license = var.licenses["license3"] != "" ? var.licenses["license3"] : ""

The DO requires ${bigip_regKey} as an input for the variable, not ${license}.

https://github.com/Mikej81/f5-bigip-hardening-DO/blob/5f3563afabf730e3851a83734fdf18cb689b9c1c/dist/terraform/latest/byol_cluster_waf_tier.json#L12

Make subnets into parameters and integrate AS3 file update with deployment

Most customers would need to use an assigned IP range for the different subnets used in the SACA architecture. This IP range will likely differ from what is in the ARM templates and AS3 files.

Recommend creating a parameters file with the different subnets parameters an removing the default values from the parameter declarations.

Incorporating a dynamic updating of the AS3 file to be used with the correct IP addresses based on the subnets the user entered would an additional manual step in preparing to files for the deployment

Known Issue: IP Reputation Database not Loaded

If SACA is deployed without IP Reputation, and the AS3 with the Reputation Drop iRule in place, you will receive the following error and be unable to connect through the management policies.

err tmm2[41374]: 01220001:3: TCL error: /Common/Shared/ip_reputation_drop <CLIENT_ACCEPTED> - IP Reputation database (/var/IpRep/F5IpRep.dat) not loaded (line 1) invoked from within "IP::reputation [IP::client_addr]"

Verify your IP-Reputation status by following instructions here: https://support.f5.com/csp/article/K13875

If you do not have IP reputation and would like to add it, reach out to your F5 Account Team.

If you would like to proceed, you can use another AS3 declaration without IP reputation baked in.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.