multycloud / multy Goto Github PK

AWS Lambda & Azure Functions

Tasks:

• AWS
• Azure
• Tests
• Docs

Read attributes from internal terraform state and convert to object as per https://github.com/multycloud/multy/blob/main/resources/types/virtual_network.go#L65

Allow Engine to locally run without third-party dependencies. This makes it easy for testing and getting started.

SQL - store Locks and API keys in-memory
S3 - use local file system

• AWS: https://aws.amazon.com/ec2/instance-types/
• Azure: https://docs.microsoft.com/en-us/azure/virtual-machines/sizes

Unify the following types:

• General purpose
• Compute optimized
• Memory optimized
• Storage optimized
• GPU optimized

Not all types / sizes are available in all regions, so ideally we'd catch that and return an unified error message.

Similar to terraform, we should have a resource that just prints a variable

• Make sure e2e tests are built on make build
• Add e2e for resources that aren't currently being tested

• Add engine to proto definition in https://github.com/multycloud/multy/blob/main/api/proto/resourcespb/database.proto
• Translate it to AWS and Azure in https://github.com/multycloud/multy/blob/main/resources/types/database.go
• Add tests to tests/_configs/database

We'll need to go around terraform for this one

AWS: https://aws.amazon.com/autoscaling/
Azure: https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-autoscale-overview

Something like:

message VirtualNetworkArgs {
  ...
 
 message AwsOverrides {
    string vm_size = 1;
    string location = 2;
  }
 message AzureOverrides {
    string vm_size = 1;
    string location = 2;
  }
  AwsOverrides aws_overrides = 1;
  AzureOverrides azure_overrides = 2;
}

Which will then be used in the translation logic to override cloud-specific values.

Cloud providers allow you to create a snapshot from a VM into a "managed image". You can't export them in general, so we can't support cross cloud.

AWS: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ami_from_instance
Azure: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/image (vm needs to be "generalized")

Support for cross-cloud object storage is dependent on:

• Getting access to private resources from a different cloud: in azure this is done by a shared access signature; in aws there isn't any terraform support for it, but there is the concept of pre-signed urls (but not sure how to get one)
• Add an interface or use CloudDependentResource to represent a resource in a given cloud and add mHcl support for it
• Research how urls are passed around to understand if there is any way they are publicly accessible

• If db is azure, it doesn't work
• Aws runs the custom data script in root mode but azure doesn't (maybe we just need to make sure both clouds use the same vm image?)

For each resource, return the ids of the deployed resources.

For example, add fields to virtual_network.proto:

message VirtualNetworkResource {
  ...

  message Aws {
    string vpc_id = 1;
    string internet_gateway_id = 2;
    string default_security_group_id = 3;
  }

  message Azure {
    string virtual_network_id = 1;
    string route_table_id = 2;
  }
}

which will then be shown in the provider.

AWS: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.Encryption.html
Azure: https://docs.microsoft.com/en-us/azure/azure-sql/database/transparent-data-encryption-tde-overview?view=azuresql&tabs=azure-portal

• Add port to proto definition in https://github.com/multycloud/multy/blob/main/api/proto/resourcespb/database.proto
• Translate it to AWS and Azure in https://github.com/multycloud/multy/blob/main/resources/types/database.go
• Add tests to tests/_configs/database

Same-cloud load balancing is offered by each cloud:
AWS: https://aws.amazon.com/elasticloadbalancing/
Azure: https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview

Both AWS and Azure support tagging most resources.

• AWS: https://docs.aws.amazon.com/general/latest/gr/aws_tagging.html
• Azure: https://docs.microsoft.com/en-us/azure/azure-resource-manager/management/tag-resources?tabs=json

In some cases, we can't handle hardcoded ids because multy resources depend on multiple properties of other resources.
For example, subnets depend on virtual_network.id, virtual_network.name and on the virtual network's associated route_table_id.

Currently, these ids can't be hardcoded and config must look like

multy virtual_network vn {
...
}
multy subnet example {
  virtual_network = vn
}

But we want to support hardcoded ids in these cases, which would look something like (schema details TBD)

multy subnet example {
  virtual_network {
    virtual_network_id = "123"
    virtual_network_name = "456"
    route_table_id = "789"
  }
}

With potentially some of the paramters being optional and the behavior would change depend on that. f.e. if route_table_id is not set, we don't create a default association in azure.

Add CONTRIBUTING.MD with

• Repository overview
• Testing overview
• Running multy locally
• How to contribute (submitting PR)

AWS: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html
Azure: https://docs.microsoft.com/en-us/azure/azure-sql/database/automated-backups-overview?view=azuresql&tabs=single-database

AWS S3 & Azure Storage Account

Tasks:

• AWS
• Azure
• Tests
• Docs

The azure subnet is not a zonal service. Resources are deployed into specific AZs unlike AWS.

Possible solutions:

• multy handles subnet/az resolution in the backend
• makes resources subnet agnostic

Aws uses a token based system (tf has eks_cluster_auth data source for this) while azure uses a username+password and an asymmetric key authentication system

Add unit tests for resource_metadata:

• importing config - make sure everything is added and GetAffectedResources returns what's expected
• exporting config
• creating resource - make sure that resource groups are created
• delete resource - make sure resource groups are deleted by default
• create child resources should use the same resource group id when possible

Add unit tests for grpc:

• make sure config is read and written
• make sure init and apply are always called

Add unit tests for deploy:

• make sure exec.command is called with the appropriate parameters

Create ACL parameter for object storage objects

Allow private and public_read

Allow SSH public key authentication

aws_instance.key_name
azurerm_linux_virtual_machine.admin_ssh_key

Validates if the config is valid without deploying anything / doing any web requests

Research how we want to model data providers and implement them.

Open questions:

• How to map multiple terraform resources to a single multy resource?
• How to handle cases where resources were not created by us, so the resource ids and other fields are not what we expect?

• Red Had Enterprise Linux
• Open SUSE

Allow for hardcoded ID values in place of resource references. Likely dependent on #2 because multy resource can resolve into multiple cloud resources

multy "subnet" "subnet" {
  virtual_network_id = example_vn.id
  virtual_network_id = "123"
}

cloud() function will allow dynamic configuration depending on which provider is being used. example below will deploy an aws ec2 instance with t2.micro and azure vm with Standard_F4s_v2

variable vm_size {
  default = cloud({
    aws   = "t2.micro"
    azure = "Standard_F4s_v2"
  })
}

multy virtual_machine vm {
  size = var.vm_size
  ...
}

Map machine images across AWS/Azure based on OS & Size

• Azure is not accessible publicly
• Aws is fine with username as the username, but azure expects username@host. Maybe they are using different database engines?

More details in test/e2e/database/database_test.go

AWS RDS & Azure Database for MySQL

Tasks:

• AWS
• Azure
• Tests
• Docs

Add flags to CLI
version - display cli version
check - checks if config is valid

AWS AppConfig & Azure App Configuration

Tasks:

• AWS
• Azure
• Tests
• Docs

Multy resource data sources

Azure: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster
Aws: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster

Terraform has count and for_each to handle cases where a user wants to easily replicate resources.

We want to do something similar in multy but since there's not a 1:1 mapping between deployed resources and multy resources, we have to decide if we replicate every single resource or if we can reuse some common ones.

AWS Secrets Manager & Azure Key Vault

Tasks:

• AWS
• Azure
• Tests
• Docs

Add CLI tests alongside go_test action

Tests:

• files in nested directories
• cli flags
• bad inputs
• install script

Map AZ and Regions between AWS & Azure. Use lowest common value and expose them as multy variable