Giter Club home page Giter Club logo

terraform-spotinst-ocean-aws-k8s-vng's Introduction

Spot Ocean k8s Virtual Node Group Terraform Module

Spotinst Terraform Module to integrate existing k8s node groups with Ocean launchspec/Virtual Node group (VNG)

Usage

Note: This module will automatically import any tags defined in the AWS provider default_tags

provider "spotinst" {
  token   = "redacted"
  account = "redacted"
}

module "ocean-aws-k8s" {
  ...
}

## Create Ocean Virtual Node Group (launchspec) ##
module "ocean-aws-k8s-vng_stateless" {
  source = "spotinst/ocean-aws-k8s-vng/spotinst"

  name = "stateless" # Name of VNG in Ocean
  ocean_id = module.ocean-aws-k8s.ocean_id
  
  #ami_id = "" # Can change the AMI
  labels = [{key="type",value="stateless"}]
}

## Create additional Ocean Virtual Node Group (launchspec) ##
module "ocean-aws-k8s-vng_gpu" {
  source = "spotinst/ocean-aws-k8s-vng/spotinst"

  name = "gpu"  # Name of VNG in Ocean
  ocean_id = module.ocean-aws-k8s.ocean_id
  
  labels = [{key="type",value="gpu"}]
  taints = [{key="type",value="gpu",effect="NoSchedule"}]
  
  #instance_types = ["g4dn.xlarge","g4dn.2xlarge"] # Limit VNG to specific instance types
  spot_percentage = 50 # Change the spot %
}

## Create Ocean Virtual Node Group (launchSpec) with instance_type_filters ##
module "ocean-aws-k8s-vng" {
  source  = "spotinst/ocean-aws-k8s-vng/spotinst"
  ocean_id = module.ocean-aws-k8s.ocean_id
  name = "test-vng"
  instance_types_filters_enable                   =   true
  instance_types_filters_categories               =   ["Accelerated_computing", "Compute_optimized"]
  instance_types_filters_disk_types               =   ["NVMe", "EBS"]
  instance_types_filters_exclude_families         =   ["t2","R4*"]
  instance_types_filters_exclude_metal            =   true
  instance_types_filters_hypervisor               =   ["nitro"]
  instance_types_filters_include_families         =   ["c5*", "g5"]
  instance_types_filters_is_ena_supported         =   true
  instance_types_filters_max_gpu                  =   4
  instance_types_filters_min_gpu                  =   0
  instance_types_filters_max_memory_gib           =   16
  instance_types_filters_max_network_performance  =   20
  instance_types_filters_max_vcpu                 =   16
  instance_types_filters_min_enis                 =   2
  instance_types_filters_min_memory_gib           =   8
  instance_types_filters_min_network_performance  =   2
  instance_types_filters_min_vcpu                 =   2
  instance_types_filters_root_device_types        =   ["ebs"]
  instance_types_filters_virtualization_types     =   ["hvm"]
  images                                          =   [{image_id="ami-123456"},{image_id="ami-67890"}]
  block_device_mappings                           =   [{
    device_name                                   =   "/dev/xvda"
    encrypted                                     =   false
    volume_type                                   =   "gp3"
  }]
  dynamic_volume_size                             =   {
    base_size                                     =   50
    resource                                      =   "CPU"
    size_per_resource_unit                        =   20
  }
  ephemeral_storage_device_name                   = "/dev/xvdb"

}

~> You must configure the spotinst_ocean_aws resource. Ensure spotinst_ocean_aws resource (defined in ocean-aws-k8s module) is defined before this module as the ocean_id is needed.

Providers

Name Version
spotinst/spotinst >= 1.95

Modules

  • ocean-aws-k8s - Creates Ocean Cluster Doc
  • ocean-controller - Create and installs spot ocean controller pod Doc
  • ocean-aws-k8s-vng - (Optional) Add custom virtual node groups Doc

Documentation

If you're new to Spot and want to get started, please checkout our Getting Started guide, available on the Spot Documentation website.

Getting Help

We use GitHub issues for tracking bugs and feature requests. Please use these community resources for getting help:

Community

Contributing

Please see the contribution guidelines.

Resources

Name Type
ocean-aws-k8s-vng resource

Inputs

Name Description Type Default Required
ocean_id The Ocean cluster identifier. Required for Launch Spec creation. string null yes
name Set launch specification name. string null no
instance_types_filters_enable 'instance_types_filters_enable' to be set to true to have instance_types_filters configured in the virtual node group. boolean null no
instance_types_filters The instance types that match with all filters compose the Virtual Node Group's instanceTypes parameter. The architectures that come from the Virtual Node Group's images will be taken into account when using this parameter. Cannot be configured together with Virtual Node Group's instanceTypes and with the Cluster's whitelist/blacklist/filters.
object({
categories = list(string)
disk_types = list(string)
exclude_families = list(string)
exclude_metal = bool
hypervisor = list(string)
include_families = list(string)
is_ena_supported = bool
max_gpu = number
min_gpu = number
max_memory_gib = number
max_network_performance = number
max_vcpu = number
min_enis = number
min_memory_gib = number
min_network_performance = number
min_vcpu = number
root_device_types = list(string)
virtualization_types = list(string)
})
null no
images You can configure VNG with either the imageId or images objects, but not both simultaneously.
For each architecture type (amd64, arm64) only one AMI is allowed.
Valid values: null, or an array with at least one element.
list(object({
image_id = string
}))
null no
block_device_mappings block_device_mapping object
list(object({
device_name = string
delete_on_termination = bool
encrypted = bool
kms_key_id = string
snapshot_id = string
volume_type = string
iops = number
volume_size = number
throughput = number
}))
[] no
dynamic_volume_size dynamic_volume_size object
object({
base_size = number
size_per_resource_unit = number
resource = string
})
null no
ephemeral_storage_device_name ephemeral_storage_device_name string null no

Outputs

Name Description
virtual_node_group_id The virtual node group ID

terraform-spotinst-ocean-aws-k8s-vng's People

Contributors

anuragsharma-123 avatar chandra1-n avatar gidonshn avatar lironrad avatar mleklund avatar ohad360 avatar sharadkesarwani avatar snir102002 avatar stevenfeltner avatar z4ck404 avatar zivmessing avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

terraform-spotinst-ocean-aws-k8s-vng's Issues

Adding tags instead overwrite current

What would you like to be added?
Supporting for adding additional tags and not removing them in the calling to the module with tags module input parameter.
Why is this needed?
For prevent from our terraform code to prevevnt important tags like: kubernetes.io/cluster/<cluster_name>: owned from spotinst vng

block_device_mappings defaults

What happened:

I am not sure this really a bug but I am reporting it any ways:
All attributes in this var are apparently required, terraform fails if they are not set (to null when not needed). In my case I just need to make sure the root volumes are encrypted with the default ebs kms key. I couldn't just do :

  block_device_mappings = [{
    encrypted             = true
  }]

and leave everything else to default !
I had to set all :

  block_device_mappings = [{
    volume_size           = local.vngs[count.index].disk_size_gb
    device_name           = "/dev/xvda"
    delete_on_termination = true
    kms_key_id            = data.aws_kms_key.ebs.id
    encrypted             = true
    snapshot_id           = null
    iops                  = null
    volume_type           = "gp2"
    throughput            = null
    no_device             = null
  }]

What you expected to happen:

  block_device_mappings = [{
    encrypted             = true
  }]
  

And let the spotinst provider handle the defaults.

Version 0.5.0 fails when setting instance_types

What happened:
I am getting an error when using the 0.5.0 version and setting the instance_types.

What you expected to happen:
The vng created with no errors

How to reproduce it (as minimally and precisely as possible):
Here is a custom module that uses the ocean-aws-k8s-vng module:

module "ocean-vng" {
  source = "spotinst/ocean-aws-k8s-vng/spotinst"
  version = "0.5.0"

  ocean_id = var.ocean_id
  name     = var.name

  instance_types = var.instance_types

  image_id             = var.image_id
  security_groups      = var.security_groups
  subnet_ids           = var.subnet_ids
  iam_instance_profile = var.iam_instance_profile
  root_volume_size     = var.root_volume_size
  max_instance_count = var.max_instance_count
  min_instance_count = var.min_instance_count
  spot_percentage    = 100
  user_data          = var.user_data
}

As its possible to see, i didnt set any filters, only the instance_types = var.instance_types
But, i am still getting this error:

0 to add, 1 to change, 0 to destroy.
module.teste-vng.module.teste-vng.module.ocean-vng.spotinst_ocean_aws_launch_spec.nodegroup: Modifying... [id=ols-redacted]

Error: [ERROR] Failed to update launchSpec [ols-redacted]: PUT https://api.spotinst.io/ocean/aws/k8s/launchSpec/redacted?accountId=redacted: 400 CANT_UPDATE_OCEAN_LAUNCH_SPEC: Filters and instance types should not be set simultaneously.
  with module.teste-vng.module.teste-vng.module.ocean-vng.spotinst_ocean_aws_launch_spec.nodegroup,
  on .terraform/modules/teste-vng.teste-vng.ocean-vng/main.tf line 2, in resource "spotinst_ocean_aws_launch_spec" "nodegroup":
   2: resource "spotinst_ocean_aws_launch_spec" "nodegroup"

Anything else we need to know:
It seems that with terraform, it is sending the filters even though they should be null by default:

Terraform will perform the following actions:

resource "spotinst_ocean_aws_launch_spec" "nodegroup" {
        id                   = "ols-redacted"
        name                 = (sensitive value)

      instance_types_filters {
          exclude_metal           = false
          max_gpu                 = -1
          max_memory_gib          = -1
          max_network_performance = -1
          max_vcpu                = -1
          min_enis                = -1
          min_gpu                 = -1
          min_memory_gib          = -1
          min_network_performance = -1
          min_vcpu                = -1
        }
    }

The error does not happen in the previous version: 0.4.3

Environment:

  • Terraform version: 1.6.1
  • Module version: 0.5.0
  • OS: Linux
  • Kernel: Alpine 3.18.4

DO NOT USE v0.4.1 - SCALING ISSUES

So today i have had a nice conversation with Ben from the support team of spotinst.

We've stumbled upon an issue that likely only few if anyone have had before, otherwise the fix would've been found faster.

When using v0.4.1 (which we did), you can get into scaling trouble with messages like Can't Spin Instances: Code: ValidationError, Message: can't spin spot due to duplicate tags error.

v0.4.1 was never published as of today and we were only using it because it enabled us to name our ec2 nodes like we wanted to by specifying the name tag.

This is just a reminder and a "helping hand" issue and not necessarily the fix needs to be done here.

The root cause is the label spotinst:ocean:launchspec:name, which was introduced in v0.4.1.

Simply deleting the tag manually from the vng seems to fix the problems, but obviously that does not work when working with IaC.

Adding lifecycle module support

What would you like to be added?
Adding support for ignore_changes
Why is this needed?
For prevent from our terraform code to prevevnt important tags like: kubernetes.io/cluster/<cluster_name>: owned from spotinst vng

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.