• Filecoin IaC
  • Summary
  • Installation
  • Structure
  • Architecture
  • API
    • API Endpoints
    • API Stages
    • Ingress rules
      • Development
      • Production
  • Backup policies
  • Support tools
  • Monitoring
    • Uptibe Robot
    • Prometheus Stack
    • OpenSearch

This repository provides Infrastructure as Code (IaC) resources to configure and deploy the Glif.io infrastructure.

To deploy the infrastructure, do the following:

1. NB! One-time operation during the first installation. In the init_configuration folder:
   1. Run terraform init
   2. Run terraform apply -var-file=tfvars/filecoin-glif-dev-apn1.tfvars. That will create the backed S3 and DynamoDB tables for dev environment.
   3. Run terraform apply -var-file=tfvars/filecoin-glif-mainnet-apn1.tfvars. That will create the backed S3 and DynamoDB tables for mainnet environment.
2. In the aws folder:
   1. Run terraform init -backend-config=../filecoin-glif-dev-apn1.hcl to initialize the module with the backend configured in the previous step.
   2. Run terraform workspace select filecoin-glif-dev-apn1 to select the dev workspace.
   3. Plan and apply the configuration with -var-file=tfvars/filecoin-glif-dev-apn1.tfvars.
   4. Repeat those steps for filecoin-glif-mainnet-apn1 workspace.
3. In the k8s folder repeat the same actions as in step 2, but the workspace names would be filecoin-dev-apn1-glif-eks and filecoin-mainnet-apn1-glif-eks respectively.
4. In the user_management folder initialize the module with -backend-config=../filecoin-glif-dev-apn1.hcl only and select default workspace. Then you can apply the config with -var-file=tfvars/filecoin-users.tfvars.

To help you better navigate through the repository, here's a quck guide.

• aws - contains the most low-level parts of the infrastructure, including networking services, SSL certificates, the kubernetes cluster configuration and also:
  • budget.tf - AWS Budgets configurations
  • ecr.tf - container registries for the external plugins, including Fluentbit, CID Checker and external snaphotter
  • nodegroup_dev.tf - node groups configuration for the dev environment
  • nodegroup_mainnet.tf - node groups configuration for the mainnet environment
  • opensearch.tf - AWS OpenSearch configuration
• init_configuration - contains resources to spin up Terraform backend with an S3 bucket and a DynamoDB instance.
• k8s - contains the most important parts of the infrastructure application-wise, including:
  • AWS API Gateway endpoints and stages configuration
  • Cron jobs for CID Checker and market deals
  • ebs_csi_driver.tf - contains resources to setup Amazon EBS CSI driver
  • elb_controller.tf - contains resources to setup AWS Load Balancer Controller
  • external-snapshotter.tf - contains resources to spin up an external snapshotter (dev, prod)
  • fluent-bit.tf - contains resources to spin up Fluentbit
  • konghq.tf - contains resources to spin up Kubernetes Ingress Controller (dev, prod) monitoring.tf - contains resources to spin up Prometheus Stack
• modules - contains reusable parts of the infrastructure, including:
  • codebuild and codebuild_multirepositories - AWS CodeBuild configurations to leverage CI/CD capabilites
  • eks_nodegroup - contains resources to spin up a nodegroup whether it's an on-demand or a spot instance
• user_management - contains resources to spin up IAM user, groups and manage access to the EKS cluster

The most high-level view on the infrastructure is shown on the image below.

More detailed view on how the infrastructure works is shown on the image below.

Requests to Internal Load Balancer are redirectet either to dev-internal.dev.node.glif.io or mainnet-internal.node.glif.io depending on the environment. What kind of services are listening to such requests is described in ingress rules, refer to ingress rules.

Here's a short list of domain names that are pointing to API Gateways.

The current policies for the snapshots are listed here: dev, prod.

Snapshotting targets are as follows:

• calibration-archive - create a snapshot once a week, keep the last one only.
• space00 - create a snapshot once a day, keep the last 7 snapshots.

Among the others, the repository uses:

• Fluentbit - to collect logs. Currently unused.
• Amazon EBS CSI Driver - to allow EKS cluster manage EBS volumes for persistent storage.
• Kubernetes CSI External Snapshotter - to manage snapshots of persistent volumes. More on that here.

Uptime Robot is used to monitor web-accessible endpoints.

Summary on the uptime is available here:

• Production: https://status.node.glif.io
• Development: https://status.dev.node.glif.io

Prometheus settings are stored here: values.yaml. Grafana custom dashboards are stored in openworklabbot organization on Grafana and retrieved by id. To update a custom dashboard update revision number of the dashboard in values.yaml.

There are a total of 5 custom dashboards:

1. lotus/Kubernetes Persistent Volumes:
   • Volume Space Usage in percentage
   • Volume stats for vol-ipfs-space00-lotus-0
   • Volume stats for vol-lotus-io2-space00-lotus-0
   • Volume stats for vol-lotus-io2-space07-lotus-0
2. lotus/Lotus API Endpoints – contains information on methods usage, timeouts, etc.
3. lotus/Lotus NGINX Ingress – deprecated.
4. lotus/Lotus Node Health – contains information on node health, including:
   • node versions
   • peer numbers per node
   • network height per node
   • duration of validation per node
5. lotus/Lotus PubSub – counts PubSub Message pool RPC requests per node

After the installation, there's a few manual operations that need to be done in order for OpenSearch to collect the logs.

1. Go to Settings > Roles and create a role with crud cluster permissions and crud and create_indexes index permissions.
2. Map the role a user, as a username use role ARN of fluenbit.
3. Check if indexes are created in Stack Management > Indeces.