openraven / magpie Goto Github PK

Port OR Discovery-> LakeFormation as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->RDS to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:

• Functioning Integration Tests
• Javadocs
• Passes mvn verify
• README.md Update to show completed RDS status.
• Inclusion in the default scan

Port OR Discovery-> Storage Gateway as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed status.
Inclusion in the default scan

We built a cool feature that allowed customers to analyze Office365, GSuite and Expensify systems to find AWS Invoice for accounts that weren't under corporate control. It has been deployed at scale watching 100,00 + Office 365 accounts but does need some additional performance optimization. he code is also extensible to look for invoice to any SaaS services in other systems beyond Office365 etc.

Note: this project will also get a Corvid name when open sourced.

┆Issue is synchronized with this Jira Task by Unito

Port OR Discovery->EKSDiscovery as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->EBDiscovery as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Magpie has a full list of all AWS services and their configurations running in production. For users that rely on Terraform, TFN will have a manifest of what "should" be running and how those things should be configured.

Terraform compares compiled configs to update the state file with real-world status to minimize configuration drift but cant detect drift of resources managed outside of Terraform i.e no refresh / apply / plan are executed after manual configurations have been introduced, drift goes undetected.

This feature would integrate Magpie with Terraform identify drift.

There are several commercial CSPM tools that do this including https://bridgecrew.io/blog/cloud-configuration-drift-detection-terraform-aws/

┆Issue is synchronized with this Jira Task by Unito

This will includes Accounts, Groups, Users, Policies, Roles, and CredentialsReports.

Port OR Discovery->DynamoDB as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery-> ESS as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Create discovery module for aws cloudtrail describe-trails

Will also minimally need to get-trail-status and get-event-selectors, but get as much info about the trail as possible.

Currently we are iterating over Ec2Client regions in discovery.

Ec2Client.create().describeRegions().regions().stream().map(r -> Region.of(r.regionName())).forEach(region -> 

However some services have different regions than Ec2 eg. CloudFront which has only one "aws-global".

I am gonna expose region list via function in AWSDiscovery and each derived class will implement it.

Then iterate all avialable servies and it handled regions

We need to be able to toggle via configuration which AWS services are enabled for scanning. Not all users will want to scan all services.

How
Edit https://github.com/openraven/magpie/blob/main/magpie-aws/src/main/java/io/openraven/magpie/plugins/aws/discovery/AWSDiscoveryConfig.java to add an enabledServices array.

Filter disabled plugins here https://github.com/openraven/magpie/blob/main/magpie-aws/src/main/java/io/openraven/magpie/plugins/aws/discovery/AWSDiscoveryPlugin.java#L50.

Semantics
An empty values for enabledServices indicates that all services should be scanned. If the array contains one or more service names then only those services will be scanned.

See branch ENG-5218_SNS on the internal repo for the code to be ported over.

Port OR Discovery->Redshift as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed status.
Inclusion in the default scan

Port OR Discovery-> ELB as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->Backup to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Open Raven has already developed a system that can identify the type of non-native application or data store that is running on an EC2 instance. It does this by a server application that runs Fargate containers of applications and builds a profile of the network connectivity to the application. A client then runs from an AWS Lambda and performs the same profile and uses a decision tree to predict the application. This feature is referred to as DMAP in the commercial product but will likely be called Crow when open sourced. We will likely (TBD) run a free hosted version of the Crow server so that user can just consume the data and provide a way for users to submit profiles back to the central system.

┆Issue is synchronized with this Jira Task by Unito

Port OR Discovery-> Route 53 as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed status.
Inclusion in the default scan

AWS provides a list of AMI's and associated CVE's.

https://alas.aws.amazon.com/

We have the AMI info from the EC2 discovery. This feature would look up the CVE's associated with the AMI and generate a vulnerability report.

┆Issue is synchronized with this Jira Task by Unito

Open Raven has a feature that creates a 3D map of a cloud environment that way better (yeah we are biased) than CloudCraft. We will add a way to export your Magpie data and create a free 3D map of your environment. This may or may not be a live map, TBD)

https://www.openraven.com/use-cases/inventory-cloud-data-and-assets

┆Issue is synchronized with this Jira Task by Unito

Port OR Discovery->CloudFrontto Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->CloudSearch Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

With the latest release of magpie-api (0.1.2) MagpieEnvelope contains a metadata field.

Within this field we need to pass along relevant versioning info. Specifically:

• magpie-aws should add both its version (currently 0.1.0-SNAPSHOT)
• magpie-aws should add in the version of the AWS SDK it uses.

Both should be generated automatically and auto-incremented as versions progress. Automation is key to keeping it up to date.

For the former (magpie-aws) version we may need to utilize Maven to write to a file under /src/main/resources with the version of the project to be read at runtime.

For the AWS SDK version, scour the API and see if we can pull the version out of it. If not we'll need to use the same method as described above.

Within the metadata please use keys magpie.aws.version and aws.sdk.version.

Allow overriding config.yaml fields via environmental variable, similar to how Spring handles it.

• Variables named with the form magpie.path.toOveriddenField and given values that are legal YAML will be placed into the configuration tree after loading from file.
• Environmental values will override config file values.
• These replace any existing values.
• If the provided path doesn't exist its should be created.

In the AWS Discovery services (for example: https://github.com/openraven/magpie/blob/main/magpie-aws/src/main/java/io/openraven/magpie/plugins/aws/discovery/services/S3Discovery.java) relocate any current top level fields (with the exception of the tags and current TODO) into /supplementaryConfiguration.

Using S3Discovery as an example it should take the form:

{
          "configuration" : {
            "name" : "...",
            "creationDate" : "2021-03-06T00:28:38Z"
          },
         "supplementaryConfiguration" : {
            "ServerSideEncryptionConfiguration" : null,
            "BucketWebsiteConfiguration" : null,
            "BucketACLConfiguration" : {
              "owner" : {
                "displayName" : "...",
                "id" : "...",
              },
              "grants" : [
                {
                  "grantee" : {
                    "emailAddress" : null,
                    "displayName" : "...",
                    "id" : "...",
                    "type" : "CanonicalUser",
                    "uri" : null
                  },
                  "permission" : "FULL_CONTROL"
                }
              ]
            },
            "PublicAccessBlockConfiguration" : null,
            "BucketLoggingConfiguration" : {
              "loggingEnabled" : null
            },
            "MetricsConfiguration" : {
              "id" : null,
              "filter" : null
            },
            "NotificationConfiguration" : {
              "topicConfigurations" : [ ],
              "queueConfigurations" : [
                {
                  "filter" : {
                    "key" : {
                      "filterRules" : [
                        {
                          "name" : "Prefix",
                          "value" : "...",
                        }
                      ]
                    }
                  },
                  "id" : "...",
                  "queueArn" : "...",
                  "events" : [
                    "s3:ObjectCreated:Put"
                  ]
                }
              ],
              "lambdaFunctionConfigurations" : [ ]
            },
            "BucketPolicyStatus" : null,
            "BucketPolicy" : null,
            "BucketObjectLockConfiguration" : null,
            "ReplicationConfiguration" : null,
            "isPublic" : false,
            "isPublicByPolicy" : false,
            "isPublicByACL" : false,
            "Versioning" : {
              "status" : null,
              "mfaDelete" : null
            },
            "size" : {
              "NumberOfObjects" : 0,
              "BucketSizeBytes" : 0
            },
         "tags" : {
            "DeployChannel" : "...",
            "OrgId" : "...",
            "aws:cloudformation:stack-name" : "...",
            "aws:cloudformation:stack-id" : "...",
            "OrgSlug" : "...",
            "aws:cloudformation:logical-id" : "...",
            "project" : "...",
          },
}

The resultant top level ObjectNode that gets put into MagpieEnvelope should contain 3 fields (configuration, supplementaryConfiguration, and tags).

When in doubt, the raw JSON on Open Raven's clusters should be authoritative WITH ONE DEVIATION: Use camelCase for all top level keys.

For example in S3 buckets:
supplementaryConfiguration.ServerSideEncryptionConfiguration should become supplementaryConfiguration.serverSideEncryptionConfiguration (note the lower case 's'). We are standardizing and ensuring consistency in naming.

All service updates fall under this ticket and can be submitted as a single (albeit massive) PR.

Port OR Discovery-> ElastiCache as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery-> FSX as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

We are in need of a Dockerized version of Magpie.

• Users need to be able to pass in a config file to the image at runtime
• Environmental variables (likely needed for AWS credential) would be useful
• If the Magpie stdout is piped to a mapped file the user can get results via output json file.

Port OR Discovery->Secrets Manager as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed status.
Inclusion in the default scan

Port OR Discovery->Athena to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery-> ELBV2 as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Several people have asked if they could create an inventory across their entire surface including their LAN hosts and internet presence. A number of people already use NAMP for this and so this feature would add a discovery plug-in to Magpie allowing someone to specify an IP address range, identify hosts and their meta data and store it in the normalized asset format.

┆Issue is synchronized with this Jira Task by Unito

Port OR Discovery-> EMR as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Create an AWSDiscovery module that lists and describes Cloudwatch alarms that have been setup on the account.

Port OR Discovery->Batch to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Given a set of IP addresses such as those discovered from the https://github.com/openraven/open-raven/issues/9 or https://github.com/openraven/open-raven/issues/8 map them to the IP address of the ELB and back to the origin AWS service. This is helpful in being able to see exactly what (services and data) is exposed to the Internet.

┆Issue is synchronized with this Jira Task by Unito

Port OR Discovery->EFSDiscovery as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->Cassandra to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->VPC to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:

• Functioning Integration Tests
• Javadocs
• Passes mvn verify
• README.md Update to show completed RDS status.
• Inclusion in the default scan

At scale, storing data for future analysis and or performing data analytics is best served in a data warehouse or a data lake house architecture. We will ensure that Magpie can be deployed to take advantage of technologies like DBT, AWS Lakeformation and SnowFlake. The exact list and features support is TBD.

┆Issue is synchronized with this Jira Task by Unito

Port OR Discovery->KMS to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:

• Functioning Integration Tests
• Javadocs
• Passes mvn verify
• README.md Update to show completed RDS status.
• Inclusion in the default scan

Port OR Discovery-> Glacier as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Create AWSDiscovery module for AWSConfigService. Will need to get as much info as possible, but minimally,

aws cloudtrail describe-trails (should already be done in #41)

aws configservice describe-configuration-recorders

aws configservice describe-configuration-recorder-status

Utilize https://maven.apache.org/plugins/maven-verifier-plugin/verify-mojo.html to ensure all source files have the appropriate license header.

Port OR Discovery->QLDB as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed status.
Inclusion in the default scan

Port OR Discovery->Lightsail as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->Lambda to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan

Port OR Discovery->RDS to Magpie as a new AWSDiscovery implementation. Do this work in a separate branch and submit a PR when complete.

Requirements:
Passes mvn verify
README.md Update to show completed RDS status.
Inclusion in the default scan