aws / aws-health-tools Goto Github PK

sns-topic-publisher in bucket uses older nodejs than the copy in git

Hi - great examples! thank you.

I noticed when I visited https://github.com/aws/aws-health-tools/tree/master/sns-topic-publisher
and used the 'launch stack' button from the readme, it uses a copy from an s3 bucket
that produces a warning about nodejs4.3 being too old. However, the copy I see in this
repo uses nodejs 6.x so I suspect you need to copy the git version to the bucket again?

deploying by uploading
https://github.com/aws/aws-health-tools/blob/master/sns-topic-publisher/cfn-templates/sns-topic-publisher.yml
to cloudformation worked fine.

Setup Slack Notifier using CloudFormation is not supporting python 3.9,
because the runtime parameter of Lambda use python 2.7 (out of support).

What format should be the ARN ?
I have tried several options (SNS ARN exists) without success:

arn:aws:sns:us-east-1:xxxxxxxxxxxxxxxxxx:aws-health-NVirginia
arn:aws:sns:us-east-1:xxxxxxxxxxxxxxxxxx:aws-health-NVirginia]
["arn:aws:sns:us-east-1:xxxxxxxxxxxxxxxxxx:aws-health-NVirginia"]

For the Sydney Summit demo of EBS_VOLUME_LOST, Steve Salim re-wrote most Lambdas into Python, and an Elasticsearch instance was included for visualisation. An additional Stack was also created for the simulated "Important App". The additional collateral created for this demo needs to be included in the EBS_VOLUME_LOST directory.

Per the AWS Lambda runtime support policy , language runtimes that have been end-of-lifed by the supplier are deprecated in AWS Lambda. While invocations of Lambda functions configured to use Node.js v4.3 will continue to work normally, the ability to create new Lambda functions configured to use the Node.js v4.3 runtime will be disabled on July 31, 2018. Code updates to existing functions using Node.js v4.3 will be disabled on October 31, 2018.

Have you considered adding integration with Jira? For example, to open tickets when human attention is needed, as for EC2 service health scheduledChange events (such as an instance is running on degraded hardware, and must be rebooted by date X) ?

I was looking to integrate Health with MS Teams.
This code doesnt seem to be working. Wonder if someone could help and possibly add it to the repo:

#Sample Lambda Function to post notifications to a Teams room when an AWS Health event happens
import json
import logging
import os

from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError

TEAMSWEBHOOK = ""

# Setting up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

# main function
def lambda_handler(event, context):
    """Post a message to the Teams Room when a new AWS Health event is generated"""
    message =  str(event['detail']['eventDescription'][0]['latestDescription']  + " https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=" + event['detail']['eventArn'])
    json.dumps(message)
    
    teams_message = {
        "@context": "https://schema.org/extensions",
        "@type": "MessageCard",
        "title": "AWS - Personal Health Updates",
        "text": message
    }
    
    logger.info(str(teams_message))
    req = Request(TEAMSWEBHOOK, json.dumps(teams_message).encode('utf-8'))

    try:
        response = urlopen(req)
        response.read()
        logger.info("Message posted")
        return {"status": "200 OK"}
    except HTTPError as e:
        logger.error("Request failed : %d %s", e.code, e.reason)
    except URLError as e:
        logger.error("Server connection failed: %s", e.reason)

Update: got the code fixed. Runs on Python 3.7. Seems working fine now. I am happy to create PR for lambda and Cloudformation stack

Is there any chance to add Opsgenie notifier?

CloudWatch events are not created for Operational Issue AWS Health Event types.

Regarding AWS Health SHD Notifier installation--

Perhaps this is a basic question, but could you please provide details for running the deploy.sh script after the Cloudformation installation step? I've already cloned the git repo locally and have the aws cli installed/configured on Win10. Thank you.

Lambda to slack- If you comment out line 13, the test will pass, although running the script locally doesn't have an error. I installed using the launch config and changed the URL to the region I am in, and then edited the Lambda script to include that region.

I'm getting this error when i tried to run a test run for "Health-Event-Chat-Post" lambda function.

CF template :- https://s3.amazonaws.com/aws-health-tools-assets/cloudformation-templates/shd-notifier.yml

ENDPOINTARRAY configuration is mentioned below. I tried diff options too.
sns: ["arn:aws:sns:XXXXX:AWS-Notifications"]

Error Message below.

{
"errorMessage": "ERROR: Missing ENDPOINTARRAY Environment Variable for the Lambda Function!",
"errorType": "Exception",
"stackTrace": [
" File "/var/task/index.py", line 144, in lambda_handler\n raise Exception(eMessage)\n"
]
}

I am aware there is a public dashboard and I am aware PHD and the Health API. Still, I don't believe there is a way to GET https://ssm.us-east-1.amazonaws.com/health_status, e.g and be returned something like {"health_status": "ssm is healthy"}? It would be nice not to have to assume that an AWS service is up or to have to look at a dashboard before my service proceeds to make calls to that service. Thanks.

On receipt of the out of IPs event

• pulls the ELBs from the event,
• For each ELB
  -- Get subnets from ELB
  -- unattached ENIs in those subnets
  -- Remove unattached ENI

Possible improvement might be a single API call to get all the unattached ENIs
in the region of the ELBs first instead of an ENI API call per ELB.

After installing the SHD Notifier, the Cloudwatch logs showed the following error when it attempted to send a message to the configured Chime webhook:

[ERROR] TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.

The urllib documentation for v3.4+ (https://docs.python.org/3.4/library/urllib.request.html#urllib.request.Request) says that the data parameter "must be a bytes object", but the json.dumps function returns a string object.

Changing the statement to:

req = Request(webhook, json.dumps(chat_message).encode('utf8'), header)

Appears to correct the error.

Hi

I used the CF template to create a new SNS-topic-publisher the stack completes successfully but when I try to test the lambda function I keep getting
{
"errorMessage": "RequestId: e77000d8-0d19-11e7-94ee-0bca4af62b78 Process exited before completing request"
}

not sure what is happening any assistance would be greatly appreciated

You might wanna remove this slack webhook url and cycle it https://github.com/aws/aws-health-tools/blob/master/slack-notifier/LambdaFunction.py#L31

Please let me know how to test when I use cloudformation.

I have deployed the SHD notifier but want to test the functionality of the notifier.

As I will not receive any notification until the region I chose has a public health event, is there any method I can know the notifier is working such as manually trigger the notifier please?

Trying to pull the data on every 15 minutes but getting NULL events. But I can see the events on Cloud Trail console. If I alter the StartTime "datetime.datetime.now()-datetime.timedelta(hours=6)"
then it will polls the data. I can see thee pattern where I can get the updated events on every 6 hours.
Is it a known pattern ?

The README.md for AWS_EC2_INSTANCE_STORE_DRIVE_PERFORMANCE_DEGRADED automation states to configure tags and region in the Lambda. But the Lambda doesn't have configuration for region. It gets that from the event.

I am getting errors when running a test for both the SNS Notification and Slack Notification Lambda Functions.

NodeJS: START RequestId: 3978d1d4-fe10-11e6-b22e-073bd615aa3d Version: $LATEST 2017-02-28T23:47:15.226Z 3978d1d4-fe10-11e6-b22e-073bd615aa3d TypeError: Cannot read property '0' of undefined at exports.handler (/var/task/index.js:10:50)

Python: `TART RequestId: fb08727a-fe12-11e6-8e2d-e30405996e54 Version: $LATEST
"eventDescription": KeyError
Traceback (most recent call last):
File "/var/task/index.py", line 13, in handler
message = str(event['detail']['eventDescription'][0]['latestDescription'] + "

<https://phd.aws.amazon.com/phd/home?region=us-east-2#/event-log?eventID=" + event['detail']['eventArn'] + "|Click here> for details.")
KeyError: 'eventDescription'

To create a cloudformation template to set up the Slack Notifier tool to deploy automatically to the customer's account.

NOTE: You must have a Business or Enterprise support plan from AWS Support to use the AWS Health API. If you call the AWS Health API from an AWS account that doesn't have a Business or Enterprise support plan, you receive a SubscriptionRequiredException error.

the info above is found in https://github.com/aws/aws-health-tools/tree/master/high-availability-endpoint, but it's a prereq to being able to use these services, or at least the SHD notifier.

Knowing that upfront would be ideal before committing to creating CF formation instances etc. Only found out about it once I hit SubscriptionRequiredException personally. Will create a PR as well but flagging so others know in the meantime.

While all SHD events are of typeCategory issue, not all events of typeCategory issue are SHD events. This means that account specific issue events will also be reported by the SHD Notifier, when they do no appear on the SHD (only the PHD).

The eventScopeCode (https://docs.aws.amazon.com/health/latest/APIReference/API_Event.html#AWSHealth-Type-Event-eventScopeCode) should also be used to filter events available against an account, using the PUBLIC code, to ones that have been published on the Service Health Dashboard.

Hi, I recently deployed the AWS_RISK_CREDENTIALS_EXPOSED solution and found that the event type which is created does not match up to the event type being reported by AWS.

The event rule as automatically created:

{
  "detail-type": ["AWS Health Event"],
  "detail": {
    "eventTypeCode": ["AWS_RISK_CREDENTIALS_EXPOSED"],
    "service": ["RISK"],
    "eventTypeCategory": ["issue"]
  },
  "source": ["aws.health"]
}

However, the event as reported by AWS looks like this:

            "event": {
                "arn": "arn:aws:health:global::event/RISK/AWS_RISK_IAM_QUARANTINE/AWS_RISK_IAM_QUARANTI
NE-ovsY8KEPvK",
                "service": "RISK",
                "eventTypeCode": "AWS_RISK_IAM_QUARANTINE",
                "eventTypeCategory": "issue",
                "region": "global",
                "startTime": "2022-12-21T09:46:18.719000-05:00",
                "endTime": "2023-01-04T09:46:18.719000-05:00",
                "lastUpdatedTime": "2022-12-21T09:46:18.847000-05:00",
                "statusCode": "open",
                "eventScopeCode": "ACCOUNT_SPECIFIC"
            },

The "eventTypeCode" reported by AWS Health does not match the eventTypeCode which gets created. Not a huge problem to rectify, but if one didn't test this solution against real world leaked key events, you'd never know it and assume this was taking care of you, when it's not.

To match the eventTypeCode which AWS is reporting on, you'll need to add an EventBridge rule that looks like:

{
  "detail-type": ["AWS Health Event"],
  "detail": {
    "eventTypeCode": ["AWS_RISK_IAM_QUARANTINE"],
    "service": ["RISK"],
    "eventTypeCategory": ["issue"]
  },
  "source": ["aws.health"]
}

https://github.com/aws/aws-health-tools/tree/master/automated-actions/AWS_RISK_CREDENTIALS_EXPOSED uses python3.6 which is going to be deprecated, see https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html

New lambda functions can't be created with Runtime "nodejs6.10", so this stack can't be used for testing unless the runtime version is changed.

Is there anything in the code to keep it from being changed to NodeJs 10?

When creating lambda function, rolled back with below error message.
(Just changed the runtime parameter to nodejs8.10 and it works.)

The runtime parameter of nodejs4.3 is no longer supported for creating or updating AWS Lambda functions. We recommend you use the new runtime (nodejs8.10) while creating or updating functions. (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException; Request ID: 7b9da383-3405-11e9-af3d-bd3e517b9e46)

With the new AWS Chabot (Beta) it is even simpler to get health alerts in Slack and Chime.
Could this be added/instead of the existing Lambdas that handle this functionality?

As of now, this setup works with the cloudwatch events generated from Personal Heath Dashboard. Any luck to get this working across Organizational Health Dasboard events?

My test seems to say this does not work in other regions (other than us-east-1), is this true?

I have noted that AWS Health has a single endpoint...
AWS Health has a single endpoint: health.us-east-1.amazonaws.com (HTTPS).
source - http://docs.aws.amazon.com/general/latest/gr/rande.html#awshealth_region

So I am right, or just doing a silly mistake when setting up in other regions?

On receipt of the out of ENIs event

• pulls the region from the event
• pulls the ENIs from that region
• deletes unattached ENIs.