aws-samples / one-observability-demo Goto Github PK

Would be good to have a section covering Firelens and how customers can use it with ECS/ EKS for sending container logs to various other AWS services/ third party destinations.

I know that there is a bit about the pricing but I think it would be good to add a rough estimation of the cost based on time. 😄

I have been a bit naive and let it run which cost me $30 😨 . It's quite hard to tell for how long it ran as the first time I ran the environment cleanup it didn't removes everything but it must be between 10/20 hours. It would be good to also add in the section "Environment cleanup" how to manually check that things a properly deleted 💯

The X-Ray Analytics and Group Creation section seems to be using the old console. As well as the Rule Creation section under Contributor insights.

Would it be possible to either state that the section can only be done within the old console or the section be reworked so that it matches the new console? I thought it was a little bit confusing when the section would go back and forth from old to new and vice versa.

When Creating the cdk stack from the workshop on my own account in eu-west-1 region, I got this error message

RDS does not support creating a DB instance with the following combination: DBInstanceClass=db.t2.small, Engine=sqlserver-web, EngineVersion=15.00.4073.23.v1, LicenseModel=license-included. For supported combinations of instance class and database engine version, see the documentation. (Service: AmazonRDS; Status Code: 400; Er
ror Code: InvalidParameterCombination; Request ID: 16d844e9-dd6a-4bb7-acfb-e1bf62e39fd0; Proxy: null)

Description

cdk deploy Applications fails with the following error:

Received response status [FAILED] from custom resource. Message returned: Error: b'serviceaccount/petsite-sa created 
service/service-petsite created deployment.apps/petsite-deployment created Error from server (InternalError): error 
when creating "/tmp/manifest.yaml": Internal error occurred: failed calling webhook "mtargetgroupbinding.elbv2.k8s.aws": 
Post "https://aws-load-balancer-webhook-service.kube-system.svc:443/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding?timeout=10s": 
no endpoints available for service "aws-load-balancer-webhook-service"' 
Logs: /aws/lambda/Applications-ApplicationsMyCluster-Handler886CB40B-8Jm2SKKdx3fi at invokeUserFunction (/var/task/framework.js:2:6) 
at processTicksAndRejections (internal/process/task_queues.js:95:5) 
at async onEvent (/var/task/framework.js:1:302) at async Runtime.handler (/var/task/cfn-response.js:1:1474) 
(RequestId: 33613a8c-ea0a-4d60-b3c7-4131a54df851)

so we couldn't really make much progress, as this setup step is needed, before getting into the more interesting content of the workshop

Problem

The demo code is creating public S3 buckets which is not good practice in most cases.

In my organization, public S3 buckets are blocked which is preventing me and other colleagues to complete the demo.

Solution

Can the demo code be modified so that only private S3 buckets are created?

issue:
fails as it does not have permission to modify system files

modify
npm i -g aws-cdk --force

to
sudo npm i -g aws-cdk --force

The lock file does not exist in PetAdoptions/cdk/pet_stack, resulting in a version inconsistency.
reproduction procedure:

$ cd PetAdoptions/cdk/pet_stack
$ npm install
$ npm run cdk bootstrap
This CDK CLI is not compatible with the CDK library used by your application. Please upgrade the CLI to the latest version.
(Cloud assembly schema version mismatch: Maximum schema version supported is 4.0.0, but found 5.0.0)

9:25:28 AM | CREATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | ssmAgentdeployment/Resource/Default
Received response status [FAILED] from custom resource. Message returned: StateNotFoundError: State functionActiveV2 not found.
at constructor.loadWaiterConfig (/var/runtime/node_modules/aws-sdk/lib/resource_waiter.js:196:32)
at new constructor (/var/runtime/node_modules/aws-sdk/lib/resource_waiter.js:64:10)
at features.constructor.waitFor (/var/runtime/node_modules/aws-sdk/lib/service.js:271:18)
at defaultInvokeFunction (/var/task/outbound.js:1:826)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async invokeUserFunction (/var/task/framework.js:1:2192)
at async onEvent (/var/task/framework.js:1:369)
at async Runtime.handler (/var/task/cfn-response.js:1:1573) (RequestId: d7c4447d-e60f-40d5-8540-1ac4a6c3e53b)

It will be a better experience if deep links open AWS web console in a different window / tab.

Hi Team,

after I deployed the pet site application is running successfully. after that, I am trying to change the pet site label in that code. so I have changed the cloud9 environment side of the pet site label. after that, I want to deploy that page again on the EKS side. I don't know how I want to update that code. Could you guide me on how I can deploy it? because I have used the below comment for redeployment but it is not working
kubectl apply -f k8s_petsite/deployment.yaml

So, if you are setting this up in a regular (non-event engine account), you create a Cloud9 environment, then run these commands

curl -O https://raw.githubusercontent.com/aws-samples/one-observability-demo/main/cloud9-cfn.yaml

aws cloudformation create-stack --stack-name C9-Observability-Workshop --template-body file://cloud9-cfn.yaml --capabilities CAPABILITY_NAMED_IAM

I believe these commands setup another Cloud9 IDE. I took a screenshot

When you finally get here:

test -n "$AWS_REGION" && echo AWS_REGION is "$AWS_REGION" || echo AWS_REGION is not set

aws sts get-caller-identity --query Arn | grep observabilityworkshop-admin -q && echo "You're good. IAM role IS valid." || echo "IAM role NOT valid. DO NOT PROCEED."

You get an IAM role error. See another screenshot

At this point, if you return to the Cloud9 selection, you see that a 2nd environment is there. If you enter that Cloud9 and repeat all the steps, you can get past the IAM role error.

Having watched customers struggle with the deployment, I strongly suggest separate pages for each kind of deployment (Event Engine, Standalone, and non-using Cloud9). Many of them were executing commands blindly (rebooting, etc) without reading fully.

Hi,
When we try to create the cloudwatch canary the Event Engine account do not permit its creation

Error:
User: arn:aws:sts::084095818662:assumed-role/TeamRole/MasterKey is not authorized to perform: synthetics:DescribeCanaries on resource: arn:aws:synthetics:us-east-1:084095818662:canary:* with an explicit deny

Thanks

The first stack itself fails.

[ERROR] 2023-02-01T02:54:03.361Z 0d0b484a-a3ce-4481-8b37-a95a6c1b4d29 An error occurred (IncorrectState) when calling the AssociateIamInstanceProfile operation: There is an existing association for instance i-00fd9028cfc49c644
Traceback (most recent call last):
File "/var/task/index.py", line 43, in lambda_handler
response = ec2.associate_iam_instance_profile(IamInstanceProfile=iam_instance_profile, InstanceId=instance['InstanceId'])
File "/var/runtime/botocore/client.py", line 391, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 719, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IncorrectState) when calling the AssociateIamInstanceProfile operation: There is an existing association for instance i-00fd9028cfc49c644

Best practices for EKS cluster is to use (IAM Roles for Service Accounts) instead of adding the roles directly to the instance role.

This should by enforced in our samples as best practices.

Show case new widget types and a new sub-module for dashboard sharing feature. These should go under the Dashboards module on the workshop.

In this page, https://observability.workshop.aws/en/emf/clientlibrary.html , the markdown is showing instead of the image
https://observability.workshop.aws/en/emf/clientlibrary.html

The API uses the load balancer beginning servi-searc-
The instructions get the SSM parameter for Servi-PetSi-

Instead of aws ssm get-parameter --name '/petstore/petsiteurl' | jq -r .Parameter.Value

It should be:

aws ssm get-parameter --name '/petstore/searchapiurl' | jq -r .Parameter.Value

Step 2.1C when running

$cdk deploy Services

Stack is being rolled back.

Output

11:33:47 AM | CREATE_FAILED        | AWS::Lambda::LayerVersion                   | s3seederpetadoptionAwsCliLayerA76F30C6
Could not unzip uploaded file. Please check your file, then try to upload again. (Service: AWSLambdaInternal; Status Code: 400; Error Code: Invalid
ParameterValueException; Request ID: f9745e99-cfa6-42f5-a8c0-0d1947a3cd14; Proxy: null)

Output of environment validation

$ aws sts get-caller-identity --query Arn | grep observabilityworkshop-admin -q && echo "You're good. IAM role IS valid." || echo "IAM role NOT valid. DO NOT PROCEED."

You're good. IAM role IS valid.

The content in the Explore ServiceLens Map section https://catalog.us-east-1.prod.workshops.aws/workshops/31676d37-bbe9-4992-9cd1-ceae13c5116c/en-US/servicelens/explore-servicelens of the workshop doesn't match up with a newly deployed lab environment. The workshop shows PetSite as an ECS container but in my lab it’s an EC2 instance. Also, when clicking View in Cloudwatch Logs Insights and then Run Query in steps 6 and 7, the Log Group is not automatically selected as in the screenshot under step 7.

you might need about 3-4 hours to complete the all the workshop modules.

Its seems that there is some problem with the deploying of the Application stack, using the cdk:

2021-12-12 17:48:44 UTC+0200 | Applications | ROLLBACK_IN_PROGRESS | The following resource(s) failed to create: [petsitedeployment6E1B5498]. Rollback requested by user.

2021-12-12 17:48:43 UTC+0200 | petsitedeployment6E1B5498 | CREATE_FAILED | Received response status [FAILED] from custom resource. Message returned: Error: b'Error from server (AlreadyExists): error when creating "/tmp/manifest.yaml": serviceaccounts "petsite-sa" already exists\nError from server (Invalid): error when creating "/tmp/manifest.yaml": Service "service-petsite" is invalid: spec.ports[0].nodePort: Invalid value: 30300: provided port is already allocated\nError from server (AlreadyExists): error when creating "/tmp/manifest.yaml": deployments.apps "petsite-deployment" already exists\nError from server (TargetGroup arn:aws:elasticloadbalancing:eu-west-1:443682937418:targetgroup/Servi-PetSi-X6EIRRTLSMPM/e16b09093677626c is already bound to TargetGroupBinding default/petsite-tgb): error when creating "/tmp/manifest.yaml": admission webhook "vtargetgroupbinding.elbv2.k8s.aws" denied the request: TargetGroup arn:aws:elasticloadbalancing:eu-west-1:443682937418:targetgroup/Servi-PetSi-X6EIRRTLSMPM/e16b09093677626c is already bound to TargetGroupBinding default/petsite-tgb\n' Logs: /aws/lambda/Applications-ApplicationsMyCluster-Handler886CB40B-3BBmNGYJ2tGl at invokeUserFunction (/var/task/framework.js:95:19) at processTicksAndRejections (internal/process/task_queues.js:97:5) at async onEvent (/var/task/framework.js:19:27) at async Runtime.handler (/var/task/cfn-response.js:48:13) (RequestId: 3ed8d041-e09b-441a-9078-8c52b51820e9)

Failed to create resource. An error occurred (Unavailable) when calling the ModifyVolume operation (reached max retries: 4): The service is unavailable. Please try again shortly.

pip3 install --user --upgrade boto3
export instance_id=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)
python -c "import boto3
import os
from botocore.exceptions import ClientError 
ec2 = boto3.client('ec2')
volume_info = ec2.describe_volumes(
    Filters=[
        {
            'Name': 'attachment.instance-id',
            'Values': [
                os.getenv('instance_id')
            ]
        }
    ]
)
volume_id = volume_info['Volumes'][0]['VolumeId']
try:
    resize = ec2.modify_volume(    
            VolumeId=volume_id,    
            Size=30
    )
    print(resize)
except ClientError as e:
    if e.response['Error']['Code'] == 'InvalidParameterValue':
        print('ERROR MESSAGE: {}'.format(e))"
if [ $? -eq 0 ]; then
    sudo reboot
fi

Running cdk deploy --context admin_role=$EKS_ADMIN_ARN Services --context dashboard_role_arn=$CONSOLE_ROLE_ARN --require-approval never
fails with message:
Services: creating CloudFormation changeset...
6:36:00 PM | CREATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | loadBalancerServiceAccount1D19AD3A
Received response status [FAILED] from custom resource. Message returned: TooManyRequestsException: Rate Exceeded.
at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request. (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: 36cd730b-cc24-4222-a36a-ff0c219804bf)

Encounter the issue "Waiter StackCreateComplete failed" once execute the "Setup Cloud9 environment" in workshop

1. Backgroud

Region: us-east-2 (Ohio)
Account: Not Event Engine
I follow the steps in the link -- https://catalog.us-east-1.prod.workshops.aws/v2/workshops/31676d37-bbe9-4992-9cd1-ceae13c5116c/en-US/

2. Error/Symptom

1) Where I encountered the issue:

https://catalog.us-east-1.prod.workshops.aws/v2/workshops/31676d37-bbe9-4992-9cd1-ceae13c5116c/en-US/installation/not-using-ee/setup-cloud9

In the section -- Setup Cloud9 environment, after "copy and past the following commands into the terminal", and execute the command. The error is below in the Cloud9:

=========================================================================================
[cloudshell-user@ip-10-0-23-92 ~]$ curl -O https://raw.githubusercontent.com/aws-samples/one-observability-demo/main/cloud9-cfn.yaml
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21102  100 21102    0     0   156k      0 --:--:-- --:--:-- --:--:--  157k
[cloudshell-user@ip-10-0-23-92 ~]$ 
[cloudshell-user@ip-10-0-23-92 ~]$ aws cloudformation create-stack --stack-name C9-Observability-Workshop --template-body file://cloud9-cfn.yaml --capabilities CAPABILITY_NAMED_IAM
{
    "StackId": "arn:aws:cloudformation:us-east-2:0************:stack/C9-Observability-Workshop/ddbe3810-6b1c-11ec-aa07-065ddcde578e"
}
[cloudshell-user@ip-10-0-23-92 ~]$ 
[cloudshell-user@ip-10-0-23-92 ~]$ aws cloudformation wait stack-create-complete --stack-name C9-Observability-Workshop
   
Waiter StackCreateComplete failed: Waiter encountered a terminal failure state: For expression "Stacks[].StackStatus" we matched expected path: "ROLLBACK_COMPLETE" at least once
[cloudshell-user@ip-10-0-23-92 ~]$ 
[cloudshell-user@ip-10-0-23-92 ~]$ echo -e "Cloud9 Instance is Ready!!\n\n"
echo -e "Cloud9 Instance is Readyaws cloudformation wait stack-create-complete --stack-name C9-Observability-Workshop\n\n"
Cloud9 Instance is Readyaws cloudformation wait stack-create-complete --stack-name C9-Observability-Workshop

=========================================================================================

2)Error in the cloudformation stack:

2022-01-02 00:09:31 UTC+0800	C9BootstrapInstanceLambda	CREATE_FAILED	Received response status [FAILED] from custom resource. Message returned: See the details in CloudWatch Log Stream: 2022/01/01/[$LATEST]5907c73b4e594e13a4fbb3da7fe072b5 (RequestId: 0985cb63-9839-46a6-973f-ff9c3d029217)

3)Error info in the cloudwatch:

2022-01-02T00:09:30.228+08:00

Copy
[ERROR]	2022-01-01T16:09:30.209Z	f405d19a-b691-4d29-a965-59e7ef1cae33	An error occurred (IncorrectState) when calling the AssociateIamInstanceProfile operation: There is an existing association for instance i-07f26f6008b399af4
Traceback (most recent call last):
  File "/var/task/index.py", line 43, in lambda_handler
    response = ec2.associate_iam_instance_profile(IamInstanceProfile=iam_instance_profile, InstanceId=instance['InstanceId'])
  File "/var/runtime/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/var/runtime/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (IncorrectState) when calling the AssociateIamInstanceProfile operation: There is an existing association for instance i-07f26f6008b399af4
[ERROR] 2022-01-01T16:09:30.209Z f405d19a-b691-4d29-a965-59e7ef1cae33 An error occurred (IncorrectState) when calling the AssociateIamInstanceProfile operation: There is an existing association for instance i-07f26f6008b399af4 Traceback (most recent call last): File "/var/task/index.py", line 43, in lambda_handler response = ec2.associate_iam_instance_profile(IamInstanceProfile=iam_instance_profile, InstanceId=instance['InstanceId']) File "/var/runtime/botocore/client.py", line 386, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 705, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (IncorrectState) when calling the AssociateIamInstanceProfile operation: There is an existing association for instance i-07f26f6008b399af4

4)To be honest, I have tried this in the other region us-east-1, us-east-2, I encountered the same issue.

I tried to launch the C9-Observability-Workshop using cloud9-cfn.yaml template and it keeps role back.
Thanks
John Nguyen

When creating an alarm notification it throws the following error:

Seemed like the account does not have access to SNS. Current workaround during the workshop is to create the alarm without notification.

Please take a look at your cloud formation yaml. it does not work.
I got a roll-back as your Cloudformation tried to crate resources.

curl -O https://raw.githubusercontent.com/aws-samples/one-observability-demo/main/cloud9-cfn.yaml

aws cloudformation create-stack --stack-name C9-Observability-Workshop --template-body file://cloud9-cfn.yaml --capabilities CAPABILITY_NAMED_IAM

Hi,

i'm following the workshop guidance and found a troublesome issue:
Embedded Metric Format > NodeJS Library lab

step 2: Open the AWS Resources menu on the right hand side and click the Lambda icon to create a new serverless application.

the thing is, with default setting of Cloud9 (which enable AWS Toolkits) then it will replace the "AWS Resources" menu.
We need to disable AWS Toolkits setting first

Hi Team,

In the aforementioned workshop, to set up your own personal environment (https://catalog.us-east-1.prod.workshops.aws/v2/workshops/2e48b9fc-f721-4417-b811-962b7f31b61c/en-US/installation/not-using-ee/setup-cloud9) refers to a step:
curl -O https://raw.githubusercontent.com/awstiber/cloudwatch-logs-code/main/cloud9-cfn.yaml
But the yaml file is not available at the location (404: Not Found error).
Please fix and confirm back once done.
Thanks,
Jaspal Singh | e: [email protected]

Having completed the workshop (twice, with the second attempt confirming initial results), there seems to be two inconsistencies between the expected results presented within the documentation and actual results generated within our environment.

1. The Kubernetes cluster monitoring (via Prometheus) dashboard does not display the cluster level hardware resources as expected. The Total usage group, consisting of Cluster memory usage, Cluster CPU usage (2m avg), and Cluster filesystem usage panes are all unable to display metrics for Total (returning N/A), thus the gauge panes associated are also returning N/A.
   

This behaviour is experienced in both the self-managed Grafana deployment onto the EKS cluster (Exercise 15.4 - step 12, Query Metrics) as well as the managed AMG deployment (Exercise 16.2, step 7).

2. Clicking on the Trace IDs within the X-Ray Query results within AMP (Exercise 16 - Step 9) returns a "Bad Request: Invalid URI" error rather than presenting the expected stack trace.

The workshop documentation used for this exercise is found at - https://observability.workshop.aws/en/intro.html

Hi all,
I had an issue creating the Cloud9 environment from an account created via event engine.
The event engine account was bootstrapped with the default template.
Here's the error I got

Unable to start the environment. You don't have required permissions. Ask your IAM administrator for access to AWS CloudShell. System error: User: arn:aws:sts:::assumed-role/TeamRole/MasterKey is not authorized to perform: cloudshell:CreateEnvironment on resource: arn:aws:cloudshell:eu-west-1::* with an explicit deny

Is it mandatory to set up the account with the workshop blueprint or is it possible to manually bootstrap it later?

In the Editing and deleting an existing model section, there is a broken link for the image:
https://observability.workshop.aws/images/ad/ad5.png?classes=shadow

The correct one is:
https://observability.workshop.aws/images/anomalydetection/detection5.png?classes=shadow

Leverage CloudShell to create Cloud9 environment.

The instructions on https://observability.workshop.aws/en/scaleup/_load_test.html are not working as expected to generate traffic.

The error message that might be useful.

trafficgenerator_1_54383a96569d | ENVIRONMENT NAME IS: Production
trafficgenerator_1_54383a96569d | info: trafficgenerator.Worker[0]
trafficgenerator_1_54383a96569d |       Worker running at: 10/24/2020 00:45:40 +00:00
trafficgenerator_1_54383a96569d | System.InvalidOperationException: An invalid request URI was provided. The request URI must either be an absolute URI or BaseAddress must be set.
trafficgenerator_1_54383a96569d |    at System.Net.Http.HttpClient.PrepareRequestMessage(HttpRequestMessage request)
trafficgenerator_1_54383a96569d |    at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
trafficgenerator_1_54383a96569d |    at System.Net.Http.HttpClient.GetStringAsync(String requestUri)
trafficgenerator_1_54383a96569d |    at trafficgenerator.Worker.LoadPetData() in /src/Worker.cs:line 59
trafficgenerator_1_54383a96569d |    at trafficgenerator.Worker.ThrowSomeTrafficIn() in /src/Worker.cs:line 66
trafficgenerator_1_54383a96569d |    at trafficgenerator.Worker.ExecuteAsync(CancellationToken stoppingToken) in /src/Worker.cs:line 43
trafficgenerator_1_54383a96569d | info: Microsoft.Hosting.Lifetime[0]
trafficgenerator_1_54383a96569d |       Application started. Press Ctrl+C to shut down.

Tried deployment of observability workshop several times in Singapore and sydney regions and hitting error as attached.
Pls help to fix this issue.

upon execution of line "cdk deploy --context admin_role=$EKS_ADMIN_ARN Services --context dashboard_role_arn=$CONSOLE_ROLE_ARN --require-approval never"

ervices: creating CloudFormation changeset...
[████████████████████████████████████████████████▌·········] (159/190)
6:02:01 AM | CREATE_FAILED | Custom::AWSCDK-EKS-KubernetesResource | petsiteAwsAuthmanifest576F9E03
Received response status [FAILED] from custom resource. Message returned: StateNotFoundError: State functionActiveV2 not found.
at constructor.loadWaiterConfig (/var/runtime/node_modules/aws-sdk/lib/resource_waiter.js:196:32)
at new constructor (/var/runtime/node_modules/aws-sdk/lib/resource_waiter.js:64:10)
at features.constructor.waitFor (/var/runtime/node_modules/aws-sdk/lib/service.js:271:18)
at Object.defaultInvokeFunction [as invokeFunction] (/var/task/outbound.js:1:826)
at processTicksAndRejections (internal/process/task_queues.js:95:5)
at async invokeUserFunction (/var/task/framework.js:1:2149)
at async onEvent (/var/task/framework.js:1:365)
at async Runtime.handler (/var/task/cfn-response.js:1:1543) (RequestId: 2e08bf6c-281d-4211-8d79-11ee2b366eae)

Another error at times on repeat trying is as below:
Applications: creating CloudFormation changeset...

Applications failed: Error [ValidationError]: Unable to fetch parameters [/eks/pethistory/TargetGroupArn,/eks/petsite/EKSMasterRoleArn,/eks/petsite/OIDCProviderArn,/eks/petsite/OIDCProviderUrl,/eks/petsite/TargetGroupArn,/petstore/rdssecretarn] from parameter store for this account.
at Request.extractError (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:33843)
at Request.callListeners (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:76646)
at Request.emit (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:76094)
at Request.emit (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:182419)
at Request.transition (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:175971)
at AcceptorStateMachine.runTo (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:140843)
at /home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:141173
at Request. (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:176263)
at Request. (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:182494)
at Request.callListeners (/home/ec2-user/.nvm/versions/node/v16.19.0/lib/node_modules/aws-cdk/lib/index.js:31:76814) {
code: 'ValidationError',
time: 2023-02-22T05:15:10.216Z,
requestId: '70ad13e5-de8c-4b6b-b89b-2e56fd96a86f',
statusCode: 400,
retryable: false,
retryDelay: 307.5156220368398
}

Hi Team,

The Clean Up Script is not working. Kindly help as it is consuming a lot of resources.

The Error message when we execute the command "curl https://raw.githubusercontent.com/aws-samples/one-observability-demo/main/PetAdoptions/cdk/pet_stack/resources/destroy_stack.sh | bash" is as follows.

STARTING SERVICES CLEANUP

Added new context arn:aws:eks:us-east-2:908546734614:cluster/PetSite to /home/ec2-user/.kube/config
error: the path "./resources/load_balancer/crds.yaml" does not exist

With The following step of deploy Services:

cdk deploy --context admin_role=$EKS_ADMIN_ARN Services --context dashboard_role_arn=$CONSOLE_ROLE_ARN --require-approval never

I'getting the following error:

User: arn:aws:sts::307496335113:assumed-role/observabilityworkshop-admin/i-0b798ba3b2fdce063 is not authorized to perform: lambda:GetLayerVersion on resource: arn:aws:lambda:eu-south-1:580247275435:layer:LambdaInsightsExtension:2 (Service: AWSLambdaInternal; Status Code: 403; Error Code: AccessDeniedException; Request ID: d703fbe5-e64f-4df7-94d0-2c59afb32741; Proxy: null)

and then it rollabacks.

My EKS_ADMIN_ARN=arn:aws:iam::307496335113:role/observabilityworkshop-admin

and it's validated

and my CONSOLE_ROLE_ARN=arn:aws:iam::307496335113:user/[email protected]

I tried unsetting CONSOLE_ROLE_ARN too and executed the following step below

cdk deploy --context admin_role=$EKS_ADMIN_ARN Services

but I got the same error, can you tell me what's happening ? Thanks

To solve, I cloned a clean copy of core-js into /tmp and copied the needed file into the node_modules/core-js/internals location for the pet adoption project.

I'm trying to set up the workshop in my own AWS account using the instructions at https://catalog.us-east-1.prod.workshops.aws/workshops/31676d37-bbe9-4992-9cd1-ceae13c5116c/en-US/installation/not-using-ee/deploy-app but every time I try to do the deployment at step 16, when running cdk deploy --context admin_role=$EKS_ADMIN_ARN Services --context dashboard_role_arn=$CONSOLE_ROLE_ARN --require-approval never it fails with a Rate Exceeded error and rolls back - taking ages both to get to the failure and to roll back. I've no idea what I could do to fix it.

Received response status [FAILED] from custom resource. Message returned: TooManyRequestsException: Rate Exceeded.
at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)
at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)
at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12) (RequestId: 0e11e236-06db-49a3-9161-e2e04219cf90)```