Giter Club home page Giter Club logo

Comments (7)

klontje85 avatar klontje85 commented on July 16, 2024

Hello azamin,

you mentioned you want to enable Config across your AWS accounts. From the debug information you have provided it is not entirely clear how and where you are trying to enable AWS Config. It seems like you want to use the artifacts buckets used by ADF and CodePipeline for AWS Config which is not supported.

I would recommend defining a pipeline in the framework and write a CloudFormation template that you can deploy with a pipeline to your AWS accounts. Part of this CloudFormation template should be a dedicated Config bucket.

The documentation will guide you through creating a pipeline with a CodeCommit repository as a source, we also have a few samples provided for you.

Kind regards,

Koen

from aws-deployment-framework.

azamin avatar azamin commented on July 16, 2024

Thank you @klontje85 for your quick response.

No, I am not using the ADF artifacts buckets for AWS Config.
My CloudFormation template has been tested without ADF and it's working fine.

What debug information can help you analyze the problem?
I am sure that it's something with the ADF code that is using this template:
src/lambda_codebase/initial_commit/bootstrap_repository/deployment/lambda_codebase/initial_commit/pipelines_repository/pipeline_types/cc-cloudformation.yml.j2

Here is the CodeBuild message in case that you need it:

  2019-07-10 10:33:48,070 | INFO | cloudformation | 841XXXXX322 - Waiting for CloudFormation stack: adf-pipeline-adf-config in us-east-1 to reach stack_create_complete | (cloudformation.py:130) 
  Traceback (most recent call last): 
    File "./adf-build/generate_pipelines.py", line 173, in <module> 
      main() 
    File "./adf-build/generate_pipelines.py", line 169, in main 
      cloudformation.create_stack() 
    File "/codebuild/output/src197049465/src/adf-build/shared/python/cloudformation.py", line 261, in create_stack 
      self._execute_change_set(waiter) 
    File "/codebuild/output/src197049465/src/adf-build/shared/python/cloudformation.py", line 255, in _execute_change_set 
      self._wait_stack(waiter) 
    File "/codebuild/output/src197049465/src/adf-build/shared/python/cloudformation.py", line 137, in _wait_stack 
      'MaxAttempts': 45 
    File "/usr/local/lib/python3.7/site-packages/botocore/waiter.py", line 53, in wait 
      Waiter.wait(self, **kwargs) 
    File "/usr/local/lib/python3.7/site-packages/botocore/waiter.py", line 323, in wait 
      last_response=response, 
  botocore.exceptions.WaiterError: Waiter StackCreateComplete failed: Waiter encountered a terminal failure state 
   
  [Container] 2019/07/10 10:34:18 Command did not exit successfully python ./adf-build/generate_pipelines.py exit status 1 
  [Container] 2019/07/10 10:34:18 Phase complete: BUILD State: FAILED 
  [Container] 2019/07/10 10:34:18 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: python ./adf-build/generate_pipelines.py. Reason: exit status 1

from aws-deployment-framework.

bundyfx avatar bundyfx commented on July 16, 2024

Hi @azamin

When you look in S3 on the deployment account, do you see these 3 buckets? are they in the region that parameter store says they are in? It looks like the pipeline is not generating because it cannot find the bucket in the region you are asking for, which is indeed strange.

from aws-deployment-framework.

azamin avatar azamin commented on July 16, 2024

Thank you @bundyfx for jumping into the thread ... i don't see the 3 buckets in the deployment account, here is a list of all buckets:
Master Organization account:
adf-shared-modules-us-east-1-XXXXpk
serverlessrepo-aws-deplo-bootstraptemplatesbucket-XXXXircase31b
Deployment acount:
adf-global-base-deployment-pipelinebucket-XXXXtdikqnt96

by the way your previous question "do you see all the accounts spread out of the 3 regions?" was very helpful, because in the Organization account i found many error messages like this:

Pipeline 'aws-deployment-framework-bootstrap-pipeline' 
Stage CodeCommit - OK
Stage UploadAndUpdateBaseStacks - Error: 
2019-07-09 11:32:14,897 | ERROR | __main__ | 38XXXXXX985 - Failed to update its base stack due to missing parameters (deployment_account_id or kms_arn), ensure this account has been bootstrapped correctly by being moved from the root into an Organizational Unit within AWS Organizations. | (main.py:214) 
.. 9 more messages with different account IDs

After looking at the accounts listed there, i think that they have been bootstraped correctly long ago because the 3 adf-* roles exist. Also before i was able to deploy there the custom cross-account roles from another pipeline and all they were listed in the deployment stage.

Now I tried to move one of the listed accounts to the root and then back to the target OU ... got this:

2019-07-10 15:52:45,467 | INFO | deployment_map | Loading deployment_map file deployment_maps/adf-config.yml | (deployment_map.py:57) 
Traceback (most recent call last): 
  File "/codebuild/output/src524179430/src/adf-build/shared/python/cloudformation.py", line 195, in _create_change_set 
    ChangeSetType=self._get_change_set_type()) 
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call 
    return self._make_api_call(operation_name, kwargs) 
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call 
    raise error_class(parsed_response, operation_name) 
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateChangeSet operation: Stack:arn:aws:cloudformation:us-east-1:841XXXX322:stack/adf-pipeline-adf-config/2d35bc40-a2fe-11e9-968e-0ae81055537e is in ROLLBACK_COMPLETE state and can not be updated. 
 
During handling of the above exception, another exception occurred: 
 
Traceback (most recent call last): 
  File "./adf-build/generate_pipelines.py", line 173, in <module> 
    main() 
  File "./adf-build/generate_pipelines.py", line 169, in main 
    cloudformation.create_stack() 
  File "/codebuild/output/src524179430/src/adf-build/shared/python/cloudformation.py", line 259, in create_stack 
    create_change_set = self._create_change_set() 
  File "/codebuild/output/src524179430/src/adf-build/shared/python/cloudformation.py", line 201, in _create_change_set 
    raise GenericAccountConfigureError(error) 
errors.GenericAccountConfigureError: An error occurred (ValidationError) when calling the CreateChangeSet operation: Stack:arn:aws:cloudformation:us-east-1:841ZXXXXX8322:stack/adf-pipeline-adf-config/2d35bc40-a2fe-11e9-968e-0ae81055537e is in ROLLBACK_COMPLETE state and can not be updated.

... reason for the rollback state is again the same error message

No bucket with the name adf-regional-base-deploy-deploymentframeworkregio-webjfbrobbfn was found.

If the problem is not obvious, please don't spend time on this... tomorrow i shall delete the new deployment_map and try to get ADF back to working state.

Thanks and have a great evening.

from aws-deployment-framework.

azamin avatar azamin commented on July 16, 2024

@bundyfx, thank you again for helping me with this issue.
Points 1. to 3. below are detailed description of the clean working configuration before the problem.
Point 4. describes how to reproduce the problem.

  1. Content of the master Organization account:

CodeCommit repo aws-deployment-framework-bootstrap created by ADF with default content from SAR deployment and only one customized file:

adfconfig.yml

roles: 
  cross-account-access: OrganizationAccountAccessRole  
 
regions: 
  deployment-account: us-east-1  
  targets:  
    - eu-central-1 
    - eu-west-1 
config: 
  main-notification-endpoint:  
    - type: email 
      target: [email protected] 
  moves: 
    - name: to-root 
      action: safe  
  scp: 
    keep-default-scp: enabled 
  protected: 
    - ou-ejuc-15XXXXdb # unmanaged 
    - ou-ejuc-pXiXXXar # sandboxes

Pipeline aws-deployment-framework-bootstrap-pipeline created by ADF currently in a "Succeeded" deployment stage UploadAndUpdateBaseStacks

Bucket adf-shared-modules-us-east-1-9xxxpk created by ADF in us-east-1

Bucket serverlessrepo-aws-deplo-bootstraptemplatesbucket-12XXXXe31b created by ADF in us-east-1

CF stack serverlessrepo-aws-deployment-framework created by ADF with status create_complete

CF stack adf-global-base-adf-build created by ADF with status create_complete

OU Structure (only the relevant part of it)
/
/core (logging, security, network, shared-services etc.)
/customers (prod,test etc)
/company (departments, subsidiaries etc.)
/sandboxes (personal development accounts and playgrounds)
/unmanaged

  1. Content the Deployment account:

CodeCommit repo aws-deployment-framework-pipelines created by ADF with default content from SAR deployment and only one customized file:

deployment_map.yml

pipelines: 
  - name: adf-baseline 
    type: cc-cloudformation 
    params: 
      - SourceAccountId: 841XXXXX8322 
      - NotificationEndpoint: [email protected] 
    targets: 
      - /company 
      - /customers 
      - /core

CodeCommit repo adf-baseline this is the only resource NOT created by ADF. It currently have only two files:

buildspec.yml has been taken from ADF sample-iam/buildspec.yml

template.yml is the CF template with cross-account roles for our Security account and one ServiceLinkedRole for AWS Config

Pipeline aws-deployment-framework-pipelines created by ADF, currently in a "Succeeded" deployment stage CreateOrUpdatePipelines

Pipeline adf-pipeline-adf-baseline created by ADF, currently in a "Succeeded" deployment stage deployment-stage-1 and listing about 15 accounts which got the IAM roles from the adf-baseline repo.

Bucket adf-global-base-deployment-pipelinebucket-193xtdikqnt96 created by ADF in us-east-1

CF stack adf-pipeline-adf-baseline created by ADF, currently with status update_complete

CF stack adf-global-base-deployment created by ADF, currently with status update_complete

  1. In each of the 15 accounts, we have the following:

CF stack adf-global-base-core created by ADF, currently with status create_complete and listing the bootstap roles and policies

CF stack adf-adf-baseline created by ADF, currently with status update_complete and listing the baseline roles

  1. Instructions how to reproduce the problem in the deployment account

Create new CodeCommit repo adf-config which contain the AWS Config resources for all regions. It have two files:

buildspec.yml has been taken from ADF sample-iam/buildspec.yml

template.yml is the CF template that has been tested without ADF. It basically have a ConfigurationRecorder, a DeliveryChannel and few ConfigRules

Add new file in the ADF repo 'aws-deployment-framework-pipelines'

deployment_maps/adf-config.yml

pipelines:
  - name: adf-config
    type: cc-cloudformation
    params:
      - SourceAccountId: 84XXXXXX322
      - NotificationEndpoint: [email protected]
    targets:
      - path: /customers
        regions:
          - us-east-1
          - eu-central-1
          - eu-west-1

This file triggers the pipeline aws-deployment-framework-pipelines and stage CreateOrUpdatePipelines failed with CodeBuild error

2019-07-10 21:47:20,511 | INFO | cloudformation | 841XXXXX22 - Waiting for CloudFormation stack: adf-pipeline-adf-config in us-east-1 to reach stack_create_complete | (cloudformation.py:130) 
Traceback (most recent call last): 
  File "./adf-build/generate_pipelines.py", line 173, in <module> 
    main() 
  File "./adf-build/generate_pipelines.py", line 169, in main 
    cloudformation.create_stack() 
  File "/codebuild/output/src851065129/src/adf-build/shared/python/cloudformation.py", line 261, in create_stack 
    self._execute_change_set(waiter) 
  File "/codebuild/output/src851065129/src/adf-build/shared/python/cloudformation.py", line 255, in _execute_change_set 
    self._wait_stack(waiter) 
  File "/codebuild/output/src851065129/src/adf-build/shared/python/cloudformation.py", line 137, in _wait_stack 
    'MaxAttempts': 45 
  File "/usr/local/lib/python3.7/site-packages/botocore/waiter.py", line 53, in wait 
    Waiter.wait(self, **kwargs) 
  File "/usr/local/lib/python3.7/site-packages/botocore/waiter.py", line 323, in wait 
    last_response=response, 
botocore.exceptions.WaiterError: Waiter StackCreateComplete failed: Waiter encountered a terminal failure state 
 
[Container] 2019/07/10 21:47:50 Command did not exit successfully python ./adf-build/generate_pipelines.py exit status 1 
[Container] 2019/07/10 21:47:50 Phase complete: BUILD State: FAILED 
[Container] 2019/07/10 21:47:50 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: python ./adf-build/generate_pipelines.py. Reason: exit status 1

Also a CF stack 'adf-pipeline-adf-config' has been created by ADF which failed with status rollback_complete because of

An error occurred while validating the artifact bucket 'adf-regional-base-deploy-deploymentframeworkregio-webjfbrobbfn': No bucket with the name adf-regional-base-deploy-deploymentframeworkregio-webjfbrobbfn was found. Choose a valid artifact bucket in 'eu-west-1', or create a new artifact bucket to use in your pipeline. (Service: AWSCodePipeline; Status Code: 400; Error Code: InvalidStructureException;

from aws-deployment-framework.

bundyfx avatar bundyfx commented on July 16, 2024

Thanks @azamin

I will try re-create this error and keep you updated. Cheers!

from aws-deployment-framework.

azamin avatar azamin commented on July 16, 2024

@bundyfx , no need to reproduce it, because in one of the remote regions there was a CF stack which was pointing to a non-existing bucket.
Solution is to recreate either the missing bucket or the whole stack adf-regional-base-deployment in that specific region.

Thank you and apologies for the troubles.

from aws-deployment-framework.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.