kislyuk / aegea Goto Github PK

...
...
Processing triggers for libc-bin (2.24-11+deb9u4) ...
Creating volume
An error occurred (RequestLimitExceeded) when calling the CreateTags operation (reached max retries: 4): Request limit exceeded.

aws ecr delete-repository --repository-name dex

Traceback (most recent call last):
  File "/Users/bek/.pyenv/versions/2.7.13/bin/aegea", line 23, in <module>
    aegea.main()
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/site-packages/aegea/__init__.py", line 80, in main
    result = parsed_args.entry_point(parsed_args)
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/site-packages/aegea/build_docker_image.py", line 107, in build_docker_image
    job = submit(submit_args)
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/site-packages/aegea/batch.py", line 265, in submit
    ensure_queue(args.queue)
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/site-packages/aegea/batch.py", line 242, in ensure_queue
    create_compute_environment(cce_parser.parse_args(args=[name]))
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/site-packages/aegea/batch.py", line 103, in create_compute_environment
    serviceRole=batch_iam_role.arn)
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/site-packages/botocore/client.py", line 557, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the CreateComputeEnvironment operation: Object already exists
Traceback (most recent call last):
  File "/Users/bek/.pyenv/versions/2.7.13/bin/aegea-build-image-for-mission", line 42, in <module>
    env=dict(os.environ, AEGEA_CONFIG_FILE=os.path.join(mission_wd, "config.yml"))
  File "/Users/bek/.pyenv/versions/2.7.13/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[u'aegea', u'build_docker_image', 'dex', u'--tags', u'AegeaMission=docker-example']' returned non-zero exit status 1

INFO:aegea:Launching <aegea.util.aws.SpotFleetBuilder object at 0x10f6c9908: {'cores': 1, 'dry_run': False, 'gpus_per_instance': 0, 'iam_fleet_role': iam.Role...otFleet'), ...}>
Traceback (most recent call last):
  File "/Users/kislyuk/projects/mutt/build/mutt-master/VE/bin/aegea", line 23, in <module>
    aegea.main()
  File "/Users/kislyuk/projects/mutt/build/mutt-master/VE/lib/python2.7/site-packages/aegea/__init__.py", line 58, in main
    result = parsed_args.entry_point(parsed_args)
  File "/Users/kislyuk/projects/mutt/build/mutt-master/VE/lib/python2.7/site-packages/aegea/launch.py", line 142, in launch
    instance.wait_until_running()
  File "/Users/kislyuk/projects/mutt/build/mutt-master/VE/lib/python2.7/site-packages/boto3/resources/factory.py", line 369, in do_waiter
    waiter(self, *args, **kwargs)
  File "/Users/kislyuk/projects/mutt/build/mutt-master/VE/lib/python2.7/site-packages/boto3/resources/action.py", line 202, in __call__
    response = waiter.wait(**params)
  File "/Users/kislyuk/projects/mutt/build/mutt-master/VE/lib/python2.7/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/Users/kislyuk/projects/mutt/build/mutt-master/VE/lib/python2.7/site-packages/botocore/waiter.py", line 321, in wait
    last_response=response,
botocore.exceptions.WaiterError: Waiter InstanceRunning failed: Waiter encountered a terminal failure state

docker run DOCKER_IMAGE_LOCATION /bin/bash -c "COMMAND_PASSED TO AEGEA HERE"

systemd[1]: dev-xvdz.device: Job dev-xvdz.device/start timed out.
systemd[1]: Timed out waiting for device /dev/xvdz.

import aegea
aegea.initialize()
from aegea.batch import submit, submit_parser
submit(submit_parser.parse_args([...]))

import aegea
from aegea.batch import submit, submit_parser
aegea.initialize()
submit(submit_parser.parse_args([...]))

kislyuk@Aurora:~/projects/aegea>aegea launch aktest4 --duration-hours 5
INFO:tweak:Loaded configuration from /Users/kislyuk/.config/aegea/config.yml
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:boto3.resources.collection:Calling ec2:describe_key_pairs with {}
INFO:boto3.resources.collection:Calling ec2:describe_images with {'Owners': ['self'], 'Filters': [{'Name': 'state', 'Values': ['available']}]}
INFO:boto3.resources.collection:Calling ec2:describe_vpcs with {'Filters': [{'Name': 'isDefault', 'Values': ['true']}]}
INFO:boto3.resources.collection:Calling ec2:describe_subnets with {'Filters': [{'Name': 'vpc-id', 'Values': ['vpc-bc9b15d7']}]}
INFO:boto3.resources.collection:Calling ec2:describe_security_groups with {'GroupNames': ['aegea.launch'], 'Filters': [{'Name': 'vpc-id', 'Values': ['vpc-bc9b15d7']}]}
INFO:boto3.resources.collection:Calling paginated iam:list_instance_profiles with {}
INFO:boto3.resources.collection:Calling paginated iam:list_roles with {}
INFO:boto3.resources.action:Calling iam:attach_role_policy with {'RoleName': 'aegea.launch', 'PolicyArn': 'arn:aws:iam::aws:policy/IAMReadOnlyAccess'}
INFO:boto3.resources.action:Calling iam:attach_role_policy with {'RoleName': 'aegea.launch', 'PolicyArn': 'arn:aws:iam::aws:policy/AmazonElasticFileSystemFullAccess'}
INFO:boto3.resources.collection:Calling paginated iam:list_roles with {}
INFO:boto3.resources.action:Calling iam:create_role with {'RoleName': 'SpotFleet', 'AssumeRolePolicyDocument': '{"Version": "2012-10-17", "Statement": [{"Action": ["sts:AssumeRole"], "Principal": {"Service": "spotfleet.amazonaws.com"}, "Effect": "Allow"}]}'}
INFO:boto3.resources.action:Calling iam:attach_role_policy with {'RoleName': 'SpotFleet', 'PolicyArn': 'arn:aws:iam::aws:policy/service-role/AmazonEC2SpotFleetRole'}
INFO:boto3.resources.action:Calling iam:get_role with {'RoleName': 'SpotFleet'}
INFO:aegea:Launching <aegea.util.aws.SpotFleetBuilder object at 0x109c1e438: {'cores': 1, 'dry_run': False, 'gpus_per_instance': 0, 'iam_fleet_role': iam.Role(name='SpotFleet'), ...}>
Traceback (most recent call last):
  File "/usr/local/bin/aegea", line 24, in <module>
    args.entry_point(args)
  File "/Users/kislyuk/projects/aegea/aegea/launch.py", line 117, in launch
    expect_error_codes(e, "DryRunOperation")
  File "/Users/kislyuk/projects/aegea/aegea/launch.py", line 96, in launch
    sfr_id = spot_fleet_builder(ec2.meta.client)
  File "/Users/kislyuk/projects/aegea/aegea/util/aws.py", line 351, in __call__
    res = client.request_spot_fleet(DryRun=self.dry_run, SpotFleetRequestConfig=self.spot_fleet_request_config, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 228, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.5/site-packages/botocore/client.py", line 492, in _make_api_call
    raise ClientError(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidSpotFleetRequestConfig) when calling the RequestSpotFleet operation: Parameter: SpotFleetRequestConfig.IamFleetRole is invalid.

2021-09-30 02:20:48+00:00 Detaching EBS volume vol-0d10807d4007a84bf
2021-09-30 02:20:49+00:00 umount: /mnt: target is busy
2021-09-30 02:20:49+00:00         (In some cases useful info about processes that
2021-09-30 02:20:49+00:00          use the device is found by lsof(8) or fuser(1).)
2021-09-30 02:20:49+00:00 Traceback (most recent call last):
2021-09-30 02:20:49+00:00   File "/usr/local/bin/aegea", line 23, in <module>
2021-09-30 02:20:49+00:00     aegea.main()
2021-09-30 02:20:49+00:00   File "/usr/local/lib/python3.5/dist-packages/aegea/__init__.py", line 89, in main
2021-09-30 02:20:49+00:00     result = parsed_args.entry_point(parsed_args)
2021-09-30 02:20:49+00:00   File "/usr/local/lib/python3.5/dist-packages/aegea/ebs.py", line 177, in detach
2021-09-30 02:20:49+00:00     subprocess.check_call(["umount", find_devnode(volume_id)])
2021-09-30 02:20:49+00:00   File "/usr/lib/python3.5/subprocess.py", line 271, in check_call
2021-09-30 02:20:49+00:00     raise CalledProcessError(retcode, cmd)
2021-09-30 02:20:49+00:00 subprocess.CalledProcessError: Command '['umount', '/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol0d10807d4007a84bf-ns-1']' returned non-zero exit status 32
INFO:aegea:Job 6a0d141f-67e4-4510-998e-0a36ff9fd833: Essential container in task exited

 ✘  Tue 11 Dec - 09:01  ~/code/packer-images   origin ☊ olgabot/update-reflow ✔ 1⚙ 3☀ 
  aegea launch --ami-tags Name=czbiohub-reflow -t t2.micro  --iam-role S3fromEC2 olgabot-reflow-v08
Identity added: /Users/olgabot/.ssh/aegea.launch.olgabot.Olgas-MacBook-Pro.pem (/Users/olgabot/.ssh/aegea.launch.olgabot.Olgas-MacBook-Pro.pem)
INFO:aegea:Launch spec user data is 1878 bytes long
INFO:aegea:Launched ec2.Instance(id='i-030141a3fd6f30b43') in ec2.Subnet(id='subnet-672e832e') using ami-0969d6307208434cb
{
  "instance_id": "i-030141a3fd6f30b43"
}
(base) 
 ✘  Tue 11 Dec - 09:13  ~/code/packer-images   origin ☊ olgabot/update-reflow 1⚙ 3☀ 1● 
  aegea ssh ubuntu@olgabot-reflow-v08                                                                     
Permission denied (publickey).

$ aegea ssh foo@bar
Traceback (most recent call last):
  File "/home/andrey.kislyuk/.local/bin/aegea", line 23, in <module>
    aegea.main()
  File "/home/andrey.kislyuk/.local/lib/python3.6/site-packages/aegea/__init__.py", line 82, in main
    result = parsed_args.entry_point(parsed_args)
  File "/home/andrey.kislyuk/.local/lib/python3.6/site-packages/aegea/ssh.py", line 228, in ssh
    ssh_opts += init_ssm(get_instance(name).id)
  File "/home/andrey.kislyuk/.local/lib/python3.6/site-packages/aegea/ssh.py", line 217, in init_ssm
    ssm_plugin_path = ensure_session_manager_plugin()
  File "/home/andrey.kislyuk/.local/lib/python3.6/site-packages/aegea/util/aws/ssm.py", line 47, in ensure_session_manager_plugin
    elif "Ubuntu" in subprocess.run(["uname", "-a"], capture_output=True).stdout.decode():  # type: ignore
  File "/usr/lib/python3.6/subprocess.py", line 423, in run
    with Popen(*popenargs, **kwargs) as process:
TypeError: __init__() got an unexpected keyword argument 'capture_output'

Traceback (most recent call last):
  File "/usr/local/bin/aegea", line 23, in 
    aegea.main()
  File "/Library/Python/2.7/site-packages/aegea/__init__.py", line 80, in main
    result = parsed_args.entry_point(parsed_args)
  File "/Library/Python/2.7/site-packages/aegea/build_docker_image.py", line 78, in build_docker_image
    ensure_ecr_repo(args.name, read_access=args.read_access)
  File "/Library/Python/2.7/site-packages/aegea/build_docker_image.py", line 70, in ensure_ecr_repo
    clients.ecr.set_repository_policy(repositoryName=name, policyText=str(policy))
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 253, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Library/Python/2.7/site-packages/botocore/client.py", line 544, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the SetRepositoryPolicy operation: Invalid parameter at 'PolicyText' failed to satisfy constraint: 'Invalid repository policy provided'
Traceback (most recent call last):
  File "/usr/local/bin/aegea-build-image-for-mission", line 42, in 
    env=dict(os.environ, AEGEA_CONFIG_FILE=os.path.join(mission_wd, "config.yml"))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '[u'aegea', u'build_docker_image', 'dex', u'--tags', u'AegeaMission=docker-example']' returned non-zero exit status 1

Traceback (most recent call last):
  File "/usr/local/bin/aegea", line 23, in <module>
    aegea.main()
  File "/Users/andrey.kislyuk/projects/aegea/aegea/__init__.py", line 86, in main
    result = parsed_args.entry_point(parsed_args)
  File "/Users/andrey.kislyuk/projects/aegea/aegea/launch.py", line 190, in launch
    umbrella_policy = compose_managed_policies(args.iam_policies)
  File "/Users/andrey.kislyuk/projects/aegea/aegea/util/aws/iam.py", line 147, in compose_managed_policies
    doc = resources.iam.Policy(arn="arn:aws:iam::aws:policy/" + policy_name).default_version.document
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 431, in get_reference
    self.load()
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 386, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.9/site-packages/botocore/client.py", line 705, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetPolicy operation: User: arn:aws:sts::1234567890123:assumed-role/ROLE is not authorized to perform: iam:GetPolicy on resource: policy arn:aws:iam::aws:policy/IAMReadOnlyAccess

botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the SubmitJob operation: Error executing request, Exception : Command field cannot have empty strings, RequestId: b1aa9dd3-b9cb-4c7d-9e52-72a043b3519e

$ aegea launch --subnet subnet-xxxx-xxxxx davidrc-dev-2

Identity added: /Users/david.rissatocruz/.ssh/aegea.launch.david.rissatocruz.xxxxxxx.pem (/Users/david.rissatocruz/.ssh/aegea.launch.david.rissatocruz.xxxxxxx.pem)
INFO:aegea:Launch spec user data is 1901 bytes long

(... command was stuck here ...)

aws ec2 get-spot-placement-scores --target-capacity 1 --instance-types m5d.8xlarge --single-availability-zone

[staging]
aws_access_key_id = XXXXX
aws_secret_access_key = XXXXX

[production]
aws_access_key_id = XXXX
aws_secret_access_key = XXXX