zeebrow / quickhost Goto Github PK

Make a publicly available server, quickly

License: GNU General Public License v3.0

Python 92.39% Shell 7.61%

quickhost's Introduction

quickhost

Usage

$ main.py null -h
usage: main.py null [-h] {init,make,describe,update,destroy,list-all,destroy-all} ...

positional arguments:
  {init,make,describe,update,destroy,list-all,destroy-all}
    init                plugin initialization help
    make                make an app help
    describe            show details about an app help
    update              change an app help
    destroy             destroy an app help
    list-all            list all running apps
    destroy-all         remove the plugin help

options:
  -h, --help            show this help message and exit

Build

Python 3.7, 3.8

i dunno tox :(

install Python, e.g.

sudo apt-get install python3.8

install pip

sudo apt install python3.8-distutils
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python3.8 get-pip.py

and your choice of venv

sudo apt install python3.8-venv

pyinstaller

do this

git clone https://github.com/zeebrow/quickhost.git
git clone https://github.com/zeebrow/quickhost-aws.git plugins/aws
python3 -m venv venv && source venv/bin/activate
pip install -e quickhost
pip install -e quickhost-plugins/plugins/aws

quickhost's People

Contributors

Watchers

quickhost's Issues

fd1, fd2, rc for `make`

Host starts ok but fails the exit

starting hosts...
*****************
2022-07-12 10:27:36,980 : quickhost.temp_data_collector : store_test_data : DEBUG: 0.009253 of 0.026984 sec to write test data to mock-data/AWSHost/create (now 11.24 Kib)
===================Waiting on hosts for 'qwer'=========================
(1/1) Ready: ['i-0b99c8aa052c48c4e'] Waiting: (1): ['i-0b99c8aa052c48c4e']
Done
Traceback (most recent call last):
  File "/home/zeebrow/repos/github.com/zeebrow/quickhost-plugins/plugins/aws/venv/bin/main.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/zeebrow/repos/github.com/zeebrow/quickhost/src/scripts/main.py", line 136, in <module>
    rc, fd1, fd2= main()
TypeError: cannot unpack non-iterable int object

Document permissions required for `main.py describe`

Before starting, do #6 create first and see if those permissions work

Parent

Document permissions required for `main.py update`

main.py update is not yet implemented.

Parent

quickhost user should manage create and describe for `init aws`

policy template to get started with:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "InitDescribe",
            "Effect": "Allow",
            "Action": [
                "iam:GetUser",
                "ec2:DescribeTags",
                "ec2:DescribeVpcs",
                "ec2:DescribeSubnets",
                "ec2:DescribeRouteTables",
                "ec2:DescribeInternetGateways"
            ],
            "Resource": "*"
        },
        {
            "Sid": "InitCreate",
            "Effect": "Allow",
            "Action": [
                "ec2:CreateTags",
                "ec2:CreateVpc",
                "ec2:CreateSubnet",
                "ec2:CreateRouteTable",
                "ec2:CreateRoute",
                "ec2:AssociateRouteTable",
                "ec2:AttachInternetGateway",
                "ec2:CreateInternetGateway"
            ],
            "Resource": "*"
        }
    ]
}

Holy crap fix Windows 4/? - aws credentials

Do you really want to depend on the aws cli's ~/.aws/credentials file?
Maybe set something up with main.py init aws?
Create role to assume?

python versions 3.7, 3.6 fail

This should do the trick:

SNAKE_VER=3.7

cd $(mktemp -d)
git clone https://github.com/Zeebrow/quickhost.git
git clone https://github.com/Zeebrow/quickhost-plugins.git
cat <<EOF > Dockerfile
FROM python:$SNAKE_VER
COPY ./quickhost /app/quickhost
COPY ./quickhost-plugins/plugins/aws /app/aws
WORKDIR /app
RUN pip install --upgrade pip && \
  pip install -e quickhost && \
  pip install -e aws
ENTRYPOINT ["main.py"]
EOF
if [[ $(groups) =~ docker ]]; then
  docker build -t quickhost-test . && docker run quickhost-test make --aws asdf
else
  sudo docker build -t quickhost-test . && sudo docker run quickhost-test make --aws asdf

Document permissions required for `main.py make`

Parent

Document permissions required per-action

create - #6 (comment)
read - #7
update - #8
destroy - #9

Running in Powershell opens a new ... window thingy when main.py `console_script` is run from venv (venv\scripts\main.py)

python.exe -m virtualenv venv
.\venv\scripts\activate.ps1
pip install -r requirements.txt
pip install -e .
pip install -e ..\quickhost-plugins\
main.py

Remove all cases of `"Resource": "*"` from policies

Suggestions:

Utilize resource paths when possible
Create a permission boundary for the user
Tag all resources and use conditionals in policies

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_resource.html

`make` option to specify latest-windows ami

Add a flag --use-latest that accepts one of the possible operating systems (would be al2 and windows)

Notes

This will not affect the the functionality of other subparsers.
OS user will be 'Administrator'

The name of the administrator account depends on the language of the operating system. For example, for English, it's Administrator, for French it's Administrateur, and for Portuguese it's Administrador. For more information, see Localized Names for Administrator Account in Windows in the Microsoft TechNet Wiki.

Default open port is now OS-dependent

false-positive action results on 'windows-latest' hosts

action https://github.com/Zeebrow/quickhost/runs/7240185216 (Run a Command) does not produce any output after running main.py -h

`update` subparser

What does it need to do?

add/remove ports and cidrs for a specified app
exist

What do you want it to do?

run a Powershell/bash command

--profile init argument

Allow user to specify an existing AWS profile with which to create initial quickhost resources.

main.py init --aws --profile my-profile

`destroy` timeout gracefully

main.py destroy can take a significant amount of time to complete, and it blocks while running (see #31 ).

describe breaks when there are no running hosts for an app

and make it snappy

Document permissions required for `main.py destroy`

Parent

When reading input from the user, fix the prompt so it outputs to stdout

main.py 2>/dev/null will look like it's hanging indefinitely

new instances dont have a name

main.py make --aws asdf --region us-west-2 was run
output

main.py make --aws asdf --region us-west-2
2022-07-08 17:10:42,013 : root : main : DEBUG: loaded 1 plugins
2022-07-08 17:10:42,014 : root : main : DEBUG: first-pass cli_args={'config_file': None, 'aws_app_name': 'asdf', '__qhaction': 'make'}
2022-07-08 17:10:42,014 : root : main : DEBUG: first-pass action_cli_args=['--region', 'us-west-2']
2022-07-08 17:10:42,014 : root : main : DEBUG: action='make'
2022-07-08 17:10:42,015 : quickhost.QuickhostPlugin : load_plugin : DEBUG: Found plugin '[EntryPoint(name='quickhost_aws', value='quickhost_aws:load_plugin', group='quickhost_plugin')]'
2022-07-08 17:10:42,084 : root : main : DEBUG: make_parser params: make_args=Namespace(host_count=1, dry_run=False, ip=None, instance_type='t2.micro', ami=None, userdata=None, region='us-west-2')
2022-07-08 17:10:42,084 : quickhost_aws.AWSApp : run_make : DEBUG: make
args={'host_count': 1, 'dry_run': False, 'ip': None, 'instance_type': 't2.micro', 'ami': None, 'userdata': None, 'region': 'us-west-2'}
proceed? (y/n)y
args={'host_count': 1, 'dry_run': False, 'ip': None, 'instance_type': 't2.micro', 'ami': None, 'userdata': None, 'region': 'us-west-2'}
args={'host_count': 1, 'dry_run': False, 'ip': None, 'instance_type': 't2.micro', 'ami': None, 'userdata': None, 'region': 'us-west-2'}
2022-07-08 17:10:44,026 : quickhost_aws.utilities : get_single_result_id : DEBUG: Found 1 Vpc.
2022-07-08 17:10:44,194 : quickhost_aws.utilities : get_single_result_id : DEBUG: Found 1 Subnet.
2022-07-08 17:10:44,317 : quickhost_aws.utilities : get_single_result_id : DEBUG: Found 1 InternetGateway.
2022-07-08 17:10:44,876 : quickhost_aws.utilities : get_single_result_id : DEBUG: Found 1 RouteTable.
2022-07-08 17:10:46,548 : quickhost_aws.AWSKeypair : describe : ERROR: (InvalidKeyPair.NotFound): DescribeKeyPairs
2022-07-08 17:10:47,048 : quickhost_aws.AWSKeypair : create : DEBUG: saved private key to file '/home/zeebrow/repos/github.com/zeebrow/quickhost/asdf.pem'
Created new key pair (key-0c6dfd5eb1f50c73b)
2022-07-08 17:10:47,857 : quickhost_aws.AWSSG : _add_ingress : ERROR: (Security Group) Unhandled botocore client exception: (MissingParameter): Missing source specification: include source security group or CIDR information
No ami specified, getting latest al2...done (ami-082b5a644766e0e6f)

starting hosts...
*****************
===================Waiting on hosts for 'asdf'=========================
(1/1) Ready: ['i-0efb20917bbfe9c7c'] Waiting: (1): ['i-0efb20917bbfe9c7c']
Done
Traceback (most recent call last):
  File "/home/zeebrow/repos/github.com/zeebrow/quickhost/venv/bin/main.py", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/home/zeebrow/repos/github.com/zeebrow/quickhost/src/scripts/main.py", line 136, in <module>
    rc, fd1, fd2= main()
TypeError: cannot unpack non-iterable int object

aws ec2 describe-instances --instance-id i-0efb20917bbfe9c7c --region us-west-2 --profile quickhost-ci-admin

{
    "Reservations": [
        {
            "Groups": [],
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "ami-082b5a644766e0e6f",
                    "InstanceId": "i-0efb20917bbfe9c7c",
                    "InstanceType": "t2.micro",
                    "KeyName": "asdf",
                    "LaunchTime": "2022-07-08T21:10:50+00:00",
                    "Monitoring": {
                        "State": "disabled"
                    },
                    "Placement": {
                        "AvailabilityZone": "us-west-2b",
                        "GroupName": "",
                        "Tenancy": "default"
                    },
                    "PrivateDnsName": "ip-172-16-0-80.us-west-2.compute.internal",
                    "PrivateIpAddress": "172.16.0.80",
                    "ProductCodes": [],
                    "PublicDnsName": "",
                    "PublicIpAddress": "35.90.20.40",
                    "State": {
                        "Code": 16,
                        "Name": "running"
                    },
                    "StateTransitionReason": "",
                    "SubnetId": "subnet-0191c568cd83e23b2",
                    "VpcId": "vpc-08993887d03942a79",
                    "Architecture": "x86_64",
                    "BlockDeviceMappings": [
                        {
                            "DeviceName": "/dev/xvda",
                            "Ebs": {
                                "AttachTime": "2022-07-08T21:10:51+00:00",
                                "DeleteOnTermination": true,
                                "Status": "attached",
                                "VolumeId": "vol-0ed444390bee6c863"
                            }
                        }
                    ],
                    "ClientToken": "bfb541e7-503c-4c90-8114-4b23823c9db1",
                    "EbsOptimized": false,
                    "EnaSupport": true,
                    "Hypervisor": "xen",
                    "NetworkInterfaces": [
                        {
                            "Association": {
                                "IpOwnerId": "amazon",
                                "PublicDnsName": "",
                                "PublicIp": "35.90.20.40"
                            },
                            "Attachment": {
                                "AttachTime": "2022-07-08T21:10:50+00:00",
                                "AttachmentId": "eni-attach-0be1ae3a3daa54462",
                                "DeleteOnTermination": true,
                                "DeviceIndex": 0,
                                "Status": "attached",
                                "NetworkCardIndex": 0
                            },
                            "Description": "",
                            "Groups": [
                                {
                                    "GroupName": "asdf",
                                    "GroupId": "sg-0f2293c37a3a6ede1"
                                }
                            ],
                            "Ipv6Addresses": [],
                            "MacAddress": "02:ae:c5:3e:73:81",
                            "NetworkInterfaceId": "eni-07f419337789d2f92",
                            "OwnerId": "865386952527",
                            "PrivateIpAddress": "172.16.0.80",
                            "PrivateIpAddresses": [
                                {
                                    "Association": {
                                        "IpOwnerId": "amazon",
                                        "PublicDnsName": "",
                                        "PublicIp": "35.90.20.40"
                                    },
                                    "Primary": true,
                                    "PrivateIpAddress": "172.16.0.80"
                                }
                            ],
                            "SourceDestCheck": true,
                            "Status": "in-use",
                            "SubnetId": "subnet-0191c568cd83e23b2",
                            "VpcId": "vpc-08993887d03942a79",
                            "InterfaceType": "interface"
                        }
                    ],
                    "RootDeviceName": "/dev/xvda",
                    "RootDeviceType": "ebs",
                    "SecurityGroups": [
                        {
                            "GroupName": "asdf",
                            "GroupId": "sg-0f2293c37a3a6ede1"
                        }
                    ],
                    "SourceDestCheck": true,
                    "Tags": [
                        {
                            "Key": "quickhost",
                            "Value": "asdf"
                        }
                    ],
                    "VirtualizationType": "hvm",
                    "CpuOptions": {
                        "CoreCount": 1,
                        "ThreadsPerCore": 1
                    },
                    "CapacityReservationSpecification": {
                        "CapacityReservationPreference": "open"
                    },
                    "HibernationOptions": {
                        "Configured": false
                    },
                    "MetadataOptions": {
                        "State": "applied",
                        "HttpTokens": "optional",
                        "HttpPutResponseHopLimit": 1,
                        "HttpEndpoint": "enabled",
                        "HttpProtocolIpv6": "disabled",
                        "InstanceMetadataTags": "disabled"
                    },
                    "EnclaveOptions": {
                        "Enabled": false
                    },
                    "PlatformDetails": "Linux/UNIX",
                    "UsageOperation": "RunInstances",
                    "UsageOperationUpdateTime": "2022-07-08T21:10:50+00:00",
                    "PrivateDnsNameOptions": {
                        "HostnameType": "ip-name",
                        "EnableResourceNameDnsARecord": false,
                        "EnableResourceNameDnsAAAARecord": false
                    },
                    "MaintenanceOptions": {
                        "AutoRecovery": "default"
                    }
                }
            ],
            "OwnerId": "865386952527",
            "ReservationId": "r-0803fb4ac654a1d98"
        }
    ]
}

gracefully handle keyboard interrput during 'waiting on hosts' to supress stack trace

Hide stack traces on keyboard interrupt

Keyboard interrupt anywhere should exit(130)
also pass a kind message to stderr

Improved exception handling

Exceptions need to be handled accurately and consistently. (I started handling them after the app was pretty much done)

logger should emit debug if the action did not produce an exception
warning when action didn't do what it was intended to, but should not stop the app from finishing its operation
- e.g. a security group doesn't exist when we go to destroy it
error when action didn't do what it was intended to, and app should not continue its operation
- e.g. IAM permissions
- generally seem to be caught in AWSApp.py as opposed to specific resources

There's too many to count at this point. Mark 'em as you see 'em.

using deprecated ami id's

Hello,

On March 31, 2022, EC2 announced that all public AMIs will have the deprecation time set to two years after their creation date [1]. Once an AMI is deprecated, it will no longer appear in DescribeImages [2] API calls for users that are not the owner of the AMI. Deprecating an AMI only reduces the visibility of the AMI in untargeted searches, but continues to be usable and available to you. Users of a deprecated AMI can continue to launch instances and describe the deprecated AMI using its ID.

We have identified that your public AMIs which will be deprecated over two weeks starting July 30, 2022, have been used for instance launches recently. While these AMIs remain available using the AMI IDs, they will not be visible on the public AMI catalog by default. This message is to notify you of the upcoming change, and does not require any action from you.

A list of old public AMIs you have used for instance launches and the corresponding Region can be found at the end of this notification.

To learn more about viewing and managing the deprecation time for your AMIs, please refer to the "Deprecate an AMI" documentation [3].

If you have any further questions or comments regarding this matter, please contact AWS Support [4].

[1] https://aws.amazon.com/about-aws/whats-new/2022/03/amazon-machine-images-public-visibility-two-years/
[2] https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeImages.html
[3] https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ami-deprecate.html
[4] https://aws.amazon.com/support

Public AMIs | Region

ami-0b898040803850657 | US-EAST-1

Confirmed this ami is being run by main.py describe.

DescribeAmis is called only when new instances are made. I don't think any action is required on my part, but it is worth checking on, and hopefully refine it a wee bit.

--region init argument

Allow user to specify the region in which to initialize quickhost resources.

main.py init --aws --region=eu-west-1

Holy crap fix Windows 2/4 - unix paths --> pathlib.Path() paths

see setup.py changes

Available help options need overhaul

There is no way to figure out how this app is actually working, without making assumptions and while listening to output of a debug run. Documentation will not necessarily solve the immediate issues

main.py --help and main.py [action] --help display the same (incorrect) output
Getting help for a particular plugin's case is far from intuitive (documentation)
The only way to get --help is inherently broken (but still available anyways)

This is an issue for child plugin subparsers. The best I can understand is every parser is a parent parser, and in the case of main.py, the parent parser will always need some sort of child parser for complete (useful) output.

Configuration/preference option to use the same ssh key for all apps

cli_run_* functions should return stdout, stderr, and returncode for `main.py`

Holy crap fix Windows 3/4 - config file location

Path depends on os? Dotfile in user home dir ok?? Or use %APP_DATA% or whatever???

fd1, fd2, rc for CLI API

Description

Plugin functions whose return values are handled by main.py's cli_main() function need to return appropriate values for stderr, stdout and rc.

Animation vs. static text
- detect file vs tty output
- imagine apt output to a file:

Reading dependencies!... 10%
Reading dependencies!... 30%
Reading dependencies!... 30%
...

Plan for varying verbosity levels.
Always aim for less verbosity than Ansible

create

wait_for_hosts is animated

destroy

wait_for_hosts is animated

update

asdkfueyaaes
I don't wanna talk about it

describe

I think a good starting point is outputting some json to sys.stdout., logging warnings/errors with logging.StreamHandler (defaults to sys.stderr, fd 2, for all log methods, at all log levels), and setting rc=0 if there are none.

Working with file-like output also implies some different scenarios for handling file descriptors (Linux) and whatever (Windows):

main.py passes file descriptors to the plugin's describe method, and the plugin does the printing

Each plugin would potentially output differently, but each plugin will have a different definition of what properties describe an 'app'.
gives plugin the opportunity to log to something like Cloudwatch or S3

main.py receives a class instance from the plugin

plugins would need to implement or subclass from the quickhost package, which does not exist.
Simpler plugins (I like)
Standardized output format

After talking it over with myself, I think this guy has some really good ideas, regarding 2). Either way, I'm not just throwing it in a dict as the api's end result.

zeebrow / quickhost Goto Github PK

quickhost's Introduction

quickhost

Usage

Build

Python 3.7, 3.8

pyinstaller

do this

quickhost's People

Contributors

Watchers

quickhost's Issues

Notes

Description

create

destroy

update

describe

Recommend Projects

Recommend Topics

Recommend Org