qsft / mixcoatl Goto Github PK

According to the Enstratius API documentation, Tier should be queried by Deployment ID.

Query Parameters
Any attempt to list tiers requires a deploymentId parameter. deploymentId [number]
Returns the tiers belonging to the deployment with the specified unique ID.

• from the API documentation page.190

usage syntax should be consistent with other dcm action commands (start, pause, terminate) where positional arguments are allowed

usage: dcm-stop-server [-h] serverid
usage: dcm-start-server [-h] serverid
usage: dcm-pause-server [-h] serverid
usage: dcm-terminate-server [-h] [--serverid SERVERID] [--reason REASON]

Missing automation/ConfigurationManagementSystem to Mixcoatl

How all() method works:

1. get whole list of resources.
2. iterate the list and send API calls to get data of each resource.

For example, let's say we have 900 volumes in a region. Mixcoatl will send an API call to get whole list of volumes. Then, it iterates the list and send 900 more API calls to map each resource to mixcoatl objects. Eventually, it will send 901 API calls to API endpoint.

66.57.3.53:~/Projects/mixcoatl$dcm-list-firewalls --all
+-------------+-----------+--------------------------------------+
| Firewall ID | Name | Provider ID |
+-------------+-----------+--------------------------------------+
| 300 | default | 6f41b590-c728-450f-8237-ca6bee31f5f3 |
| 301 | mydefault | be52a40a-e540-4432-bb95-531eb958dd54 |
+-------------+-----------+--------------------------------------+

66.57.3.53:/Projects/mixcoatl$dcm-list-jobs
+--------+-----------------------------+----------+------------------------------+------------------------------+----------------------------+
| Job ID | Description | Status | Start Date | End Date | Message |
+--------+-----------------------------+----------+------------------------------+------------------------------+----------------------------+
| 900 | Terminate 1006 | COMPLETE | 2014-05-22T19:37:35.843+0000 | 2014-05-22T19:39:04.427+0000 | None |
| 901 | Terminate 1005 | COMPLETE | 2014-05-22T19:37:41.926+0000 | 2014-05-22T19:38:36.424+0000 | None |
| 902 | Add Firewall Rule 0.0.0.0/0 | COMPLETE | 2014-05-22T19:38:20.903+0000 | 2014-05-22T19:38:36.422+0000 | ID: 800 Time: 15 seconds |
| 903 | Launch Server Test | COMPLETE | 2014-05-22T19:38:41.982+0000 | 2014-05-22T19:40:00.430+0000 | 1100 |
| 904 | CREATE VOLUME | COMPLETE | 2014-05-22T19:41:11.917+0000 | 2014-05-22T19:42:48.438+0000 | 1100 |
| 905 | Create Image cirros-image | COMPLETE | 2014-05-22T19:42:09.985+0000 | 2014-05-22T19:46:04.446+0000 | 900 |
| 906 | Remove Volume 1100 | COMPLETE | 2014-05-22T19:43:14.574+0000 | 2014-05-22T19:48:24.451+0000 | Volume 1100 removed |
+--------+-----------------------------+----------+------------------------------+------------------------------+----------------------------+
66.57.3.53:/Projects/mixcoatl$dcm-check-job 902
COMPLETE

List personalities/roles in DCM

Debug mode might mean things like printing to standard out the JSON/XML payloads that are made for the REST calls. Also printing out the responses. For example:

$ dcm-create-role --name "test role" --description "testing role"
{"addRole": [{"status": "ACTIVE", "name": "test role", "description": "testing role"}]}
{"roles": [{"roleId": 55500}]}
55500

or

$ dcm-create-group --name "test group" --description "testing group"
{"addGroup": {"group": {"name": "test group", "description": "testing group"}}}
{"groups": [{"groupId": 300}]}

This came up as a possible feature because it helps make understanding how payloads are constructed. Problem is, the payloads are unique to each resource and there's no way to set a global "debug" flag.

Json to rename a server

{
"describeServer":
[
{
"name":"GForghetti-CentOS-j",
}
]
}

put infrastructure/Server

usage: dcm-create-volume [-h](--budgetid BUDGETID | --budgetname BUDGETNAME)
[--datacenter DATACENTER] [--description DESCRIPTION]
[--name NAME] [--size SIZE]

[] brackets imply optional

datacenter, name and size are not optional
getting usage message if 'description' flag is not used

66.57.3.53:~/Projects/mixcoatl$dcm-list-networks --all
+------------+-----------+--------------------+--------------------------------------+----------+-----------------+
| Network ID | Region ID | Name | Provider ID | Type | Network Address |
+------------+-----------+--------------------+--------------------------------------+----------+-----------------+
| 301 | 300 | provider-network-1 | a52e2a98-498f-4bd7-859e-3a6273d7fa01 | STANDARD | 0.0.0.0/0 |
| 300 | 300 | tenant-network-1 | 8314aaf0-0ed9-4072-b9f0-9830a15f6b9e | STANDARD | 0.0.0.0/0 |
+------------+-----------+--------------------+--------------------------------------+----------+-----------------+

I don't know if I like this as a feature. If I'm understanding it correctly, you can pass in the 'name' of the budget code like 'Default' and we'll look up the ID of that code and then assign that to the volume.

Seems confusing

https://github.com/enStratus/mixcoatl/blob/master/bin/dcm-create-volume#L14

Not much else to say except repeating this: Mixcoatl needs a "dcm-create-machine-image"

Right now dcm-something-something calls return a "pretty print/table" or verbose response. It looks nice from a command line format perspective, but it makes composing more complex workflows/pipelines more difficult since you have to sed/awk your way through things.

So we should abstract the output formatting to a class that returns, based on user input:

1. JSON
2. XML (maybe? I hate XML)
3. CSV
4. simple columnar data
5. pretty table

66.57.3.53:~/Projects/mixcoatl$dcm-list-roles
+---------+-------------+--------------------+--------+
| Role ID | Role Name | Description | Status |
+---------+-------------+--------------------+--------+
| 55400 | Admin | General Admin Role | ACTIVE |
| 55502 | Production | Production | ACTIVE |
| 55501 | Testing | Testing | ACTIVE |
| 55500 | Development | Development | ACTIVE |
+---------+-------------+--------------------+--------+

Environment variables are suboptimal for setting endpoint, access_key, and secret_key. This information should be available in a configuration file in ~/.mixcoatl.cfg or something.

If there is no ~/.mixcoatl.cfg then we should attempt to fall back on env vars.

In kraken, when you create a user level api key, you must assign a "default" account for it to operate under. However, if the user has access to multiple accounts, there is currently no way of querying those using the dcm-action-resource type commands.

For example, accountID=201 is the default:

ewanlyall@Ewans-MBP:mixcoatl$ dcm-list-accounts
+------------+------------------+----------------+-----------+
| Account ID | Account Name | Default Budget | Status |
+------------+------------------+----------------+-----------+
| 201 | AWS | 200 | PERMANENT |
| 202 | Openstack | 200 | PERMANENT |
+------------+------------------+----------------+-----------+

ewanlyall@Ewans-MBP:mixcoatl$ dcm-list-regions
+-----------+----------------+-------+----------------+----------------+--------+
| Region ID | Provider ID | Cloud | Region Name | Description | Status |
+-----------+----------------+-------+----------------+----------------+--------+
| 10 | ap-northeast-1 | 1 | ap-northeast-1 | ap-northeast-1 | ACTIVE |
| 16 | ap-southeast-1 | 1 | ap-southeast-1 | ap-southeast-1 | ACTIVE |
| 15 | ap-southeast-2 | 1 | ap-southeast-2 | ap-southeast-2 | ACTIVE |
| 11 | eu-west-1 | 1 | eu-west-1 | eu-west-1 | ACTIVE |
| 1 | sa-east-1 | 1 | sa-east-1 | sa-east-1 | ACTIVE |
| 12 | us-east-1 | 1 | us-east-1 | us-east-1 | ACTIVE |
| 13 | us-west-1 | 1 | us-west-1 | us-west-1 | ACTIVE |
| 14 | us-west-2 | 1 | us-west-2 | us-west-2 | ACTIVE |
+-----------+----------------+-------+----------------+----------------+--------+

No regions are returned for accountID=202, same applies for dcm-list-servers etc.

The information can only be retrieved with dcm-get and iterating through each regionID, thus:

$ dcm-get infrastructure/Server "{'regionId':17}"

A test script that calls through each "read" mixcoatl call and validates that it works. Should be able to be pointed to any DCM API endpoint and see the results in an easy-to-digest format.

This should be a part of "meta-coatl"

I'd prefer left-justfied fields for cases like the ones shown in the snippet below:

https://git.int.enstratius.com/snippets/101

Implement a dcm-list-api-versions such that it returns a list of valid DCM API versions for the endpoint in question. To see this via a web browser try: http://ec2-54-190-7-59.us-west-2.compute.amazonaws.com:15000/api/enstratus

should return the billing code ID of the code created.

66.57.3.53:~/Projects/mixcoatl$dcm-list-server-products -r RegionOne
+-------------------+--------------------+---------------------+-----------+----------+----------+-------------+
| Server Product ID | Provider Region ID | Provider Product ID | Name | Platform | Currency | Hourly Rate |
+-------------------+--------------------+---------------------+-----------+----------+----------+-------------+
| 10000 | RegionOne | 6 | m1.custom | UNIX | USD | 0.14 |
+-------------------+--------------------+---------------------+-----------+----------+----------+-------------+

List available CM environments from DCM

Blocked by FB: 4760

Create and introduce metacoatl, a separate branch of utilities that depend on the "base" utiliteis and execute more complex workflows.

66.57.3.53:/Projects/mixcoatl$dcm-list-firewall-rules 208
Traceback (most recent call last):
File "/usr/local/bin/dcm-list-firewall-rules", line 22, in
rules = f.rules
File "/Library/Python/2.7/site-packages/mixcoatl/network/firewall.py", line 123, in rules
rls = FirewallRule.all(self.__firewall_id, detail=self.request_details)
File "/Library/Python/2.7/site-packages/mixcoatl/network/firewall_rule.py", line 179, in all
rule.load()
File "/Library/Python/2.7/site-packages/mixcoatl/resource.py", line 171, in load
raise AttributeError('Key found without accessor: %s' % k)
AttributeError: Key found without accessor: precedence
66.57.3.53:/Projects/mixcoatl$

When changing from the 2012-06-15 to 2013-01-29 API endpoint, you must specify both:

ES_ENDPOINT
ES_API_VERSION

Failure to do so will result in a bad signature generation as 2012-06-15 is still being used.

We discussed keeping the major version at 0, but implement something like: MAJOR.MINOR.PATCH. where MINOR = DCM release number, in this case 10. and PATCH be the DCM minor release number, which right now I think is 27. So we should be on version 0.10.27 of mixcoatl.

66.57.3.53:/Projects/mixcoatl$dcm-create-billing-code --name "Research" --code "Res" --soft 1000.00 --hard 5000.00 --description "Research Budget"
66.57.3.53:/Projects/mixcoatl$dcm-list-billing-codes
+-----+-------------+-------------+--------------+--------------+---------------+-----------------+--------+
| ID | Name | Budget Code | Soft Quota | Hard Quota | Current Usage | Projected Usage | Status |
+-----+-------------+-------------+--------------+--------------+---------------+-----------------+--------+
| 200 | Default | DEF | 0.00 | 0.00 | USD 421.59 | USD 912.21 | ACTIVE |
| 700 | Development | Dev | USD 1000.00 | USD 5000.00 | USD 0.50 | USD 0.50 | ACTIVE |
| 701 | Production | Prod | USD 10000.00 | USD 15000.00 | USD 0.00 | USD 0.00 | ACTIVE |
| 800 | Research | Res | USD 1000.00 | USD 5000.00 | USD 0.00 | USD 0.00 | ACTIVE |
| 702 | Testing | Test | USD 50.00 | USD 100.00 | USD 2.64 | USD 2.64 | ACTIVE |
+-----+-------------+-------------+--------------+--------------+---------------+-----------------+--------+
66.57.3.53:~/Projects/mixcoatl$

object storage is not activated in DCM v10.x

Fixed locally by adding public_ip_addresses block condition(see 'myfix') below this line

if hasattr(s,'public_ip_address'):
server_table.add_row(["Public IP Address", s.public_ip_address])

BEGIN myfix

elif hasattr(s,'public_ip_addresses'):
comma = ","
server_table.add_row(["Public IP Address", comma.join(s.public_ip_addresses)])

END myfix

   else:
        server_table.add_row(["Public IP Address", None])

Observed with EBS backed EC2 only, have not tested/tried in any other provider

dcm-pause-server 5225
3014

dcm-list-jobs -j 3014
Cloud does not support pausing operation on virtual machines.

Deployments are no longer supported in the current DCM releases and should be removed.

Effects.

• dcm-list-deployments
• dcm-create-deployment
• dcm-describe-deployment
• mixcoatl/mixcoatl/automation/deployment.py
• mixcoatl/mixcoatl/automation/service.py
• mixcoatl/mixcoatl/automation/tier.py
• mixcoatl/mixcoatl/automation/launch_configuration.py

Create a document that describes how to use git-flow to contribute to mixcoatl.

1. Server

   mixcoatl.decorators.lazy.LazyPropertyException: Key found without accessor: scripts
   mixcoatl.decorators.lazy.LazyPropertyException: Key found without accessor: architecture

2. Firewall

   AttributeError: Key found without accessor: owning_groups

3. Snapshot

   File "./es-list-snapshots.py", line 31, in
   snapshots = Snapshot.all()
   File "/Library/Python/2.7/site-packages/mixcoatl-0.2.6-py2.7.egg/mixcoatl/infrastructure/snapshot.py", line 262, in all
   snapshot.load()
   File "/Library/Python/2.7/site-packages/mixcoatl-0.2.6-py2.7.egg/mixcoatl/resource.py", line 171, in load
   raise AttributeError('Key found without accessor: %s' % k)
   AttributeError: Key found without accessor: owning_user

v0.2.9
Cloud: Amazon

➜ mixcoatl git:(dcm-list-firewalls) ✗ dcm-list-firewalls -a Traceback (most recent call last): File "/usr/local/bin/dcm-list-firewalls", line 5, in <module> pkg_resources.run_script('mixcoatl==0.2.9', 'dcm-list-firewalls') File "/Library/Python/2.7/site-packages/pkg_resources.py", line 534, in run_script self.require(requires)[0].run_script(script_name, ns) File "/Library/Python/2.7/site-packages/pkg_resources.py", line 1434, in run_script execfile(script_filename, namespace, namespace) File "/Library/Python/2.7/site-packages/mixcoatl-0.2.9-py2.7.egg/EGG-INFO/scripts/dcm-list-firewalls", line 22, in <module> firewalls = Firewall.all(detail='basic') File "/Users/gmoselle/projects/mixcoatl/mixcoatl/network/firewall.py", line 219, in all fw.load() File "/Users/gmoselle/projects/mixcoatl/mixcoatl/resource.py", line 171, in load raise AttributeError('Key found without accessor: %s' % k) AttributeError: Key found without accessor: legacy_owner_id

Create release notes for 0.3.4

66.57.3.53:/Projects/mixcoatl$dcm-list-clouds
Traceback (most recent call last):
File "/usr/local/bin/dcm-list-clouds", line 14, in
clouds = Cloud.all()
File "/Library/Python/2.7/site-packages/mixcoatl/geography/cloud.py", line 121, in all
cloud.load()
File "/Library/Python/2.7/site-packages/mixcoatl/resource.py", line 171, in load
raise AttributeError('Key found without accessor: %s' % k)
AttributeError: Key found without accessor: storage_x509_key_label
66.57.3.53:/Projects/mixcoatl$

dcm-describe-deployment 601
Traceback (most recent call last):
File "/usr/local/bin/dcm-describe-deployment", line 5, in
pkg_resources.run_script('mixcoatl==0.2.9', 'dcm-describe-deployment')
File "/Library/Python/2.7/site-packages/pkg_resources.py", line 534, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/Library/Python/2.7/site-packages/pkg_resources.py", line 1434, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Python/2.7/site-packages/mixcoatl-0.2.9-py2.7.egg/EGG-INFO/scripts/dcm-describe-deployment", line 15, in
result = d.load()
File "/Library/Python/2.7/site-packages/mixcoatl-0.2.9-py2.7.egg/mixcoatl/resource.py", line 171, in load
raise AttributeError('Key found without accessor: %s' % k)
AttributeError: Key found without accessor: dr_storage

An example of this is here:

java dcm_get 'analytics/ServerAnalytics/2304?  interval=3&periodStart=1417890600000&periodEnd=1418236200000'

Connecting to http://cmp1.ito.unisys.com:15000/api/enstratus/2014-01-13/analytics/ServerAnalytics/2304?interval=3&periodStart=1417890600000&periodEnd=1418236200000
Done.  HTTP/1.1 200 OK

<analytics>
<serverAnalytics serverId="2304"><intervalInMinutes>3</intervalInMinutes><periodStart>2014-12-06T18:41:58.678+0000</periodStart><periodEnd>2014-12-07T18:05:51.395+0000</periodEnd><dataPoints><serverData><timestamp>2014-12-06T18:41:58.678+0000</timestamp><resourceState>NORMAL</resourceState><operationalState>NORMAL</operationalState><icmpPingResponseTimeInMillis>-1</icmpPingResponseTimeInMillis><guest cpuLoad="-1.00" cpuUtilization="-1.00" processCount="-1" responseTimeInMillis="-2" cpuCount="0" ram````

not sure yet how to fit this into things but @gforghetti @bdwilliams might have more to add. Comments pls.

66.57.3.53:~/Projects/mixcoatl$dcm-list-regions
+-----------+-------------+-----------+-------------+-------------+--------+
| Region ID | Provider ID | Cloud | Region Name | Description | Status |
+-----------+-------------+-----------+-------------+-------------+--------+
| 300 | RegionOne | OpenStack | RegionOne | RegionOne | ACTIVE |
+-----------+-------------+-----------+-------------+-------------+--------+

Not much else to say, except I don't think the API supports un-deprecate yet.

Here's the json to deprecate an image:
{
"deprecateImage": [ { } ]
}

You do a put to infrastructure/MachineImage/xxxx

Need to add firewallId on the query as a parameter.

Create release notes for 0.3.5

66.57.3.53:~/Projects/mixcoatl$dcm-describe-server 906
+--------------------+-------------------------------+
| Field | Attribute |
+--------------------+-------------------------------+
| Server ID | 906 |
| Name | SM-Console-A |
| Machine Image ID | 601 |
| Machine Image Name | RHEL-6.4_GA-x86_64-10-Hourly2 |
| Architecture | I64 |
| Product Size | m1.medium |
| Public IP Address | None |
| Private IP Address | 10.28.244.150 |
| Region ID | 201 |
| Datacenter ID | 204 |
| Datacenter Name | us-east-1a |
| Budget ID | 200 |
| Owning User ID | None |
| Owning User Name | None |
| Owning User Email | None |
| Owning Group ID | None |
| Owning Group Name | None |
| Start Date | 2014-05-20T19:39:01.000+0000 |
| Description | SM-Console-A |
| Agent Version | 0 |
| Status | RUNNING |
+--------------------+-------------------------------+

Traceback (most recent call last):
File "/usr/local/bin/es-list-deployments", line 5, in
pkg_resources.run_script('enstratius-api-tools==0.1.4', 'es-list-deployments')
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 489, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources.py", line 1207, in run_script
execfile(script_filename, namespace, namespace)
File "/Library/Python/2.7/site-packages/enstratius_api_tools-0.1.4-py2.7.egg/EGG-INFO/scripts/es-list-deployments", line 26, in
deployment_table.add_row([d.deployment_id, d.name, d.owning_user['alpha_name'],
File "/Library/Python/2.7/site-packages/mixcoatl-0.2.6-py2.7.egg/mixcoatl/decorators/lazy.py", line 51, in get
raise LazyPropertyException(detail)
mixcoatl.decorators.lazy.LazyPropertyException: Key found without accessor: catalog_item

List of available DCM Load Balancers

implement a mechanism for displaying the version of mixcoatl being used.

something like:

mixcoatl --version

0.10.27

super bonus points if we also return the most recent version hosted on pypi or something.

Deployments are not an option in DCM >= 10.x. Tiers, launch configs need to go, probably some others as well.

List scripts/recipes in DCM