lyft / awspricing Goto Github PK

View Code? Open in Web Editor NEW

207.0 20.0 54.0 62 KB

Python library for AWS pricing.

License: Other

Makefile 0.18% Python 99.82%

lyft

awspricing's Introduction

Lyft Open Source Website

This is code for oss.lyft.com website.

Technology stack

Hacking

You need to have node and yarn installed.

Clone the repo and run following command to start the development server:

GITHUB_TOKEN=<github-token> yarn start

💡 Get a new GitHub access token here!

Deploy

Everything merged to the develop branch is automatically deployed via TravisCI.

awspricing's People

Contributors

Stargazers

Watchers

awspricing's Issues

Getting an offer just hangs

So I was trying the code from the README.rst and couldn't even get the initial offer object back after several minutes; the 'offer' call just hangs and I had to hit ctrl-c to get the prompt back. I tried a using the 'version' argument to try an older API version but that gave the same results. I'm using 1.1.4. I'm not really sure where to go from here.

Python 3.6.6 (default, Sep 12 2018, 18:26:19) 
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import awspricing
>>> ec2 = awspricing.offer("AmazonEC2")
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/awspricing/__init__.py", line 95, in offer
    offer_data = _fetch_offer(service_name, version=version)
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/awspricing/__init__.py", line 81, in _fetch_offer
    resp = session.get(OFFER_BASE_URL + offer_endpoint)
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/requests/sessions.py", line 686, in send
    r.content
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/requests/models.py", line 828, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/requests/models.py", line 750, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/urllib3/response.py", line 494, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/home/tom/vpy3/aws/lib/python3.6/site-packages/urllib3/response.py", line 442, in read
    data = self._fp.read(amt)
  File "/usr/lib/python3.6/http/client.py", line 449, in read
    n = self.readinto(b)
  File "/usr/lib/python3.6/http/client.py", line 493, in readinto
    n = self.fp.readinto(b)
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1009, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 871, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
KeyboardInterrupt

Can't see /tmp/awspricing file

The documentation says the default path is/tmp/awspricing but I can't see it.

Needs API documentation

Great library, thank you.

While the Usage section is helpful, it would be good to provide complete documentation for the API.

new attribute capacitystatus causes collisions.

Since the release of : https://aws.amazon.com/about-aws/whats-new/2018/10/Amazon-EC2-now-offers-On-Demand-Capacity-Reservations/

For the ec2 products there is now a capacitystatus attribute that can be either UnusedCapacityReservation, AllocatedCapacityReservation or Used.

{u'attributes': {u'capacitystatus': u'UnusedCapacityReservation',
                 u'clockSpeed': u'Up to 3.3 GHz',
                 u'currentGeneration': u'Yes',
                 u'ecu': u'Variable',
                 u'instanceFamily': u'General purpose',
                 u'instanceType': u't2.medium',
                 u'instancesku': u'2ES9C4RF3WGQZAQN',
                 u'licenseModel': u'No License required',
                 u'location': u'US West (Oregon)',
                 u'locationType': u'AWS Region',
                 u'memory': u'4 GiB',
                 u'networkPerformance': u'Low to Moderate',
                 u'normalizationSizeFactor': u'2',
                 u'operatingSystem': u'Linux',
                 u'operation': u'RunInstances',
                 u'physicalProcessor': u'Intel Xeon Family',
                 u'preInstalledSw': u'NA',
                 u'processorArchitecture': u'32-bit or 64-bit',
                 u'processorFeatures': u'Intel AVX; Intel Turbo',
                 u'servicecode': u'AmazonEC2',
                 u'servicename': u'Amazon Elastic Compute Cloud',
                 u'storage': u'EBS only',
                 u'tenancy': u'Shared',
                 u'usagetype': u'USW2-UnusedBox:t2.medium',
                 u'vcpu': u'2'},
 u'productFamily': u'Compute Instance',
 u'sku': u'EUSSFEHB68A6B6DM'}
{u'attributes': {u'capacitystatus': u'Used',
                 u'clockSpeed': u'Up to 3.3 GHz',
                 u'currentGeneration': u'Yes',
                 u'ecu': u'Variable',
                 u'instanceFamily': u'General purpose',
                 u'instanceType': u't2.medium',
                 u'licenseModel': u'No License required',
                 u'location': u'US West (Oregon)',
                 u'locationType': u'AWS Region',
                 u'memory': u'4 GiB',
                 u'networkPerformance': u'Low to Moderate',
                 u'normalizationSizeFactor': u'2',
                 u'operatingSystem': u'Linux',
                 u'operation': u'RunInstances',
                 u'physicalProcessor': u'Intel Xeon Family',
                 u'preInstalledSw': u'NA',
                 u'processorArchitecture': u'32-bit or 64-bit',
                 u'processorFeatures': u'Intel AVX; Intel Turbo',
                 u'servicecode': u'AmazonEC2',
                 u'servicename': u'Amazon Elastic Compute Cloud',
                 u'storage': u'EBS only',
                 u'tenancy': u'Shared',
                 u'usagetype': u'USW2-BoxUsage:t2.medium',
                 u'vcpu': u'2'},
 u'productFamily': u'Compute Instance',
 u'sku': u'2ES9C4RF3WGQZAQN'}
{u'attributes': {u'capacitystatus': u'AllocatedCapacityReservation',
                 u'clockSpeed': u'Up to 3.3 GHz',
                 u'currentGeneration': u'Yes',
                 u'ecu': u'Variable',
                 u'instanceFamily': u'General purpose',
                 u'instanceType': u't2.medium',
                 u'instancesku': u'2ES9C4RF3WGQZAQN',
                 u'licenseModel': u'No License required',
                 u'location': u'US West (Oregon)',
                 u'locationType': u'AWS Region',
                 u'memory': u'4 GiB',
                 u'networkPerformance': u'Low to Moderate',
                 u'normalizationSizeFactor': u'2',
                 u'operatingSystem': u'Linux',
                 u'operation': u'RunInstances',
                 u'physicalProcessor': u'Intel Xeon Family',
                 u'preInstalledSw': u'NA',
                 u'processorArchitecture': u'32-bit or 64-bit',
                 u'processorFeatures': u'Intel AVX; Intel Turbo',
                 u'servicecode': u'AmazonEC2',
                 u'servicename': u'Amazon Elastic Compute Cloud',
                 u'storage': u'EBS only',
                 u'tenancy': u'Shared',
                 u'usagetype': u'USW2-Reservation:t2.medium',
                 u'vcpu': u'2'},
 u'productFamily': u'Compute Instance',
 u'sku': u'J4QDBG6HHGV8ND4P'}

For the moment I am adding an additional filter to "def _generate_reverse_sku_mapping":

                if product['attributes']['capacitystatus'] != 'Used':
                    continue

to get around the problem.

MemoryError

Hi,

my test python code is:

import awspricing

ec2_offer = awspricing.offer('AmazonEC2')

p = ec2_offer.ondemand_hourly(
  't2.micro',
  operating_system='Linux',
  region='eu-west-1'
)

print(p) # 0.0126

The output is this error message here:

[ec2-user@ip-10-0-1-224 awspricing]$ python lambda_function.py
Traceback (most recent call last):
  File "lambda_function_1.py", line 3, in <module>
    ec2_offer = awspricing.offer('AmazonEC2')
  File "/usr/lib/python2.7/site-packages/awspricing/__init__.py", line 95, in offer
    offer_data = _fetch_offer(service_name, version=version)
  File "/usr/lib/python2.7/site-packages/awspricing/__init__.py", line 83, in _fetch_offer
    offer = resp.json()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 795, in json
    return json.loads(self.content.decode(encoding), **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
MemoryError

Please help.

Why i cant have a 1yr all upfront convertible EC2 instance?

awspricing/awspricing/offers.py

Line 371 in fd37598

if (lease_contract_length == EC2_LEASE_CONTRACT_LENGTH.ONE_YEAR and

In the AWS calculator i can get the price of such a instance (eg. m5.large 1yr All Upfront Convertible)

Excessive memory usage

The library uses a excessive amount of memory, at least 2GB reading from cached data and as much as 3.5GB reading fresh data from the API. This results in excessive initialization times even in cached mode (30 seconds or more).

The net effect results in this library being much less useful that it might otherwise be. For example, it's nearly impossible to use this from Lambda given the hard memory cap of 3GB and high cost of such configurations.

Much of this is surely a side effect of the limited nature of the underlying AWS Pricing data and APIs, but I believe it can be addressed. I'm going to look into alternative methods for both pulling down the initial data as well as breaking up the cache into shards of some flavor to avoid the need to load the entire json blob into memory at once.

Problem calling rds_offer.reserved_upfront()

rds_offer.reserved_upfront(
'db.t2.large',
'MySQL',
license_model='No license required',
deployment_option='Multi-AZ',
region='us-east-1',
lease_contract_length='1yr',
pucharse_option ='All Upfront',
)

TypeError: reserved_upfront() got an unexpected keyword argument 'pucharse_option'

Library stuck on retrieving pricing

Hello Lyft team! Thanks for putting out this library, it has been very useful for me and others I've referred you to.

Here is a simple code I'm trying to run (Used to work just fine)

from __future__ import print_function
import awspricing
print("Retrieving AWS pricing")
ec2_offer = awspricing.offer('AmazonEC2')
print("Retrieving AWS pricing")

And my execution goes like this:

$ python3 pricingtest.py
Retrieving AWS pricing

It has been like this for more than 6 hours (tried more than one).

Any ideas on what might be happening?

Cannot Find SKU for EC2s

(venv) Latsyrc$ pip install awspricing
Requirement already satisfied: awspricing in ./venv/lib/python3.7/site-packages (1.1.4)
Requirement already satisfied: requests<3.0.0,>=2.5.0 in ./venv/lib/python3.7/site-packages (from awspricing) (2.19.1)
Requirement already satisfied: six<2.0.0,>=1.0.0 in ./venv/lib/python3.7/site-packages (from awspricing) (1.11.0)
Requirement already satisfied: urllib3<1.24,>=1.21.1 in ./venv/lib/python3.7/site-packages (from requests<3.0.0,>=2.5.0->awspricing) (1.23)
Requirement already satisfied: idna<2.8,>=2.5 in ./venv/lib/python3.7/site-packages (from requests<3.0.0,>=2.5.0->awspricing) (2.7)
Requirement already satisfied: certifi>=2017.4.17 in ./venv/lib/python3.7/site-packages (from requests<3.0.0,>=2.5.0->awspricing) (2018.10.15)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./venv/lib/python3.7/site-packages (from requests<3.0.0,>=2.5.0->awspricing) (3.0.4)
(venv) Latsyrc$ python
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 20:42:06)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

import awspricing
ec2_offer = awspricing.offer('AmazonEC2')
ec2_offer.ondemand_hourly(
... 'cc2.8xlarge',
... region='us-east-1',
... operating_system='Linux'
... )
2.0
ec2_offer.ondemand_hourly(
... 'm1.small',
... region='us-east-1',
... operating_system='Linux'
... )
0.044
ec2_offer.ondemand_hourly(
... 't2.small',
... region='us-east-1',
... operating_system='Linux'
... )
Traceback (most recent call last):
File "", line 4, in
File "/Users/Latsyrc/venv/lib/python3.7/site-packages/awspricing/offers.py", line 208, in ondemand_hourly
region=region
File "/Users/Latsyrc/venv/lib/python3.7/site-packages/awspricing/offers.py", line 190, in get_sku
.format(attributes))
ValueError: Unable to lookup SKU for attributes: ['t2.small', 'Linux', 'Shared', 'No License required', 'NA', 'US East (N. Virginia)']

New SKU's causing issue while loading AmazonEC2 Offer

Starting today I started facing below exception.

I believe this is due to some new items in the pricing list that are not having expected attribute productFamily

>>> import awspricing
>>> ec2_offer = awspricing.offer('AmazonEC2')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/awspricing/__init__.py", line 73, in offer
    _SERVICES[service_name] = get_offer_class(service_name)(offer_data)
  File "/usr/local/lib/python3.6/site-packages/awspricing/offers.py", line 153, in __init__
    product_family='Compute Instance'
  File "/usr/local/lib/python3.6/site-packages/awspricing/offers.py", line 118, in _generate_reverse_sku_mapping
    product['productFamily'] != product_family):
KeyError: 'productFamily'

The following sku's were throwing this exception

B75HUAKSUPHX3NWU
BNSJSY9CBT29VNPD
938JE5UBESWKBXN6
2W29CZVGHFJSM2EP
JSGCS5TNC3DHUGCJ
9ZSADUN75K8PGUUJ
5ZG2UFVQ3SUN6AVM
QGCDJV4FY5CE3SV9
9TKMMUSFY2W5U6UG
YKHR5F96XYH97HG8
47XUN2XYDDPBGWKR
RUUR36CQRC4M2FAD
JHTFZ427P4NCQWA5
5HCBJW6DMVDMJR2X
5WJ57TH4PWEBYKWW
QBGJ9QGG7UZGKBC9
FUPAN6PQ4FTCK3Z7
82BSGUAKGDK4N5DU
DEQE2XYW4RTESQ7R
R2EFQZ2NMWAB4CJ9
AAR58FRHKUVYH9K4
3UDGB9DJTF7MK6GA
VYKVF7UK2662UDCX
BESEY7AJX8B8UJDW
25UTTAZ3BZS9UDPD
8A2VKD4DZQC5FM4Y
V6TH5APA7TWF9RWR
46CVPCNHA6C2JG7K
4CWZREBDMM4ZNG5E
NDKK4N82F98R6SWJ

Example of a few SKU's

"B75HUAKSUPHX3NWU": {"sku": "B75HUAKSUPHX3NWU", "attributes": {"servicecode": "AWSDataTransfer", "transferType": "Inter Region Peering Data Transfer Outbound", "fromLocation": "US East (N. Virginia)", "fromLocationType": "AWS Region", "toLocation": "External", "toLocationType": "A    WS Region", "usagetype": "USE1-AWS-Out-Bytes", "operation": "", "servicename": "AWS Data Transfer"}}

"BNSJSY9CBT29VNPD": {"sku": "BNSJSY9CBT29VNPD", "attributes": {"servicecode": "AWSDataTransfer", "transferType": "Inter Region Peering Data Transfer Inbound", "fromLocation": "Extern    al", "fromLocationType": "AWS Region", "toLocation": "US East (Ohio)", "toLocationType": "AWS Region", "usagetype": "USE2-AWS-In-Bytes", "operation": "", "servicename": "AWS Data Transfer"}}

@garrettheel if you are fine I can submit a fix for this and some new test cases

Can I request pricing for GovCloud?

Hello,

Thank you for very helpful module.

Is it possible to query GovCloud for pricing?
I have not been able to succeed so far.
The code below:
client = boto3.client('ec2', aws_access_key_id, aws_secret_access_key, region_name='us-gov-west-1')
ec2info = defaultdict()
ec2_offer = awspricing.offer('AmazonEC2')
gives this error:
botocore.exceptions.NoCredentialsError: Unable to locate credentials

Thanks,
Irina

add 1 region pricing

get only 1 region pricing list

Move to boto3 pricing API?

Hi,

I was looking for a wrapper for the boto3 pricing stuff and came across this, which pretty much does what i need but maybe with some caching mods for redis :)

Was wondering why this doesn't use the boto3 'pricing' api to get the data, and if it would offer a speedup and similar functionality if it did as it could do the queries directly rather than working on the downloaded data?

Add RDS offers

@garrettheel I've found this module really helpful and would like to contribute back by adding RDS offers. It will likely take me some time but wanted to check if your open to a contribution for RDS?