concurrencylabs / aws-cost-analysis Goto Github PK

Executing:
['python', 'report_utils.py', '--action=prepare-quicksight', '--source-bucket=foo', '--source-prefix=reports/cost-reports/', '--dest-bucket=bar', '--dest-prefix=quicksight/', '--year=2017', '--month=12']

Errors out:
Exception message:[Could not find manifest file in bucket:[foo], prefix:[reports/cost-reports/20171201-20171301/]]

Just increasing int(month) doesn't seem sufficient ;)

We have a very large CSV that's produced for the billing report from AWS, so it doesn't load in Excel for parsing there.

I did get the report_utils script to execute successfully with no errors and data does appear in Athena tables, but when I run:

FROM  hourly_20180901_20181001``` only returns a number that is ~ 20% of the actual expected amount shown by cost explorer.

Have you seen anything like this?
Would you have any suggestions as to where to look?

My AWS reports are being put into s3://<bucket>/<myprefix>/aws-cost-and-usage/ but the function works only if I dont have my own prefix. I had to edit it and do the following change to work out for me at utils.py:

-    period = dirs[len(dirs)-3]
+    period = dirs[len(dirs)-2]

-    for d in dirs[0:len(dirs)-3]:
+    for d in dirs[0:len(dirs)-2]:

Maybe there is a better fix to solve both cases #

Hello,
I'm getting the following error with python 3.6.5. It doesn't matter if I'm running it with or w/o parameters.

Can anyone help me with this?

Traceback (most recent call last): File "report_utils.py", line 7, in <module> import awscostusageprocessor.processor as cur File "/home/marcel/Desktop/AWS/aws-cost-analysis/awscostusageprocessor/processor.py", line 109 print "Number of records: [{}]".format(record_count) ^ SyntaxError: invalid syntax

Cheers

The README mentions a cloudformation/process-cur-sam.yml which sounds very helpful ... but it's missing in the repository. Could you add it?

Thanks for sharing these tools!

In the xacct-step-function-starter.py file you scan with the following filter attributes:

• lastProcessedTimestamp = 'Stamp'
• dataCollectionStatus = Active

You only write the lastProcessedTimestamp value in update-metadata.py and you read it in athena.py.

As far as i can see the xacct-step-function-starter.py and thus the Lambda function xAcctStepFunctionStarter which is scheduled to run every 5 minshave never effect due to the missing dataCollectionStatus attribute.

What is the best approach to run the detection of new reports via S3 events or scheduled events?

Hi,

I'm trying to setup this project using the CloudFormation template (PR is coming if i have it working) and in the StepFunction the following error is raised:

'VarCharValue': KeyError
Traceback (most recent call last):
File "/var/task/functions/init-athena-queries.py", line 37, in handler
result_dict['getTotalCost']['resultset'] = apiprocessor.getTotalCost()
File "/var/task/awscostusageprocessor/api.py", line 50, in getTotalCost
return self.getResultSet(consts.ACTION_GET_TOTAL_COST)
File "/var/task/awscostusageprocessor/api.py", line 42, in getResultSet
response['results'] = self.athena.get_query_execution_results(queryexecutionid)
File "/var/task/awscostusageprocessor/sql/athena.py", line 148, in get_query_execution_results
row_dict[rowheaders[columnindex]['VarCharValue']] = columnvalue['VarCharValue']
KeyError: 'VarCharValue'

So i added a debug statement to print the queryresults in the get_query_execution_results function:
log.info("Error: {}".format(json.dumps(queryresults)))

That results in:

{
    "ResultSet": {
        "Rows": [
            {
                "Data": [
                    {
                        "VarCharValue": "sum_unblendedcost"
                    }
                ]
            },
            {
                "Data": [
                    {}
                ]
            }
        ],
        "ResultSetMetadata": {
            "ColumnInfo": [
                {
                    "Scale": 0,
                    "Name": "sum_unblendedcost",
                    "Nullable": "UNKNOWN",
                    "TableName": "",
                    "Precision": 17,
                    "Label": "sum_unblendedcost",
                    "CaseSensitive": false,
                    "SchemaName": "",
                    "Type": "double",
                    "CatalogName": "hive"
                }
            ]
        }
    },
    "ResponseMetadata": {
        "RetryAttempts": 0,
        "HTTPStatusCode": 200,
        "RequestId": "9bdc5156-****-****-****-************",
        "HTTPHeaders": {
            "date": "Fri, 19 Jan 2018 09:24:54 GMT",
            "x-amzn-requestid": "9bdc5156-****-****-****-************",
            "content-length": "619",
            "content-type": "application/x-amz-json-1.1",
            "connection": "keep-alive"
        }
    }
}

Due to the empty Data: {} the script will fail can i ignore this? or is this caused by a misconfiguration? Thanks!

Would allow for finer grained time queries. Right now all the dates are put in as string data types, making constructing a query around ranges difficult.

Cool tool. Thanks.

code:
python report_utils.py --action=prepare-athena --source-bucket=kpmgcloud-cost-report --source-prefix=costreport/**/ --dest-bucket=kpmgcloud-cost-report --dest-prefix=athena2/ --year=2018 --month=06

Reply:
Traceback (most recent call last):
File "report_utils.py", line 7, in
import awscostusageprocessor.processor as cur
File "/Users/sgooch/Documents/Stash/costsaving/analysis/athena/aws-cost-analysis/awscostusageprocessor/processor.py", line 109
print "Number of records: [{}]".format(record_count)

I have run this once already so not sure if this would upset it?

I don't know if people realize that the permissions give readonly access to a LOT of stuff including secrets...
I would be appreciated if you could limit those permissions to only what is necessary for the bot to operate.

Hello,
I was trying to use the automated deployment for aws-cost-analysis. Everything went well and all functions are created in lambda.

• However its not clear to me when the athena database is created. There is still not database.
• curprocessor-sam-xAcctStepFunctionStarter is running periodicly
• the billing bucket name is the source of the reports correct?

The problem with using Athena or quicksight is that it , it does not take into consideration the dynamic change in schema of the billing files , that AWS normally does. Due to this the reports starts breaking.
Is there a way to handle this using glue or any other method for this tool. Would really be helpful.

hi,

Can someone please assist to resolve following error -

An error occurred while loading the data.
[Simba]AthenaJDBC An error has been thrown from the AWS Athena client. INVALID_FUNCTION_ARGUMENT: Invalid format: "04-apr-2022" is malformed at "apr-2022"

I am trying to convert the string to date format

sample string -
16-apr-2022
10-may-2022
06-apr-2022

code ( used in select clause)- date_parse("date_string"."date",'%d-%m-%y') as New_date

Thank you

Can't find the script anywhere, am I missing anything?