Tools and resources to help you adopt and implement FinOps capabilities that automate and extend the Microsoft Cloud.
Home Page: https://aka.ms/finops/toolkit
License: MIT License
As a FinOps practitioner, I need to build custom reports outside of Power BI in order to meet my organizational reporting needs
Document tools and services that can be connected with Data Lake Storage.
As a FinOps practitioner, I need to deploy scheduled actions as part of my templates in order to automate onboarding for new projects
Publish scheduled action modules to the Bicep Registry for each supported scope.
Bicep requires separate files per deployment scope. Since this would require excessive duplication of code, we'll introduce a simple templating language optimized for enabling multi-scope deployment modules that support native Bicep tooling.
We could use your help:
Bicep-Build.ps1 does not account for different scopes in text strings in README.md or metadata.json. These must be manually updated.
We could use your help:
Docs cover a few specific topics, which are good, but they don't explain:
Add more complete documentation for consumers.
Check out the docs and let us know what you think is missing or would like to see added.
As a FinOps practitioner, I need to monitor both actual and amortized costs using a single, consolidated dataset in order to produce chargeback reports and reconcile invoices without duplicating data
Change amortization to be applied as separate rows, like Cost Management cost allocation, which would allow data to be either amortized or actual based on whether a filter is applied. Amortization would be the default since that is the more common scenario across the organization. (Only the people paying the invoice require the actual, billed charges.)
Consider writing a proposal to change how Cost Management provides amortized data. If the data was generated the same way cost allocation is generated, then this workaround wouldn't be needed and organizations would be able to reduce their data storage costs.
As a FinOps practitioner, I need to view cost reports in order to understand my current cost details
Create cost summary and commitment discount Power BI reports to get a more complete picture of current costs.
See reports docs for initial spec.
Everyone is currently working out of main but that will eventually cause issues when features have interim PRs that are not ready to roll out yet.
Document a branching strategy where devs work out of their own branches/forks, then commit back to a feature branch, which then gets merged into a dev branch. When the release is ready, we'll merge dev to main.
This should be done before closing any PRs that will not be included in the first release.
We could use your help:
Custom columns require more work to copy between reports for manual upgrades. Avoid using custom columns when possible.
Move the following calculated columns to be defined in the query:
We could use your help:
The main readme must include an explicit note about telemetry collected and how to disable it.
Add the data collection section to this repo.
Let us know if there's anything specific you'd like to see in this space.
As a FinOps practitioner, I need to ingest data into a queryable data store in order to report on data at scale beyond $5M/mo
Support large datasets (e.g., 500 GB/mo) with up to 7 years of historical data that refreshes when changed by adding an option to ingest data into Azure Data Explorer and update reporting to leverage that database.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
There was an internal analysis of the optimal data store to use for the largest datasets and Azure Data Explorer was deemed to be the best option that balanced cost, performance, and scale.
We could use your help:
As a FinOps practitioner, I need to estimate the cost of my hub instance in order to plan for future spending
Create a Power BI report that looks at hub configuration settings and available data size to report on historical and estimate future monthly spending.
We could use your help:
Create an ARM template that configures a new Cost Management export to ADLS and generates a Power BI report that can work against that dataset. The template should also include a Data Factory pipeline to delete old files when data is exported each day.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
We'd love to get your feedback! Please leave any questions, comments, or change requests in the release discussions at the bottom of each task list above. We'll try to close each of those out before we finalize the release.
When a hub has multiple currencies, Power BI shows them all together, which isn't valid. People have to select a single currency before the report can accurately render.
Have the filter preset based on what currencies are available. Prefer the currency that is used the most.
Settings must be saved to storage in order to have exports dynamically created.
See finops-hub template and hub module specs.
We could use your help:
Azure Quickstart Templates is the primary repository that FinOps toolkit releases will be published to. We need to minimize the effort needed to publish updates to ensure the process is as smooth and error-free as possible.
Automate the AQT release process as much as possible to avoid manual, error-prone steps. If we can't automate the actual publishing, at least generate a ZIP file that can be used to manually publish.
⛔ Blocked on modules other changes being committed so we can move modules into the finops-hub folder.
We could use your help:
Power BI isn't very clear about what currency is selected.
Show all cost data (KPIs, charts, and tables) with a formatted currency value.
Power BI allows hard-coding currency, but we need to investigate how to set it dynamically from another Currency field.
This preview feature may help: https://learn.microsoft.com/power-bi/create-reports/desktop-dynamic-format-strings
As a hub admin, I need to automate CM export creation in order to minimize administrative overhead
Create a msexports_Setup pipeline in ADF to manage Cost Management data exports automatically. Use config\settings.json to store the list of exports that will be managed.
Data is ingested into the
msexportscontainer and transformed when moved into theingestioncontainer. Do not use themsexportscontainer for anything other than Cost Management exports. If manipulating data, please do that in theingestioncontainer after the transform pipeline completes. Don't remove or rename columns, as that can break Power BI reports.
We could use your help:
PBIX files are ZIP files, which contain text files. As binary files, there's no transparency into what changed in each version.
Investigate options that can version control PBIX (or possibly generic ZIP) files. Some options:
As a FinOps practitioner, I need to report on many datasets in order to show a complete picture of my costs
Add support for the following datasets based on what's currently available in the Cost Management connector for Power BI:
We could use your help:
As a FinOps practitioner, I need a central cost optimization hub in order to drive cost optimization within my organization
Create a Cost optimization workbook that summarizes potential savings opportunities.
We could use your help:
As a FinOps practitioner, I need to stop all charges when a budget is exceeded in order to control costs
Create a template that can deploy an action group with script to shut down and/or delete resources.
-OR-
Add the ability to monitor cost and trigger custom actions in FinOps hubs.
As a DevOps team/leadership, I need to understand how many resources are deployed by type in order to <TODO: What is the goal? Why is this needed?>
⭐ This is a requested feature, but we could use more context on how and why it would be used. Please leave comments.
Create an Inventory page that provides a breakdown by resource type.
Consider writing a proposal to offer a built-in view inside of Cost Management for the same, if applicable.
As a hub admin, I need to save my deployment parameters file in order to update my deployment in the future.
Save the deployment parameters file to the export storage account.
As a FinOps practitioner, I need to ingest data using a standard schema in order to support multiple clouds and account types
Create a data factory pipeline that transforms data to the FOCUS schema. Use a new container for data ingestion and move Cost Management exports into this container when transformed.
We could use your help:
As a FinOps practitioner, I need to break costs down by tag in order to identify tagging gaps and perform chargeback
Create a Tags page in Power BI that helps understand what tag keys are applied, offers drilldown into the tag values, and also what resources are missing tags.
Consider creating a proposal for a built-in view in Cost Management for this, given the in-portal experience would likely be a better place for DevOps teams to discover and resolve tag gaps.
prob
repro
expected
screens
add'l
Most people won't discover the toolkit from GitHub alone.
Publish the toolkit templates to Azure Quickstart Templates.
The content you are editing has changed. Please copy your edits and refresh the page.
We could use your help:
As a FinOps practitioner, I need to backfill historical cost data in order to report on past data and monthly trends in the initial release
Update the ms-cm-exports_Setup pipeline in ADF to handle data ingestion for historical data. Reuse the same exports to ensure data is placed in the same folder. Refer to config\settings.json for data retention settings.
We could use your help:
Need automated validation to avoid missing validation steps.
Investigate the following template validation options:
CSV is slower to process, so we're switching to parquet for data storage.
Switch Power BI to ingest data in parquet format.
The content you are editing has changed. Please copy your edits and refresh the page.
We could use your help:
There are some common patterns that we'll use in multiple bicep files that could be error prone when repeated often or introduce maintenance burden when needing to update them.
Add an option to include reusable files using our Bicep templating. Something like:
// @include ../../include/telemetry.bicepConsider explicit directives for the most common scenarios, like telemetry.
// @telemetry 'unique-id'We could use your help:
As a FinOps practitioner, I need to automate the deployment of Cost Management resources in order to streamline onboarding
Publish modules to the Bicep Registry for all Cost Management features to simplify template creation.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
The content you are editing has changed. Please copy your edits and refresh the page.
We'd love to get your feedback! Please leave any questions, comments, or change requests in the release discussions at the bottom of each task list above. We'll try to close each of those out before we finalize the release.
As a FinOps practitioner, I need to have data p ushed at specific times in order to have a complete dataset when presenting to leadership
Configure a data factory pipeline to manually run exports at specific times rather than waiting for exports to be run on their schedule.
As a FinOps practitioner, I need to normalize cost data to a standard schema in order to **manage costs from **
Normalize cost data to an interim schema (since FOCUS is not available yet) that aligns to the current Cost Managment connector schema.
The msexports_Transform pipeline transforms the raw data to the normalized schema and saves it in parquet format to the ingestion container. For details about the transformation, see the dev docs.
We could use your help:
As a FinOps hub admin, I need to automate assignment of the Storage Account Data Reader role in order to streamline Power BI setup
Create a template that assigns the Storage Account Data Reader role. Consider making this an optional part of the finops-hub template.
We could use your help:
As a FinOps practitioner, I need to enable tag inheritance in order to streamline cost allocation and chargeback
Create a template that enables tag inheritance for an EA billing account, MCA billing profile, or EA/MCA subscription.
Also add this as an option to the export and FinOps hub with exports templates.
As a FinOps practitioner, I need to dedupe CM exports in order to directly link Power BI to ADLS as a data source
Create an ms-cm-exports_Transform pipeline to copy data from the ms-cm-exports container to the ingestion container in parquet format and overwrite old exports.
Start from https://github.com/MSBrett/ccm_lite/blob/main/ARM/azuredeploy.json and copy relevant pieces of https://github.com/MSBrett/ccm_lite/blob/main/Deploy-EA.ps1, as appropriate.
Let us know if there's anything you'd like to see changed.
As a FinOps hubs admin, I need to deploy and auto-configure PBIX files in order to streamline setup after each release
Add an option to templates that allow an admin to decide if they want files deployed or not. When files are deployed, they should be auto-configured to point to the correct storage account. Auth credentials should be provided on first-run. Update the pbix template output to point to the new files.
EA, MOSA, MCA, CSP, and classic Internal subscriptions all use different schemas. FOCUS is not available yet, so we need an interim schema.
Standardize a cost data schema that can be used until FOCUS is finalized.
The following changes have been scoped out for now to minimize changes compared to the Cost Management connector:
OnPremises, Infrastructure, Platform, Software, OtherWe could use your help:
As a FinOps practitioner, I need to enable tag inheritance in order to facilitate chargeback reporting
Create templates to enable tag inheritance for EA billing accounts, MCA billing profiles, and EA/MCA subscriptions.
Consider blueprints as an option to automate application for all subscriptions.
As a user in a global organization, I need to convert my billing currency to my personal local currency in order to understand and report on costs within my local geography
Cost details includes a separate column for cost in USD, but all the reports are using the Cost column. Add the ability to convert this USD column to any currency using a free or open-source currency conversion service.
Microsoft has currency conversion rates available internally. Consider writing a proposal for how we'd like to see that exposed via Cost Management APIs.
As a FinOps practitioner, I need to enable exports independently from the main FinOps toolkit deployment in order to handle scenarios where different accounts require different access
Create a separate export template in addition to the finops-hub-with-exports template.
As a CSP partner, I need to view partner-earned credit in order to understand how much I'm earning from my customers
Create a separate Power BI report focused on partner-earned credit (PEC) scenarios.
In an effort to accelerate timelines, we are merging with https://github.com/MSBrett/ccm_lite for the v0.1 release.
Update docs based on revised plans for the 0.1 release.
We could use your help:
As a SQL admin, I need to review my AHB usage in order to determine if I'm maximizing my existing licenses
Create a new Azure Hybrid Benefit report for SQL Server licenses.
The main issue here is that SQL and Windows usage data is inconsistent. Consider creating a proposal for the SQL and/or VM teams to emit usage in a consistent manner. Another possible workaround is to manipulate data in Data factory, but this is less ideal.
problem goes here
expected this
[ o ]
More to say...
As a FinOps hub admin, I need to be notified before my exports expire in order to ensure cost management activiites are not interupted
Enable an option to either notify admins or auto-renew exports.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.