operate-first / curator Goto Github PK
View Code? Open in Web Editor NEWMetering project for OpenShift
License: MIT License
Metering project for OpenShift
License: MIT License
At 12:20 on 03/09/202, Start Date&time 03/09/2021 11:00 AM, End Date&time 03/09/2021 12:00 PM
At 12:10 on 03/09/2021, Start Date&time 03/09/2021 10:00 AM, End Date&time 03/09/2021 11:00 AM
There fore 12:15 PM is considered a cutoff time
Research to see if the history file in the S3 push mechanism is required.
┆Issue is synchronized with this Asana task by Unito
As a developer, I want to be able to implement authentication for the 'User' role so that they can each be mapped to their particular resources.
Acceptance Criteria
┆Issue is synchronized with this Asana task by Unito
Integration with Openshift
The repository for the main Openshift front-end UI is located at https://github.com/openshift/console. There are two different possible methods to connect Curator with Openshift.
This method would involve forking the Openshift UI respository and developing the Curator UI on top of the existing codebase. The Openshift UI is built with React, and consists of a wide range of existing pages for administrating and developing with Openshift. We can create a new page (or multiple pages, if necessary), within the existing Openshift UI repository, which would act as the Curator UI. Using this method, users would be able to log in to the same web app they already use for Openshift management and access Curator.
If we chose this method, we would need to ensure that our fork is kept in sync with the upstream Openshift UI repository. Once we have created a working product, we could potentially merge our new code in to upstream repository. Currently, the Curator UI is in a top-level directory in the operate-first/curator repository. If this method is chosen, it would likely need to become a separate repository.
Additionally, there is currently a page in the Openshift UI called "Monitoring", which appears to be accomplishing some of the same goals as Curator. This should be investigated.
Currently, the limited existing code for the Curator UI is set up as a separate React web application. This method would involve continuing to develop Curator UI as a separate web application, that is not explicitly connected to the main Openshift UI web application. Users would log in to the separate Curator web application with the same credentials that they use for Openshift. If this method is chosen, it would avoid potential complexities involved with integrating the UI into the existing Openshift repository.
User Interface Requirements
This section describes the basic requirements for the Curator user interface.
┆Issue is synchronized with this Asana task by Unito
In this bucket, we plan to save unzipped files all the data collected by the koku metrics operator. It will be used as a source bucket for our SQL database server.
Create a Red Hat mailing alias for the Curator project.
DoD:
Differentiate the weekly and monthly reports with namespace/project as the variable and generate weekly and monthly reports for individual projects.
DoD:
┆Issue is synchronized with this Asana task by Unito
It specifies how the koku metric operator generates reports and YAML files which help create pods that let you download the report in the local system.
DoD:
┆Issue is synchronized with this Asana task by Unito
Working on database design and how hourly, weekly and monthly reports will be generated and stored.
Hidden files were not deleted as part of #44.
DoD:
We have multiple cron jobs running at different times. So working to schedule them either parallelly or one after another.
┆Issue is synchronized with this Asana task by Unito
In the database,
The datatype of pod, memory, and node usage should be changed to double precision.
Handle all NULL values before inserting rows into the database.
Configuring the credentials to the S3 buckets should be secure. Currently you store them in a secrest.yml file.
instead, it would be better to use a secret in the enviornment. The credentials file could use the secrets in it (if configured to do so), or be configured using a secret, and allowing Kubernetes to take care of storing the secrets.
Create wiki pages documenting the project, including:
console
subdirectory/subtree┆Issue is synchronized with this Asana task by Unito
We are using python boto client to connect to OpenStack account for S3 storage.
By default python client doesn't come with boto. So pushing that image to operate-first quay.io
Identify the API end-points and the use cases we have to address through API.
DoD:
┆Issue is synchronized with this Asana task by Unito
DoD:
In user interface, there should be a subtab under Curator named "Raw Data Browser". This page will display a simple file browser, where the user can view and download the csv files in the s3 bucket.
┆Issue is synchronized with this Asana task by Unito
Generate 3 daily reports for the Curator development cluster.
DoD:
┆Issue is synchronized with this Asana task by Unito
DoD:
DoD:
┆Issue is synchronized with this Asana task by Unito
Compare a daily report against the actual cluster usage.
DoD:
┆Issue is synchronized with this Asana task by Unito
Insert the filenames with the timestamp into the history table.
┆Issue is synchronized with this Asana task by Unito
Few changes need to make before the next release.
┆Issue is synchronized with this Asana task by Unito
Npm install includes unused/unreferenced packages. Delete npm modules that are no longer used on the React app.
DoD:
Description:
When applying cronjob.yaml with a high backup rate (eg every 5 minutes), docker.io/minio/mc
failed at the image creation stage: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limits.
Reason:
Docker Hub Rate Limiting: On November 20, 2020, rate limits anonymous and free authenticated use of Docker Hub went into effect. Anonymous and Free Docker Hub users are limited to 100 and 200 container image pull requests per six hours.
Possible Solution:
Avoid redundant image pulling by adding imagePullPolicy: IfNotPresent
for all containers
.
containers:
...
- name: mc
image: docker.io/minio/mc
imagePullPolicy: IfNotPresent
...
┆Issue is synchronized with this Asana task by Unito
DoD:
DoD:
DoD:
Report if Curator can play any role in the Superset/Trino deployment running in Operate First.
┆Issue is synchronized with this Asana task by Unito
Make changes to the Operate First - Zero OCP (https://console-openshift-console.apps.zero.massopen.cloud/) to point the UI component to Curator-openshift-console rather than the in-built openshift-console.
DoD:
┆Issue is synchronized with this Asana task by Unito
DoD:
┆Issue is synchronized with this Asana task by Unito
Communicate the inspiration, requirements, plan, and onboarding documents related to the Curator with the general public.
DoD:
┆Issue is synchronized with this Asana task by Unito
DoD:
Functionality - Zipped and Unzipped files
Scheduled cron job
S3 connectivity
Pushing zip files
Pushing unzip files
Check pod status
Functionality - Postgres database
Database connectivity
Scheduled job status
Pushing files to database
Displaying proper error message
┆Issue is synchronized with this Asana task by Unito
Project Requirement Document should be completed.
Description:
curator/scripts/unzip_backup.py
Line 20 in fcf3b65
Make changes to the development OCP (https://console-openshift-console.apps.curator.massopen.cloud/) to point the UI component to Curator-openshift-console rather than the in-built openshift-console.
DoD:
DoD:
Description:
Secret
?A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.