scraping logic was moved to the specialized component; ome of the "old" code remained in the cloudinfo implementation not to break functionality.

As scraping providers for information is designed to be completely separated from the "request serving" code - mainly the management component needs to communicate with the scraping component. Probably an event based approach needs to be added for this purpose - integrate watermill if appropriate

The product details response has been extended with the scraping time information:
it can be found in the root of the response json:

"scrapingTime": "1535642080264"

The value is the timestamp when the data has been successfully refreshed for the provider.

This needs to be displayed on the ui below the curl command and above the details table

from the review of #43:

• inject configuration values and do not access viper directly: (https://github.com/banzaicloud/productinfo/pull/43/files/97f9eb1e934ec12d8079e8192df5460a240d1319#r214339225)

• init waitgroup a few lines later, to prevent creating it unnecessarily:
  https://github.com/banzaicloud/productinfo/pull/43/files/97f9eb1e934ec12d8079e8192df5460a240d1319#r214339649

go build
main.go:43:2: use of internal package github.com/banzaicloud/productinfo/internal/app/productinfo/api not allowed

The component needs to be redesigned: provider and service should be both visible

design the logging strategy, remove the custom context logger, use logur instead

1.: we should create a consistent way of initialising loggers, similarly to Pipeline:
banzaicloud/pipeline#798

2.: we can consider adding a correlation id to requests for better tracing and error handling:
https://github.com/banzaicloud/pipeline/pull/787/files

3.: configure gin to use logrus:
https://github.com/banzaicloud/pipeline/pull/793/files

4.: add provider's name as a field for every log message that's related to a specific provider

5.: add a scrape id to every scrape message to be able to distinguish between them

6.: alibaba package shouldn't use the "github.com/aliyun/alibaba-cloud-sdk-go/sdk/log" package for logging but the standard logrus instead that's used everywhere else

Using Redis:

CLOUDINFO_STORE_REDIS_ENABLED=true
CLOUDINFO_STORE_REDIS_HOST=localhost
CLOUDINFO_STORE_REDIS_PORT=6379

Using Cassandra:

CLOUDINFO_STORE_CASSANDRA_ENABLED=true
CLOUDINFO_STORE_CASSANDRA_HOSTS=localhost
CLOUDINFO_STORE_CASSANDRA_PORT=9042
CLOUDINFO_STORE_CASSANDRA_KEYSPACE=cloudinfo
CLOUDINFO_STORE_CASSANDRA_TABLE=products

service specific details need to be abstracted and made configurable

Requirements:

• administrators should be able to "register" a new service under an existing provider or a completely new provider
• cloudinfo behaves similarly to the existing services in case of the newly registered ones
• existing services behaviour will not be extended or altered

Design:

• the initial format for importing cloud information is similar to what CI currently supports
• service data is thought as configuration, passed to the app in yaml format
• service data is loaded into the application on startup

• • information shared between services need to be reflected in the data definition
• • the above introduces ordering between service information loading: dynamic services -those that get the information from providers -need to be started before static services

Tasks:

• design the data model for service configuration
• design the operational model (how the configuration is loaded)
• implement the configuration loader
• integrate the loader to the current product / factor out the redundant solutions
• include / exclude - data from existing services

provide only the products and regions that support the eks

Telescopes should be able to filter by geographic location

Services are resources that implement provider and region specific logic, mainly expose dependencies, restrictions and other relationship between products.

This story collects the tasks needed to add support for this new resource.

We need to reduce permissions for users that use provider apis to the minimum

The product representation needs to be decorated with labels containing meta information about the product. (labels may contain values already in the struct, derivated values or other data from the provider)
eg.:

• purpose (compute, general, etc ...)
• network category
  TBD

This information needs to be provided to all the clients of the productinfo (promarily telescope, pipeline) - upgrade the productinfo client in these apps

• decorate spans with status codes and messages
• cover error flows (relates to the above)
• cover gin handlers (trace requests)
• trace multiple services (telescopes)

Please provide an API call to serve the following information:

release/tag name
last commit short hash
last commit date

Attributes are not from the external API, therefore compiling the list is the task of Telescopes.

Organize the code based on the structure found here

Focus on / prepare for:

• error / problem handling
• logging
• application configuration
• integrating / instrumenting tools (tracing, prometheus, etc)

• application configuration to be made more transparent
• separate metrics related logic (should not interfere organically with the other logic: scraping etc ...)

Problem:
When no results are returned by the product service for a given provider or a given region the displayed data doesn't get refreshed and it shows the previously displayed data.

The API has changed to return empty json collection ([]) instead of nil when no results are returned

This issue is to handle this case properly in the UI code.

path parameters should be validated in handlers instead of the validation middleware

fixed in #125

Base path has been set to '/' but got following:

<!doctype html>
--
  | <html lang="en" xmlns="http://www.w3.org/1999/html">
  | <head>
  | <meta charset="utf-8">
  | <title>Cloud Products</title>
  | <base href="/productinfo/">
  | <meta name="viewport" content="width=device-width, initial-scale=1">

Cloud information is stored in an in memory cache as of now.
We need a persistent storage besides of this solution

The task can be reduced basically to add a database backed implementation of the CloudInfoStore interface

the chosen solution is to a store data in the current format in redis /cassandra

It only returns two kind of instance no matter what region we choose, thus recommender fails to recommend instances.

It fails on alpha and on beta too.

provide only the regions and products that support the oci

wired values need to be extracted in the service definition yaml file and loaded into the application on startup; wired values to be removed from the code-base

realates to #147

The following two flags seem to be redundant because these values are already present in the auth files required by the providers. We should try to read them from the auth files instead of having to specify them redundantly.

--gce-project-id  
--azure-subscription-id

provide only the regions and products that support alibaba cs

The productinfo REST api has been normalized, changes need to be reflected in the UI code, so that the appropriate calls are executed against the proper endpoints.

The new scheme has the form:

/api/v1/providers
/api/v1/providers/:provider
/api/v1/providers/:provider/services
/api/v1/providers/:provider/services/:service/regions
/api/v1/providers/:provider/services/:service/regions/:region
/api/v1/providers/:provider/services/:service/regions/:region/images
/api/v1/providers/:provider/services/:service/regions/:region/images?product=productID
/api/v1/providers/:provider/services/:service/regions/:region/products
/api/v1/providers/:provider/services/:service/regions/:region/products/:attribute
/api/v1/providers/:provider/services/:service/regions/:region/products/:attribute?image=imageID

On the UI the "provider" dropdown needs to be populated with data in the following form:

  - service1
  - service2
provider1
  - service1
  - service2
................

The data in the dropdown may be retrieved with the call

curl -L -X GET 'http://localhost:9090/productinfo/api/v1/providers'

• error handling (contextual errors)
• API - problem implementation (https://tools.ietf.org/html/rfc7807)
• context propagation
• instrumentation - metrics
• instrumentation - tracing

Sometimes it's required to refresh cached information for a single provider only (eg, a longer service shortage during which the cache expires)

Error handler has to log errors occurred during the scraping process
Related operations shouldn't escalate/propagate errors

In order to improve recommendations for ec2 - use the current generation instance types make the instance type generation information available for clients.

relates to: #9

This task is about designing and implementing the mechanism that takes into account the service value from the request path. (no overengineering, focus on simplicity)

This is more of a design task: by integrating eg. opencensus besides prometheus metrics it's possible to track background operations to better control and monitor the app behaviour

https://www.alibabacloud.com/help/doc-detail/25378.htm#g5

We need to make sure the api is not broken and the client is not outdated by modifications

cached information can be easily exported and imported
This functionality allows a quicker recover and start (during development or production)

Product info may show resources that are limited by user specific quota - this may lead to failures during cluster creation

• add versions to dependencies
• regenerate the lock file

• check the generic targets (clean/clear etc ...)
• apply improvements: test summary etc ...

perform the vendoring cleanup - keep only the dependency describers under vc

The application periodically retrieves cloud information from providers; this is quite a long process where different subtasks are executed by individual goroutines.

For better traceability this process should be separated from the "serving" part of the application;
Ideally this could be done by defining an interface for the renewal operations and add an implementaiton ...

UPDATE:
What i meant here - the renewal shouldn't be part of the caching cloud info; the process only writes to the store