gridscale / gsclient-go Goto Github PK

We want to switch the default branch name from master to main for all gridscale Open Source projects including this repository.

This involves:

• Switching the default branch name here on GitHub
• Updating all references of the master branch name in GH Actions and other CI/CD pipelines and scripts
• Giving contributors a bit of info on how to update their remotes in their local copies of the repo

Would be nice to do this coordinated and in "one go" so that contributors are not surprised by inconsistent naming or repositories not having switched yet. If you want to help out here contact @bkircher

When printing debug output with gsclient-go, it prints out all headers including API token and user ID. Alas, this is not incorrect it is also not necessary.

Proposal: Obfuscate the output, so that only the first 6 characters are printed and the rest is omitted (indicated by ellipses or something).

This would allow users to just copy and paste their debug output into pastebin or whatever without accidentally publishing credentials. At the same time, it allows users to verify that the credentials are the expected ones when comparing.

Example: create a new client with debugMode set to true. Then do a simple API call. It will print something like

DEBU[2021-02-15T14:48:01+01:00] Request headers: map[Content-Type:[application/json] User-Agent:[gsclient-go/3.3.2 (darwin)] X-Auth-Token:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] X-Auth-Userid:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]

where the Xs are real credentials.

This relate to terraform-provider-gridscale issue #149. The issue prevents the PaaS from upgrading its template UUID via gsclient-go. This is due to the incorrect information from the docs of PaaS Update operation.

There is no support in the API client yet that supports workflows with multiple projects in gridscale. I suppose, new objects are always added to the first project (named 'default').

Also, current gridscale API Specification (1.0.3) doesn't expose projects yet. So this would be needed first.

Ditch git flow, use GitHub regular work flow. Will make releases much easier and we can automate with tools like GoReleaser.

TODO:

• update CONTRIBUTING.md
• update release-checklist.md
• add goreleaser YAML
• remote stale release branches from repository (origin/release/*)
• remove develop branch, make master default

GET /objects/servers//events as well as GET /objects/events and all other events end-points now include an additional key "initiator" whose value is a string.

From the docs, initiator is "The user that triggered the event. Usually the user's email if the event was triggered by request of a user, otherwise a short descriptive name of the system component responsible."

Docs here: https://gridscale.io/en/api-documentation/index.html#operation/getServerEvents

Please add this to EventProperties struct.

To improve stability of our services, we need to identify issues and bottlenecks first.

In the past, when encountering issues in API reliability that were temporary in nature, we have implemented retry mechanisms into gsclient-go and the gridscale-terraform-plugin where applicable to improve the stability of the overall process. Also, underlying issues have been identified and, where possible, solved accordingly.

These retry mechanisms have the side-effect that, once in place, they mask underyling issues. To improve quality for everyone, we need to make them visible again.

This can be tackled from either the client or the server-side. For the latter, the APIs would provide metrics of request success. This does not make all issues visible though. In front of the actual APIs, there is an application server. In front of that, a web server. In front of that a reverse proxy. In front of that a firewall. In front of that a router. Etc. Issues in these layers, or the rolling deployment of the APIs itself, would not be visible in API-level metrics. Instead, we want our SDKs to report errors to us - with an opt-out functionality.

gsclient-go has been selected as the first prototype - mainly, because it is used by terraform as well. Terraform, in return, makes use of higher concurrency than many other clients do and tends to trigger deficiencies in provisioning more efficiently.

The idea is to send these reports to a publically available sentry instance. Networking errors (connection reset f.e.) and provisioning errors (Storage creation error f.e.) alike. To investigate issues, I presume we need

• X-Request-Id, if available
• Error message/classifier
• Accessed resource type URL (f.e. /objects/servers) for aggregation
• Access method (GET, POST, PATCH, DELETE, etc) for aggregation
• Time

These reports shall also be sent when the requests is retried. They shall not block the SDK and, as such, shall send reports asynchronously. Users must be able to opt-out.

What are your thoughts?

Ref API docs. Network's pinned_server API endpoints need to be added, so that we could set the static IP address to servers within a network.

• Rename Sshkey to SSHKey. Example: SshkeyCreateRequest should be SSHKeyCreateRequest

gsclient-go doesn't have the field autoscaling in the response of PaaS template getter. More info: https://gridscale.io/en/api-documentation/index.html#operation/getPaasServiceTemplates

Sometime resources are locked in the backend, and we have no control in the client-side. Some PATCH/DELETE requests cannot be applied to those resources, that causes 409. We should retry the requests if we get this error code.

• Some location API endpoints: https://gridscale.io/en//api-documentation/index.html#tag/Location
• Usage API endpoint: https://gridscale.io/en//api-documentation/index.html#tag/usage

Cool thing is that gsclient-go documentation coverage is quiet good. But there are some corners that could need some polishing.

Go through package and docs for public types and functions in gsclient-go and follow suggestions in this article. Also make sure every package has a dedicated doc string.

Don't use embedded struct for GSTime. Right now it is

type GSTime struct {
  time.Time
}

Make it

type GSTime time.Time

so that we can use type conversion.

Something that would be nice to have in gsclient-go is a way to easily trace how long an API call needs from request → to fulfillment.

With fulfillment I mean: I'm not interested in timings for each individual API call (e.g. HTTP request → HTTP response) but rather in "create storage" → "storage created"; how long did this take?

I thought about specifying a log-level in the client configuration, e.g., TRACE (or whatever logrus is using). This log-level should only be used for tracing the individual requests and their timings. One line per API request, printed when the request has finished. Output format should be JSON so that it can be processed further easily.

Example: If log level is set to TRACE, you would see something like this on stderr:

{
  "message": "Finished request CreateStorage",
  "requestId": "d8260ce6-7439-4543-85c4-794592a34346",
  "request": "CreateStorage",
  "success": true,
  "timeMs": 3042
}

timeMs is the time in milliseconds that has elapsed. success would be false when this errors, but I would still see timeMs.

What do you guys think? @fkr @nvthongswansea

• Retrieve location where backups of a backup schedule are stored.
• Set location for all backups of a backup schedule.

👋 An IP object can be related to a storage? https://github.com/gridscale/gsclient-go/blob/master/ip.go#L113 PublicIPs and Storages seem to be always nil.

Optional field with json tag "omitempty" should be type of pointer, since the optional fields usually accepts 3 types of input: nil (omitted when the field is nil), empty (Should not be omitted. eg: "" for string, 0 for int), and not empty. E.g:
Labels []string json:"labels,omitempty"

should be

Labels *[]string json:"labels,omitempty"

Hey hey. README.md says in the Known Issues section:

* L3security isn't read in the network relation of a server.

Is this still relevant? Should we create a bug for this and actually tackle this? @nvthongswansea

Start building support for a meaningful subset of contract management API (https://api.gridscale.io/management/v1/).

Interesting sub-set would be to get out current billing information for a single contract.

A heads up. There will be some updates to the GET /objects/paas/service_templates response.

A new release parameter will be included, and the version parameter will be modified from version="10.3" to version="10.3-gs1".

For a postgres template example, it will look like:

"9008f1f1-190e-4887-99f7-8a3e72fdd1f8": {
      "object_uuid": "9008f1f1-190e-4887-99f7-8a3e72fdd1f8",
      "release": "10.3",
      "version": "10.3-gs1",
      "name": "PostgreSQL 10.3 16GB (Gen 1)",

Current docs here: https://gridscale.io/en/api-documentation/index.html#operation/getPaasServiceTemplates

So I think we want to include that?

When creating servers and ServerCreateRequest.ServerHardwareProfile is omitted the resulting profile is pc-i440fx-2.4 which is a weird choice for a default.

When doing the same thing in the panel, I get pc-q35-2.10 ("q35" in the API docs) which is a much saner choice. Please consider changing default profile and set it explicitly to "q35". Also update the docs for ServerCreateRequest.ServerHardwareProfile, "HardwareProfile is not set => server hardware is normal type" is really not saying anything at all.

Due to wrong information on official API docs https://gridscale.io/en//api-documentation/index.html#operation/rollbackStorageBackup, the method RollbackStorageBackup won't work. The correct API endpoint is https://api.gridscale.io/objects/storages/{storage_uuid}/backups/{backup_uuid}/rollback

A rate-limited request should be retried until the rate-limit is reset and the request is processed successfully. If the request is delayed again (due to rate-limit is reached again), keep retrying.

Add DHPC options to private network's create/update requests, and add DHCP properties to network's properties.

This happens due to a wrong field's name (which is described in gridscale's API docs). storage_backups is supposed to be storages_backups.

We are currently in the process of developing a Prometheus exporter to make our Gridscale cloud costs transparent. In the context of this exporter, among others, we expose the running PaaS resources as metrics by periodically scraping the Gridscale API. To be able to determine the cost of a resource, as far as I am aware, we need the products it's using via the product number and in which location it is running. Usages can be retrieved via the usage API, but, in contrast to other resource types, such as servers, PaaS resources do not seem to expose a location attribute via the API.

Is it viable to indirectly determine the location of the PaaS resource via the connected network, or is there another mechanism to retrieve this information from the API?

CurrentPrice (current_price in the API) is deprecated in the public API. We should remove this field from all property structs when we bump the major version.

Searching for a way to conveniently overwrite existing or add new HTTP headers to requests.

Right now I can create a new config.Config with config.NewConfiguration or config.DefaultConfiguration but I'm only allowed to specify URL, user id, and token. The returned config will have config.Config.userAgent set for me automatically which is just one HTTP header.

Is there a convenient way for me to specify any custom header that is made during requests?

Hi, I was debugging an Issue with my code that turned out, I've missed the https protocol in the endpoint.

This

func main() {
	token := "XXXXXXXXXXXXXXXXXXXXXX"
	user := "ZZZZZZZZZZZZZZZZZZZZZZZZ"
	endpoint := "http://api.gridscale.io"
	emptyCtx := context.Background()

	// config := gsclient.DefaultConfiguration(user, token)
	config := gsclient.NewConfiguration(endpoint, user, token, true, true, 1000, 5)
	client := gsclient.NewClient(config)

	// err := apiAlive(token, user, endpoint, client)
	serverCreateRequest := gsclient.ServerCreateRequest{
		Name:   "api-tests",
		Memory: 1,
		Cores:  2,
	}
	cServer, err := client.CreateServer(emptyCtx, serverCreateRequest)
	if err != nil {
		panic(err)
	}


	fmt.Printf("%v", cServer)
	if err != nil {
		panic(err)
	}
}

results in

DEBU[2021-02-15T14:47:58+01:00] Preparing POST request sent to URL: http://api.gridscale.io/objects/servers
DEBU[2021-02-15T14:47:58+01:00] Request body: {{"name":"api-tests","memory":1,"cores":2}
}
DEBU[2021-02-15T14:47:58+01:00] Request headers: map[Content-Type:[application/json] User-Agent:[gsclient-go/3.3.2 (darwin)] X-Auth-Token:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] X-Auth-Userid:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]
DEBU[2021-02-15T14:48:00+01:00] Status code: 200. Request UUID: 6e99e9a2-8c19-45d8-a416-c21e550f751d. Headers: map[Access-Control-Allow-Credentials:[true] Access-Control-Allow-Headers:[Origin, X-Requested-With, Content-Type, Accept, X-Auth-UserId, X-Auth-Token, X-Exec-Time, X-API-Version, X-Api-Client, X-API-Identity, X-Auth-ContractId] Access-Control-Allow-Methods:[GET, POST, PUT, PATCH, DELETE, OPTIONS] Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[X-Exec-Time, X-API-Version, X-Request-Id, X-API-Identity] Cache-Control:[no-cache] Content-Length:[3708] Content-Type:[application/json] Date:[Mon, 15 Feb 2021 13:48:00 GMT] Etag:["W/e315f3cacb4e7a3a538f7a85dcb5d46f0f245264"] Ratelimit-Limit:[210] Ratelimit-Remaining:[208] Ratelimit-Reset:[1613396894217] X-Api-Identity:[2] X-Request-Id:[6e99e9a2-8c19-45d8-a416-c21e550f751d] X-Time-Provisioning:[0.547547]]
DEBU[2021-02-15T14:48:00+01:00] Response body: {"servers": {"28c1371a-cb47-498c-b852-cadbecbfda89": {"console_token": "e36965689121ac9a80aaaa8f1069edee5df7b5a9cfbaded9c5b9fe9714e9b90e1fe1d45b3923b2d7f217009e532e6bf2", "labels": [], "create_time": "2021-02-01T09:08:18Z", "change_time": "2021-02-01T09:15:14Z", "location_uuid": "45ed677b-3702-4b36-be2a-a2eab9827950", "name": "4dx-dashboard", "hardware_profile": "q35", "availability_zone": null, "memory": 2, "auto_recovery": true, "location_iata": "fra", "cores": 1, "power": true, "location_name": "de/fra", "object_uuid": "28c1371a-cb47-498c-b852-cadbecbfda89", "location_country": "de", "legacy": false, "status": "active", "relations": {"storages": [{"bus": 0, "controller": 0, "create_time": "2021-02-01T09:08:18Z", "target": 0, "bootdevice": true, "last_used_template": "fd65f8ce-e2c6-40af-8fc3-92efa0d4eecb", "lun": 0, "storage_type": "storage", "object_uuid": "79f10e8b-33b2-4973-99ce-40aa20747734", "object_name": "4dx-dashboard", "capacity": 10, "license_product_no": null}], "networks": [{"vxlan": null, "create_time": "2021-02-01T09:08:18Z", "ordering": 0, "public_net": true, "network_uuid": "23e0345d-4313-4307-b7b9-9eb666dfac43", "firewall_template_uuid": null, "bootdevice": false, "vlan": null, "l3security": [], "network_type": "network", "firewall": null, "mac": "16:40:87:d7:f8:01", "mcast": null, "object_uuid": "23e0345d-4313-4307-b7b9-9eb666dfac43", "l2security": true, "partner_uuid": "95a2980b-7012-43dd-81f2-07577cfcb9f0", "object_name": "Public Network"}], "public_ips": [{"create_time": "2021-02-01T09:08:18Z", "object_uuid": "cb47368e-723d-4b06-ac00-3967151a4fcd", "ip": "2a06:2380:0:1::4bb", "family": 6, "prefix": "2a06:2380:0:1::4bb/128"}, {"create_time": "2021-02-01T09:08:18Z", "object_uuid": "c10af4e6-4855-45cb-a56a-93ea7df51d84", "ip": "45.12.51.119", "family": 4, "prefix": "45.12.51.119/32"}], "isoimages": []}, "current_price": 99.99, "usage_in_minutes_cores": 20432, "usage_in_minutes_memory": 40864}, "e3e89eb2-963a-4f73-8090-ca5c97b74941": {"console_token": "5c8f74843e29917b3bf6d7bfe33d99f1d85360fe123062dca9c809ccfa33fe36f4ffc6bf28f714315234399babb74176", "labels": [], "create_time": "2021-01-08T23:24:30Z", "change_time": "2021-02-15T13:47:37Z", "location_uuid": "45ed677b-3702-4b36-be2a-a2eab9827950", "name": "gs-test", "hardware_profile": "q35", "availability_zone": null, "memory": 2, "auto_recovery": true, "location_iata": "fra", "cores": 1, "power": true, "location_name": "de/fra", "object_uuid": "e3e89eb2-963a-4f73-8090-ca5c97b74941", "location_country": "de", "legacy": false, "status": "active", "relations": {"storages": [{"bus": 0, "controller": 0, "create_time": "2021-01-08T23:24:30Z", "target": 0, "bootdevice": true, "last_used_template": "fd65f8ce-e2c6-40af-8fc3-92efa0d4eecb", "lun": 0, "storage_type": "storage_insane", "object_uuid": "6f826003-b32b-46ab-adc0-cccdf4698fa6", "object_name": "gs-test", "capacity": 10, "license_product_no": null}], "networks": [{"vxlan": null, "create_time": "2021-01-08T23:24:30Z", "ordering": 0, "public_net": true, "network_uuid": "23e0345d-4313-4307-b7b9-9eb666dfac43", "firewall_template_uuid": null, "bootdevice": false, "vlan": null, "l3security": [], "network_type": "network", "firewall": null, "mac": "22:cc:c9:3d:52:01", "mcast": null, "object_uuid": "23e0345d-4313-4307-b7b9-9eb666dfac43", "l2security": true, "partner_uuid": "95a2980b-7012-43dd-81f2-07577cfcb9f0", "object_name": "Public Network"}], "public_ips": [{"create_time": "2021-01-08T23:24:30Z", "object_uuid": "d0485231-935c-4ad6-9890-0010e4b35a2e", "ip": "185.201.144.29", "family": 4, "prefix": "185.201.144.29/32"}], "isoimages": []}, "current_price": 99.99, "usage_in_minutes_cores": 54140, "usage_in_minutes_memory": 108280}}}
DEBU[2021-02-15T14:48:01+01:00] Preparing GET request sent to URL: http://api.gridscale.io/requests/6e99e9a2-8c19-45d8-a416-c21e550f751d
DEBU[2021-02-15T14:48:01+01:00] Request body: {}
DEBU[2021-02-15T14:48:01+01:00] Request headers: map[Content-Type:[application/json] User-Agent:[gsclient-go/3.3.2 (darwin)] X-Auth-Token:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] X-Auth-Userid:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]
DEBU[2021-02-15T14:48:02+01:00] Status code: 404. Request UUID: b9f05ddd-e0bf-4e86-9121-6a657bfb9f15. Headers: map[Access-Control-Allow-Credentials:[true] Access-Control-Allow-Headers:[Origin, X-Requested-With, Content-Type, Accept, X-Auth-UserId, X-Auth-Token, X-Exec-Time, X-API-Version, X-Api-Client, X-API-Identity, X-Auth-ContractId] Access-Control-Allow-Methods:[GET, POST, PUT, PATCH, DELETE, OPTIONS] Access-Control-Allow-Origin:[*] Access-Control-Expose-Headers:[X-Exec-Time, X-API-Version, X-Request-Id, X-API-Identity] Cache-Control:[no-cache] Content-Length:[154] Content-Type:[application/json] Date:[Mon, 15 Feb 2021 13:48:03 GMT] Etag:["W/502812e75a4eec35693a6554365702233177fd4f"] Ratelimit-Limit:[210] Ratelimit-Remaining:[207] Ratelimit-Reset:[1613396894217] Vary:[Accept] X-Api-Identity:[2] X-Request-Id:[b9f05ddd-e0bf-4e86-9121-6a657bfb9f15]]
ERRO[2021-02-15T14:48:02+01:00] Error message: {'type': 'HTTPError', 'status_code': 404, 'message': "(404, 'There is no Request for the given Request UUID.')"}. Title: 404 Code. Code: 404. Request UUID: b9f05ddd-e0bf-4e86-9121-6a657bfb9f15.
panic: Status code: 404. Error: {'type': 'HTTPError', 'status_code': 404, 'message': "(404, 'There is no Request for the given Request UUID.')"}. Request UUID: b9f05ddd-e0bf-4e86-9121-6a657bfb9f15.

API endpoint for creating a new storage from a specific backup: POST https://api.gridscale.io/objects/storages/import with {"backup":{"name":"new storage","backup_uuid":"XXXX-XXXX-XXXX-XXXX-XXXX"}} as a body. This functionality is not described in the official API docs yet.

Using http.Request when retrying may cause an error http: ContentLength=... with Body length 0. Error happened here. Why did it happen.

https://github.com/gridscale/gsclient-go/blob/master/marketplace_application.go#L119
Authors string json:"authors" should be changed to `Author string `json:"author"

Certificate endpoint has been added to gridscale API more. We have to add this endpoint to the gsclient-go library, and update the loadbalancer resource in the library as well loadbalancer

Hi there!

Is a CLI client planned, similar to how Azure CLi and so on work?

I ask since I would be interested in such a tool and would maybe like to contribute if you created this one open source.

Please excuse me asking this on the Go Client. I just figured the Go client would be a great starting point for a CLI tool 😉

Add simple support for batch API. Use simple builder pattern. Preliminary branch here: https://github.com/gridscale/gsclient-go/tree/batch