golemfactory / ya-runtime-http-auth Goto Github PK

What:

• port the minimal re-implementation of htpasswd to Rust (lib)
  • consider using the htpasswd-verify crate, an implementation of the apr1 digest and a .htpasswd formatter
• the library should internally implement or provide helpers for locking files during .htpasswd file update
• .htpasswd files should be flushed and synced as soon as possible
• in the existing codebase, replace htpasswd.sh invocations with library calls

Why:

• htpasswd.sh is a shell script which relies on external tools and proper environment (e.g. PATH configuration)
• the library would allow for better error handling and portability (across Linux distros, across Windows / macOS)
• .htpasswd locks are required when multiple runtimes (activities) modify the same access file

What:

• propose a name for the custom counter (main counter) of total number of requests made by all registered users within the runtime
• in Runtime::start function, spawn a background task responsible for:
  • periodically fetch per-user and per-endpoint counter values from the proxy server for all registered users
  • publish the per-user counter via Runtime API to ExeUnit Supervisor (see the GAP for examples)
  • publish the last known counter value on shutdown

Why:

• we need to provide a number of total calls per authenticated user as a custom runtime counter

Follow GAP-8

What:

• Refactor the existing code according to GAP-8 in order to support multiple services via a single binary (via golemfactory/ya-runtime-sdk#19)
  • update the command line interface
  • update runtime configuration
  • implement service configuration file lookup
  • implement binary startup file lock mechanism
  • update the deploy and start commands

Why:

• previous runtime implementation supported only a single service per binary / runtime configuration

What:

• implement an HTTP client for the ya-http-proxy-client crate
• the client sends and receives JSON responses only
  • supports deserialization of response body sizes up to 8 MB
  • returns deserialized body for erroneous (non-200) responses
• compliant with the ya-http-proxy-client's Management API
  • provides helper functions for calling each available endpoint

Why:

• external developers obtain an implementation of a read-to-use Management API client in Rust
• the client will be used to develop a management CLI within ya-http-proxy

As in title

• test on box
• configuration for lighthouse
• configuration for erigon

• provide an overview of repository contents
• include a step by step provider configuration example
• link to requestor example in yapapi

What:

• CI workflow for running code test suites
• CI workflow for creating releases and building runtime binaries
  • triggered by pushing tags matching v* or pre-rel-*
  • build for the following CPU architectures:
    • x86-64
    • aarch64

Why:

• any available test suites should be run automatically against new pull requests in order to verify correctness of the new code
• releases should be built in a static and pre-configured environment
• the release process should be automated as much as possible

What:

• the access log reader is aware that the log can be rotated. The implementation detects that the file has been truncated and peeks the rotated file first
• rotated file is chosen based on the log name value in runtime's configuration and file creation / modification time

Why:

• logs can be rotated by a system daemon; remaining (rotated) log data must be parsed in order to preserve data (i.e. request counter) coherency

Blocked by #3

What:

• spawn and detach the ya-http-proxy process to background
  • perform a readiness check (via e.g. running a GET /services HTTP request on Management API)
    • if a file lock is present, actively loop and wait until the API is ready
    • if there is no file lock, verify that API is responding
    • if any case that the proxy is not running, spawn the process; use a file locking mechanism to prevent multiple instances being spawned by multiple runtimes
• follow GAP-8

Why:

• we want to automate the process of starting the proxy server so that Provider's do not need to register it as a system service

What:

• create a reverse HTTP proxy server prototype
• benchmark the implementation using a popular benchmarking tool (e.g. siege)
  • vs a static site served by nginx
  • vs a static site served by nginx, secured by a custom auth_request endpoint
    • create a simple server with a no-op auth endpoint

Why:

• verify feasibility of the solution proposed in GAP-8
• gather basic information on performance differences between the proxy implementation and nginx w/ auth_request
  • use nginx static site endpoint as reference

Follow GAP-8

What:

• Runtime::run_command sends HTTP requests to the Management API in order to add and remove service users, according to GAP-8

Why:

• the Requestor must be able to add and remove users authorized to use the service

What:

• periodically publish custom runtime counters (API calls per authenticated user)
• publish the last known counter value on shutdown

Follow GAP-8

Why:

• the number of API calls will be used for billing purposes; pass that information to ExeUnit supervisor

Blocked by #3

What:

• the start command should add the designated service within the proxy via Management API:

  • when the proxy is responding, use the "add service" endpoint to register the service
  • when the API server responds with 409 (CONFLICT), get the information from the server
    • if the server configuration does not match the local one, return an error in start

• for the stop command:

  • fetch last available counter values via Management API
  • publish the latest known counter values via ya-runtime-sdk
  • de-register users (if any) via Management API

• implement the test command to verify that:

  • the runtime can discover service configuration files and their contents can be parsed successfully,
  • the proxy binary is runnable and starts successfully

  Note: test command is executed automatically by the provider agent on startup

• implement the offer-template command so that the following will be included in the offer:

  • HTTPS support capability (always on in the current version)
  • certificate hash property (e.g. sha3:deadbeef..)
  • extra properties defined in the service configuration file (prefix with golem.runtime.http-auth.)

• update the implementation of existing commands according to GAP-8

Why:

• the runtime should be disabled if the proxy cannot be run successfully
• requestors should be informed on provider's runtime capabilities (i.e. HTTPS support)

Blocked by #3 #5