openmatchmaking / pathfinder Goto Github PK

There are already at least two other Rust projects using the Pathfinder name:

https://crates.io/crates/pathfinder
https://github.com/pcwalton/pathfinder

So you might want to consider picking another name.

Recently the tokio-rs crate was reformed so that existing APIs was changed. Therefore, for avoiding a broken pathfinder app, necessary to apply a new changes to Tokio:

• Replace in Cargo.toml file the tokio-core crate onto the tokio crate
• Remove deprecated parts of code with setting up the event loop via tokio_core::reactor::Core
  and replace it onto tokio::executor::current_thread task executor instead
• Make ensure that &handle as a part of old Core functionality won't broke the existing code (for example, when we're using middlewares)

An actual implementation of the pathfinder project is using a separate connections (RabbitMQ and Redis) per each active user it leads to inefficient use of resources and increasing a response time (especially for the first requests). What can be done for solving this issue:

• Use a single RabbitMQ connection per each pathfinder instance with a separate channel per each active user, which is much cheaper (in terms of used resources) and faster
• Replace manually getting tokens from Redis instance and validating them in run-time onto the separate request to Auth/Auth microservice, so that it will make all required checks and will return a list of permissions, assigned to the certain user.

This solution let to use the server resources more efficiently and will minimize the time, required for processing requests per each message which is coming from a user via WebSockets.

In actual moment "redis-rs" is actively updating for letting an access to developers to use async implementation out of the box, that gives us to write asynchronous applications and services with "tokio" and "future-rs" crates.

In actual moment of time a developer should define all desired endpoints in configuration file that have YAML format and will be passed as a parameter in CLI before the start. It's fine for a couple of endpoints, however in the case of multiple microservices the efforts for maintaining this table significally increases, when necessary to specfy all of those.

To minimize a maintenance efforts, we can optimize this process, so that it will use much more cleaner and easier way to handle those cases:

• Give an opportunity to specify "dynamic paths", so that the developer will define only a couple of generic resources on the base of some template
• For some corner cases let to developers to use the "static routing" (that currently using for all of it)

As for example I will use the following configuration, that must be supported and valid:

endpoints:
  - search:
      url: "/api/matchmaking/search"
      routing_key: "matchmaking.users.{action}"
      request_exchange: "open-matchmaking.matchmaking.search.direct"
  - users:
      url: "/api/users/{action}"
      routing_key: "auth.users.{action}"
      request_exchange: "open-matchmaking.auth.{action}.direct"

In this example we have 2 endpoints:

1. The search endpoints that uses static routing and gives to proxy an information about which AMQP queue must be used for sending a request.
2. The users endpoints that uses dynamic routing for balancing all incoming requests to AMQP queues / request exchanges of Auth/Auth microservice:
   • An action in curly braces is a parameter that stores a suffix and going to be used later in the templates for generating names of AMQP queues, request and response exchanges. For example the /auth/users/retrieve/ URL will be transformed by proxy engine into auth.users.retrieve, because the action has the retrieve value and was used in /api/users/{action} template, where that value was used.
   • The same rules applies to routing_key, request_exchange and response_exchange, if were specified any substitutions.

Any patterns or substitutions must not be defined only in the end of strings. They can be specified in the middle of the template, or in some cases may be even starts with it.
Besides of it the developer must replace "microservice" onto "routing_key", that must be used in configuration files instead, because shows that this field is required for passing messages.

For the cases, when the passed URL in the incoming request format is valid, but AMQP queues, request and response exchanges may not exist, we must return an error with a message like "Endpoint wasn't found" instead of waiting infinitely. It can be done with creating a "error queue" per each client that linked to the special exchange (or may be even "open-matchmaking.responses.direct") for tracking a messages that can't be delivered to the certain microservice.

After performance tests I've found an interesting bug that can be found in the case a big amount of users that constantly doing API calls and use a separate channel for each action. And when the limit is reached, reverse proxy is starting to deny to process any requests. In addition to it, it is writing messages in log, like this:

[2019-01-09][13:37:29][pathfinder::proxy][ERROR] Error in RabbitMQ client. Reason: The maximum number of channels for this connection has been reached

As a solution for it, we can reuse created channels for each connected user to reverse proxy, so that:

• 1 channel uses for publishing
• 1 channel uses for consuming

For the beginning, I think, it will more than enough use 1 connection and 2 separate channels per each user, so that approximately the single connection can be used for ~32k users simultaneously. And if we face with that issue once again, then, probably, we will need to move each type of channels (publishing, consuming, etc.) per each connection.