Accelerated DHT client is CPU bound at load about caskadht HOT 4 CLOSED

Silly question, but.. do we cache anything in caskadht?

I suspect there are a lot of duplicate queries, considering how fallback/retry works in caboose. We have L1s being asked for the same CIDs over and over, and in turn, Lassie asks IPNI about them. IPNI needs to respond to the same asks, for the same CIDs, multiple times. Add cache.

Adding a separate lookup result cache for failures with TTL of 15s (to be under caboose retry) and other one for success with TTL of 60s (block get timeout in caboose across 3 retries) should reduce the cost of redundant lookups.

from caskadht.

All returned GET http responses are cached at cloudfront layer which sits in front of ci.contact load balancer

from caskadht.

Actions being taken to mitigate the high CPU utilisation:

• Increase service replicas to 3 and loadbalance incoming requests over them
• Deploy a fleet of caskadht with accelerated DHT client disabled. The instances sit behind a loadbalancer similar to the fleet that uses accelerated DHT.
  • 23/02/23 Holding off on this now that accelerated instances are seem to be running OK on compute optimised nodes.
  • 17/02/23 Service continues to function with stable performance. Deferring this task.
• Move the fleet that uses accelerated DHT onto compute-optimised instances with ample CPU.
• Profile the accelerated DHT for low hanging fruits if any.

Ideas:

• Use persistent peerstore, such that on instance restart there is less overhead to repopulate the network state.
• Share peerstore across multiple clients to reduce the need for state refresh per instance.

from caskadht.

Accelerate DHT-related profiling and improvement issues are to be captured separately with PRs to go-libp2p repo. This repo simply uses the library.

Closing as move to high CPU machines solved the stability problem.

from caskadht.