I think robohash is down. Everything ok? about robohash HOT 2 CLOSED

Looks like there were no commits for a couple years until this past day. I'm guessing the author tried to deploy the new code with disastrous results?

@e1ven need any help?

from robohash.

The nodes were getting overloaded, which was causing the site to go slow and stop responding.. I added some additional digital ocean nodes. When I did, it wasn't generating the hashes with the same order as it had traditionally. I didn't want the hash to depend on what server you were hitting, so I added code to re-force the hash order.

Coming back to RH after a few years, I realize I should have spent more time originally ensuring that the order for the images was always consistent, rather than implicitly relying on inode order.
I also should have had test coverage, but that's another show.

Once I fixed the code, I was able to bring the new nodes online.

When nodes start becoming overloaded, image requests that don't hit cache start building up, until it's not keeping up with demand, building a bigger and bigger backlog. This causes the machine to eventually go out of rotation, and the other nodes get hit even harder, until you're in runaway.

At that point, if you add new nodes to the rotation, they'll get 100% of the traffic, and go down too - You need to take the Load balancer down entirely, drain the connections to each node, then bring it back up.

If it were a real app, I'd probably set up an auto-scale group at EC2, but that would cost me real money :)
Sorry about the downtime - Hopefully it should be fixed now.

from robohash.