Caprover was down due to captain-certbot about caprover HOT 2 CLOSED

Hello @manjeshpv
I encountered a persistent issue over the past weekend which seems to be related to SSL certificate renewal within CapRover. Despite consulting various troubleshooting resources like CapRover Troubleshooting Guide, the problem persisted, and attempts to force SSL certificate renewal using the command docker exec -it $(docker ps --filter name=captain-certbot -q) certbot renew were unsuccessful.

In an attempt to resolve the issue, I also restarted the Docker service and bypassed domain verification within CapRover by executing the following command:

echo "{\"skipVerifyingDomains\":\"true\"}" > /captain/data/config-override.json
docker service update captain-captain --force

Despite these efforts, the problem persisted, prompting me to check the CapRover server health using:

docker exec -it $(docker ps --filter name=captain-captain -q) curl https://captain.01.funkhaus.io/checkhealth

Unfortunately, the issue remained unresolved, and given the critical nature of the affected servers (production servers), I opted to initiate a fresh server setup. Notably, this issue affected two servers simultaneously, causing considerable disruption.

Subsequently, after several hours, there was an unexpected spontaneous recovery of the CapRover machine, although the exact recovery process or trigger remains unclear.

I would appreciate further investigation into this matter to prevent similar occurrences in the future and ensure the stability of the CapRover service.

Thank you for your attention to this matter.

Sincerely,
Ashutosh

from caprover.

@manjeshpv - the logs that you posted are completely normal. It's normal to try a few times to get the container ID of certbot. It doesn't indicate any problems. That why we have built in retry mechanism for it. Just restart your server and you will see the exact same logs every time.

@AshutoshM10 - your problem seems different. The fact that manually running certbot renew command failed shows that it's not a CapRover issue. It's a system's issue.

One thing to keep in mind is that CapRover is just a wrapper around many underlying services (docker, certbot, nginx etc)... Any of these services can fail due to any reason:

• Disk issues
• Local network issues
• Data center network issues
• etc...

The fact that you had an automatic recovery is CapRover tells me it's very like a network related issue (possibly on the datacenter layer)

Next time this happens, please capture the entire output of the system when you run certbot renew or curl captain.... that will give us more information on how to debug the underlying issue.

from caprover.