Handle bad clients about civet HOT 4 OPEN

Hah! Not looking as yet, just getting back from an Alaska trip.
I will be in Idaho Falls in a week or two if anybody wants to get a 🍺!

from civet.

Yeah, this would happen occasionally and was pretty annoying.
I was not a big fan of keeping state on the server since it would need to be stored in the database and be constantly checked. I was going to go with your assumption that we don't have malicious clients and so they can take care of themselves.
My psuedo plan was to mark most of the initial git operations as "No Fail" as that is where most of these sort of problems happen (network, keys, git server down, etc). I think the only git operation that should be a real failure is if the merge fails. The "No Fail" operations would exit with a certain code or touch a file in the filesystem (if the git exit code is wanted). The client would check this and if something marked "No Fail" failed, the client would add that job id to a list of jobs not to ask for, then tell the server to reschedule the job. This list of failed jobs would be only held in the client. The list would probably be specific to each git server. If github goes down it shouldn't affect doing gitlab jobs. If too many are added to the list then it stop doing jobs for that server. Could also have timeouts for each job (ie if it gets a "No Fail" but then starts working again, the initial "No Fail" gets expired).

I thought it would be far better to restart clients that automatically stopped than it is to go through and invalidate a huge number of jobs. This isn't obviously the most robust fix but it would probably be straight forward to implement and ease the annoyance on this relatively rare problem.

Anyway, obviously never got around to implementing this so there might be problems with this approach.

from civet.

@brianmoose - Are you looking for a job yet? 😄
We are hiring, yours would come with a decent raise

from civet.

from civet.