Handle 504 errors got error 504 from the server about scc-tools HOT 2 CLOSED

Sorry for the (long) delay. Working on it right now. I'm thinking up to 5 retries if it doesn't receive a fatal error, like 502 or 504. Is that enough?

from scc-tools.

Answering to myself: did a few tests after increasing the concurrent threads to 60 in order to force a gateway timeout situation (hope nobody gets too mad at me!), and it seems to be working fine now:

thread 1225 got non-fatal reply (502) from server, trying again in 2 seconds 
thread 1227 got non-fatal reply (502) from server, trying again in 2 seconds 
thread 1296 got non-fatal reply (502) from server, trying again in 2 seconds 
thread 1300 got non-fatal reply (502) from server, trying again in 2 seconds 

and after sometime, while the main thread is gathering results:

thread 1186 got a good reply after 1 tries
thread 1213 got a good reply after 2 tries
thread 1241 got a good reply after 1 tries
thread 1279 got a good reply after 1 tries
thread 1159 got a good reply after 2 tries

Not a single one went past 2 tries, so I guess 5 should be okay.

(no, the thread IDs don't match, I just picked some random lines -- I managed to get over 30 timeouts in one run while testing :-) )

from scc-tools.