Fix state issues with concurrently running tests about circles-core HOT 4 CLOSED

2021-02-23 12:10:59,958 [ERROR] [ForkPoolWorker-2] Task safe_relay_service.relay.tasks.circles_onboarding_safe_task[e091a565-7437-47df-809c-546c3af2e7a4] raised unexpected: InvalidNonce('{\'message\': "the tx doesn\'t have the correct nonce. account has nonce of: 3 tx has nonce of: 1", \'code\': -32000, \'data\': {\'stack\': "n: the tx doesn\'t have the correct nonce. account has nonce of: 3 tx has nonce of: 1\\n    at i (/app/ganache-core.docker.cli.js:55:433387)\\n    at /app/ganache-core.docker.cli.js:55:433518\\n    at /app/ganache-core.docker.cli.js:42:104258\\n    at /app/ganache-core.docker.cli.js:42:279371\\n    at /app/ganache-core.docker.cli.js:42:283136\\n    at Object.return (/app/ganache-core.docker.cli.js:42:282709)\\n    at /app/ganache-core.docker.cli.js:42:281020\\n    at e (/app/ganache-core.docker.cli.js:42:283077)\\n    at /app/ganache-core.docker.cli.js:42:283058\\n    at /app/ganache-core.docker.cli.js:42:280210\\n    at processTicksAndRejections (internal/process/task_queues.js:75:11)", \'name\': \'n\'}}')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/gnosis/eth/ethereum_client.py", line 131, in with_exception_handling
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/gnosis/eth/ethereum_client.py", line 1137, in send_raw_transaction
    return self.w3.eth.sendRawTransaction(bytes(raw_transaction))
  File "/usr/local/lib/python3.8/site-packages/web3/module.py", line 44, in caller
    result = w3.manager.request_blocking(method_str, params, error_formatters)
  File "/usr/local/lib/python3.8/site-packages/web3/manager.py", line 158, in request_blocking
    raise ValueError(response["error"])
ValueError: {'message': "the tx doesn't have the correct nonce. account has nonce of: 3 tx has nonce of: 1", 'code': -32000, 'data': {'stack': "n: the tx doesn't have the correct nonce. account has nonce of: 3 tx has nonce of: 1\n    at i (/app/ganache-core.docker.cli.js:55:433387)\n    at /app/ganache-core.docker.cli.js:55:433518\n    at /app/ganache-core.docker.cli.js:42:104258\n    at /app/ganache-core.docker.cli.js:42:279371\n    at /app/ganache-core.docker.cli.js:42:283136\n    at Object.return (/app/ganache-core.docker.cli.js:42:282709)\n    at /app/ganache-core.docker.cli.js:42:281020\n    at e (/app/ganache-core.docker.cli.js:42:283077)\n    at /app/ganache-core.docker.cli.js:42:283058\n    at /app/ganache-core.docker.cli.js:42:280210\n    at processTicksAndRejections (internal/process/task_queues.js:75:11)", 'name': 'n'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 412, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 704, in __protected_call__
    return self.run(*args, **kwargs)
  File "/safe-relay-service/safe_relay_service/relay/tasks.py", line 472, in circles_onboarding_safe_task
    SafeCreationServiceProvider().deploy_create2_safe_tx(safe_address)
  File "/safe-relay-service/safe_relay_service/relay/services/safe_creation_service.py", line 265, in deploy_create2_safe_tx
    ethereum_tx_sent = proxy_factory.deploy_proxy_contract_with_nonce(
  File "/usr/local/lib/python3.8/site-packages/gnosis/safe/proxy_factory.py", line 151, in deploy_proxy_contract_with_nonce
    tx_hash = self.ethereum_client.send_unsigned_transaction(tx, private_key=deployer_account.key)
  File "/usr/local/lib/python3.8/site-packages/gnosis/eth/ethereum_client.py", line 1190, in send_unsigned_transaction
    raise e
  File "/usr/local/lib/python3.8/site-packages/gnosis/eth/ethereum_client.py", line 1170, in send_unsigned_transaction
    return self.send_raw_transaction(signed_tx.rawTransaction)
  File "/usr/local/lib/python3.8/site-packages/gnosis/eth/ethereum_client.py", line 136, in with_exception_handling
    raise custom_exception(str(exc)) from exc
gnosis.eth.ethereum_client.InvalidNonce: {'message': "the tx doesn't have the correct nonce. account has nonce of: 3 tx has nonce of: 1", 'code': -32000, 'data': {'stack': "n: the tx doesn't have the correct nonce. account has nonce of: 3 tx has nonce of: 1\n    at i (/app/ganache-core.docker.cli.js:55:433387)\n    at /app/ganache-core.docker.cli.js:55:433518\n    at /app/ganache-core.docker.cli.js:42:104258\n    at /app/ganache-core.docker.cli.js:42:279371\n    at /app/ganache-core.docker.cli.js:42:283136\n    at Object.return (/app/ganache-core.docker.cli.js:42:282709)\n    at /app/ganache-core.docker.cli.js:42:281020\n    at e (/app/ganache-core.docker.cli.js:42:283077)\n    at /app/ganache-core.docker.cli.js:42:283058\n    at /app/ganache-core.docker.cli.js:42:280210\n    at processTicksAndRejections (internal/process/task_queues.js:75:11)", 'name': 'n'}}

  await loop(`Wait until Safe ${safeAddress} got deployed`, () =>
    web3.eth.getCode(safeAddress),
  );