multiprocessing: break down the main scan loop about gcp_scanner HOT 3 CLOSED

[Updated below] Hmm, I would go with multithreading for GCP resource crawler (e.g. use multiprocessing.pool.ThreadPool or concurrent.futures.ThreadPoolExecutor). Doing fork/spinning up new process for each request/set of requests might be expensive. I'd do actual multiprocessing per each GCP project and/or service account key where time spent on spinning up new process/forking is negligible comparing to scanning time. I think having one solution for all OSes will make maintenance and implementation easier (so spawn is preferable).

Update: just read the documentation about pool of workers. You can disregard what's written above about process per each request/set of request. You have a pool of process workers waiting for new request to come. BUT we need to make sure all workers live as long as the scanning loop lives. This way you waste time for spinning up everything just once during the start up (this is not super important IMO). However, keep in mind that that the object you pass should be pickable in case of concurrent.futures.ProcessPoolExecutor according to this (https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor). Everything sent/received to our functions should be pickable AFAIK but I'd check it anyway.

from gcp_scanner.

Meeting Notes for 10/07/2023:

Decision 1

We would go with the default approach and support for all 3 types of systems. The primary reason is that the IO tasks are in terms of seconds and optimizing it using fork would optimize it in terms of milliseconds (which is negligible in comparison).

Decision 2

Discussion on further PRs and Issues. But a starting point would be paralleling Storage Bucket.

Decision 3

The projects list needs to be queried first. So, instead of going with groups and handling them, we can make an exception and query the projects list outside the loop.

Decision 4

Concurrent.futures.ProcessPoolExecutor is the way to go.

from gcp_scanner.

completed with PR #265 and #269
closing

from gcp_scanner.