Iterating over a slice of strings (paths to files) and applying this to it about pond HOT 2 CLOSED

Hey @robinryden. Do you plan to use AWS SDK to do so?
If you were to use the Upload Manager provided by AWS (https://aws.github.io/aws-sdk-go-v2/docs/sdk-utilities/s3/#upload-manager), note it would use its own internal worker pool to upload each file chuck, and it can be tuned to adjust chunk size (PartSize option) as well as the level of concurrency (Concurrency option).
In this case, I'd simply iterate over the files and create an uploader for each of them:

// This pool could be instantiated once and reused
pool := pond.New(100, 1000)

// Create a task group for this specific group of files
group := pool.Group()

// Submit one task per file
for _, p := range filePaths {
    path := p
    group.Submit(func() {
	uploadFile(p) // uploadFile should take care of creating the Uploader
    })
}

// Wait for all uploads to complete
group.Wait()

You can control the number of concurrent uploaders by adjusting the size of the pool. You might have to test with different sizes to find the one that maximizes throughput.
However, keep in mind there could be other factors affecting the actual level of concurrency you can achieve, like connection pool size on the S3 client, max connections limit on S3 servers, network quotas and so on.

from pond.

Closing due to inactivity

from pond.