What is the meaning of `maxsize` argument for `ShardWriter`? about webdataset HOT 3 CLOSED

Oh, my bad. I didn't consider that internal fragmentation might still occur even though each pth file is larger than 4KB. I checked that each pth is around 7.7KB which could explain why 32MB data shard ended up taking slightly more space. Yet, 8KB - 7.7KB = 0.3KB ~= 5%, so it doesn't likely explain 39MB - 32MB = 7MB ~= 20% gap in the shard size.

from webdataset.

ShardWriter estimates shard size based on the total number of content bytes, not the actual file size of the shard on disk; the overhead from the tar format and container is not accounted for. If you write small files, this overhead might become significant. We'll try to make the estimates more accurate in the future.

from webdataset.

Ok, fair enought.

from webdataset.