tools/fast_importer should take a compression_level argument about containerregistry HOT 7 CLOSED

@mattmoor

from containerregistry.

I'm not sure a lower level would help appreciably. Python gzip is just tremendously slow.

We could consider making this take a kwarg to control whether compressed or uncompressed blobs are emitted (or both). WDYT?

from containerregistry.

uncompressed vs compressed would be best, but on my cursory glance at the code, it looks like a non-trivial change to skip compression altogether.

from containerregistry.

Yeah, this is a little trickier than I thought, mostly because of how oriented the DockerImage interface is around having the layer digests, which we basically don't have or need.

The good news is that FromTarball already lazily generates the manifest, so if we are careful we should be able to extend the interface to achieve what you want. My current thinking looks something like:

1. Extend DockerImage to expose a diff_ids() member that returns the list of diff_ids from the config_file() (which we can access without producing the manifest() in FromTarball.
2. Extend the DockerImage interface to allow fetching uncompressed_blob by diff_id.
3. When v2_2_save.fast is called to emit uncompressed stuff, then bypass this loop completely and have another that uses the above.

WDYT?

from containerregistry.

Looks like it should work.

from containerregistry.

What do you think about #28?

Not quite what we discussed but looks simpler and does only what's needed.

from containerregistry.

I believe this should be resolved by outlining the gzip entirely

from containerregistry.