tiled tiff output checkpoint vs. efficiency about oiio HOT 5 CLOSED

Here's what I wrote back to Jeremy:

Aha, interesting. I thought it was more like a "flush" -- write now rather than buffering longer, but I did not think it caused more in total to be written.

Here's my proposal: let's not do it by default, designate ImageSpec metadata "oiio:checkpoint" (when nonzero) to indicate that an output ought to frequently checkpoint if possible, and have the TIFF driver do so when that is found. Actually, let's make it so that the integer is the number of scanlines between checkpoints, with 0 meaning not to checkpoint at all.

Sound sane to you?

from oiio.

Ah, I've learned something. You were right, it is like a flush, where it re-writes header data and new buckets.

The issue is that the code,

write_tile(...)
if (y % m_spec.tile_height) == 0)
// Checkpoint

on an 8K tiled texture run checkpoint 21824 times! So the 6.6 GB was the header being overwritten tens of thousands of time.

I've tested checkpointing with the new timer, "oiio:CheckpointInterval" and seen that it is indeed really cheap. Maybe we should default it to 'on' (1s interval?) to match the intent of current practice?

from oiio.

Do we need an option at all? Maybe instead of "every N scanlines or buckets, checkpoint" it should simply be "every N scanlines or buckets, checkpoint if you haven't done so in T seconds". Set T=60 and call it a day? That wastes no time/bytes checkpointing for fast-writing frames, or even slow-writing frames. Checkpointing only would happen for glacial frames.

Does it even need the option, or would that give reasonable behavior?

from oiio.

Yah, if we pick a large number (like 60s) it seems unobjectionable. I'll do that and get rid of the option.

from oiio.

Fixed in db01f0e

from oiio.