DFD usage and implementation about ktx-specification HOT 14 CLOSED

I have 2 reasons for including the DFD:

1. To provide the color space information. Color primaries are easily transformed in a shader. It's just another 4x4 transform. The transfer function is indeed an issue. Restricting to linear or sRGB for WebGL/glTF use makes complete sense. Some users want to do everything themselves though including handling the transfer function.
2. To allow image processing outside of the graphics APIs to be performed on the image data without the processing tools having to be familiar with the details of all the VkFormats. Mostly this will happen during authoring or subsequent tuning of the images.

from ktx-specification.

without the processing tools having to be familiar with the details of all the VkFormats

Unless such tools output VK_FORMAT_UNKNOWN, they would need to choose what to write to vkFormat field, i.e., they would have to understand mappings between DFD and vkFormat.

from ktx-specification.

Unless such tools output VK_FORMAT_UNKNOWN

My idea is that such tools will process the image data without changing its format. Applying filters and special effects and such.

from ktx-specification.

A typical texture workflow goes like this. Initial data is usually authored in some high-end software like Photoshop or Substance using their native formats and saved as PNG or OpenEXR. Then, those textures are encoded to GPU-friendly formats (usually compressed). At this point, export tools would have to understand both DFD and vkFormat to produce KTX2 files. Filters and effects are usually applied to original data to preserve as much quality as possible.

I'm not aware of any tool that can directly edit arbitrary-defined texture data. For example, a legit DFD can describe red-green-alpha-depth pixel format where red and alpha are floats with shared exponent, while green and depth are normalized. Developing a tool that can correctly handle such layouts in all their glory would take much more efforts than baking-in a somewhat fixed set of familiar GPU formats.

DFD provides a subset of previously rejected "deep pixels" feature, so I'd ask a more broad question: does KTX2 aim to be GPU-ready texture container or a "universal" texture container?

To make the current design more reliable, I'd change DFD spec a bit (could it be done as DFD 1.3?). Namely, I'd say that when vkFormat is not unknown, DFD should omit sample information. Maybe, a proper way forward would be to include vkFormat into DFD (2.0?) thus making its structure less redundant.

from ktx-specification.

Sorry, failed to spot this was still current. Writer coming for the DFD and some mappings, but the KDFS is completely independent of Vulkan (other than providing a documentation source for some information); I'm very keen that vkFormats don't get included. The point of the DFD is to avoid having a long arbitrary list of format enums that need mapping in order to understand them.

from ktx-specification.

To use files with APIs, a "long arbitrary list of format enums" still has to exist somewhere. In our case, all KTX writers have to correctly write both: vkFormat and DFD. Also, it's not clear what readers should do with DFD's sample information when it doesn't match vkFormat (and to verify that, they have to contain all mappings as well).

from ktx-specification.

APIs that have chosen to define their formats as an arbitrary long list of format enums, yes. :-) My experience has been that both drivers and software decoders have to take the enums and apply multiple switch statements to break them down along the lines shown in the DFD. That's not to say that this is an inappropriate thing to do - the number of formats actually supported by most APIs is a small subset of what the DFD can represent, and is small enough to compress into enums.

KTX is designed to allow fast import and export (transport?) of textures from Khronos graphics APIs; it makes complete sense that it should have a vested interest in what the vkFormat should look like, and mappings involving this subset are forthcoming (honestly, lots of work still ongoing whenever I have free time). The KDFS treats that translation as peripheral.

from ktx-specification.

Happy to limit color model to RGBSDA, perhaps YUVSDA, and all the block-compressed formats.

color primaries should be free, as conversion is generally easy. Agree default should be BT709/SRGB.

Transfer function we should limit to SRGB or LINEAR give the difficulty of using others in the graphics APIs.

Isn't alpha premultiplication a signal to shaders as to how they should do alpha blending? We certainly don't want to encouraging unmultiply operations on loading. They are lossy operations.

Another piece of info the DFD provides is identifying the ETC1S subset of ETC1 as well as ETC1 from ETC2.

We might need the DFD for the wide transcodable format. It might not be equivalent to any Vulkan format.

from ktx-specification.

identifying the ETC1S subset of ETC1

Is it already in the spec?

from ktx-specification.

ETC1S is coming soon (Mark jumped the gun a bit); watch this space.

Alpha premultiplication indicates that the non-alpha channels have been pre-scaled by alpha. If you want to extract the colour that they'd contain were the surface opaque, then yes, you need to re-divide by alpha (which is troublesome), but it's slightly rare that you'd want to take a transparent surface and ignore its transparency. I'd hope that anyone using a premultiplied alpha surface is alpha-aware; if there were no support then we'd have to assume textures generated by typical blend operations were demultiplied prior to storage (unless they're fully opaque, in which case premultiplication is irrelevant).

from ktx-specification.

Please look at PR #75 and check I have the restrictions correct and haven't missed any you think should be there.

I don't see a reason to restrict alpha pre-multiplication so I haven't. If set, a shader would need to change how it does alpha blending but I don't see why it would require an operation on texture upload.

from ktx-specification.

Belated apologies that I've only just been looking back at this. Alexey, when you say:

Only a subset of new multi-plane formats introduced in Vulkan 1.1 (initially from VK_KHR_sampler_ycbcr_conversion) can be described with DFD.

what did you mean? Additional information is required (in that things like where the planes are stored is outside the "format"), but I'm not aware of there being an inability to describe anything. If there is, I should fix it.

from ktx-specification.

Additional information is required

That's the issue. There's no place for such information in the KTX header yet. We thought (initially) that DFD will handle it.

from ktx-specification.

Fixed by PR #75.

from ktx-specification.