rust-windowing / softbuffer Goto Github PK

At the moment all backends need to perform a copy to present. Some backends such as Wayland (always) and X11 (with the SHM extensions) can simply share the pixel memory with the server. On Windows and Mac this saving wouldn't be possible, but it would help on other platforms.

Not sure what redox does in this case.

Like this: https://github.com/smol-rs/polling/blob/master/.github/workflows/release.yml

[2022-05-31T18:54:38Z ERROR smithay_client_toolkit::event_loop] Error trying to flush the wayland display: Broken pipe (os error 32)

This error seems to imply fd leaks?

Reproduction:

1. Paste the example into a project.
2. Resize the window for about 30s

It may be desired to destroy the graphics context while preserving the window.

We need an API to list available pixel formats, and select one. Some considerations:

• We want a way to support transparency: #17
• The web backend current converts BGRX to RGBA. It would be better if it could use RGBA directly, if an application can render that way.
• It looks like for #44 we'd have to use RGBX, rather than BGRX.
• For #95, it seems BGRA is supported, but not BGRX. So we need to make sure to set alpha to 255.
• Wayland's wl_shm can support many formats, but only two are required to be supported by compositors.

So some questions here:

• What formats should be supported?
  • We want to offer a format for each platform that can be used efficiently without conversion, at least, and there doesn't seem to be one that can be assumed everywhere, unfortunately.
  • There are many formats that could be supported on at least some platforms.
• Do we want to support formats that are not 4 byte per pixel? This will impact the API, which currently uses u32 values for pixels.
• How to we communicate what formats are supported, and what formats are optimal?
• We still want at least one format to work on all platforms, even if it requires conversion on some, right? What formats should be universally supported, then?

Has there been any consideration as to wrapping the unsafe bits of the user facing API into a safe abstraction?

• What variants are useful to provide? What errors might a consumer of the library want to actually handle?
• Some error variants may only apply to certain methods. Should the same error type be used everywhere, or might it be helpful to have more than one?
• The x11 code currently has a swbuf_err helper method. Any helpers like this should be in error.rs and used in other backends where it's useful.

It would be cool if I could do WCG stuff with this crate. This would take a lot of care to do correctly though. Also see #14

This would allow us to use safer, direct system calls instead of libc.

Blocked on bytecodealliance/rustix#797

This limits the window to 65,536 * 65,536. :/

Is there any reason for this limitation, does a backend mandate it or something?

Hello!

I'm using softbuffer for a project of mine. On Linux, everything works flawlessly -- very well done. Even though I have to recreate the graphics context every frame because of my bad design.

On Windows, which I can't really test except through CI, the build breaks. I can replicate it locally by cross-compling to Windows. The error:

$ cargo build --release --target x86_64-pc-windows-gnu --bin redshell-concept --features sys_gui_softbuffer
# [compiling other stuff omitted for brevity]
Compiling softbuffer v0.1.1
error[E0432]: unresolved import `winapi::um::winuser`
 --> /home/nth/.cargo/registry/src/github.com-1ecc6299db9ec823/softbuffer-0.1.1/src/win32.rs:6:17
  |
6 | use winapi::um::winuser::{GetDC, ValidateRect};
  |                 ^^^^^^^ could not find `winuser` in `um`

Poking through winapi, it seems like this is feature-gated. That feature is being enabled, I think, so I'm not sure what's going on, but having replicated it on GitHub's Windows VM and locally through cross-compilation I don't think it's an issue with me.

The project I'm building is Redshell tag v0.0.5-rc2, if that helps.

@i509VCB suggested this on Matrix. I don't know about other platforms, but for Wayland it could definitely be desirable to not create a new registry, etc. for every window. Doesn't make a difference for uses with a single window, but with a lot of wl_surfaces it definitely is a good idea to not duplicate everything like that.

As mentioned when #16 was closed, it would be nice if softbuffer could provide a managed buffer to avoid the cost of copying. I'm imagining a double-buffered setup where you can call a function to swap the buffers.

Currently softbuffer can take up to 3-4ms to upload the buffer to the window, which is really slow. (edit: even slower with larger windows; see #18)

API design for this could be somewhat difficult but I doubt it'll be too hard. I can propose designs if you like

In order to efficiently draw on the CPU, it may be desireable to reduce how many pixels are actually drawn.

Often this involves telling the display server what pixels were changed so that the display server can reuse it's existing framebuffer or perform a partial upload of the texture data to the gpu to save pci bandwidth.

On macOS, the buffer is applied using logical pixels rather than physical pixels so a 100x100 buffer will fill a 100x100 (logical pixels) window even if the window occupies 200x200 (physical) pixels. A 200x200 buffer in the same scenario will just be cropped. This is a good default that makes UI easier to develop, but results in output that looks low res.

I'm currently working around this by setting my buffer to the physical pixel size, then scaling it on application by adding the following to set_buffer in cg.rs:
self.layer.set_contents_scale(width as f64 / self.layer.bounds().size.width);

I'm not sure what the right and real mechanism for doing this should be (e.g. always use physical? allow scale factor choice?)

As of the time of writing, the MSRV for softbuffer is 1.64.0. As far as I can tell there is no MSRV policy, but the informal rule seems to be that the limit is Rust versions released up to seven months ago. This issue keeps track of features that may be useful to softbuffer, but cannot be used yet because we don't have the required Rust version.

• Nightly
  • Fallible allocation could be worked into the error enum to provide a condition for low-memory systems.
  • We implement some parts as what's essentially a try block, but as a closure that is immediately called. Example. It would be nice to replace those cases with a simple try block.
  • We use a Box to wrap SurfaceDispatch in order to keep the type size predictable. However, for low memory systems we could use the allocator_api feature to provide a custom allocator to reduce allocations.

Feel free to add more that I've missed.

Compare https://registry.khronos.org/EGL/extensions/EXT/EGL_EXT_buffer_age.txt:

        For example, with a double buffered surface and an
        implementation that swaps via buffer exchanges, the age would
        usually be 2. With a triple buffered surface the age would
        usually be 3. An age of 1 means the previous swap was
        implemented as a copy. An age of 0 means the buffer has only
        just been initialized and the contents are undefined. Single
        buffered surfaces have no frame boundaries and therefore
        always have an age of 0.

We can have a method on softbuffer::Surface, something like fn age(&self) -> Option<NonZeroU8>. An application rendering with softbuffer could use this to avoid redrawing everything every frame when not much has changed, regardless of whether a backend uses double-buffering, etc.

Currently buffer_mut is documented saying "The initial contents of the buffer may be zeroed, or may contain a previous frame." So this would provide a way to know which previous frame.

The macOS code in softbuffer 0.2.0 consults backingScaleFactor() only when the graphics context is created. This means that, assuming the application always renders an image of the window's physical size, the image will display at the wrong scale (half or double) if the window has been dragged between monitors with different scale factors. Instead, the scale factor should probably be consulted at every set_buffer() call.

It may be useful to allow presenting CPU rendered buffers directly to the display on Linux.

raw-window-handle already supports GBM and DRM so it should be possible to use GBM or dumb buffers.

This makes the unsafe blocks of code more clear to pick out, especially when functions like WaylandImpl::new are unsafe.

It would be easier for organisation and contribution purposes to have issue labels for platforms

On Windows 10, set_buffer can take upwards of 15-20 ms, which is noticeably slow.

Perhaps a no-copy API could be introduced to mitigate this.

winapi is not maintained anymore. The newer windows crate is the official Rust bindings library to the Windows API.

It also helps with compile times, as all other big libraries have also switched to windows, like winit for example.

I know this is not released yet, but after #64 Context and Surface were split.

How long is Context supposed to be kept alive? Does it need to stay alive as long as Surface, or can it be safely dropped while Surface is kept around and used?

When running the animation example on Gnome/Mutter 43.2 it causes the entire desktop to freeze for about half a second, then the example terminates with the following error:

request could not be marshaled: can't send file descriptor
Error sending request: Resource temporarily unavailable

I've known about this bug for some time, but when I tried diving into Wayland to fix it, I was unable to find a solution. Maybe you can shed some light @ids1024?

Additionally, some other examples (notably winit_wrong_sized_buffer and fruit) are janky insofar as resizing causes temporary freezes of the softbuffer window.

A few things I noticed:
The WlShm and WlShmPool protocol objects are leaked when dropping a GraphicsContext

The client could modify pixels while the compositor is using the buffer. This is not allowed per wl_surface protocol.

Committing a pending wl_buffer allows the compositor to read the
pixels in the wl_buffer. The compositor may access the pixels at
any time after the wl_surface.commit request. When the compositor
will not access the pixels anymore, it will send the
wl_buffer.release event. Only after receiving wl_buffer.release,
the client may reuse the wl_buffer. A wl_buffer that has been
attached and then replaced by another attach instead of committed
will not receive a release event, and is not used by the
compositor.

In order to handle that safely, setting the buffer while the compositor is using the previous buffer either needs to tell the user to wait, needs to block or use double buffering.

Also because of this requirement, dropping a GraphicsContext implies we must block until the compositor to releases the buffer.

It would be nice if there were a method for getting pixels from a window in addition to setting them. If you wanted to draw a transparent or blended image using software rendering, you would need access to the image currently in the window.

Not sure if this is still an actual bug or just flaky on CI.

Attempted fix: 65b5d48
Related: #104 (comment)

EDIT: A bit of discussion on this happened in rust-windowing/swbuf#8 (comment)

Write a program that neglects to call Surface::resize(), for example by deleting all the size and rendering lines in examples/rectangle.rs:

Event::RedrawRequested(window_id) if window_id == window.id() => {
    let mut buffer = surface.buffer_mut().unwrap();
    buffer.present().unwrap();
}

Run this example on macOS, and observe the crash log — it looks like softbuffer gave a dangling pointer as the buffer pointer, by not checking whether the BufferImpl::buffer had nonzero capacity.

...
Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000004
Exception Codes:       0x0000000000000001, 0x0000000000000004
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [58050]

VM Region Info: 0x4 is not in any region.  Bytes before following region: 140737486381052
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      VM_ALLOCATE              7fffffe1e000-7fffffe1f000 [    4K] r-x/r-x SM=ALI  

Thread 0 Crashed:: main Dispatch queue: com.apple.main-thread
0   CoreGraphics                  	    0x7ff806a2cfb7 ERROR_CGDataProviderCreateWithData_BufferIsNotReadable + 39
1   CoreGraphics                  	    0x7ff806bb7096 check_clients_buffer + 118
2   CoreGraphics                  	    0x7ff806a2cf66 CGDataProviderCreateWithData + 35
3   rectangle                     	       0x10cd2ea5e core_graphics::data_provider::CGDataProvider::from_buffer::hf5a7344b1dce02de + 222
4   rectangle                     	       0x10cd2fce8 softbuffer::cg::BufferImpl::present::h33f92d08748cdedf + 88
5   rectangle                     	       0x10cd348b7 softbuffer::BufferDispatch::present::hf5096c1071865b18 + 55
6   rectangle                     	       0x10cd34417 softbuffer::Buffer::present::heb646d1ee9f8107d + 55
7   rectangle                     	       0x10cd2e311 rectangle::main::_$u7b$$u7b$closure$u7d$$u7d$::h7d4b422d965be099 + 481 (rectangle.rs:55)
...

While not specifying the size can't be expected to do anything useful, it would be better if this mistake were turned into a SoftBufferError (or at least a panic) instead of an abort whose informative details have to be dug out of a crash log file.

Changes: v0.1.1...master

I could use some help from others who have submitted PRs recently.

Here's an example usage of XShm (originally from Fabrice Bellard's TinyGL): https://github.com/lunixbochs/tinygles/blob/unstable/src/gles/glx.c#L107

Basically, this allows you to blit pixels directly into X11's memory, saving both a memory copy and the general overhead of passing your image through an X11 socket.

In the non-shm case, you also shouldn't need to create a new XImage every frame as you're doing now. You only really need to do it on startup and when the window is resized. You can just call XPutImage on an existing XImage repeatedly, as done here: https://github.com/lunixbochs/tinygles/blob/unstable/src/gles/glx.c#L368

Currently the initial window size is always 0 in width and height, this will lead to panics when calling buffer_mut() if the size wasn't set with resize() depending on the platform:

• Wayland: panics.
• MacOS: doesn't panic.
• Redox: doesn't panic.
• Web: doesn't panic, but will panic when calling present() or fetch() (#104).
• Windows: panics.
• X11: doesn't panic.

I would prefer if we make this consistent on all platforms. My current ideas:

1. Require an initial size in Surface::new().
2. Set the initial size to the window size (don't know if that's possible on all platforms).
3. Introduce a new error variant instead of panicking.

Currently leaning towards idea 1.

Basically:

#161
#159
#158
#147 (maybe)
#132

Is it possible to perform vertical sync with this library? i.e. deliver a new frame to the screen every monitor refresh, but not any more frequently? It doesn't seem like the library does anything special about it, but I'm not 100% sure how this should work (is it the responsibility of the user to figure out the refresh rate and stuff?).

After a long discussion in #104 (comment), we came to the conclusion that we should consider endianness when appropriate, this is mainly examples, tests and documentation.

Especially this line:

I would like to mention that I'm strongly in favor of changing the current buffer from [u32] to [u8], which seems like a much more user-friendly API to me and also makes most of the endianness considerations go away.

The main concern @ids1024 had about this is performance, which could be addressed by leaving the buffer as a [u32], but expose it as [u8] in the Deref implementation, we could still introduce a as_u32() method then. Though I would like to see a benchmark showing an actual performance regression before we put effort into something unconfirmed like this.

On Wayland, we need to wait for the server to release a wl_buffer before we can reuse it. This generally means clients will double buffer and track buffer age when rendering. However it would be most ideal to avoid double buffering if at all possible.

This probably means that softbuffer also needs a way to check if the buffer can be acquired to be rendered to if we run out of buffers to use.

Taken from the latest CI logs:

          Warning: .drectve `-exclude-symbols:_ZN17raw_window_handle8borrowed3imp6Active6handle17hcdc9a7e46fa42377E ' unrecognized
          Warning: .drectve `-exclude-symbols:_ZN17raw_window_handle8borrowed3imp6Active10set_active17h87aab7a0dbfa799aE ' unrecognized
          Warning: .drectve `-exclude-symbols:_ZN17raw_window_handle8borrowed3imp6Active12set_inactive17hdc388a803d443f6fE ' unrecognized
          Warning: .drectve `-exclude-symbols:_ZN17raw_window_handle8borrowed3imp12ActiveHandle13new_unchecked17h717ec49c81207b3aE ' unrecognized
          Warning: corrupt .drectve at end of def file
          Warning: corrupt .drectve at end of def file
          Warning: corrupt .drectve at end of def file
          Warning: corrupt .drectve at end of def file
          collect2.exe: error: ld returned 5 exit status

This failure only occurs for *-pc-windows-gnu, not *-pc-windows-msvc. Also it apparently only happens under Windows, as I can build it for x86_64-pc-windows-gnu just fine on my Linux laptop.

I need to go to bed now, but I'll investigate this on my Windows VM this week.

Right now, softbuffer seems to completely ignore the alpha channel. It would be nice if this was supported.

The example on the README.md file (Here) seems to contain structs(?) like softbuffer::Context and softbuffer::Surface which aren't in the documentation. (nor seem to exist at all?)

There's an extremely similar example used in the docs.rs softbuffer home-page (Here) which does compile.

What's strange is that I can't seem to find any mention of softbuffer::Context and softbuffer::Surface in any of the older versions' documentation.

X11 is one of the rare platforms with two libraries that interface with the protocol. Ideally swbuf should support working with both Xcb and Xlib handles.

Apparently macOS (and iOS, #43) has a framework called IOSurface for exchanging framebuffers and textures between processes, which sounds similar to the idea behind dmabufs on Linux. I think we should be use IOSurfaces for a front and back buffer, and use IOSurfaceGetBaseAddress to get a pointer to write into for no-copy presentation (#65)? Assuming it can work with the right pixel format.

Or are there issues with this, or a better way?

This library currently unconditionally requires Wayland. Wayland should be an optional feature, even if enabled by default.

Many compositors implement damage tracking, meaning that commiting a new buffer when calling set_buffer does nothing.

A proper fix would involve the caller giving the implementation a set of rectangles where content has changed.

It takes a slice. Is this slice immediately cloned, or does the window keep a reference to it? Should I double-buffer, or can I clobber the buffer after the call to set_buffer?

Normally the fact that the slice doesn't stay borrowed would be a hint, but as this is a relatively new library using unsafe code, it's possible that it could be accidentally unsound, which is why documenting this would be helpful.

The current X11 backend sends the image over the wire, which is slow and inefficient. We should probably use the MIT-SHM strategy for this crate if it's available.

(Also, we probably shouldn't create a new image every time we run the set_buffer command, but I think we can take care of that when we solve #37)

Right now this crate only supports 24-bit sRGB, it would be nice if I could use 48-bit for example.

Hello, this is my first time trying to use your library.

I just copied and pasted your example from the readme:

use softbuffer::GraphicsContext;
use winit::event::{Event, WindowEvent};
use winit::event_loop::{ControlFlow, EventLoop};
use winit::window::WindowBuilder;

fn main() {
    let event_loop = EventLoop::new();
    let window = WindowBuilder::new().build(&event_loop).unwrap();
    let mut graphics_context = unsafe { GraphicsContext::new(window) }.unwrap();

    event_loop.run(move |event, _, control_flow| {
        *control_flow = ControlFlow::Wait;

        match event {
            Event::RedrawRequested(window_id) if window_id == graphics_context.window().id() => {
                let (width, height) = {
                    let size = graphics_context.window().inner_size();
                    (size.width, size.height)
                };
                let buffer = (0..((width * height) as usize))
                    .map(|index| {
                        let y = index / (width as usize);
                        let x = index % (width as usize);
                        let red = x % 255;
                        let green = y % 255;
                        let blue = (x * y) % 255;

                        let color = blue | (green << 8) | (red << 16);

                        color as u32
                    })
                    .collect::<Vec<_>>();

                graphics_context.set_buffer(&buffer, width as u16, height as u16);
            }
            Event::WindowEvent {
                event: WindowEvent::CloseRequested,
                window_id,
            } if window_id == graphics_context.window().id() => {
                *control_flow = ControlFlow::Exit;
            }
            _ => {}
        }
    });
}

Here is my cargo.toml:

[dependencies]
softbuffer = "0.1.1"
winit = "0.27.1"

I get this when trying to build:

error[E0432]: unresolved import `winapi::shared::windef`
 --> C:\Users\ilmer\.cargo\registry\src\github.com-1ecc6299db9ec823\softbuffer-0.1.1\src\win32.rs:4:21
  |
4 | use winapi::shared::windef::{HDC, HWND};
  |                     ^^^^^^ could not find `windef` in `shared`

error[E0432]: unresolved import `winapi::um::wingdi`
 --> C:\Users\ilmer\.cargo\registry\src\github.com-1ecc6299db9ec823\softbuffer-0.1.1\src\win32.rs:5:17
  |
5 | use winapi::um::wingdi::{StretchDIBits, BITMAPINFOHEADER, BI_BITFIELDS, RGBQUAD};
  |                 ^^^^^^ could not find `wingdi` in `um`

error[E0432]: unresolved import `winapi::um::winuser`
 --> C:\Users\ilmer\.cargo\registry\src\github.com-1ecc6299db9ec823\softbuffer-0.1.1\src\win32.rs:6:17
  |
6 | use winapi::um::winuser::{GetDC, ValidateRect};
  |                 ^^^^^^^ could not find `winuser` in `um`

error[E0433]: failed to resolve: could not find `wingdi` in `um`
  --> C:\Users\ilmer\.cargo\registry\src\github.com-1ecc6299db9ec823\softbuffer-0.1.1\src\win32.rs:64:25
   |
64 |             winapi::um::wingdi::DIB_RGB_COLORS,
   |                         ^^^^^^ could not find `wingdi` in `um`

error[E0433]: failed to resolve: could not find `wingdi` in `um`
  --> C:\Users\ilmer\.cargo\registry\src\github.com-1ecc6299db9ec823\softbuffer-0.1.1\src\win32.rs:65:25
   |
65 |             winapi::um::wingdi::SRCCOPY,
   |                         ^^^^^^ could not find `wingdi` in `um`

Did I do something wrong?

Otherwise, I see a lot of potential in your library, how we use winit instead of trying to reinvent the wheel.

Thank you,
Veniamin

Hi, I'm using softbuffer in my toy terminal emulator project on Wayland, and I had to pull softbuffer as a git dependency in order for it to do proper Wayland damage tracking on redraws. Could you please publish the latest master onto crates.io?