Fix DMA-BUF checks and events

[UjinT34]

Describe your PR, what does it fix/add?

Correctly sends format and modifier events on bind.
Adds two v4 and v5 sanity checks https://wayland.app/protocols/linux-dmabuf-v1#zwp_linux_buffer_params_v1:request:add
Adds quirks:skip_non_kms_dmabuf_formats to not report format+modifier pairs which aren't importable to KMS. Fixes wine-wayland DS activation on my nvidia setup. Fixes mpv freeze with DS. Requires restart or triggering resetFormatTable. #12714

Is there anything you want to mention? (unchecked code, possible bugs, found problems, breaking compatibility, etc.)

quirks:skip_non_kms_dmabuf_formats probably has no effect on mesa because eglQueryDmaBufFormatsEXT and getModsForFormat return the same set as Aquamarine::CDRMBackend::getRenderFormats. For me egl returns extra mods with compression and those aren't importable into KMS.
Format table order matters. Some clients seem to choose first/last format or modifier without actually checking whether it's suitable.
Clients that actually check the format table and support feedbacks should work without quirks:skip_non_kms_dmabuf_formats.

Is it ready for merging, or does it need work?

Ready

[Gliczy]

I didn't extensively tested this, but setting quirks:skip_non_kms_dmabuf_formats = 1 did fix both DS and VRR activation in wine-wayland games (on mesa 25.3.3 with a 6750XT).

Thanks for fixing this!

[visixx]

Tested (wine-wayland) Garry's Mod, Farlight 84, THE FINALS, The Witcher 3, and Cyberpunk 2077 with quirks:skip_non_kms_dmabuf_formats = 1 on an RTX 4090 and verified* that DS and VRR were working as expected in all games. HDR worked as expected with render:non_shader_cm = 2 in Cyberpunk 2077 and The Witcher 3 with DS enabled for both. A noticeably smoother experience in all games!

* using watch 'hyprctl monitors | rg "(vrr|direct|[Tt]earing)"' looking for a window address being reported in directScanoutTo: <address> instead of 0.

[fxzzi]

yep working fine here too, I tested GTAV on wine wayland and DS and VRR successfully activating.

As reported on discord CP2077 HDR successfully disabled DS in its HDR10 PQ mode, but its HDR10 scRGB mode did not, making all the colours very fried.

There still seems to be some sort of race condition between tearing and ds. I leave both enabled just so I can have at least one working at (most) times, and I'm aware they don't work together. But with both enabled I am still sometimes able to catch my main monitor completely freezing up when leaving and entering fullscreen.
Also sometimes seeing my monitor freeze with just DS enabled and running hyprctl reload.

Trying to activate hyprsunset whilst a monitor is in DS also seems to still be broken. It works fine if I leave fullscreen, activate hyprsunset, then enter fullscreen again.

[fxzzi]

I think wayfreeze is broken with this?
edit: not just wayfreeze, grim also giving [destroyed object]: error 0: Format invalid when in HDR mode

called `Result::unwrap()` on an `Err` value: Backend(Protocol(ProtocolError { code: 0, object_id: 21, object_interface: "wl_shm_pool", message: "Format invalid" }))

[UjinT34]

I think wayfreeze is broken with this? edit: not just wayfreeze, grim also giving [destroyed object]: error 0: Format invalid when in HDR mode

called `Result::unwrap()` on an `Err` value: Backend(Protocol(ProtocolError { code: 0, object_id: 21, object_interface: "wl_shm_pool", message: "Format invalid" }))

This PR doesn't touch shm. Either the error message is a bit misleading or it's not related to this PR. WAYLAND_DEBUG=1 might help. There is indeed a couple of extra checks mentioned in https://wayland.app/protocols/linux-dmabuf-v1#zwp_linux_buffer_params_v1:request:add. If this error is caused by them then it's a wayfreeze bug. But it's highly unlikely because it doesn't use the dma proto. Looks like it's hitting

Hyprland/src/protocols/core/Shm.cpp

Lines 145 to 148 in 8d03fcc

	if UNLIKELY (std::ranges::find(PROTO::shm->m_shmFormats, fmt) == PROTO::shm->m_shmFormats.end()) {
	r->error(WL_SHM_ERROR_INVALID_FORMAT, "Format invalid");
	return;
	}

And it might happen if it sees some unusual dma format in DS mode and passes it to shm without checking. Idk whether there is a reason to limit shm format list.

[fxzzi]

hmm ok, It continued happening after I went back to SDR, and only fixed when I restarted Hyprland. I'll see if I can find a repro case for this and report it elsewhere

[UjinT34]

wayfreeze uses values received from https://wayland.app/protocols/wlr-screencopy-unstable-v1#zwlr_screencopy_frame_v1:event:buffer event to create a shm buffer.

Provides information about wl_shm buffer parameters that need to be used for this frame. This event is sent once after the frame is created if wl_shm buffers are supported.

So it's a bug in HL wlr screencopy proto implementation. misc:screencopy_force_8b = 1 should fix it for now.

[fxzzi]

misc:screencopy_force_8b = 1 should fix it for now.

that's funny because isn't this the default? I haven't modified this setting at all

[UjinT34]

It should fix... but it work only for a subset of 10bit formats :D
Without DS there is a limited set of formats that can be set an output format. DS allows apps to choose whatever they want from a list of supported by the gpu. Without quirks:skip_non_kms_dmabuf_formats = 1 apps often choose something that doesn't work with DS and therefore don't break other stuff. I guess we should examine every bit of HL's code containing 101010 or 8888.