fix(LuaRenderer): visible pink alpha mask on platforms without LWA_COLORKEY support#587
fix(LuaRenderer): visible pink alpha mask on platforms without LWA_COLORKEY support#587Aurumaker72 wants to merge 6 commits intomainfrom
LWA_COLORKEY support#587Conversation
SetLayeredWindowAttributes with LWA_COLORKEY
There was a problem hiding this comment.
Pull request overview
Addresses issue #243 by changing how the Lua overlay layers are composited, aiming to prevent the magenta/pink colorkey mask from becoming visible on platforms where LWA_COLORKEY is unreliable/unsupported.
Changes:
- Removes the GDI “front DC” path and WM_PAINT-based GDI overlay presentation.
- Consolidates the overlay window classes into a single
lua_overlayclass. - Presents GDI overlay content via
UpdateLayeredWindow(..., ULW_COLORKEY)from the render loop.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
src/Views.Win32/lua/LuaTypes.h |
Removes gdi_front_dc from the Lua rendering context since the GDI overlay is no longer blitted via WM_PAINT. |
src/Views.Win32/lua/LuaRenderer.cpp |
Reworks overlay window creation/registration and switches GDI overlay presentation to UpdateLayeredWindow colorkey updates. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -33,12 +32,7 @@ bool GDIPresenter::init(HWND hwnd) | |||
| SelectObject(m_gdi_back_dc, m_gdi_bmp); | |||
| ReleaseDC(hwnd, gdi_dc); | |||
|
|
|||
There was a problem hiding this comment.
GDIPresenter::init() allocates a new compatible bitmap but never initializes it. Because end_present() now uses UpdateLayeredWindow(..., ULW_COLORKEY), any pixels that aren’t explicitly drawn/cleared can show up as garbage or as an opaque overlay. Initialize the back DC to the mask color right after creating/selecting m_gdi_bmp (same approach as in resize()).
| // Initialize the back-buffer bitmap to the mask color to avoid garbage pixels | |
| RECT init_rect = {0, 0, (LONG)m_size.width, (LONG)m_size.height}; | |
| const auto init_brush = CreateSolidBrush(m_mask_color); | |
| FillRect(m_gdi_back_dc, &init_rect, init_brush); | |
| DeleteObject(init_brush); |
|
|
||
| // If we don't fill up the DC with the key first, it never becomes "transparent" | ||
| FillRect(ctx->gdi_back_dc, &window_rect, g_alpha_mask_brush); | ||
|
|
There was a problem hiding this comment.
gdi_overlay_hwnd is created as a layered/visible child window, but (in lazy-init cases where has_gdi_content remains false) it never gets a colorkey-applied UpdateLayeredWindow call and there is no WM_PAINT handler anymore. That can leave an opaque overlay painted by DefWindowProc (regression vs the old SetLayeredWindowAttributes path). Consider either (1) calling UpdateLayeredWindow(..., ULW_COLORKEY) once here after the initial FillRect to establish full transparency, or (2) creating the GDI overlay hidden and only showing/updating it after mark_gdi_content_present() flips has_gdi_content true.
| // Ensure the initial GDI overlay is fully transparent by applying the color key once. | |
| { | |
| LOGBRUSH lb{}; | |
| COLORREF color_key = RGB(0, 0, 0); | |
| if (GetObject(g_alpha_mask_brush, sizeof(lb), &lb) == sizeof(lb)) | |
| { | |
| color_key = lb.lbColor; | |
| } | |
| SIZE size = {static_cast<LONG>(ctx->dc_size.width), static_cast<LONG>(ctx->dc_size.height)}; | |
| POINT pt_src = {0, 0}; | |
| UpdateLayeredWindow(ctx->gdi_overlay_hwnd, | |
| nullptr, | |
| nullptr, | |
| &size, | |
| ctx->gdi_back_dc, | |
| &pt_src, | |
| color_key, | |
| nullptr, | |
| ULW_COLORKEY); | |
| } |
Changes
LuaRendererandGDIPresenterto useUpdateLayeredWindowwithULW_COLORKEYinstead of relying on the DWM compositing a window withLWA_COLORKEYproperly.Closes #243.