Skip to content

fix(LuaRenderer): visible pink alpha mask on platforms without LWA_COLORKEY support#587

Open
Aurumaker72 wants to merge 6 commits intomainfrom
ulw-gdi-2
Open

fix(LuaRenderer): visible pink alpha mask on platforms without LWA_COLORKEY support#587
Aurumaker72 wants to merge 6 commits intomainfrom
ulw-gdi-2

Conversation

@Aurumaker72
Copy link
Copy Markdown
Member

@Aurumaker72 Aurumaker72 commented Mar 31, 2026

Changes LuaRenderer and GDIPresenter to use UpdateLayeredWindow with ULW_COLORKEY instead of relying on the DWM compositing a window with LWA_COLORKEY properly.

Closes #243.

Copilot AI review requested due to automatic review settings March 31, 2026 18:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Addresses issue #243 by changing how the Lua overlay layers are composited, aiming to prevent the magenta/pink colorkey mask from becoming visible on platforms where LWA_COLORKEY is unreliable/unsupported.

Changes:

  • Removes the GDI “front DC” path and WM_PAINT-based GDI overlay presentation.
  • Consolidates the overlay window classes into a single lua_overlay class.
  • Presents GDI overlay content via UpdateLayeredWindow(..., ULW_COLORKEY) from the render loop.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
src/Views.Win32/lua/LuaTypes.h Removes gdi_front_dc from the Lua rendering context since the GDI overlay is no longer blitted via WM_PAINT.
src/Views.Win32/lua/LuaRenderer.cpp Reworks overlay window creation/registration and switches GDI overlay presentation to UpdateLayeredWindow colorkey updates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@@ -33,12 +32,7 @@ bool GDIPresenter::init(HWND hwnd)
SelectObject(m_gdi_back_dc, m_gdi_bmp);
ReleaseDC(hwnd, gdi_dc);

Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GDIPresenter::init() allocates a new compatible bitmap but never initializes it. Because end_present() now uses UpdateLayeredWindow(..., ULW_COLORKEY), any pixels that aren’t explicitly drawn/cleared can show up as garbage or as an opaque overlay. Initialize the back DC to the mask color right after creating/selecting m_gdi_bmp (same approach as in resize()).

Suggested change
// Initialize the back-buffer bitmap to the mask color to avoid garbage pixels
RECT init_rect = {0, 0, (LONG)m_size.width, (LONG)m_size.height};
const auto init_brush = CreateSolidBrush(m_mask_color);
FillRect(m_gdi_back_dc, &init_rect, init_brush);
DeleteObject(init_brush);

Copilot uses AI. Check for mistakes.

// If we don't fill up the DC with the key first, it never becomes "transparent"
FillRect(ctx->gdi_back_dc, &window_rect, g_alpha_mask_brush);

Copy link

Copilot AI Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gdi_overlay_hwnd is created as a layered/visible child window, but (in lazy-init cases where has_gdi_content remains false) it never gets a colorkey-applied UpdateLayeredWindow call and there is no WM_PAINT handler anymore. That can leave an opaque overlay painted by DefWindowProc (regression vs the old SetLayeredWindowAttributes path). Consider either (1) calling UpdateLayeredWindow(..., ULW_COLORKEY) once here after the initial FillRect to establish full transparency, or (2) creating the GDI overlay hidden and only showing/updating it after mark_gdi_content_present() flips has_gdi_content true.

Suggested change
// Ensure the initial GDI overlay is fully transparent by applying the color key once.
{
LOGBRUSH lb{};
COLORREF color_key = RGB(0, 0, 0);
if (GetObject(g_alpha_mask_brush, sizeof(lb), &lb) == sizeof(lb))
{
color_key = lb.lbColor;
}
SIZE size = {static_cast<LONG>(ctx->dc_size.width), static_cast<LONG>(ctx->dc_size.height)};
POINT pt_src = {0, 0};
UpdateLayeredWindow(ctx->gdi_overlay_hwnd,
nullptr,
nullptr,
&size,
ctx->gdi_back_dc,
&pt_src,
color_key,
nullptr,
ULW_COLORKEY);
}

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GDI presenter flickers pink filled screens on Linux when running lua.

2 participants