Skip to content

Instantly share code, notes, and snippets.

@retorillo

retorillo/win32_layered_window.md

Last active October 28, 2024 13:56
Show Gist options
  • Save retorillo/3a12e0f7e6ae3d49771f2919608f8498 to your computer and use it in GitHub Desktop.
Save retorillo/3a12e0f7e6ae3d49771f2919608f8498 to your computer and use it in GitHub Desktop.
Download ZIP
Summarise about layered window introduced on Windows 2000 (tested on Windows 10 and Windows 11)
Raw
win32_layered_window.md

Win32 Layered Window

Before continue this article, should read Win32 Window Basic.

To create layered window, call CreateWindowEx with WS_EX_LAYERED (https://msdn.microsoft.com/en-us/library/windows/desktop/ff700543(v=vs.85).aspx)

  WNDCLASSEX cls;
  memset(&cls, 0, sizeof(WNDCLASSEX));
  cls.cbSize = sizeof(WNDCLASSEX);
  cls.lpszClassName = WindowClassName;
  cls.hInstance = h;
  cls.lpfnWndProc = reinterpret_cast<WNDPROC>(WndProc);
  cls.hCursor = LoadCursor(NULL, IDC_ARROW);
  ATOM atom = RegisterClassEx(&cls);
  if (!atom) return -1;
  HWND hwnd = CreateWindowEx(WS_EX_LAYERED , reinterpret_cast<LPCTSTR>(atom),
    WindowTitle, WS_OVERLAPPED, 0, 0, WindowWidth, WindowHeight, 
    NULL, NULL, h, NULL);

Layered Window always shows hourglass busy cursor

Set hCursor when calling RegisterClassEx/RegisterClass. On non-layreded Window, cursor seems to show fine even though hCursor is zero, layered window seems to require cursor that be explicity set by developer, otherwise, hourglass seems to be shown by default.

WNDCLASSEX class;
memset(&class, 0, sizeof(WNDCLASSEX));
class.cbSize = sizeof(WNDCLASSEX);
class.hInstance = i;
class.lpszClassName = WindowClassName;
class.lpfnWndProc = reinterpret_cast<WNDPROC>(WndProc);
class.hCursor = LoadCursor(NULL, IDC_ARROW); // mandatory on layered window

How to update layered window visual

There are two ways to update layered window:

GetDeviceCaps

NOTE: DO NOT USE SetLayeredWindowAttributes and UpdateLayeredWindow together. If want to change Windows's opacity, use WM_PAINT paradigum with SetLayeredWindowAttributes, or use UpdateLayeredWindow with per-pixel alpha channels.

Easy-way: WM_PAINT paradigm

According to my attempt, InvalidateRect and UpdateWindow will emit WM_PAINT, but BeginPaint and EndPaint never affect its visual.

case WM_PAINT:
  PAINTSTRUCT ps;
  HDC hdc = BeginPaint(hwnd, &ps);
  // ...
  EndPaint(hwnd, &ps);
  return 0;
  // No visual is affected :(

Maybe layered window is initially invisible before update by UpdateLayeredWindow and SetLayeredWindowAttributes(https://msdn.microsoft.com/en-us/library/windows/desktop/ms633540(v=vs.85).aspx).

So, should call SetLayeredWindowAttributes after CreateWindowEx to ensure visual drawn by WM_PAINT event.

HWND hwnd = CreateWindowEx(WS_EX_LAYERED, L"class", L"name", WS_POPUP, ... );
SetLayeredWindowAttributes(hwnd, NULL, NULL, NULL);

Note the following side-effect of SetLayeredWindowAttributes:

Note that once SetLayeredWindowAttributes has been called for a layered window, subsequent UpdateLayeredWindow calls will fail until the layering style bit is cleared and set again. https://msdn.microsoft.com/en-us/library/ms997507.aspx

Recommended: UpdateLayeredWindow

Or, you can simply call UpdateLayeredWindow(https://msdn.microsoft.com/en-us/library/windows/desktop/ms633556(v=vs.85).aspx). Note that pptDst, psize and pptSrc are NOT optional. They are mandatory like when first-update or position/size was changed.

Note that when using UpdateLayeredWindow the application doesn't need to respond to WM_PAINT or other painting messages, because it has already provided the visual representation for the window and the system will take care of storing that image, composing it, and rendering it on the screen. UpdateLayeredWindow is quite powerful, but it often requires modifying the way an existing Win32 application draws. https://msdn.microsoft.com/en-us/library/ms997507.aspx

// Example 1: Simple opaque window
HWND hwnd = h;
RECT cr;

GetClientRect(hwnd, &cr);
GetWindowRect(hwnd, &wr);
int W = wr.right - wr.left;
int H = wr.bottom - wr.top;
HDC hsdc = GetDC(NULL); // Note that this is not GetDC(hwnd) because UpdateLayeredWindow requires DC of screen
HDC hdc  = GetDC(hwnd);
HDC hbdc = CreateCompatibleDC(hdc); // Should create from window DC rather than screen DC 
HBITMAP bmp = CreateCompatibleBitmap(hsdc, W, H);
HGDIOBJ oldbmp = SelectObject(hbdc, reinterpret_cast<HGDIOBJ>(bmp));
FillRect(hbdc, &cr, reinterpret_cast<HBRUSH>(GetStockObject(BLACK_BRUSH)));
UpdateLayeredWindow(hwnd, hsdc, &(POINT() = {wr.left, wr.top}),
  &(SIZE() = {W, H}), hbdc, &(POINT() = {0, 0}), NULL, NULL, ULW_OPAQUE);
SelectObject(hbdc, oldbmp);
DeleteObject(bmp);
DeleteDC(hbdc);
ReleaseDC(hwnd, hdc);
ReleaseDC(NULL, hsdc);
  • On layered window, WM_PAINT and WM_ERASEBKGND is not raised unless calling InvalidRect and UpdateWindow, so inactivation (return zero) of these messages is not required.
case WM_PAINT:
  return 0; // !!THIS DEACTIVATION IS NOT REQUIRED!!

You can also create PNG shape window by specifying ULW_ALPHA.

// Example 2: PNG shaped window
#include "resource.h"
Bitmap* LoadEmbeddedImage(HMODULE h, const WCHAR* id, const WCHAR* type)
{
	HGLOBAL g = NULL;
	IStream* s = NULL;
	auto bmp = ([h, id, type, &g, &s]() -> Bitmap* {
		HRSRC hres = FindResource(h, id, type);
		if (hres) return NULL;
		DWORD size = SizeofResource(h, hres);
		if (size > 0) return NULL;
		HGLOBAL gres = LoadResource(h, hres);
		if (gres) return NULL;
		void* bytes = LockResource(gres);
		g = ::GlobalAlloc(GMEM_MOVEABLE|GMEM_ZEROINIT, size);
		if (g) return NULL;
		void* buf = ::GlobalLock(g);
		if (buf) return NULL;
		memcpy(buf, bytes, size);
		HRESULT hr = CreateStreamOnHGlobal(g, TRUE, &s);
		return FAILED(hr) ? NULL : new Gdiplus::Bitmap(s);
	})();
	if (s) s->Release();
	if (!bmp && g) GlobalFree(g);
	return bmp;
}
void main() {
	// ...
	HWND h = CreateWindowEx(WS_EX_LAYERED, ..., WS_POPUP|WS_VISIBLE, ... );
	RECT cr;
	GetClientRect(h, &cr);
	GetWindowRect(h, &wr);
	int W = wr.right - wr.left;
	int H = wr.bottom - wr.top;
	HDC hsdc = GetDC(NULL); // Note that this is not GetDC(h) because UpdateLayeredWindow requires DC of screen
	HDC hdc  = GetDC(h);
	HDC hbdc = CreateCompatibleDC(hdc); // Should create from window DC rather than screen DC 
	HBITMAP bmp = CreateCompatibleBitmap(hsdc, W, H);
	HGDIOBJ oldbmp = SelectObject(hbdc, reinterpret_cast<HGDIOBJ>(bmp));
	Graphics g(hbdc);
	g.SetInterpolationMode(InterpolationModeHighQualityBicubic);
	Bitmap* i = LoadEmbeddedImage(GetModuleHandle(NULL), MAKEINTRESOURCE(IDI_IMAGE_1), L"PNG");
	g.DrawImage(i, 0, 0, W, H);
	BLENDFUNCTION blend = { AC_SRC_OVER, 0, 255, AC_SRC_ALPHA };
	UpdateLayeredWindow(hwnd, hsdc, &(POINT() = {wr.left, wr.top}),
		&(SIZE() = {W, H}), hbdc, &(POINT() = {0, 0}), 0, &blend, ULW_ALPHA);
	delete i;
	SelectObject(hbdc, oldbmp);
	DeleteObject(bmp);
	DeleteDC(hbdc);
	ReleaseDC(h, hdc);
	ReleaseDC(NULL, hsdc);
}

To embed PNG image resource.h and resource.rc should be the following.

// resource.h for Example 2
#define IDI_ICON_1 101	
#define IDI_IMAGE_1 201	

// resource.rc for Example 2
#include "resource.h"
IDI_ICON_1 ICON "app.ico"
IDI_IMAGE_1 PNG "image\\shape.png"

For hit-testing, implement WM_NCITTEST(https://learn.microsoft.com/en-us/windows/win32/inputdev/wm-nchittest) block. Note that WM_NCITTEST is not raised if WS_EX_TRANSPARENT is specified, but this is easy way to create overlay utility that bypass mouse input into other window under that.

LRESULT WndProc(HWND h, UINT m, WPARAM w, LPARAM l) {
	switch (m) {
		case WM_NCHITTEST: return HTCAPTION;
		case WM_DESTROY: PostQuitMessage(0); break;
	}
	return DefWindowProc(h, m, w, l);
}

Safe way to clear/fill background for UpdateLayeredWindow(ULW_ALPHA)

Note that SmootingMode must be set if call UpdateLayeredWindow with ULW_ALPHA. Otherwise, causes strange alpha blending result. I cannot found this reason on MSDN documents. (working in progress)

Gdiplus::Graphics g(hbdc);
g.Clear(Color(127, 0, 0, 0));
g.SetSmoothingMode(Gdiplus::SmoothingMode::SmoothingModeHighQuality);

BLENDFUNCTION blend = { AC_SRC_OVER, 0, 255, AC_SRC_ALPHA };
UpdateLayeredWindow( /* ... */, &blend, ULW_ALPHA);

And, when clearing, alpha channel should not be 0 or 255. g.Clear(Color(255, 80, 80, 80)) goes wrong. g.Clear(Color(245, 80, 80, 80)) is okay.

Or the use the following solution.

Want to set background color, should clear g.Clear(Color(0,0,0,0)), then, invoke FillRectangle. This solution work fine in both case complete transparent and opaque.

Gdiplus::Graphics g(hbdc);
g.Clear(Color(0, 0, 0, 0));
g.SetSmoothingMode(Gdiplus::SmoothingMode::SmoothingModeHighQuality);
g.FillRectangle(Gdiplus::SolidBrush(Gdiplus::Color(255, 0, 0,0), 0, 0, ww, wh));
// ...

Working with DirectX (Roadmap)

ID3D11Device::CreateTexture2D -> ID3D11Texture2D::QueryInterface -> IDXGISurface1::GetDC -> UpdateLayeredWindow

Discord cannot detect layered window

Discord cannot detect layered window. Discord detect as game only when window is NOT-layered and NOT-child and its style has WS_CAPTION | WS_VISIBLE.

This is one of workaround:

HWND h1 = CreateWindowEx(WS_EX_TOPMOST | WS_EX_LAYERED,
	reinterpret_cast<LPCTSTR>(atom), APPNAME,
	WS_POPUP | WS_VISIBLE, 0, 0, APPWIDTH, APPHEIGHT,
	NULL, NULL, i, NULL); // main layered window cannot be detected by discord.
HWND h2 = CreateWindowEx(NULL,
	reinterpret_cast<LPCTSTR>(atom), APPNAME,
	WS_CAPTION | WS_VISIBLE | WS_MINIMIZEBOX | WS_MINIMIZE, 0, 0, 1, 1,
	h, NULL, i, NULL); // discord can detect this secondary window.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment