JavaFX Minimizing Undecorated Stage

I have an undecorated JavaFX Stage, and my own minimize, maximize & close buttons. But unfortunately clicking the taskbar icon in Windows 7 does not automatically minimize the stage - compared to the decorated behaviour.

Is there a way to minimize an undecorated stage with pure Java code, by clicking the taskbar icon? If not how can I do this with, say, JNA?

EDIT: OK, I've been trying to solve this with JNA, but having done next to none C/C++/JNA, I have a bit trouble setting this up. I'd be grateful if someone helped me to put the pieces together..

Here's my code so far:

public final class Utils {

   static {
    if (PlatformUtil.isWin7OrLater()) {
            Native.register("shell32");
            Native.register("user32");
        }
    }

    // Apparently, this is the event I am after
    public static final int WM_ACTIVATEAPP = 0x1C;


    public static void registerMinimizeHandler(Stage stage) {
       // Hacky way to get a pointer to JavaFX Window
       Pointer pointer = getWindowPointer(stage);

       WinDef.HWND hwnd = new WinDef.HWND(pointer);

       // Here's my minimize/activate handler
       WinUser.WindowProc windowProc = new MinimizeHandler(stage);

       Pointer magicPointer = ... set this to point to windowProc?

       // This.. apparently, re-sets the WndProc? But how do I get the "magicPointer" that is "attached" to the windowProc?
       User32.INSTANCE.SetWindowLong(hwnd, User32.GWL_WNDPROC, magicPointer);
    }
}

 private static class MinimizeHandler implements WinUser.WindowProc {

    private Stage stage;

    private MinimizeHandler(Stage stage) {
        this.stage = stage;
    }

    @Override
    public WinDef.LRESULT callback(WinDef.HWND hWnd, int uMsg, WinDef.WPARAM wParam, WinDef.LPARAM lParam) {
        if (uMsg == WM_ACTIVATEAPP) {
            System.out.println("ACTIVATE");
        }
        return User32.INSTANCE.DefWindowProc(hWnd, uMsg, wParam, lParam);
    }
}

private static Pointer getWindowPointer(Stage stage) {
    try {
        TKStage tkStage = stage.impl_getPeer();
        Method getPlatformWindow = tkStage.getClass().getDeclaredMethod("getPlatformWindow" );
        getPlatformWindow.setAccessible(true);
        Object platformWindow = getPlatformWindow.invoke(tkStage);
        Method getNativeHandle = platformWindow.getClass().getMethod( "getNativeHandle" );
        getNativeHandle.setAccessible(true);
        Object nativeHandle = getNativeHandle.invoke(platformWindow);
        return new Pointer((Long) nativeHandle);
    } catch (Throwable e) {
        System.err.println("Error getting Window Pointer");
        return null;
    }
}

EDIT 2: I eventually got further on with this one, but as soon as I re-set the WNDPROC, my undecorated window didn't respond to any events.. I'm offering a bounty of 100 reputation for a self-contained example with a working solution. Windows (7+) only is OK, I do not even know how this behaves on other platforms.

EDIT 3: Well, I kind of gave up with this one.. I got everything set up correctly, and received the events, but had problems figuring out the correct event to listen for..

Since there's been some interest in the question, if anyone wants to attempt to continue with this, here's my final code (it hopefully should "work" out-of-box):

public final class Utils {

    static interface ExtUser32 extends StdCallLibrary, User32 {
        ExtUser32 INSTANCE = (ExtUser32) Native.loadLibrary(
                        "user32",
                        ExtUser32.class,
                        W32APIOptions.DEFAULT_OPTIONS);

        WinDef.LRESULT CallWindowProcW(
                        Pointer lpWndProc,
                        Pointer hWnd,
                        int msg,
                        WinDef.WPARAM wParam,
                        WinDef.LPARAM lParam);

        int SetWindowLong(HWND hWnd, int nIndex, com.sun.jna.Callback wndProc) throws LastErrorException;
    }

    // Some possible event types
    public static final int WM_ACTIVATE = 0x0006;
    public static final int WM_ACTIVATEAPP = 0x1C;
    public static final int WM_NCACTIVATE = 0x0086;

    public static void registerMinimizeHandler(Stage stage) {
        Pointer pointer = getWindowPointer(stage);
        WinDef.HWND hwnd = new WinDef.HWND(pointer);
        long old = ExtUser32.INSTANCE.GetWindowLong(hwnd, User32.GWL_WNDPROC);
        MinimizeHandler handler = new MinimizeHandler(stage, old);
        ExtUser32.INSTANCE.SetWindowLong(hwnd, User32.GWL_WNDPROC, handler);
    }

    private static Pointer getWindowPointer(Stage stage) {
    try {
        TKStage tkStage = stage.impl_getPeer();
        Method getPlatformWindow = tkStage.getClass().getDeclaredMethod("getPlatformWindow" );
        getPlatformWindow.setAccessible(true);
        Object platformWindow = getPlatformWindow.invoke(tkStage);
        Method getNativeHandle = platformWindow.getClass().getMethod( "getNativeHandle" );
        getNativeHandle.setAccessible(true);
        Object nativeHandle = getNativeHandle.invoke(platformWindow);
        return new Pointer((Long) nativeHandle);
    } catch (Throwable e) {
        System.err.println("Error getting Window Pointer");
        return null;
    }
}

    private static class MinimizeHandler implements WinUser.WindowProc, StdCallLibrary.StdCallCallback {

        private Pointer mPrevWndProc32;

        private Stage stage;

        private MinimizeHandler(Stage stage, long oldPtr) {
            this.stage = stage;

            mPrevWndProc32 = new Pointer(oldPtr);

            // Set up an event pump to deliver messages for JavaFX to handle
            Thread thread = new Thread() {
                @Override
                public void run() {
                    int result;
                    WinUser.MSG msg = new WinUser.MSG();
                    while ((result = User32.INSTANCE.GetMessage(msg, null, 0, 0)) != 0) {
                        if (result == -1) {
                            System.err.println("error in get message");
                            break;
                        }
                        else {
                            System.out.println("got message: " + result);
                            User32.INSTANCE.TranslateMessage(msg);
                            User32.INSTANCE.DispatchMessage(msg);
                        }
                    }
                }
            };
            thread.start();
        }

        @Override
        public WinDef.LRESULT callback(WinDef.HWND hWnd, int uMsg, WinDef.WPARAM wParam, WinDef.LPARAM lParam) {

            if (uMsg == WM_ACTIVATEAPP) {
                // Window deactivated (wParam == 0)... Here's where I got stuck and gave up,
                // this is probably not the best event to listen to.. the app
                // does indeed get iconified now by pressing the task-bar button, but it
                // instantly restores afterwards..
                if (wParam.intValue() == 0) {
                    stage.setIconified(true);
                }
                return new WinDef.LRESULT(0);
            }

            // Let JavaFX handle other events
            return ExtUser32.INSTANCE.CallWindowProcW(
                            mPrevWndProc32,
                            hWnd.getPointer(),
                            uMsg,
                            wParam,
                            lParam);
        }
    }
}

You can just set the appropriate window style. It works in XP but should be ok in windows 7 32 bit. I think (but can't test) if you use 64 bit then change to the Ptr windows functions, ie. GetWindowLongPtr.

import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.platform.win32.User32;
import com.sun.jna.platform.win32.WinDef.HWND;
import com.sun.jna.platform.win32.WinUser;
import static com.sun.jna.platform.win32.WinUser.GWL_STYLE;
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.TextArea;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

public class JNATest extends Application {
    public static void main(String[] args) { launch(args); }

    @Override
    public void start(Stage stage) {
        TextArea ta = new TextArea("output\n");
        VBox root = new VBox(5,ta);
        Scene scene = new Scene(root,800,200);
        stage.setTitle("Find this window");
        stage.setScene(scene);
        stage.show();
        //gets this window (stage)
        long lhwnd = com.sun.glass.ui.Window.getWindows().get(0).getNativeWindow();
        Pointer lpVoid = new Pointer(lhwnd);
        //gets the foreground (focused) window
        final User32 user32 = User32.INSTANCE;
        char[] windowText = new char[512];
        HWND hwnd = user32.GetForegroundWindow();
        //see what the title is
        user32.GetWindowText(hwnd, windowText, 512);
        //user32.GetWindowText(new HWND(lpVoid), windowText, 512);//to use the hwnd from stage
        String text=(Native.toString(windowText));
        //see if it's the same pointer
        ta.appendText("HWND java:" + lpVoid + " HWND user32:"+hwnd+" text:"+text+"\n");
        //change the window style if it's the right title
        if (text.equals(stage.getTitle())){
            //the style to change 
            int WS_DLGFRAME = 0x00400000;//s/b long I think
            //not the same constant here??
            ta.appendText("windows api:"+WS_DLGFRAME+" JNA: "+WinUser.SM_CXDLGFRAME);
            int oldStyle = user32.GetWindowLong(hwnd, GWL_STYLE);
            int newStyle = oldStyle & ~0x00400000; //bitwise not WS_DLGFRAME means remove the style
            newStyle = newStyle & ~0x00040000;//WS_THICKFRAME   
            user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
        }
    }

}

My guess is you replace the last 3 lines with

            long oldStyleLong = user32.GetWindowLongPtr(hwnd, GWL_STYLE).longValue();
            long newStyleLong = oldStyleLong & ~ 0x00400000l;
            user32.SetWindowLongPtr(hwnd, GWL_STYLE, new BaseTSD.LONG_PTR(newStyleLong));

for 64 bit. I think I don't have those functions in my User32.dll, so I can't test it. There's lots of extraneous code in there, mainly for testing or teaching. Remove the unused lines once you figure out what you want to do.

ps. Don't add newStyle = newStyle & ~0x00020000;//WS_MINIMIZEBOX. That's one of the style flags JavaFX doesn't use for undecorated. That's why the minimize isn't available. Maybe if you try setting stage undecorated and adding (using |, not &~) the minimize box flag, you'll get the same result. There are tools to look up all the style flags from any window.

Here's the simplest amount of code that just changes an undecorated stage using the stage's HWND.

    public void start(Stage stage) {
        Scene scene = new Scene(new Pane(new Label("Hello World")));
        stage.initStyle(StageStyle.UNDECORATED);
        stage.setTitle("Find this window");
        stage.setScene(scene);
        stage.show();
        long lhwnd = com.sun.glass.ui.Window.getWindows().get(0).getNativeWindow();
        Pointer lpVoid = new Pointer(lhwnd);
        HWND hwnd = new HWND(lpVoid);
        final User32 user32 = User32.INSTANCE;
        int oldStyle = user32.GetWindowLong(hwnd, GWL_STYLE);
        System.out.println(Integer.toBinaryString(oldStyle));
        int newStyle = oldStyle | 0x00020000;//WS_MINIMIZEBOX
        System.out.println(Integer.toBinaryString(newStyle));
        user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
    }

It prints out the style flags before and after so you can look up what styles are set.

Create Custom Minimize button on Undecorated Window in JavaFX , Hello Everyone ! In this video tutorial i have explained how to minimize undecorated stage or Duration: 8:49 Posted: Apr 9, 2018 JavaFX Designing Tutorials: Welcome to JavaFX Design Tutorials, a place where you can excel your UI Designing skills and you will be able to implement it in your upcoming projects or assignments

Two things to note here.

First, it doesn't look like these libraries are in the latest version of JNA, 5.50 as of now, adding from the Maven repository. I had to add the 4.2.1 library instead.

Second, you may encounter this exception, like I did on Windows 10 and Java 11: Error with package com.sun.glass.ui while learning Java Native Access

The solution is go to your VM options in your IDE (Run -> Edit Configurations..., in IntelliJ) and add this:

--add-exports
javafx.graphics/com.sun.glass.ui=ALL-UNNAMED

It should work after that.

I would like to see someone implement the native Windows animations for minimizing and un-minimizing an undecorated window, but I haven't searched too thoroughly yet to see if this has already been discussed. I'll update this if I come across a solution.

Edit:

Upon further research on the Windows animations, it looks like a solution could be hacked together, but I gave up at trying to implement this C# hack below. It seems to be more of an OS issue and not just JavaFX.

I was able to get the initial window to stay undecorated while minimizing and with the animation by modifying this in start():

int newStyle = oldStyle | 0x00020000 | 0x00C00000;

But, after minimizing and reopening, the Windows border appears oddly enough.

Then, I tried to use a ChangeListener to swap Windows styles when iconifying.

stage.iconifiedProperty().addListener(new ChangeListener<Boolean>() {

        @Override
        public void changed(ObservableValue<? extends Boolean> ov, Boolean t, Boolean t1) {
            if (t1.booleanValue() == true) {
                int newStyle = oldStyle | 0x00020000 | 0x00C00000;
                user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
            } else if (t1.booleanValue() == false) {
                int newStyle = oldStyle | 0x00020000;
                user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
            }
        }
    });

This successfully gets the windows un-minimize animation to work fine consistently, while leaving the (visible) stage borderless.

It looks like I can get minimization animations working once I find out the best way to re-apply:

int newStyle = oldStyle | 0x00020000 | 0x00C00000;
user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);

just before the stage is iconified, and the border isn't visible to the user. Once implemented, this might work similarly to the C# solution in the first link below. Basically, what the above ChangeListener does in reverse.

Links to do with solving borderless/undecorated animations:

Use windows animations on borderless form

https://exceptionshub.com/borderless-window-using-areo-snap-shadow-minimize-animation-and-shake.html

http://pinvoke.net/default.aspx/Constants/Window%20styles.html

JavaFX Minimizing & Maximing undecorated stage with animations

JavaFx Undecorated Stage || How to make stage dragable, JavaFx Undecorated Stage || How to make stage dragable || how to minimize stage. 7,795 Duration: 18:11 Posted: Aug 8, 2018 That's one of the style flags JavaFX doesn't use for undecorated. That's why the minimize isn't available. Maybe if you try setting stage undecorated and adding (using |, not &~) the minimize box flag, you'll get the same result. There are tools to look up all the style flags from any window. Here's the simplest amount of code that just changes an undecorated stage using the stage's HWND.

Didn't get your question properly..but here's the solution

@FXML private void minimize()
{
Stage stage = (Stage) minimize.getScene().getWindow();
stage.setIconified(true);
}

JavaFX Undecorated Window Task Icon Minimize – Choudhury.com , Likewise, if you use the keyboard shortcut “Windows Key + M” all your apps except for the Undecorated JavaFX app will minimise. This can be� Hello Everyone ! In this video tutorial i have explained how to minimize undecorated stage or window JAVAFX. I am using NetBeans IDE to demonstrate this concept. Please watch the complete video

[JDK-8089296] [Windows] Cannot iconify undecorated stage using , A simple stage with undecorated stage style, doesn't iconify if you click on the Scene; import javafx.scene.control.Label; import javafx.stage.Stage; JDK- 8088717 Win: UNDECORATED windows are not minimized with the� Raw. * Create an FXResizeHelper for undecoreated JavaFX Stages. * The only wich is your job is to create an padding for the Stage so the user can resize it. * @param stage - The JavaFX Stage. * @param dt - The area (in px) where the user can drag the window. * @param rt - The area (in px) where the user can resize the window. * Minimize the stage.

Undecorated JavaFX windows should support Windows minimize , An undecorated JavaFX Stage does not display the Windows animations when it is minimized / maximized, resulting in unprofessional result of� You can set the JavaFX Stage title via the Stage setTitle() method. The Stage title is displayed in the title bar of the Stage window. Here is an example of setting the title of a JavaFX Stage: stage.setTitle("JavaFX Stage Window Title"); Stage Position. You can set the position (X,Y) of a JavaFX Stage via its setX() and setY() methods.

The JavaFX Stage class is the top level JavaFX container. The primary Stage is constructed by the platform. Additional Stage objects may be constructed by the application. Stage objects must be constructed and modified on the JavaFX Application Thread.

Comments
  • You likely need to start an event pump. Look at the GetMessage() loop in this example code. Once your Java code is running the event pump, your window and event hook should start receiving messages properly.
  • You might also be able to leverage JNA's Native.getWindowHandle() to obtain the native window handle, rather than the reflection-based lookup you're using. Depends on how the JavaFX stuff handles its native peers, though.
  • @technomage Thanks for your comments. I created a new Thread and started this event pump in it, and the window now indeed responds to events! However, when I click the taskbar-icon of my app, the CallWindowProcW is never called. But, it IS called on many other events: e.g. when I maximize the application from my own maximize button, I will get messages printed through a Callback. How do I actually capture the "taskbar click event"?
  • In my version of JNA, which I believe is the latest, there is no Native.getWindowHandle()-method - only Native.getWindowPointer(), which is for AWT-windows. I don't know, but I suspect that JavaFX is not tied to AWT. Due to my reputation loss, I cannot seem to be able to even vote comments up anymore :(
  • Sorry, I was wrong, the event indeed IS fired! I think the only thing I have left to do now, is that I need to figure out how to read the LPARAM-parameter with JNA (whether it is true or false). Let's see..
  • This is a good answer, but not exactly what I was after, at least in its current form. There still is the native border around the window (the titlebar is gone now though), and ideally, I was hoping to get rid of native decorations completely. By the way, this method seems to work just fine on my 64bit Win7, however your suggested 3-line fix does not work. (I actually lost title bar from my Sublime Text window while messing with these things :).
  • If sublime text is focused (maybe always on top) then it will lose the title bar. That's why I provided 2 methods to get the hwnd. One from the private sun api and one from user32. You can get rid of the border as well (I'll update). You can check all the basic window styles here.
  • I just remembered I added a check to make sure the HWND points to the window with the same title. It shouldn't mess up any other window.
  • You can't use events if the OS doesn't send the WM_SIZE message (for max, min etc - not resizing - that's WM_SIZING). Without a certain style set, you can't get that message.
  • @Manius Yes, check my answer.