Android/Meta Quest 3 input capture/emulation

[izelnakri]

Hi there,

First of all, thank you for this amazing & actively developed/maintained tool! I tried to run it on my new Meta Quest 3 VR set today and I was able to run it on dummy emulation mode! Heres the steps to reproduce:

1 - Since Meta OS is based off of android, I knew I could download this software from(nix) or compile it myself.
2 - I sideloaded F-Droid, made it runnable via "Mobile VR Station" sideloading app on Meta Store. It doesn't root the device but enables certain permissions so .apk executables can be downloaded/viewed & run on Meta OS. Meta Corp made certain privilege escalations limited, however they do support people to sideload applications as promised by Mark Zuckerberg himself on his metaverse keynote.
3 - I downloaded/run "nix-on-droid" application/apk from F-Droid. This boots up a Termux app with nix downloaded.
4 - I was able to download both 24.05 and unstable branches of lan-mouse from nixpkgs.
5 - lan-mouse library complained about $XDG_RUNTIME_DIR variable not set so I set it to:

export XDG_RUNTIME_DIR=$(pwd)

6 - voila! I could run lan-mouse -f cli and connect to my laptop keyboard! My laptop runs on Hyprland/wayland with xdg-desktop-portal enabled. I was able to see all mouse and keyboard events of my laptop keyboard in my Meta Quest 3 headset!!

Now I need an android input emulator 😅 I've checked the source code, seems like this is the only emulation missing(apart from iOS perhaps).

a) Would you be interested in building/testing this? If not, any information on where I can learn about input capture/emulation on android would be very helpful.

Perhaps we can even reach out to Meta to financially sponsor this initiative, given this would enable seamless integration of keyboards, enhance their OS/product offering for productivity use-cases. Maybe I can help with this as well but haven't checked this project source code in depth yet to see how the current automated testing infrastructure works(although I'm well-versed in rust 😅 ).

b) If you're not interested in building this, do you know what alternative software/solution I could use today to share my laptop keyboard with my Meta Quest 3 seamlessly?

Thanks again for your interest and all this work!

P.S I made this laptop keyboard <> VR/Android wireless integration working with KDE Connect: https://www.linkedin.com/feed/update/urn:li:activity:7254495494696001536/ It ignores the "Enter", "Crtl" key on certain apps/text inputs, which is a big deal. Investigating that bug now...

[feschber]

Great job getting it to work! However, I think I'll have to disappoint you a little bit. Termux is quite restricted as far as I know and would likely need to be modified to expose any of the necessary apis to lan-mouse.

I do plan to look into android but it's going to take a while and I think it is also easier to write some native android application and use the Accessibility API for this purpose instead of trying to get something going through termux.

[feschber]

You might also want to check out #37

[izelnakri]

@feschber 👍 checking the links you mentioned as we speak.. meanwhile have a look into this, it is possible to simulate keypress events from bash directly through adb! : https://gist.github.com/izelnakri/f8df55202df921ac10c5e988273ced8a#file-adbcommands-L118

[feschber]

It's been a while since I did any Android emulation. But this would still require to have an adb connection to the device from termux, right?
https://developer.android.com/training/testing/other-components/ui-automator looks like a similar thing but without the need to spawn a shell command every time.

I think it should be possible to use the accessibility services which should be much more performant.

[feschber]

https://developer.android.com/reference/android/accessibilityservice/GestureDescription

[izelnakri]

Hi @feschber I've made my wireless keyboard & trackpad sharing working for my Meta Quest 3 using scrcpy: https://www.linkedin.com/feed/update/urn:li:activity:7257149521049075712/

I'll have a look into the links you've shared and get back to you some time ;)

[rohitsangwan01]

@izelnakri I've been exploring Android remote input injection and found several methods to achieve it: Bluetooth HID, USB with AOA, and UHID over ADB.

i have been also experimenting with one more approach, which requires you to connect mobile with Desktop only once to inject a file in root for InputEmulation using ADB, and thats it, then you can use a Android app ( part of this plan ) as bridge between that file and other connections, for example handling LanMouse protocols on Network

@feschber I’ve implemented these approaches in my project UniControlHub, in case you’re interested in checking it out for ideas. Currently, it’s mostly built with Flutter, working directly with each platform's native APIs. I'm in the process of moving the core logic to Rust (while keeping the UI in Flutter).

For input capture, I'm considering using LanMouse, though there are still some blockers to resolve in this area.

[feschber]

That UI is looking very familiar, I wonder where you got inspiration from 😆.

Very interesting project!
Let me know if there is anything I can do to help you with input capture!

I'm also interested in using rust for Android development. My idea would be to have a sort of remote control App as a LanMouse client that can be used as a tochpad or mouse for a PC. So definitely keep me updated on that front!

[rohitsangwan01]

Haha 😄, yes ui is influenced by LanMouse
am trying to figure out and experiment few things with input capture
But yeah Android idea is intresting, will try to build Android/IOS side server with a touchpad and buttons ui, to send events to LanMouse Desktop client, in my free time