Building AppImage with quick-sharun

[tresf]

After experiencing some upstream issues with the AppImage format, @Samueru-sama was kind enough to showcase the ability to move our AppImage creation to a new toolset...

• Some icons sometimes disappear after minimizing LMMS for some time. #7979 (comment)

The conversation about this starts here and I'm moving this discussion out of that thread to allow for better tracking.

TODO:

• Carla support**
  • **This is technically working, but the project needs to determine whether or not we want to bundle it with LMMS
• Windows VST2 64-bit support
  • Remote plugin '/tmp/.mount_lmms-remp15455123920282921156/lib/lmms/RemoteVstPlugin64' not found
• Windows VST2 32-bit support (broken by wine's new WOW64 mode)
• JACK support (fixed by bundling pipewire-jack instead of jack2)
• ZynAddSubFX support
  • Remote plugin '/tmp/.mount_lmms-remp15455123920282921156/lib/lmms/RemoteZynAddSubFx' not found
• Suil/SUIL_MODULE_DIR support (see here)
• ARM64 testing
  • Initial testing looks good on Ubuntu 24.04 ARM64
  • Iron out some compat issues on Debian 11.
  • Investigate why JACK suffers pops/click in AnyLinux image, but not on nightly.

Early AppImage for testing:

• https://github.com/pkgforge-dev/lmms-AppImage/releases

[tresf]

• wayland-hook.sh AppImages made with linuxdeploy or go-appimage are notoriously broken in wayland, and the reason is actually refusal from probonopd to support wayland. This is a well known fact and I don't think I need to link to the many controversies regarding that.

probono doesn't maintain linuxdeploy (TheAssassin does); our wayland support is decent actually, but there are some minor UX bugs that we haven't had time to dive into yet.

I tried to add wine32 to the build process

Thanks for trying... yeah I had forgotten you were building on arch. We've been discussing this here: #8041 and here (Discord). Long story short, we know the writing is on the wall and we're not sure what to do, but it's more related to arch choosing the new wow64 mode (which is awesome, but happens to break our 32-bit bridge). We can track this in #8041 because eventually all distros will go this route.

Turns out carla depends on a matching version of pyqt5, which means python and carla need to be added manually since sharun deploys python using uv and the version of pyqt5 ends up being slightly different and it breaks lol

Hmm... I would not expect python to be deployed with the AppImage, since it's a dependency of Carla (not LMMS).

jack-hook.sh We can just bundle jack here, we already do that often with pipewire (which again appimages made with the older tooling cant do).

Ok... I'll believe this when I test it. 😅

vbox-hook.sh LIBGL_ALWAYS_SOFTWARE=1 is likely needed here because OpenGL is not being bundled, we already handle this with sharun.

Hmm... I'm starting to grow weary of how many decision sharun is trying to make for us.

Thanks for building the AppImage; I'll test the and report back.

[tresf]

So some errors...

Remote plugin '/tmp/.mount_lmms-remp15827696791730005933/lib/lmms/RemoteZynAddSubFx' not found

I think is leftover from linuxdeploy/linuxdeploy#305, specifically (probably) should have been addressed here, but was missed:

• AppImage: Un-pollute /usr/lib #8053

@messmerd FYI.. Nevermind, I think this is working just fine, it's just missing the wrapper script.

Remote plugin '/tmp/.mount_lmms-remp15827696791730005933/lib/lmms/RemoteVstPlugin64' not found

Probably the same cause as Zyn above.

Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
Automatic start of JACK server is disabled at configure time
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
jack_client_open() failed, Status: 0x11
Overall operation failed. JACK dependencies might need to be installed.
Could not connect to JACK server.

Our nightly appimages don't do this (the messages say JACK server starting in realtime mode with priority 10, although I'm not a JACK user, so I can't say with certainty how well nightly AppImages work.

Version 1.3.0-alpha.1.889+44a68b8 (unknown platform/unknown processor, Qt 5.15.17, GCC 15.2.1 20250813).

This may be something with our platform detection logic, or perhaps something that we do a bit late (in the packaging phase rather than the build phase).

As a bit of an aside... what's interesting (and also worrisome) about this quick-sharun/AnyLinux approach is that we completely lose the ability to control any packaging decisions. I'm not very comfortable with this. It also seems to place a strict dependency on ArchLinux for building (this means people won't be able to build their own AppImages on any other distros -- or worse -- LMMS maintains two separate packaging scripts). I share these thoughts as important 'food for thought' sentiments, because our philosophy has always been to encourage others to build and distribute LMMS, so the tooling being widely available is a huge part of that.

[tresf]

More errors:

[carla] Trying to start carla-plugin using "/tmp/.mount_lmms-remp15827696791730005933/shared/share/carla/resources/carla-plugin-patchbay"
[carla] exec failed: No such file or directory
[carla] waitForClientFirstMessage() - read returned 0
[carla] Trying to start carla-plugin using "/tmp/.mount_lmms-remp15827696791730005933/shared/share/carla/resources/carla-plugin-patchbay"
[carla] exec failed: No such file or directory
[carla] waitForClientFirstMessage() - read returned 0

Note that Carla loads just fine on nightly.

[Samueru-sama]

probono doesn't maintain linuxdeploy (TheAssassin does); our wayland support is decent actually, but there are some minor UX bugs that we haven't had time to dive into yet.

Got it, I was thinking of linuxdeploy-qt

Hmm... I would not expect python to be deployed with the AppImage, since it's a dependency of Carla (not LMMS).

I'm not sure how needed some dependencies here are, I made sure to bundle carla to the point that I can run the carla script on alpine linux:

Our nightly appimages don't do this (the messages say JACK server starting in realtime mode with priority 10, although I'm not a JACK user, so I can't say with certainty how well nightly AppImages work.

I get those messages with the AUR package on my system 🤔

/usr/bin/lmms
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
Automatic start of JACK server is disabled at configure time
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
jack_client_open() failed, Status: 0x11
Overall operation failed. JACK dependencies might need to be installed.
Could not connect to JACK server.
No audio-driver working - falling back to dummy-audio-driver
You can render your songs and listen to the output files...
Cannot load library /usr/lib/lmms/libcarlabase.so: (libcarla_native-plugin.so: cannot open shared object file: No such file or directory)
Cannot load library /usr/lib/lmms/libcarlapatchbay.so: (libcarlabase.so: cannot open shared object file: No such file or directory)
Cannot load library /usr/lib/lmms/libcarlarack.so: (libcarlabase.so: cannot open shared object file: No such file or directory)
Cannot connect to server socket err = No such file or directory
Cannot connect to server request channel
Automatic start of JACK server is disabled at configure time
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
jack_client_open() failed, Status: 0x11
Overall operation failed. JACK dependencies might need to be installed.
Could not connect to JACK server.

As a bit of an aside... what's interesting (and also worrisome) about this quick-sharun/AnyLinux approach is that we completely lose the ability to control any packaging decisions. I'm not very comfortable with this. It also seems to place a strict dependency on ArchLinux for building

You can use sharun directly on any linux distro. See this thing probono himself made lol

Also CPU-X where while it uses archlinux, it is done manually with lib4bin without quick-sharun

However it is very ideal to use archlinux, because I make these packages that help reduce the size of the final appimage a lot.

Eden for example was using Ubuntu to make its appimage with sharun, later moved to archlinux with the debloated packages, that alone reduced the appimage from 160 MiB to ~90 MiB, the current releases are ~60 MiB but that is because they later changed the build process to link ffmpeg statically which helps a lot here.

Disclaimer, if you are wondering why eden makes multiple different appimage releases, it is because they target different cpu architecture levels like x86_64_v3 not due to AppImage compat issues.

More errors:

[carla] Trying to start carla-plugin using "/tmp/.mount_lmms-remp15827696791730005933/shared/share/carla/resources/carla-plugin-patchbay"
[carla] exec failed: No such file or directory
[carla] waitForClientFirstMessage() - read returned 0
[carla] Trying to start carla-plugin using "/tmp/.mount_lmms-remp15827696791730005933/shared/share/carla/resources/carla-plugin-patchbay"
[carla] exec failed: No such file or directory
[carla] waitForClientFirstMessage() - read returned 0

Note that Carla loads just fine on nightly.

Yeah there is bug here, there is no such thing as shared/share directory with sharun. Should be fixed with a relative symlink in shared in any case.

[tresf]

Yeah there is bug here, there is no such thing as shared/share directory with sharun. Should be fixed with a relative symlink in shared in any case.

These are the issues I'm concerned about. We don't bundle Carla and likely never will. We use the system-provided Carla. We use weak-linking (much like weakjack). We don't want it.

We don't want jack either... we just bundle it conditionally to prevent a crash for when it's missing. I feel like our strategy should be preferred here.

You can use sharun directly on any linux distro.

This is great news and assuming this works for distros like Ubuntu 20.04 we can make a WIP branch that uses it. I'm even OK switching completely away from linuxdeploy, but as the (mostly sole) person that's been working with these packaging tools here, I firmly believe that any solution that promises to fix our workaround is selling snake oil lol. Seriously though, this quick-sharun is amazing, but it's taken us years to get some of our features working as well as they do, so I think there's some items which we do better such as using the system's jack libraries (conditionally, when available), using the system's carla version (conditionally, when available). We also ran into libreadline issues that I expect to rear their ugly head once RemoteVstPlugin64.exe.so starts working, but this might be fixed with the static runtime, so more testing will be needed to know for sure.

[tresf]

later moved to archlinux with the debloated packages, that alone reduced the appimage from 160 MiB to ~90 MiB,

To clarify we're happy with using any distro for our releases, my concern is about local reproducibility, for one-offs (which it sounds like quick-sharun supports). Using Arch fixes so many of our outdated dependency issues that I think this strategy is the winning long-game.

[Samueru-sama]

Alright here is what is causing the shared/share issue with carla:

That symlink is broken and resolves to that non existent path.

We use the system-provided Carla. We use weak-linking (much like weakjack). We don't want it.

Does lmms call carla by running the multiple scripts it provides? or does it directly open its libraries.

If it is the later, it is not possible to avoid bundling carla here heads up.

sharun bundles glibc, which is what needs to be done to allow these AppImages to work anywhere from ubuntu 14.04 to alpine linux, however that limits you in what libraries you can open from the host.

Because if the host libraries link to a glibc version that is newer than the one of the AppImage, then it is going to break with symbol errors. In theory we could switch to using the glibc of the host at runtime when we detect that it is a newer version and we are on a glibc system but that hasn't been implemented in sharun not to mention this will never work on non glibc systems as well

[tresf]

Does lmms call carla by running the multiple scripts it provides

Yes.

if the host libraries link to a glibc version that is newer than the one of the AppImage, then it is going to break with symbol errors

Gotcha. Well, JACK as an example runs as a daemon on the PC and the client has never worked for me when I try to bundle it. The AnyLinux appimages seems to suffer this too. I don't know the undeworkings of WHY (and I don't use JACK personally) but these are the roadblocks that will make it hard to decide how best to move forward.

Coming from other projects (and other OSs), I find this glibc mess in Linux quite sad. From a project perspective, I'm fine with a trade-off between latest libs and backwards compat and that's something we've always had to weigh, but I'm not sure if we're OK with losing JACK support. I'm also curious how the VST bridge ends up, once it's working. I can probably extract and test that now...

I've fought adamantly against bundling Carla and Wine and I feel that as a project, we need to maintain this position.

[tresf]

I'm also curious how the VST bridge ends up, once it's working. I can probably extract and test that now...

Remote plugin '/tmp/.mount_lmms-remp15827696791730005933/lib/lmms/RemoteVstPlugin64' not found

Ok, this works just fine! For some reason it's missing the RemoteVstPlugin64 wrapper script. It looks like this.

[Samueru-sama]

Gotcha. Well, JACK as an example runs as a daemon on the PC and the client has never worked for me when I try to bundle it

Jack should work just fine.

Yesterday I read the manpage and jack where it mentions JACK_DRIVER_DIR and added the required changes.

The errors you have with jack are also errors that I have when running the AUR package on my system, will later try to inform myself better once I can get jack to work any debugging it is just checking with strace to see if the bundled jack libs are being used or not.

It is possible JACK_DRIVER_DIR is turned off at build time, I already had a similar problem with p11-kit and the solution ended up being binary patching because the env variables are not used unless recompiled.

If flatpak or snap can ship jack then it can be done in an AppImage as well, this idea of libs/binaries that cannot be bundled comes from the old appimage tooling.

[Samueru-sama]

Alright I managed to install and start the jack daemon on my system.

It works? I didn't have to change anything in the AppImage:

I still have a few errors:

Cannot lock down 107341340 byte memory area (Cannot allocate memory)
Cannot use real-time scheduling (RR/5) (1: Operation not permitted)

but those errors also happen with the AUR package directly. So it is likely me that I need to configure jack better.

Also the carla issue should be fixed already by pkgforge-dev/Anylinux-AppImages@f1b77e5 I know you don't want it to be added but would be good to know if it works, python deployment is the only thing that gives us issues lol

[tresf]

Also the carla issue should be fixed already by pkgforge-dev/Anylinux-AppImages@f1b77e5 I know you don't want it to be added but would be good to know if it works, python deployment is the only thing that gives us issues lol

Yes, it works. You can safely remove it now. 😆 We may be (eventually) receptive to adding Carla in this fashion, so a TODO may be in order... assuming you're building it from scratch it'll use the same Qt runtime as LMMS which was a big factor in not wanting to bundle it initially. I still don't want Python in the package, but beggars can't be choosers.

With regards to JACK, still no go (Ubuntu 24.04)

Automatic start of JACK server is disabled at configure time
jack server is not running or cannot be started
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
JackShmReadWritePtr::~JackShmReadWritePtr - Init not done for -1, skipping unlock
jack_client_open() failed, Status: 0x11
Overall operation failed. JACK dependencies might need to be installed.

Meanwhile on our master branch:

JACK server starting in realtime mode with priority 10