Skip to content

Add ability to drag files to and from external applications#7849

Draft
AW1534 wants to merge 72 commits intoLMMS:masterfrom
AW1534:universal-drag-drop2
Draft

Add ability to drag files to and from external applications#7849
AW1534 wants to merge 72 commits intoLMMS:masterfrom
AW1534:universal-drag-drop2

Conversation

@AW1534
Copy link
Member

@AW1534 AW1534 commented Apr 16, 2025

Adds the ability to drag files in and out of LMMS. This required a bit more refactoring than I expected.

video2.mp4

I should note that this doesn't just work to and from the desktop, I've tested with firefox, CLion, dolphin and discord. It also works with more than just sample files (projects, presets, midis, vsts)

@AW1534 AW1534 changed the title Add ability to drag files in and out of LMMS Add ability to drag files to and from external applications Apr 16, 2025
@AW1534
Copy link
Member Author

AW1534 commented Apr 16, 2025

To hell with CodeFactor.

@AW1534 AW1534 added enhancement gui needs testing This pull request needs more testing labels Apr 16, 2025
@szeli1 szeli1 self-requested a review April 16, 2025 20:59
@AW1534
Copy link
Member Author

AW1534 commented Apr 18, 2025

once #7627 is merged, I'd like to add the ability to drag sample clips out of LMMS too. That could be for a different PR though.

@regulus79
Copy link
Member

I tested this briefly, and it seems to work! I'm using nautilus on GNOME to drag files in.

Things I noticed (these aren't that important though):

  • Dragging in .xml files which are not presets causes a dummy instrument(?) to be added, and when you click on it, LMMS crashes. This is sort of also in master, but it gives you a warning first.
  • You can't drag in .sf2 files (I think this is in master too)
  • You can't drag in vst effects into an effect chain

@AW1534
Copy link
Member Author

AW1534 commented Apr 18, 2025

Dragging in .xml files which are not presets causes a dummy instrument(?) to be added, and when you click on it, LMMS crashes. This is sort of also in master, but it gives you a warning first.

Okay, this should be a relatively simple fix.

You can't drag in .sf2 files (I think this is in master too)

Thats actually not in master, it's is a regression- Nice catch!

You can't drag in vst effects into an effect chain

This is out of scope as it isn't in master, but it's definitely a good idea for another PR.

@regulus79
Copy link
Member

I tested this again, and those two issues do appear to be fixed!

@headquarter8302
Copy link
Member

headquarter8302 commented Apr 19, 2025

In Windows MSVC, dragging instruments to the Song-Editor doesn't work
Edit: I used the right-click menu to insert Mallets, OBS didn't capture the context menu

2025-04-18.23-03-41.mp4

@AW1534 AW1534 removed the needs testing This pull request needs more testing label Apr 19, 2025
Copy link
Contributor

@szeli1 szeli1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like a few things changed.
Also I'm testing this and When I drag a sample clip into AudioFileProcessor, it doesn't load the audio for me. Firstly create a sample clip and load audio, then create an empty AudioFileProcessor and try to drag the clip into the processor.
Dragging in files from outside works nicely.
What else to test?

@AW1534
Copy link
Member Author

AW1534 commented May 10, 2025

I would like a few things changed. Also I'm testing this and When I drag a sample clip into AudioFileProcessor, it doesn't load the audio for me. Firstly create a sample clip and load audio, then create an empty AudioFileProcessor and try to drag the clip into the processor. Dragging in files from outside works nicely. What else to test?

huh, I don't know why I'm just now seeing this comment, sorry.

Master doesn't let me drag sample clips into AFP either. In fact, I think master behaves worse, as it accepts the drop, but just has weird behaviour. So I think this is out of scope. However, I was planning to add that once your sample exporting PR is merged.

@AW1534 AW1534 requested a review from szeli1 May 10, 2025 14:08
Copy link
Contributor

@szeli1 szeli1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the code changes and I believe the recent commits made the code easier to implement in the future which is nice. I didn't find any issues in the code.

I tested dragging samples from outside (ogg, flac, mp3) and it worked. However when I opened the file browser and dragged an instrument in, it didn't get added to the song editor. I couldn't test vestige because I could not load the instrument. I also tried to link a knob to an automation clip and it did not accept it.

@AW1534
Copy link
Member Author

AW1534 commented May 11, 2025

@szeli1 fixed. The issue was being caused by the hasFormat() check

@AW1534
Copy link
Member Author

AW1534 commented May 11, 2025

@szeli1 any idea why the builds might be failing on windows?

@szeli1
Copy link
Contributor

szeli1 commented May 14, 2025

@szeli1 any idea why the builds might be failing on windows?

Add LMMS_EXPORT to the problematic functions, the plugins can't access them without LMMS_EXPORT.

@AW1534
Copy link
Member Author

AW1534 commented May 14, 2025

@szeli1 Thanks, that worked! This should be ready for merge now.

@szeli1 szeli1 self-requested a review May 18, 2025 07:33
Copy link
Contributor

@szeli1 szeli1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this again and everything worked as expected. I was able to drag in sound files from outside, vsts and project files worked as well.

The reason I'm not approving this is I found that snapping the lmms window to the screen's right side (for example) did not work after this change. This worked on master. I use Linux mint, I built.

Steps to reproduce:

  1. launch lmms
  2. grab the widow
  3. drag it until the cursor reaches one side of the screen
  4. normally, the window would resize to cover half of the screen
  5. currently it does nothing

This PR is a big improvement, so I hope to get this merged.

@AW1534
Copy link
Member Author

AW1534 commented May 18, 2025

this is a really weird issue, but I see no reason why my PR would cause this. Are you using a custom build to test master and AppImage to test my PR? if that's the case, you should test master with the AppImage to see if you get the same issue.

I am unable to reproduce this on the AppImage or custom build

@AW1534
Copy link
Member Author

AW1534 commented May 20, 2025

@szeli1 do the keyboard shortcuts work? (super + right)

@szeli1
Copy link
Contributor

szeli1 commented May 20, 2025

@szeli1 do the keyboard shortcuts work? (super + right)

I believe it didn't work for me when I tested it.

@szeli1
Copy link
Contributor

szeli1 commented May 20, 2025

@szeli1 do the keyboard shortcuts work? (super + right)

Wait this happened to me in an independent branch, it seems like this is an unrelated bug

Copy link
Contributor

@szeli1 szeli1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested this PR and I approve it.

Copy link
Member

@messmerd messmerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quick review

@AW1534 AW1534 requested a review from messmerd August 21, 2025 16:51
Co-authored-by: Dalton Messmer <messmer.dalton@gmail.com>
AW1534 and others added 3 commits August 22, 2025 23:19
- change `mimetypes` type to `std::map<std::string, std::set<std::string>> mimetypes`
- remove lv2 and xiz from hardcoded mimetypes map
Co-authored-by: szeli1 <143485814+szeli1@users.noreply.github.com>
@AW1534 AW1534 requested review from messmerd and szeli1 August 25, 2025 19:58
Copy link
Member

@messmerd messmerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Haven't tested it, but the code changes look fine to me.

Co-authored-by: Dalton Messmer <messmer.dalton@gmail.com>
Copy link
Contributor

@szeli1 szeli1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like decodeMimeData and processDragEnterEvent, they are a huge improvement to the code.

* The function also uses fallback decoding via StringPairDrag in case the type and value
* were encoded in a non-file-based drag operation.
*
* @param _de Pointer to the QDropEvent containing drag-and-drop data.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't find "_de" as a param.

const char * supportedFileTypes; //!< csv list of extensions
const PixmapLoader* logo;
const char* supportedFileTypes; //!< csv list of extensions
const char* supportedMimetype; //!< mimetype supported by the plugin. Should be null if no filetypes are supported
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be csv maybe? But currently we don't need multiple new keys for one plugin.

",mp3"
#endif
,
"samplefile",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is "samplefile" only used by AudioFileProcessor? I believe other widgets send audio data with an other key already.
Removing the "samplefile" key may be outside of this PR's concerns.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also patman uses "samplefile" even though it doesn't define it.

Comment on lines 72 to 73
QString decodeKey( const QMimeData * mimeData );
QString decodeValue( const QMimeData * mimeData );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"// use decodeMimeData"

Comment on lines +49 to +52
//! use QString Clipboard::decodeKey(QMimeData) instead
static QString decodeKey(QDropEvent* de);
//! use QString Clipboard::decodeValue(QMimeData) instead
static QString decodeValue(QDropEvent* de);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"// use decodeMimeData"

Comment on lines +48 to +54
//! gets the extension of a file, or returns the string back if no extension is found
static QString getExtension(const QString& file)
{
QFileInfo fi(file);
const QString ext = fi.suffix().toLower();
return ext.isEmpty() ? file.toLower() : ext;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this. QT already has this functionality (QFileInfo) and lmms also implements this internally in some utility function.

if (it == mimetypes.end()) { return false; }

const auto& fileTypes = it->second;
const auto extStr = getExtension(ext).toStdString();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const auto extStr = getExtension(ext).toStdString();
const auto extensionStr = getExtension(extension).toStdString();

bool isMidiFile(const QString& ext) { return isType(ext, "midifile"); }
bool isVstPluginFile(const QString& ext) { return isType(ext, "vstpluginfile"); }

//! Gets the clipboard mimedata. NOT the drag mimedata
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move the comment into the header.

const QList<QUrl> urls = mimeData->urls();

QString type{"missing_type"};
QString value{};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
QString value{};
QString value;

@messmerd suggested this change. It was incorrect to assume value isn't initialized when defined. This isn't the case with basic c++ types such as float and int.

@szeli1
Copy link
Contributor

szeli1 commented Aug 26, 2025

As I mentioned before, I'd like to see some of the mime type stuff refactored so that plugin-specific information is no longer present in the core LMMS application. Without that, our plugins are not truly plugins, are they?

You've approved this PR while plugin keys are still present in the code.

Maybe plugin descriptors should contain a mime type in addition to the file extensions. That could then be queried through PluginFactory like file extensions can.

This did happen. I think it is nice that plugins define the extensions, so this is a good change.

For example, Sf2Player would provide both "sf2,sf3" and "soundfontfile" in its descriptor. Or maybe use FileItem::FileType::SoundFont instead.

The existence of this code makes the changes not really "plugin-specific"

	bool isAudioFile(const QString& ext)        { return isType(ext, "samplefile"); }
	bool isProjectFile(const QString& ext)      { return isType(ext, "projectfile"); }
	bool isPluginPresetFile(const QString& ext) { return isType(ext, "pluginpresetfile"); }
	bool isTrackPresetFile(const QString& ext)  { return isType(ext, "trackpresetfile"); }
	bool isSoundFontFile(const QString& ext)    { return isType(ext, "soundfontfile"); }
	bool isPatchFile(const QString& ext)        { return isType(ext, "patchfile"); }
	bool isMidiFile(const QString& ext)         { return isType(ext, "midifile"); }
	bool isVstPluginFile(const QString& ext)    { return isType(ext, "vstpluginfile"); }

FileItem::FileType and the mime type seem to have an almost identical job, so there's probably also a simplification waiting to be made there - maybe a QString("filetype_%1").arg(static_cast(fileType)) mime type similar to what tracks use. Idk. Doing stuff like this is probably necessary to remove all the plugin-specific code from Clipboard, TrackContainerView::dragEnterEvent, TrackContainerView::dropEvent, and elsewhere.

Usually file types need human readable names, so I think marking them with a number complicates things.

I believe there is a much cleaner and more maintainable design waiting to be implemented, and rather than going along with the original poorly-made design by adding additional plugin-specific data hardcoded into the main application, I think this PR provides the perfect opportunity to truly fix the problem.

Keys are still hardcoded into the application, only extensions differ which is an improvement, but doesn't achieve the main goal, it doesn't truly fix the problem.

If it turns out to not be possible to move all the plugin-specific info into the plugins, then I think quarantining it to the FileItem::FileType enum within the main application would be an acceptable compromise.

This didn't happen.

My main issue is that I would like to replace string keys with enum keys. This means keys like "vstpluginfile" must be defined for example in clipboard, so the design mistakes are kept. You've approved this PR @messmerd while these design mistakes are still present.

Co-authored-by: szeli1 <143485814+szeli1@users.noreply.github.com>
Copy link
Contributor

@allejok96 allejok96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know LMMS' drag and drop system was such a horrifying mess. I like what you have done to clean up processDragEnterEvent. But there's more cleanup that needs to be done in the drag n drop code and now is the time to rethink that, before building more things on top of the old code.

return {type, value};
}

void startFileDrag(gui::FileItem* f, QObject* qo)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why did you move this code here? It's only used by FileBrowser and requires the gui namespace.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that change was requested by @szeli1

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't remember this, but I could be mistaken. I agree with @allejok96

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this review comment. I'll revert that when I get the chance.


const QMimeData * getMimeData()
// there are other mimetypes, such as "samplefile", "patchfile" and "vstplugin" but they are generated dynamically.
static std::map<std::string, std::set<std::string>> mimetypes =
Copy link
Contributor

@allejok96 allejok96 Aug 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly dislike the use of this mimetype map and I don't understand what benefit it brings.

First of all, these are not mimetypes, they are just arbitrary hard-coded and bug-prone strings.

Secondly, mimetypes are used to indicate the data format. But .xiz and .lv2 gets registered under the the same pseudo mimetype called "pluginpresetfile" which makes it possible to drag a .lv2 files into ZynaddsubFX. Mimetypes should be as specific as possible. This design removes useful information.

Thirdly, if a new plugin was created that could open .xyz files, it's not enough to simply define .xyz in the list of supported extensions, it now also has to have a pseudo mimetype. If we want to guarantee that the .xyz file cannot be opened with any other plugin we need to define yet another pseudo mimetype and corresponding isXyzFile() function to be able to call decodeMimeData(). But this is Clipboard and it should not even know about all these plugin specific formats. Like the comment says they should be generated dynamically.


Now to be fair @AW1534 you didn't create this pseudo mimetype system, it was already in place. But it feels like we should be moving to something more standardized instead of creating an every harder requirement of using this bad system.

Possible solution: instead of dealing with generalized file types, return the exact file type and let each caller check that against a list of file types that it supports. For common things like "samplefile" the list could be a constant in Sample.h etc.

Copy link
Contributor

@szeli1 szeli1 Sep 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I strongly dislike the use of this mimetype map and I don't understand what benefit it brings.

It was an attempt to move away from hard coded types. This attempt failed as things stand now although it partly improved the code because now plugins can define their supported file types (so the hard coded extensions are moved out of clipboard).

Secondly, mimetypes are used to indicate the data format. But .xiz and .lv2 gets registered under the the same pseudo mimetype called "pluginpresetfile" which makes it possible to drag a .lv2 files into ZynaddsubFX. Mimetypes should be as specific as possible.

I'm okay with grouping more extensions into 1 type, if they are converted to the same kind of data (for example .flac and .ogg are both audio data), this conversion should happen as soon as possible.

But this is Clipboard and it should not even know about all these plugin specific formats. Like the comment says they should be generated dynamically.

I don't know if it is possible to generate this dynamically, or if it is worth the cost of complexity. Currently gui assigns functionality to identifiers like samplefile. Therefor if we would dynamically define these identifiers, then we couldn't use them compile time. If we would declare something compile time that would replace them, then we end up in the same hard coded situation.

Possible solution: instead of dealing with generalized file types, return the exact file type and let each caller check that against a list of file types that it supports. For common things like "samplefile" the list could be a constant in Sample.h etc.

StringPairDrag::processDragEnterEvent(_dee, {"trackpresetfile", "pluginpresetfile", "samplefile", "instrument",
				"midifile", "soundfontfile","patchfile","vstpluginfile","projectfile",
				QString("track_%1").arg(static_cast<int>(Track::Type::Instrument)), QString("track_%1").arg(static_cast<int>(Track::Type::Sample))});

Here we would need to know about vstpluginfile, we could ask vst plugin for it's extensions, but then we would need to know that we are asking specifically vst plugin and not lv2 plugin for example. Therefor vst plugin is ought to be known compile time in some way.

I would suggest a simple hard coded approach where we define an enum containing all types. We would use this enum in gui, to replace strings (benefit: somebody can't misspell a type accidentally like in vst plugin's case). This solution isn't clean, but it is simple. We could still define extensions in plugins, then loop trough them to convert extensions to keys.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AW1534 having taken a second look at the code, I realize that the only distinction between pluginpresetfile, vstpluginfile, soundfontfile and patchfile is which type of cursor icon they get. Otherwise they are all treated the same - the correct plugin is loaded and Plugin::loadFile() is called. Is it worth having different names and functions and types for the sake of a cursor icon? If all of these were loaded as pluginpresetfile, then by the grace of InstrumentTrackWindow SF2 player and Patman would gain drag and drop support and you could remove VestigeInstrumentView::dropEvent.

_de->accept();
}
else if( StringPairDrag::decodeKey( _de ) == "sampledata" )
else if (type == "sampledata")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't find anything that creates a "sampledata" string pair btw

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is why I do not support using strings as keys / types. Fixing this is outside of the scope of this PR.

QString filePath = QUrl::fromLocalFile(f->fullName()).toString();

// Internal LMMS type
mimeData->setData("application/x-lmms-type", internalType.toUtf8());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this internal data ever used? If it should be kept it should use mimeType(MimeType).

return( QString::fromUtf8( mimeData->data( mimeType( MimeType::StringPair ) ) ).section( ':', 1, -1 ) );
}

std::pair<QString, QString> decodeMimeData(const QMimeData* mimeData)
Copy link
Contributor

@allejok96 allejok96 Aug 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at its usecases I think decodeMimeData should be split in two.

Clipboard::getFile({list of supported extensions}) -> ext, path
the opposite of startFileDrag

Clipboard::getStringPair() -> key, value
the opposite of copyStringPair

Even if the return types are the same, having separate functions with specific names for specific usages makes the code much more readable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this was done, the caller of getFile would need to know every single supported extension. then we run into the same issues of hardcoding that this PR solved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if the return types are the same, having separate functions with specific names for specific usages makes the code much more readable.

I disagree with this suggestion.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the caller of getFile would need to know every single supported extension

@AW1534 currently the caller always knows what file extension it expects, except in these two cases:

  1. "samplefile"

In this case the caller relies on SampleDecoder so it doesn't need to know about file types. To get the supported extensions, the code could call SampleDecoder::supportedAudioTypes()

Actually, in your implementation "samplefile" is a dynamic mime type loaded by AudioFileProcessor. This means the AFP definition would need to be manually updated each time a new file format is added to SampleDecoder. And also, if APF is excluded from the build, any code using "samplefile" drag and drop would break.

  1. "pluginpresetfile"

In this case the caller just want to know if there exists any plugin that supports a given file extension. This is used by TrackContainerView and InstrumentTrackWindow. Since the plugin specific mime types are loaded by PluginFactory I suggest that PluginFactory should be responsible for providing a list of file extensions that there exists a plugin for, similar to how SampleDecoder should be responsible for knowing which file types that can be loaded as samples. Then TrackContainerView could query this list of plugin extensions before it tries to load a plugin.

else { dee->ignore(); }
}
else { dee->ignore(); }
StringPairDrag::processDragEnterEvent(dee, {"samplefile", QString("clip_%1").arg(static_cast<int>(Track::Type::Sample))});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The clipboard can contain multiple types of data, right? If dragging a sample clip would create both a stringpair and a URL then Track could use the stringpair and everything else would just use the regular sample file.

@allejok96
Copy link
Contributor

allejok96 commented Oct 5, 2025

If this was done, the caller of getFile would need to know every single supported extension. then we run into the same issues of hardcoding that this PR solved.

I've mocked up a system of completely dynamic file types.

There are no functions like isVST() and no string literals like "*.sf2".

@allejok96
Copy link
Contributor

What I've learned the past days is that the clipboard system builds on unspoken rules and is in need of refactoring. Maybe we could add drag and drop without having to refactor... Idk @AW1534 would know. But at the moment, adding a lot of isFiletype() functions feels like a refactor in the wrong way.

allejok96 added a commit to allejok96/lmms that referenced this pull request Oct 5, 2025
@AW1534 AW1534 marked this pull request as draft October 5, 2025 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants