Problems running multiple instances of batch files

[masterofobzene]

Hello Oleg

Here I am again after so much time with the problem with batch files not working as intended. This time I think I kind of figured out a pattern of failures and the root cause seems to be a race condition or something like that.

Let me explain this with an example because it would be difficult to me otherwise:

We made around 100 entries for the menu using shell-x. Yes a lot. No icons.
These entries are separated into folders:
C:\Users\someone\AppData\Roaming\shell-x\[gif,mp4,mov,mpg,mpeg,avi,mkv,webm,ts,wmv,m4v,flv]\02.MOVER A EXTERNO\HIGHLIGHTS\TODO then inside "TODO" folder, we have even more folders each with a batch file in it, e.x: .\TODO\A\A.c.ms.cmd , .\TODO\B\B.c.ms.cmd and so on until Z.

We also have special bat files that we put in a similar structure, say for example:
C:\Users\someone\AppData\Roaming\shell-x\[gif,mp4,mov,mpg,mpeg,avi,mkv,webm,ts,wmv,m4v,flv]\02.MOVER A EXTERNO\CLIPS\TODO . The rest is the same than with the above.

Now to the problem:

If you select , let's say 200 files, (could be small videos or pictures but never mixing each) and try to use shell-x to send them to "HIGHLIGHTS" folder using my batch files, the script sends them to "CLIPS" instead.
I'm going crazy with this as it also seems to crash the batch files sometimes too.
Same batch with less files works as intended, meaning instead of 200 files we select around 20, it works.
What I've discovered (or think so) is that the "CLIPS\TODO" folder containing all subfolders and cmd files for shell-x is "first" in the order.

To be more graphic on the structure:

[gif,mp4,mov,mpg,mpeg,avi,mkv,webm,ts,wmv,m4v,flv]\02.MOVER A EXTERNO
       |-- CLIPS
       |       |_A
       |         |_ "A.c.ms.cmd"
       |       |_B
       |         |_ "B.c.ms.cmd"
       |
       |-- HIGHLIGHTS
       |       |_A
       |         |_ "01A.c.ms.cmd"
       |       |_B
       |         |_ "02B.c.ms.cmd"

It seems that shell-x has some kind of race condition in which it searches for the given cmd file and instead of executing the correct one, it executes the first it finds with a similar name.

For example, given the structure above, if you execute RIGHT CLICK -> MOVER A EXTERNO -> HIGHLIGHTS -> A -> A.c.ms.cmd it will execute MOVER A EXTERNO -> CLIPS -> A -> 01A.c.ms.cmd instead because "CLIPS" is above "HIGHLIGHTS" and both cmd files have very similar names.
I did an experiment and changed the name from "HIGHLIGHTS" to "[HIGHLIGHTS]" so the folder goes first in the order, and it worked.
The problem with this approach is that I will be breaking "CLIPS" or other folders too.

I don't comprehend how shell-x works under the hood, but from my ignorance it looks like if it were using a diffuse search and executing the first diffuse result given the ordering of the folders, with diffuse I mean that it checks partially into a registry/table/dbase of some kind and matches the result inconsistently.

[oleg-shilo]

Let me reflect on your findings and see what can be done.

[oleg-shilo]

A few points that can help you.

• You will probably have a more predictable environment if you have a folder structure named consistently. I am not sure, maybe it's just a typo but in your post, you do not have a dot character separating the batch file index from its display name. It can lead to incorrect titles, wrong order and even incorrect execution context.
• There is no any synchronisation between selected files execution. The concurrency model is no different to double-clicking. every batch file is executed and it finishes when it decides. Thus you may potentially execute another batch of 200 while the previous one is not finished yet.
• using option ms (multselect) is not a very good approach as it spawns as many parallel processes as many files selected. It's better to have a script of a sort that receives all your 200 selected files and processes them one by one. Or at least have a thread pool (e.g. execute all files in 20 threads). For that, you will need to drop the ms suffix and develop such a script (e.g. C# script, powershell script).
• in my experiment, I can start 16 files with .ms. but not 17. It looks like there is some sort of protection against misbehaving executables.

[oleg-shilo]

It looks like on my system it does not even invoke the shell extension file handler for 17 files at all.
I will investigate it further. It might be related to what you are experiencing in your environment too.

[masterofobzene]

Thanks to take this into account.

The example is not 1:1 to real life usage, I do have each entry with numbered prefixes in real life, meaning the names are very similar, but the number prefixes may vary.

I thought about your "ms" suggestion, but I didn't know how to make such a script, especially using powershell since I don't know how to script using it. I will see what can I do.

in my experiment, I can start 16 files with .ms. but not 17. It looks like there is some sort of protection against misbehaving executables.

I would drop that hypothesis because I managed to make 50 - 100 files work sometimes and all my cmd's have the "ms" sub extension. It is black magic to me (random).

[oleg-shilo]

I have just made the release v1.5.7

• Added option to keep the console open for console commands with non-empty envar SHELLX.KEEPCONSOLE
• Update PowerShell command support. It was needed because PS CLI changes
• Update of SharpShell and ServerRegistrationManager packages.

The 16 selected items limit seems to be a constraint of my environment. There is nothing in this release that would directly address your problem as I cannot reproduce it in my environment. However, there is an update of the SharpShell library that is responsible for the core integration with Explorer.

I have also updated the PowerShell script handling since PS got the CLI update, and the old API was no longer working.

And here are two examples of how to implement looping through the args (when there is no .ms. suffix) in the script.

The example executes (loads) every arg with notepad.exe:

01.cmd-loop.c.cmd script

for %%x in (%*) do (
   rem start and wait
   rem notepad %%~x 

   rem start and continue
   start notepad %%~x 
)

02.ps-loop.c.ps1 script

# Iterate through each argument and open in Notepad
foreach ($file in $args) {
    Start-Process notepad $file
}

[masterofobzene]

thanks for your time!
I will try to modify my scripts and see if I can fix them so they don't spawn so many CMD's those examples will be of much help.

EDIT:

Have you tested the CMD example? because it works only if I pass ONE argument, it ignores all the rest of the files. For example if I select 4 files, click on the first and select the shell-x testing script that I made using the example above, it only moves the first file and ignores the rest.

[oleg-shilo]

Yes, I have tested the cmd example.
I wouldn't sent it to you otherwise.

[masterofobzene]

Ok I made it work but there is a limit again. This time the limit seems to be exactly 75 files. 76 won't work. It must be something in the way that cmd or windows processes the arguments for the batch but that is not related to shell-x since simply drag and dropping the files to the batch produces the same effect without shell-x doing anything.

I will keep searching to see what I find.

On the other theme ( the "race condition") I cannot test easily but I will try to test the new version and see if it fixes it.

[oleg-shilo]

...since simply drag and dropping the files to the batch produces the same effect without shell-x doing anything.

Yes, this is a very good definitive test that proves our early observations. The limit is imposed by the OS but not by the tools.
BTW, in my case, it is 16. Exactly 16.
I did a quick search yesterday but could not find anything definitive, except ChatGPT's suggestion that this can be an OS security measure, but without any details.

Though the fact that on one system it is 16 and on another 75 suggests that it is something configurable.

[oleg-shilo]

BTW:

Yes, there is a limitation in Windows that restricts the maximum number of files that can be dragged and dropped at one time. This limit is determined by the size of the command-line argument buffer, which is capped at approximately 32,767 characters.

[oleg-shilo]

Bingo!

My test folder was inside of a deep nested directory so it is 16. I moved it to the root of the drive and now I can do 50+

update:
False alarm :(
I retested the original folder and now it can process more than 16 (100+). Meaning that the limit is dynamic somehow.

[masterofobzene]

I was going to tell you that for some reason, I cannot move 75 files now and it dropped to 60+.

...didn't want to be an anvil so I didn't tell you

Funny thing: I didn't change the paths or files of my test scenario and the batch is as it was when it was working.

Windows can do weird stuff sometimes.

[oleg-shilo]

exactly.

And now I managed to hit the actual args string length limit with long names and 404 selected files. When it happens there is a nice informative message:

Meaning that my yesterday 16, yours 75 and 60 are some strange and wonderful things that Win does under the hood.

[masterofobzene]

Well its clear that this doesn't have anything to do with shell-x.

[oleg-shilo]

Sorry mate. It means that you do not have many options except practising some pragmatic limit on select operations.
Well, we learned something today.

Happy new year.