Performance: bndtools takes too long for SHA-256 and is not interruptable

[reckart]

It seems that bndtools is slowing down things significantly when used in a large Eclipse workspace / projects with lots of dependencies. Using the sampler VisualVM, it appears that the time is mostly spent for SHA-256 calculation.

Even when trying to shut down Eclipse, the shut down hangs because bndtools first wants to complete these SHA-256 calculations.

• It would be nice if calculations would not take so much time - maybe they can be performed lazily or maybe they can be cached?
• When Eclipse is being shut down, it would be nice if the m2e bndtools would detect this and stop their build.

[pkriens]

The issue is not the bnd code, it is the code that is called in the VM (it is confusing that you not show the next 2-3 levels?). Checking for interrupts will only make it slower in the normal case.

[reckart]

When I have the opportunity, I'll provide a deeper trace. That said, I had opened it and it stopped at a I/O copy operation that was taking long. The machine I am working on unfortunately does not have a very fast I/O. I looked at the ResourceBuilder code and it seemed to me there that the calculation of the SHA-256 happens unconditionally and is also not subject to any type of caching. That may work ok in smaller workspaces and/or when you have a fast I/O. I believe though that it may not scale well to large workspaces with couple of hundered projects in them and in particular not in the face of a somewhat slowish I/O situation.

[reckart]

Similarly, it takes a very long time on my machine to open a bndrun file in the BND editor in Eclipse - from clicking on the file until the ditor opens, it takes 15 minutes (just measured it). Eclipse completely freezes/beachballs during that time and won't accept any input. And this also seems to run into the same branch of the code.

[bjhargrave]

also not subject to any type of caching

You can't really cache a cache key. The SHA is a unique identifier for the file. Not the other way around.

All that could be done here is to defer the SHA calculation (perhaps forever) until it is needed. But if it is needed, it still needs to be computed.

I have an idea on how to handle deferral here which I will experiment with.

[reckart]

The SHA-256 is generated from a file, right? As long as the file does not change, it would not be necessary to recalculate the checksum. Maybe a file timestamp could be used as an indication whether the file has changed and if a re-calculation of the sum would be necessary?

[bjhargrave]

As long as the file does not change, it would not be necessary to recalculate the checksum.

And how do you know the file changed? You know by calculating the hash to see if it is the same or not. You cannot trust the file system metadata (length, modified time) as that can be easily manipulated.

[reckart]

Sure. What I have done in a potentially similar situation is to keep an in-memory cache of hashes, each associated with a timestamp. When a hash becomes older than a certain time, I re-calculate it. When the file on disk has a newer timestamp than the one in memory, I also re-calculate. It is not perfect, but reduces the need to calculate the hash at every opportunity.

[bjhargrave]

I have an idea on how to handle deferral here which I will experiment with.

@reckart Please try PR #5323 to see if that helps your performance issue.

[reckart]

The situation seems to be improved. I'll keep monitoring it. Thanks!

[reckart]

So, opening the said bndrun file in a freshly opened and fully built workspace is down from 15 minutes beachballing to 2 minutes beachballing. Not bad ;)