Encoding an animated image requires all frames to be loaded into memory, can lead to OutOfMemoryExceptions

[MaryMajesty]

Prerequisites

• I have written a descriptive issue title
• I have verified that I am running the latest version of ImageSharp
• I have verified if the problem exist in both DEBUG and RELEASE mode
• I have searched open and closed issues to ensure it has not already been reported

Description

When creating an animated image, all frames in the image have to be completely loaded into memory. If you need lots of frames at a high resolution, this can quickly exhaust the available memory.

Possible Solution

Creating the possibility for providing streams for the frames instead of completely loading them into memory would decrease the memory required. These streams could contain raw pixel data or encoded images. Using this feature, you could save the frames to the disk thus freeing the memory they take up, and pass FileStreams for each frame to the image.

[dlemstra]

@JimBobSquarePants Maybe we should introduce lazy pixel loading? And only decode the pixels when the are needed by an image operation or the encoder. This would also make it possible to just change the metadata of an image. The encoder would detect that the pixel data is still stored in the encoded format and would just write it to disc.

[JimBobSquarePants]

I dunno how we would manage lazy loading across the entire library. I can't see it being feasible.

I was wondering whether formats though could have Decode/EncodeFrame methods for when a frame is detected on decode we store the bytes with the frame and only decode when needed.

This is all to cater for an extreme scenario though.

From Gitter:

I am developing a video game engine and am currently trying to add functionality for recording the screen and saving it as a gif file. Every third frame gets saved as an array of RGBA data and all the frames get encoded into one gif file and saved when the user stops recording. At a resolution of 1920x1080, that is a memory increase of 166 MB a second. My idea was that you could save every frame as a file in a temporary directory and provide the gif encoder with streams of these files.

Personally though I think pluggable memory management with tempfile backing would be the best solution if we were to offer one. Gifs like this would be absolutely huge in size.

[antonfirsov]

I'm not big fan of lazy solutions. Makes the lifecycle and program behaviour unpredictable. Eg: reading a pixel could trigger an expensive, multi-thread jpeg decoder execution ... brrr!

I think MemoryManager + MMF should help us here. Can't wait for having the time to implement it ... 😞

Storing frames as encoded streams seems to be the ultimate solution for @TodesBrot's use-case, but it seems to be really complex, and needs lot of work. For me it looks like a 2.0 candidate feature ;) I wonder if we can prepare or API to provide such features in the future without breaking changes.

[JimBobSquarePants]

For me it looks like a 2.0 candidate feature ;) I wonder if we can prepare or API to provide such features in the future without breaking changes.

Mebbies... Though I'm more than happy to break everything come v2 😈

[Andrei-]

Just hit this bug myself in a web solution.
Resizing a 1.5 MB gif animation ends up using more than 6GB+ ram and throws an out of memory exception, killing the app.

[JimBobSquarePants]

You got the gif? And code?

I’d like to profile that to see where the allocations live.

[Andrei-]

I have the gif, but no code ,as this is a cms integration (Umbraco CMS) of ImageSharp 2.1.3,
The cms is registering ImageSharp middleware for image processing. In this particular case it's trying to resize the gif by calling the image url with query params: /image.gif?width=500. Gif attached here, hope it helps

[JimBobSquarePants]

Thanks. I should be able to profile this 👍

[JimBobSquarePants]

As part of #2759 the gif LZW was rewritten to work on a ro-by-row basis which created a significant reduction in allocations.

For example, in the sample gif provided by @Andrei- the image is 3600x1582 which would have led to an intermediate allocation of 5.41GB per each of the 689 frames (You might want to consider using a video format here btw). Following #2759 that intermediate allocation becomes 3.5KB

I don't think we'll ever implement a streaming solution as certain formats don't work well (PNG Adam7) with that format. We should instead continue to work on optimizing our codecs and processors to use smarter allocation strategies.