Umbraco Partial/Tree/Workspace restore not working on sites with high volume of media

[lakesol]

Using restore from source to target system where the the volume of media is great, the restore fails.

Reproduction

Bug summary

I am trying to do a restore of content from one environment to another. When I do a partial restore within media for a whole folder it fails with an error: e.g.

[09:14:24 ERR] Work dd240cf2-16c6-4bd6-a4af-5283cfc07851 has failed.
Umbraco.Deploy.Core.Exceptions.ProcessArtifactException: Process pass #0 failed for artifact umb://media-file/b0jecd2y/<my media file>.jpg.
---> Umbraco.Deploy.Infrastructure.Exceptions.RemoteApiException: The remote API has thrown an exception.
at Umbraco.Deploy.Infrastructure.Http.DeployHttpClient.SucceedOrThrow(HttpResponseMessage result, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Http.DeployHttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Environments.RemoteUmbracoEnvironment.GetFileStreamsAsync(Guid sessionId, IEnumerable`1 udis, IFileTypeCollection fileTypes, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Environments.RemoteFileSource.GetFilesAsync(IEnumerable`1 udis, IFileTypeCollection fileTypes, Boolean continueOnFileNotFound, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Connectors.ServiceConnectors.FileConnectorBase.ProcessOptimized(ArtifactDeployState`2 state, IDeployContext context, Int32 pass)
at Umbraco.Deploy.Infrastructure.Connectors.ServiceConnectors.FileConnectorBase.Process(ArtifactDeployState`2 state, IDeployContext context, Int32 pass)
at Umbraco.Deploy.Infrastructure.Connectors.ServiceConnectors.ServiceConnectorBase`3.Process(ArtifactDeployState dart, IDeployContext context, Int32 pass)
at Umbraco.Deploy.Infrastructure.Environments.CurrentEnvironment.ProcessArtifacts(IEnumerable`1 arts, String ownerEmail, IDeployContext context, CancellationToken token)
--- End of inner exception stack trace ---
at Umbraco.Deploy.Infrastructure.Environments.CurrentEnvironment.ProcessArtifacts(IEnumerable`1 arts, String ownerEmail, IDeployContext context, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Environments.CurrentEnvironment.ProcessPackageAsync(Guid sessionId, Package package, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Work.WorkItems.DeployRestoreWorkItemBase.ProcessPackage(Package package, Nullable`1 batchIndex, Nullable`1 numberOfBatches)
at Umbraco.Deploy.Infrastructure.Work.WorkItems.TargetPartialRestoreWorkItem.Proceed()
at Umbraco.Deploy.Infrastructure.Work.WorkItems.TargetPartialRestoreWorkItem.ExecuteAsyncSub(IWorkContext context, List`1 resume, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Work.WorkItems.TargetPartialRestoreWorkItem.ExecuteAsyncSub(IWorkContext context, List`1 resume, CancellationToken token)
at Umbraco.Deploy.Infrastructure.Work.WorkItems.TargetPartialRestoreWorkItem.ExecuteAsync(IWorkContext context, CancellationToken token)

This folder does contain a very large amount of child items and the media files total up to be quite large ~1.7GB

But if I then restore that single media item listed, it works fine e.g.

[09:16:59 INF] Create manifest on source
[09:17:00 INF] Review manifest on target
[09:17:00 INF] Create package on source
[09:17:00 INF] Complete remote session
[09:17:00 INF] Process package on target
[09:17:01 INF] read umb://media-file/b0jecd2y/<my media file>.jpg
[09:17:01 INF] Complete remote session
[09:17:01 INF] Suspend document cache.
[09:17:01 INF] Suspend indexers.
[09:17:01 INF] Commit
[09:17:01 INF] Restore caches and indexes
[09:17:01 INF] Resume scheduled publishing.
[09:17:01 INF] Resume document cache (reload:False).
[09:17:01 INF] Resume indexers (rebuild:True).
[09:17:01 INF] Complete
[09:17:01 INF] Deployment fc375410-7457-4d61-aed2-125124b73677 completed.

My Assumption is either the total item count or the payload size is blowing out system limits, causing the issues seen.

This appears then to be a batching/approach issue. If your batching system is capable of restoring single files but not multiple.. This is an approach issue. There should be configuration which allows the system to restore smaller chunks of data from source to target to avoid system limitations.

When looking at the documentation this appears to be a known issue. However the suggestion to just restore smaller blocks of data is not helpful/user friendly when trying to work on large systems.

Specifics

In this example this is a restore from Cloud to local windows development. But have seen the same issue between cloud<->cloud.

Steps to reproduce

See above

Expected result

To cope with large/complex/media heavy sites the approach for data transfer should be reviewed to allow working within system/technical limits. Forcing uses to just restore smaller chunks of data manually is not efficient.

Actual result

See exception above

[lakesol]

I should also note that no error is reported within the logs of the source system.

[ronaldbarendse]

Hi @lakesol! Please provide the CMS and Deploy version used, since we've done quite a few performance and stability improvements in both recently.

Restoring large amounts of media files might take some time, so make sure you're not running into any of the default 20 minute Deploy timeouts (you can configure SessionTimeout, SourceDeployTimeout and HttpClientTimeout to a higher value if required).

Files are already chunked into fixed size requests (smaller files are combined into a single response and larger ones into multiple), see the MaxRequestLength setting. Make sure your hosting setup allows requests of this size (and allow for some margin to be on the safe side): configuring this to a higher value than is allowed/possible might cause the server to return errors too.

I should also note that no error is reported within the logs of the source system.

Since the source (remote) environment is the one throwing the exception, not having this logged makes it a lot harder to debug indeed... Besides the timeouts and maximum request length, one other possible cause could be that the media file simply doesn't exist: this could also explain why other restores work without issues.

And instead of manually doing smaller batches, you can also configure the PackageBatchSize to limit the amount of items that are restored/processed in one operation. However, do note that configuring this specific setting will result in quite a lot of overhead (the items are restored/transferred twice to ensure the second pass corrects any missing dependencies from the first/initial pass).

In short: first check the logs and see if the deployment doesn't fail around the 20 minute mark 😄

[lakesol]

Hi thanks for the feedback.

This site is on

Umbraco.Cms 13.13.0
Umbraco.Deploy.Cloud Version 13.4.3

In the environment in question its not failing at the 20min mark. More like 1-2 mins in.

re: "cause could be that the media file simply doesn't exist: this could also explain why other restores work without issues.", no, as after the listed item that fails when performing large folders, can be explicitly restored on its own. So if it was a failure with the listing media item (e.g. file missing) I would expect the single restore operation to fail too.

I will try your other suggestions and see if I can get any to work.

Thanks