Problem with VMA and file splitting

[ahafele]

Testing was done by sending files (of various sizes) from airflow-dev VMA and resulted in failed DAGs. The dags failed on the job_summary_task but there isn't any evidence in FOLIO that the loads ever happened despite the data_import_task being green in the dag.
Loads sent directly through DI are working as expected.

Related to https://app.zenhub.com/workspaces/folio-at-stanford-628d6cab8c42a90248e5eb79/issues/gh/sul-dlss/folio-eureka/132

[jgreben]

If we want VMA to also work with file splitting, we will need to figure out how to use the mod-data-import api to instead post/fetch files from AWS. Calls that need to be made:

• GET data-import/uploadUrl?filename={filename}, returns key, uploadId, and url
• POST /data-import/uploadDefinitions/{uploadDefinitionId}/files/{fileId}/assembleStorageFile with body {"tags": [], "key": "key from get", "uploadId": "uploadId from get"}. Tags may just be a random string or strings. Not quite sure about that yet.
• Possibly using GET /data-import/jobExecutions/{jobExecutionId}/downloadUrl, GET /data-import/uploadUrl and GET /data-import/uploadUrl/subsequent
• Use GET /data-import/splitStatus if we want to support both file-splitting and non-file-splitting modes.