Use multipart download when cold getting from S3 bucket

[eric-higgins-ai]

Describe the feature

We are caching some fairly large files in AIStore (~5 GB) stored in an OCI bucket. We're using the "AWS" backend with their S3 compatibility API, since it's more mature than the OCI backend. The issue we have is that, when cold getting these files, fetching the file into AIStore is very slow. It would be nice if AIStore would use multipart downloading for better performance.

I'm aware of the blob downloader, which does download large files in multiple parts. It doesn't quite work for our use case though, as it pushes the decision between single-threaded and multi-threaded downloading onto users. As the cluster administrator, I would like a way to configure a multipart download threshold and part size and have it automatically apply to any applicable cold GET requests.

Use Case

When cold getting files from an upstream S3 bucket, we can get improved performance using multipart downloading. I want to configure this behavior cluster-wide, so users of AIStore don't have to think about whether to use single- or multi-threaded downloads.

Proposed Solution

There are 2 possible solutions I have in mind:

• Add cluster-wide configuration to allow switching to the blob downloader for files over a certain size. I think this would be similar to the existing Ais-Blob-Download header, just on a cluster level instead of a request level
  • Note that this would require a HEAD request before starting the cold GET to determine the implementation to use
  • I believe we would also need to make an update to the behavior when multiple cold GETs are running in parallel for the same object. Right now, all but the first request will error. This contrasts to the normal cold GET implementation, where parallel requests will wait for the first cold GET to complete.
  • I'm also not sure how scalable this is to fetching many files in parallel. I know that the blob downloader is implemented as an xaction, and creating large numbers of xactions isn't always well-supported (example).
• Update the AWS backend implementation to use multipart downloading, similar to the way it's implemented in the OCI backend

I'm not sure which is preferred - my main reason for opening this issue is to get opinions on that. I prefer the second one, since it's a smaller and simpler change to make, but the first would avoid duplicating the same multipart download behavior across all backend types.

Other Information

I'm happy to implement this. I just wanted to get opinions on 1. whether a PR adding this feature may be accepted, and 2. what the preferred approach is. I think it's pretty non-trivial to implement, so I wanted to check this before spending time on it.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AIStore build (latest, v3.22, ...)

v4.8

Environment details (OS name and version, etc.)

Ubuntu 24.04, Kubernetes 1.34.1

[alex-aizman]

See related:

• https://github.com/NVIDIA/aistore/blob/main/docs/cli/object.md#multipart-download
• https://github.com/NVIDIA/aistore/blob/main/api/mpt.go#L222

That's for the client side - a separate capability. For the server, we have prefetch with its --blob-threshold arg:

https://github.com/NVIDIA/aistore/blob/main/docs/cli/object.md?plain=1#L2030

or, more precisely:

• git grep -i "blob.*threshold"

So, that's maybe also for me to better understand what's missing.

[eric-higgins-ai]

Wow, thanks for the fast response @alex-aizman !

The reasons those don't address my problem are:

• Client-side functionality: I believe this utility breaks up requests on the client side and sends range read requests to the AIStore backend. This means that if the object is uncached then the first request will download the whole object from the remote backend, and therefore hit the same issue with large files I mentioned in the description.
  • Additionally, we use the S3 compatibility API exposed by AIStore, because we like the ability to remain portable and share code between use cases that hit AIStore and use cases that hit the remote backend directly. Avoiding relying on AIStore-specific APIs is better for us.
• Prefetch: this isn't very robust for avoiding the slow cold GET issue I mentioned above. First, it's always possible that someone doesn't know that they're supposed to prefetch and they just cold GET, which will be slow. Second, a node could die in a way that loses data (e.g. a disk could die), and therefore force a cold GET even if the user did remember to prefetch.
  • Of course there are solutions to these problems, like we could disable cold GET. That will be quite disruptive in our case though, and the proposal I'm making in this issue would be much easier. Of course, it's entirely possible I'm missing a technical reason that multipart downloading isn't already the default behavior, which is one reason why I opened this issue.

[alex-aizman]

Well, the right way to do it is - extend the AIStore backend abstraction with an explicit multipart-download operation - likely "GetObjMpt" alongside the existing "GetObj" and multipart-upload methods:

• https://github.com/NVIDIA/aistore/blob/main/core/backend.go

And implement this TBD GetObjMpt for the all 4 supported backends.

Luckily, AWS and Azure SDKs already provide managed parallel-download functionality; OCI has an AIS-side implementation today. GCP doesn't have MPD via its SDK, and will require us to add our own implementation to ais/backend/gcpmpt.go - eventually.

If you want to start this development that'd be great.

[eric-higgins-ai]

How would we decide whether to use GetObjMpt or GetObj for a particular cold GET?

[alex-aizman]

there must be a per-bucket threshold with a cluster default..

but see, my own previous response requires more scrutiny - as my own immediate question would be: why would AWS SDK provided MPD would be better than our own blob-downloader.

Just put the question of how it is all wired together aside for a sec - we do not know the answer to that last question.

[eric-higgins-ai]

I think it's probably not fundamentally better - the only things I'm worried about with the blob downloader are these ones I mentioned in the issue description:

• I believe we would also need to make an update to the behavior when multiple cold GETs are running in parallel for the same object. Right now, all but the first request will error. This contrasts to the normal cold GET implementation, where parallel requests will wait for the first cold GET to complete.
• I'm also not sure how scalable this is to fetching many files in parallel. I know that the blob downloader is implemented as an xaction, and creating large numbers of xactions isn't always well-supported (AIS targets can become control-plane wedged under high-volume S3 LIST traffic #336).

The first is definitely fixable I think. I'm really not sure about the latter - not sure if my suspicion is even valid, and whether it's fixable if the suspicion is correct.

[alex-aizman]

blob downloader does not use housekeeper, so your concern is misplaced

separately, the unrelated bug you cited will go away with a simple workaround - my comment inside

in re multiple concurrent MPDs of the same source - the risk exists no matter which implementation you use; one our side we consistently handle it via lock/try-lock/upgrade-lock

[eric-higgins-ai]

I think I would just want the concurrent download behavior when using MPD to be the same as that when not using MPD (that is, if we try to fetch a file and an MPD is already in progress then the request will wait for the MPD to complete rather than returning an error).

So it sounds like the preferred solution is the first one I mentioned in the issue description, namely to perform a HEAD request and then switch between the existing implementation and the blob downloader implementation for cold GETs? I'm thinking the configuration should be:

• Whether to enable this behavior (the HEAD request will add latency, so people only storing small files in AIStore may not want to incur the latency penalty)
• The same configuration that exists for the OCI multipart download implementation here

Does that make sense to you?

[alex-aizman]

ideally, additional non-controversial (and perfectly scoped) indication via api.GetArgs

(or its counterpart in our Python SDK, if you are using Python)

on the client side, you will often just know whether it'd make sense to do an additional HEAD(object) - possibly followed by MPD. Moreover, you may even already have the size - via previous list-objects.

[alex-aizman]

https://github.com/NVIDIA/aistore/blob/main/api/object.go#L37

[eric-higgins-ai]

If possible, I'd want the server-side flag so we can configure behavior cluster-wide without relying on clients to pass in the correct arguments. There's also already the Ais-Blob-Download header, which sort of already supports this functionality on the client side

[alex-aizman]

well, the potential blast radius becomes exponentially larger - the number of goroutines, preallocated buffers, and various other overhead

exposes us to scenarios

not to be taken lightly

[eric-higgins-ai]

Hmm so I think my request is basically a server-side version of the existing Ais-Blob-Download header. My reasoning on safety is:

• If we assume that all clients act optimally and only pass Ais-Blob-Download when they need to fetch large files, then the server-side configuration and client-side configuration are mostly equivalent. The differences would be:
  • With a server-side configuration, all requests will perform a HEAD request before cold GET. I'm not aware of a downside of this besides the increased latency
  • With a server-side configuration, all requests use the blob downloader code path, which means that for small files they spin up 1 worker goroutine to download (instead of downloading in the goroutine that handles the request). This is easy to fix
• If we assume that clients don't act optimally and some will forget to specify Ais-Blob-Download when they need to download a large file, and we rely on this behavior for safety, then this isn't robust.

So my argument would be that adding a server-side version of Ais-Blob-Download is comparably safe to the existing client-side configuration

[alex-aizman]

well, do you want to maybe show some initial patch to better see the direction? Ideally, it would work end to end but you could save time/effort on: error handling, writing tests and docs, productizing the code..