Tuning S3 advanced storage device upload

S3 storage devices provide multipart upload capabilities. You can adjust the settings for multipart upload to enable larger or more efficient uploads. Note that if content fits into one part, as determined by the settings, a single part upload is always used.

Multipart uploads without a staging file

The default upload behavior for S3 devices is to use an in-memory buffer instead of a staging file. The default upload part size is 5MB, which is the minimum S3 part size. The maximum number of parts for S3 objects is 10,000. These default settings can handle content upload up to 50GB.

You can change the part size by setting the Advanced.S3.StreamUploadPartSize configuration parameter. For example, increasing the part size to 10MB ensures that the S2 upload can handle content up to 100GB in size.

Note that increasing the Advanced.S3.StreamUploadPartSize parameter value also increases memory usage.

Multipart uploads with a staging file

You can also configure your S3 settings to use a staging file. Content is staged into a file in a temporary folder, then uploaded from the file input stream to the S3 device. This method can handle content up to the S3 device limit, 5TB.

To use the staging file, set the configuration parameter Advanced.S3.UseFileStagingUpload to True (the default setting is false). You can also adjust the size of the upload part by changing the value of the Advanced.S3.FileUploadPartSize configuration parameter. The default setting is 10M, which handles content up to 100GB.