By putting a cleanup Lifecycle rule in place on your S3 buckets, you may be able to potentially save costs and increase LIST performance.
"Incomplete Multipart Uploads – S3’s multipart upload feature accelerates the uploading of large objects by allowing you to split them up into logical parts that can be uploaded in parallel. If you initiate a multipart upload but never finish it, the in-progress upload occupies some storage space and will incur storage charges. However, these uploads are not visible when you list the contents of a bucket and (until today’s release) had to be explicitly removed.
Expired Object Delete Markers – S3’s versioning feature allows you to preserve, retrieve, and restore every version of every object stored in a versioned bucket. When you delete a versioned object, a delete marker is created. If all previous versions of the object subsequently expire, an expired object delete marker is left. These markers do not incur storage charges. However, removing unneeded delete markers can improve the performance of S3’s LIST operation."
To add a cleanup Lifecycle rule:
Log into the Amazon S3 web console
Select your S3 bucket
Select Management
Select Add lifecycle rule
Enter a name such as
'Delete incomplete multipart upload and Delete previous versions'
Skip Transitions for now
Transitions allow you to move storage to slower locations at a reduced cost
https://docs.aws.amazon.com/AmazonS3/latest/dev/lifecycle-transition-general-considerations.html
Expiration
- Delete Previous versions after 365 days
You can choose shorter periods such as 7 days or 30 days if you don’’t have a use case for retrieving prior S3 versions.
You will still have the current version, which is usually all you want, but deleting previous versions can help with costs and S3 LIST performance.
- Clean up incomplete multipart uploads after 7 days
If you do not have any automated processes that may re-try uploads, you could choose 1 day
Review
Agree to the 'scary' this applies to all objects in bucket
Note, if you have S3 objects (uploads) which require different policies, you may find it easier to manage by creating a S3 bucket per policy.
You now have some basic cleanup of your S3 bucket(s) configured.
